All About Beta Testing

[Jacic]

@Doug

Perhaps it’s just the writing group mentality. I work with my group quite often, and one of our golden rules is ‘descriptive, not prescriptive.’ We listen to our readers when they tell us what they did/didn’t like, but we politely ignore them when they try to suggest how to improve it. I could have something of a conditioned response.

I’m surprised they teach that to be honest. (That’s not what I’ve been taught which is more along the lines of have your own ideas of how the story is to progress but listen to feedback and decide whether to act on it. It’s not to say you shouldn’t have your own work and style, but there’s nothing wrong with listening to what people want. You’re talking about making it into a commercial product after all rather than just a personal piece. If the advice doesn’t fit your story line, you’re under no compulsion to fix it in that exact way, but you still know it needs to be changed. That’s especially true if you have multiple people telling you the same thing, you should definitely listen. If that’s the case you need to justify why you can ignore the advice rather than the other way around. It can sometimes also give you some great ideas for story lines and character interactions. Sometimes people will see my characters or the MC in quite a different but still very valid light to me.

I find IF to be quite different to standard novels where you can get away with less feedback in my opinion because they’re your characters, acting how you decide the story is going to play out. IF is interactive, people want to be able to have a particular amount of freedom which testers can pick up for you. For example you may have decided the story needs to go in a particular direction but a lot of readers hate that words are being put into mouth so to speak. A few extra choices or a bit of rewording to show why something has to be that way will result in a much happier audience long term.

Anyway, good luck with it

[Szaal]

@Jacic

If the advice doesn’t fit your story line, you’re under no compulsion to fix it in that exact way, but you still know it needs to be changed. That’s especially true if you have multiple people telling you the same thing, you should definitely listen.

TL;DR @Doug
Listen to any comments, suggestions, and criticisms. But you as the writer/developer still hold the wheel to decide how you want your story goes.

[Havenstone]

@Doug

I was hoping that by sourcing answers to my commonsense concerns (Does it limit my options? Aren’t I poaching my own market?) I’d better understand the process, and that once I understood it, it wouldn’t bother me anymore.

Just wanted to affirm that your concerns are commonsense ones, and this isn’t the first time they’ve been raised. Which is probably why so many people are quick to whip out the things we’ve said before, especially me, Eric, and Eiwynn.

We had a similar conversation a couple of years ago, when the requirement was still new and the open/closed issue more negotiable:

Regarding Hosted Games beta-testing

There’s not anything about this on the CoG website, so far as I can find… I hear that a public beta test of a game is required before a Hosted Game will be published. I’d like to inquire about that in more detail. For example, all of the CoG and Hosted Games seem to have a free demo version of the first few chapters, then pay for the rest of the game. Is this a proper format for the forums? Having a demo version open to the public and a private beta for the full game… or, how is it best to hand…

And since then:

Open Beta Tests or Closed Beta Tests?

So I have lately wondering what would be best to have when testing a game and getting over people’s opinion on the game. What’s the difference, What would be better, Extc Questions like that. Closed beta. Open beta. Or both? Am asking you experienced coggers your opinion and I hope you can help . Thanks in advance! Ps… I hope I put it in the right section

There’s some on it here:

Feedback about the Beta Process

Continuing the discussion from Empyrean: Take down your father with his own secret airship!: Not sure I’m doing this right, @FairyGodfeather but here is a new topic to discuss the beta process.

This thread had some derails, but here’s a post from an established HG author who doesn’t like open beta tests:

Open vs closed betas. Thoughts?

I think I’ve mastered the Art of the Beta Test. I don’t like Open Betas. For one thing, you get a hundred people catching the same first couple of errors and then missing the rest. Me, I prefer to run Closed Beta Tests in waves. I give the link to about 10 people who can reply quickly. After I fix their issues, I slowly invite more people as my inbox becomes empty. This way, I can keep getting fresh eyes instead of having everyone spot the same couple of errors. Those that…

And finally, regarding CoG “crowdsourcing” HG proofreading and editing, here’s a recent post from Jason that I believe @Lys may have had in mind above:

Should Hosted Games be vetted more thoroughly?

One of the key things that we do not want to be accused of–or mistaken for–is being a “vanity press.” A vanity press is a pejorative term for a small press that offers a suite of services for amateur and semi-pro authors that result in the author never seeing revenue for their work or even finding themselves in debt to the publisher for “services rendered.” For example, “sure, we’ll publish your 100,000 word fantasy epic! And even give you a $500 advance against 25% royalties! But you’re going…

[Doug]

@Havenstone, Wow! Thank you. Those links are awesome. It’s clear that I’m diving into an established conversation here. It’s especially helpful that you’ve linked to @Lucid as well, it’s great to see my sentiments echoed by an established HG author.

Thanks @Szaal, that’s definitely the approach I’m familiar with.

Hahaha, @Jacic, I might have shot myself in the foot there. I’m one of the people who introduced that rule. We based it off the Brandon Sanderson lectures available on YouTube. The idea isn’t that you discount reader feedback, but more that you take everything they say with a grain of salt. If you keep getting the response that a particular scene is boring–you should add in more action–you go away with the understanding that ‘people don’t like this scene. It could be boring (or there could be some other problem at the root of that feedback), and it may be that adding some action would help (but I should also consider other options).’

It’s more or less the same process, but we kind of mentally reduce feedback to a thumbs up or a thumbs down. ‘Characterisation? Thumbs up! Pacing? Thumbs doooown.’ That way we still get targeted feedback, but there’s not struggle to retain the creative vision in a sea of 'you should’s.

So I’m thinking I might chuck in some ‘tester friendly’ infrastructure (frequent save points comes to mind) and then put it up for the open beta in a day or so. It’s a very experimental piece, and definitely not the usual CoG style, so I’ll be interested to see what y’all think. (Sorry for the clunky Americanism. I’m an Aussie.)

[poison_mara]

I think authors has to switch the perspective from conventional novels. In a novel is your characters and your vision. Yes, you could listen feedback. But in inner stance is your story. Here there is the main elephant in the room interactivity aka choice. Story has to be a team between a player (not reader, player that’s a key to understanding the situation) If you don’t give enough room to player agency and decision. You could be done a literary masterpiece and still a terrible Cog , a souless cog. Because what makes Cog a cog are Choices and could creating something with the author.
So if many people asking why I can’t choosing here in that scene or why my character has to be with settled personality. That normally is good idea to hear. Because is what mayor part of people expect. If they can’t choose certain amount of things. Casuals never will bought and read past the demo. Anyway , just my two cents

[Havenstone]

Glad it was helpful. Let me also pull out from one of those threads a note by another of our most prominent HG authors:

Feedback about the Beta Process

I doubt I will ever do another open beta. I receive tons of feedback that boils down to requests for customizations (like @Mary_Duffy mentioned in the other thread). I also received reviews stating they were upset there was not much more content in the published game than the beta (50k words isn’t much, I guess). I still run a core closed beta group while I write and will run short-term betas at milestones. Otherwise, open betas are a thing of the past.

By contrast with Lucid (and Jim’s future intentions), we of course have Cataphrak, whose “open betas” produce threads that are the stuff of legend. Rambling, almost-impossible-to-keep-up-with, forum-breaking legend.

https://forum.choiceofgames.com/t/guns-of-infinity/2656/39896

And I just want to give another shout out to my writing community on the XoR thread, who over almost four years of open alpha/beta testing have strengthened my game immeasurably – even though I’ve also taken many things with a healthy grain of salt.

Choice of Rebels Part 1 WIP thread

For the past couple of years, I’ve been working on a fantasy CoG called Choice of Rebels. Like all self-respecting epic fantasy, it’s going to be a series… say, four or five games, when all’s said and done? That kind of scale seems to be working out well enough for JimD. Anyway, I’ve now got more than two-thirds of the first game written. It’s enough to give a pretty good flavor of the world, characters, and plot – but there’s enough unfinished that I can still make major course corrections.…

[poison_mara]

You hear me out lol, Even of your first intention was launch me out of your face. But I think you did a great job dealing with feedback. And with me. I was really rude back then, but I think I was in the right track. Is something lot of authors forget, A reader or tester could be an ass , and still giving you the best advice for your own work. People look more at forms, that what they really are saying.

[Jacic]

@Doug

I’m an Aussie

No problems, so am I lol

[JimD]

@Doug

I’m unsure what function the open beta is supposed to serve. Is it a market test? Is it crowdsourcing part of the editing process? Will the results of the open beta inform CoG’s decision to accept or reject my game?

Two questions for you:

1. Have you had choice game fans test your game? A reading group/workshop is a different set of readers.
2. Do you have a following outside of this community?

For a game to be " good", it needs to be tested with scrutiny. Not just spelling/grammar. I’m talking continuity, impact of choices, player agency – all that good stuff that makes choice games the bee’s knees.

Part of the requirement for open testing is to gather feedback. Most new game writers don’t have a dedicated group of choice-game fans to call on to test their work. Since there are numerous WIPs at any one time, there is competition for testers. Therefore, open betas provide the most exposure to testers.

People like me, Cataphrak, Lucid, and Allen Gies all have success in the Hosted Games arena and we all did open betas until we assembled our Avengers of testing teams. That’s not to say our way is right but it’s right.

[Lycoris]

@Doug

In fact, I have a writing group who have seen my story already.

The catch of having groups like that (where you know each other and are friends, I guess) is that they will tone down any criticism they have. What I mean is, if there’s something that sucks in your story (hypothetically speaking), your writing group may be more forgiving/lenient, but that won’t do you any favors once your game is out in the market. The good thing about having testers who don’t know you is that they will be 100% honest and tell you what they see wrong without watering it down.

[hallofmirrors]

Also, a writing group might not be testing for the same things, especially if it’s not an interactive fiction-specific group. Plus, beta testing on the forums lets our diverse community take a look at your game and see if it’s reaching as wide an audience as it could; I know I and several others usually push pretty hard for asexual/aromantic and nonbinary options, which are fairly simple to implement, but can make a big difference in who wants to buy your game.

[Vertigo]

I’ll help you out. I haven’t put my game out yet, so take it with a grain of salt.[quote=“Doug, post:1, topic:26284”]
2: I’m confused–doesn’t this poach the market?
[/quote]

As many people commented already, not really. The forum is the smaller slice of the pie when it comes to the readership you might get. Besides, having a beta test with seven or eight people hardly means the whole forum. Launching a game (a Hosted Game, I should add) that is well-received might get you some more purchases from here. And I wouldn’t count on people just playing the demo forever and not buying it, from my experience in the Guns of Infinity thread, the smashing amount of peple who tested it also bought it. Maybe it’s because it’s a series, but I’d disagree.

And if you ever have another game on your mind, you’ll already have some people willing to help you or who’d be intereste din giving your works a try (a few Hosted authors, like MultipleChoice and Cataphrak, have developed a fanbase-esque).

@Doug

I’m unsure what function the open beta is supposed to serve. Is it a market test? Is it crowdsourcing part of the editing process? Will the results of the open beta inform CoG’s decision to accept or reject my game?

I think it’s mostly bug catching and language correction for spelling mistakes.

I tend to treat to see betas this way: go for it. Nine pairs of eyes work better than one to get a mood or a scene right. You can also get people with different tastes, from different walks of life to help you out, and I believe that’ll enrich your game, regardless of opnions on it.

[Doug]

@JimD

Two questions for you:

1. Have you had choice game fans test your game? A reading group/workshop is a different set of readers.
2. Do you have a following outside of this community?

The answer to #1 is: sort of. I’ve had IF fans play the game, and one of them was familiar with CoG. It’s probably worth noting that I’m talking about ‘playing’ (or ‘reading’) as opposed to testing. The game that I’d like to submit was written as an experimental creative piece to complement my honours thesis. It’s written in choicescript, but has little in common with most CS games. In particular, the total number of choices offered in the game is very few. As a result of that, I’ve been able to create stable builds for each round of beta playing. I’ve had readers come to me with continuity errors in the past, but that’s always been when I jumped a production cycle and gave them a not-yet-proofread draft.

The answer to #2 is ‘oh man, that’d be awesome.’ Most of what I’ve written in the last year or so has been to test genre or digital literacy theories. They’re usually half-formed skeletons of projects that get abandoned as soon as I’ve learned what I needed. This particular piece needed to be assessed as a creative work as well as an experimental piece, so it got to live to see completion.

@Lycoris

The catch of having groups like that (where you know each other and are friends, I guess) is that they will tone down any criticism they have.

Mmmm, that’s one of the challenges we’ve struggled with. We’re a fairly large group (I run a student society at my university), so you can often find someone you don’t know pretty well to give you some honest feedback. I personally like the James Portnow manoeuvre: ‘hey, I’m playing this game that a friend of mine made. I don’t think it’s really good, but I’d like a second opinion just to be sure. Can you tell me what you think?’

Of course you can only use that a few times until people start catching on, but it’s good while it lasts…

@hallofmirrors That’s certainly true, and I’m definitely interested in listening to that kind of feedback. Unfortunately I’m not sure if I’ll be able to accommodate any substantive changes. I’ve started my next creative project, and I’m kind of going by the finished, not perfect/fail faster mantra. I’m definitely going to polish (line and copy edit), but I want to avoid any substantive edits. I could probably create an asexual option fairly easily, but most other changes will probably just be put down as tips for next time.[quote=“Vertigo, post:30, topic:26284”]
I tend to treat to see betas this way: go for it. Nine pairs of eyes work better than one to get a mood or a scene right. You can also get people with different tastes, from different walks of life to help you out, and I believe that’ll enrich your game, regardless of opnions on it.
[/quote]

I can certainly understand that logic. It’s a very different approach to the one I’m used to taking, and I’ll be interested in seeing how it goes.

Barring any unexpected problems (I’m working on a small project due this evening, and I’m not sure if my last wave of edits left my piece in a stable build), my draft should be up sometime today (AEST).

[Eiwynn]

@Doug

The answer to #1 is: sort of. I’ve had IF fans play the game, and one of them was familiar with CoG. It’s probably worth noting that I’m talking about ‘playing’ (or ‘reading’) as opposed to testing.

This is proof enough to me the Open Beta is needed for your submission. Good luck.

[Havenstone]

@Doug, just a quick note of clarification that the open beta doesn’t have to be quite as open as Cataphrak’s and mine have been.

Quoting @jasonstevanhill here, the open beta COG requires for Hosted Games does not mean “you have to post it so any rando can play without contacting the author.” It means at a minimum that the beta has to be publicly announced on the forum, and forum members who express interest are allowed to join a private message thread where they’re given access to your draft.

That will give you a better sense of how many testers are getting free access to your beta and how many of them are actually giving feedback.

[Doug]

@Havenstone Oh! That’s good to hear, but the cat’s a little out of bag already… I’ll have a think, and look at the kind of responses I get. It may be that it’s more productive to have a closed beta.

Thanks for letting me know!

[Szaal]

Well, at least ppl will know about the work and start giving the feedback (although I’m yet to click your thread )

[jasonstevanhill]

To follow up to Havenstone, if you say “I did a beta!” but you mean, “my three best friends and my mom played it and they all told me it was awesome!” that doesn’t mean it was given a beta test. We want to see the community given a real opportunity to engage an author in a dialogue.

In particular, when we at Hosted Games are trying to decide if we should expend money/capital on a game (putting it on Steam, putting it in a mailer, tweeting about it), Rachel gives her input, and then we look at the beta thread to see how the community has responded. Thus, whether the game is freely accessible or not, letting us see the feedback of the beta testers helps us decide how we market it.

[poison_mara]

Say that someone has read a game and hypothetically speaking not talking about this project per se, believe it is far lower anything resembling a quality bar. If that member of forum say so probably would get flag or censorship for saying so. Could that negative feedback seen directly to Cog staff because I think author doesn’t gain from it. But if quality is terrible People should have some place to say it.

[Fennik]

Um I just wanted to ask how long estimately would a game be released after beta testing was announced done? I’ll close this thread if the question is not allowed…