All About Beta Testing

General
1

Looking to submit a game under the ‘hosted’ title, but not 100% sold on the public beta test. Clearly a lot of people have found it acceptable (because otherwise nobody would be published), so I wonder if any of those people could answer some of my concerns.

1: I’m concerned that it will limit my options. If for whatever reason my game doesn’t get published by CoG (maybe they have too many in their queue. Maybe I find a fantastic opportunity elsewhere. Maybe they suddenly close business.), if I try to take it elsewhere I have to admit that the full version has already been available online, and played by an unknown number of players.

2: I’m confused–doesn’t this poach the market? I imagine that a great many of the people on this forum are avid CoG readers, exactly the kind of people I’d want to sell my game to. What stops everyone from simply playing the open betas as they are released, and never buying a single one? (Is the site traffic just negligible in the greater scheme of things–and if so, why?)

3: I’m unsure what function the open beta is supposed to serve. Is it a market test? Is it crowdsourcing part of the editing process? Will the results of the open beta inform CoG’s decision to accept or reject my game?

I’ve tried to anticipate some common responses so we can skip the initial ‘have you turned it off and on again’ style interaction:

I’ve asked CoG whether the open beta was negotiable, and they advised me that it is not.
I recognise that writers share their unfinished work all the time: In fact, I have a writing group who have seen my story already. I have no problem with the ‘beta’ part, but a few concerns surrounding ‘open’.
I understand that choicescript games can be a mammoth task for one editor to test. Mine passes randomtest and quicktest, and since it’s rather short and extremely dialogue heavy, I’m quite confident that it’ll be bug free by the time I’m ready for submission (which isn’t just yet, but is very soon).
I understand that choicescript can’t invest in proofreading, and that they don’t want to be associated with games that have poor spelling/grammar. I’m just not convinced that crowdsourcing a proofread (if indeed that’s the function of the beta test) is the best approach, and since my story will be professionally copy edited before submission I don’t think that’ll be a problem.
I understand that if I were to take the game elsewhere I’d have to port it to another scripting language. While that certainly isn’t an attractive option, it is still an option, and I like having it in my back pocket if CoG doesn’t work out.

I’d really love some input here folks. I’m excited to submit my work, and I’d like to do so without any lingering doubts.

2 Likes
2

Question from a newbie: does the open beta test involve showing the whole, complete game to the public? Showing unfinished WIPs can make the audience hungry for the final story.

But if showing them the open beta involves showing them the ENTIRE work, then yeah, the only remaining draws I can think of are a) the satisfaction of having your achievements in your collection, b) if your game has extremely high replay value, or c) whatever shiny new features you add in the final version.

I actually don’t know if proofreading is that big a factor. There have been some hosted games with less-than-perfect spelling or grammar, although they generally make up for it with good story, characters, etc.

1 Like
3

I’ll focus on 2.

The short answer is no, it does not poach the market. Let’s say you have 100 people read it via open beta. And let’s assume that, because they read it, they will decide not to buy it. (This is not accurate, but let’s just say it for sake of argument). And let’s say your HG is released for $2.99. You just lost the princely sum of $52.33. Seriously, that’s all you lost.

$2.99 x 100 x .175 = 52.325

And in reality, many beta testers here DO buy games upon release because they want to support authors and because it’s cool getting the “official” version of a product you helped test.

You’ll need to sell 1000s of copies of your HG to make decent money, so losing 100 sales just doesn’t hurt you that much, especially since those beta testers (at least some of them) probably supplied feedback to strengthen your gamebook.

11 Likes
4

I did get the impression that the number of beta players would be negligible when compared to total sales (I mean, hopefully), but even though I know that, I’d feel better if I understood why that was the case.

I mean, in any other setting if you had fans of a medium/genre/artist who regularly have the opportunity to experience the full, completed work free-of-charge… Well, unfortunately you’d see almost no sales.

Perhaps it all comes down to market metrics. I figured around half the purchasers of my gamebook would be CoG fans, and probably half of those visiting the forums semi-regularly. That would mean a quarter of my target market has easy access to my full game for free prior to publication.

Are those assumed figures just faulty? It is perhaps that the percentage of purchasers who are also regular forum visitors is much lower?

5

Have you considered doing a closed beta? Maybe put a chunk of your work out for open beta, let a few comments, criticisms, etc roll in and feel out who you think would be well suited to a closed beta to help fine tune the finished product? I think that’s how a lot of people who post works in progress do it–I’ve only very rarely seen a full game available for testing

1 Like
6

For 1, I’d say that the only one that should concern you is the “find a fantastic opportunity elsewhere.” Unless your game contains highly objectionable content or the open beta gives feedback that it’s near-incomprehensible due to poor spelling/grammar, CoG will publish it under the HG label, and they’re a thriving, growing business.

For 2, I whole-heartedly agree with Eric. Most CoG fans don’t frequent the forums; the site traffic is, as you said, negligible in the scheme of things. We definitely don’t have anything close to half of CoG customers engaged on the forum; the active forum members at any given time are far fewer than the number of people who buy CoG games. And many forum beta testers buy the games they test (assuming they liked them at beta stage).

I’ve been writing a game for the CoG label for the past six years, and have had it open-tested on the forums at every stage – including now, when it’s basically complete. It’s strengthened the game significantly, and I’m confident that when we do publish later this year, whatever customers I lose because they played it at beta will be a cost well worth the benefits of the test. (If I’m wrong and it’s a flop, then I’ll make sure to update this thread and PM you a warning.)

I can’t speak authoritatively for 3, but my impression is that it’s both to catch abysmal spelling/grammar and to make sure you’ve had some feedback from regular COG readers on how they like your game.

@HomingPidgeon, when a WiP gets to the point where it’s actually going to be a Hosted Game, it needs an open beta, per CoG rules. Whether or not Doug has done a closed beta, he’s got to put the full thing out for beta before it’s published. See the category here: https://forum.choiceofgames.com/c/hosted-games/hosted-games-betas

6 Likes
7

First point: My main thought is that if you’re at the open beta stage, you’ve already probably done/should have done your homework about other “opportunities”. At that point, you’ve built the game to CoG’s distinct style/specifications, and with the intention of getting it published through them. Any change in publication platform will require substantial rewrites/changes to fit the bill at another publisher, so you won’t have posted the “Full version” but a prior build.

Not saying it wouldn’t limit your chances, but honestly? If your game’s solid and the testing goes well, CoG is a good business with a solid market base. So I’d say it’s worth the risk.

Second point: Not especially. The beta-ers will be drawn from the portion of the fans that A) use the browser version of the game as opposed to just downloading apps, B) also enjoy going on forums C) are active during the time you have the beta open and D) notice the beta. So… enough to get your game tested, but nowhere near the final market size.

And most people on site are there to be supportive, not cheap out on games. Not that there aren’t people, but… it’s a small subset of a small part of the market. And many will buy the game based on principle, if not excitement for the release. Or to, you know, replay the games after the beta is taken down.

Third point: Not really, in a sense but please remember that there’s more to proofing than just being glorified spellcheck, and I’m not sure but it’s probably more a “If the crowd hates it, it’s probably not for the site anyway” thing… so correlation rather than cause and effect?

Keep in mind that the games are meant to be branched and winding, so having many people playing many routes will help you find problems that a copy editor will miss. Like when there’s a problem with one fragment of flavor text not fitting into the narrative (either because it breaks flow or because you forgot a *line_break and it’s a jumbled disaster) that was missed in editing because no one played the character as “grumpy and rude but friends with James and wearing a blue shirt”.

It’s sort of a “monkey with a typewriter” type deal, where the more people see your game, the better odds all combinations are tried.

Also, CoG forums are populated by fans familiar with the style and rhythm of these games. They’ll notice if something is off and that can really make or break things.

4 Likes
8

Most CoG fans don’t frequent the forums; the site traffic is, as you said, negligible in the scheme of things.

… enough to get your game tested, but nowhere near the final market size.

These are both good to hear. I was under the impression that it was a bit of a niche readership, and a very connected community. Glad to hear to most sales come from outside the core group.

…my impression is that it’s both to catch abysmal spelling/grammar and to make sure you’ve had some feedback from regular COG readers on how they like your game.

It’s sort of a “monkey with a typewriter” type deal, where the more people see your game, the better odds all combinations are tried.

It’s a little disheartening to hear that CoG might be crowdsourcing proofreading and testing, but it’s also very understandable when considering the hosted label business model.

Thanks so much for your responses folks, it’s great to hear authors’ understanding of the open beta process. I’ve got to admit that I’m still not totally sold on an open beta, but it might be time to bite the bullet. (CoG indicated that I should put up my current build now. It’s still waiting on an external line and copy edit, but there are unlikely to be any substantive changes beyond this point.)

I’ll have to go have a think about it. Thanks again for your responses.

1 Like
9

I can’t seem to find the link, but the main reason for this is CoG is still a small business, and they can’t afford to hire full time beta-testers, etc. Hell, even the big companies don’t do as much beta-testing as they should, allowing people to buy ‘Early Accces’ and pay for the privilege.

Not to mention there are legal reasons CoG also has a relatively hands-off approach for testing (mostly due to if someone would be considered a hired employee, etc.)

4 Likes
10

The community here is very connected to the individuals that participate - but when it comes to the numbers that participate in the app stores and on Steam (if you get earmarked as a Steam author) the actual numbers is small.

The thing is, those that help in the betas are very versed in what those market places look for and comment on in reviews and such. One of the most prolific testers here can tell you that in the Spanish market, your game will do well or have issues because of specific things as can others that have experience elsewhere.

Even as a niche market, the cross-section of people found here represent a good swath of the consumer market in general and if the developer in general does not have a regular reading group or circle of private testers, the public beta is a resource that few can afford to skip - even if they knew about it from experience.

5 Likes
11

Do the beta test. Not only is it a requirement for publishing, but as Eric pointed out it’s costing you peanuts in the scheme of things. (Look up how much it will be to get it professionally edited if you want to go that way, even then it is still worth the beta test as an extra check.) The number of readers that can potentially buy your book on the stores is far larger than the regular forum community.

To be frank it’s silly not to take the help that is offered by the folks around here, it’s one of the really good things about this community. You might think your work is perfect, but trust me, there’ll likely be things that have been missed that others will pick up on. It’s not just a spell check, it’s also how the book is put together including character interactions, obvious railroading, quality of the choices, coding mistakes and whether the way the stats are being used is good or frustrating your readers to peices. I’m not saying your book has any of these faults, but letting a few people read it won’t do any harm and may do some good.

I think you can have a closed beta if you’re that worried? (You’d have to double check that in the rules). If you haven’t published anything before though to have a following and there’s no WIP that people have been reading, it may be much harder to get people willing to test for you. You’d probably need at least a few chapters up to let people look and decide.

As for a “fantastic opportunity elsewhere” I haven’t seen your work and it may be high standard, but if you haven’t published anything before, you could do much worse than publishing something though HG to distribute it. It offers a good deal for new authors and has a good existing reader base. Unless you’ve got other parties knocking on your door already, the work involved to completely change a finished book over to a different format is probably not worth it to you. Write something new instead. If you think someone is going to ask about this particular one, hold off submitting it. If you are a published writer with successful books already, discuss the official line of games with COG.

Edit: I checked the rules and it does say public beta. You would need to talk to COG about whether the entire game has to be open or if you can source enough testers from the forums if the ending could be closed. I wouldn’t sweat it to much either way though and regardless the entire game needs to go through a beta test.

5 Likes
12

I’m glad to see so many responses, and I’m a little surprised to find that nobody else seems to have had any reservations about the open beta.

I was hoping that by sourcing answers to my commonsense concerns (Does it limit my options? Aren’t I poaching my own market?) I’d better understand the process, and that once I understood it, it wouldn’t bother me anymore.

Well, I think I’ve got a good understanding of it now. It’s a sound, logical practice. It may not be the best suited to my particular game, but it’ll still help, and it certainly won’t hurt.

Excellent. We can wash our hands of this issue…

…

Except it still feels, in some inexplicable, undefinable, ineffable way… sorta wrong?

Perhaps it’s just the writing group mentality. I work with my group quite often, and one of our golden rules is ‘descriptive, not prescriptive.’ We listen to our readers when they tell us what they did/didn’t like, but we politely ignore them when they try to suggest how to improve it. I could have something of a conditioned response.

Or maybe it’s the flourish. I quite like the idea of pulling a completed piece out like a rabbit from a hat and saying ‘ta-da, what do you think?’ Perhaps the open beta isn’t entirely appealing because it dilutes that a little.

It could be hubris. Maybe there’s a part of me that’s offended by the suggestion that there’d be any faults in my piece. Putting it out in an open beta means a mental downgrade from ‘perfect–almost complete’ to ‘work in progress,’ which can be a difficult adjustment to swallow.

It’s possibly fear. If I publish it and it flops, it flops. I can believe that it’s perfect right until the moment that it’s out of my hands, and then I can just move on to something else. If it goes up in open beta and is universally reviled I’ll have to engage with the criticism from then until publication.

Maybe it’s possessiveness. I quite like the idea that my game is a little brain child, nestled and nurtured in the open spaces of my cranium. Being asked to share even a small portion of the creative input with a vast sea of unknown faces is a little horrifying.

It could be pretension. Maybe I secretly think that my work is above all you hoi-polloi, and an open beta couldn’t possibly add anything to it.

Whatever the case , I think my reluctance around open beta is more an emotive than a logical reaction at this point. They’re all valid emotions to feel (though some of them aren’t particularly nice), but it would be foolish to let them get in the way of my work, whether it is ultimately successful or not. There’s no real rush, the line edit isn’t scheduled for another few weeks. I’ll take some time to grow accustomed to the idea (whether that’s a day, or a week).

Thanks again, folks, for giving me a better understanding of the dynamics around the open beta. If there are any authors out there who can remember being in the same boat before putting their work out in open beta, I’d absolutely love to hear from you.

5 Likes
13

I am not a good writer , I am still learning English. But when two years ago I tried to do a open beta. They destroy me and I ended in a depression because it happened when my grandmother died by cancer. Barely no one cares about it and the people posting was mostly trolls. Nobody gave me feedback at all. Then I tried a closed one and again absolutely nothing reply back.

During two years I slowly keep working scared to even tell my friends here that I was working on it.

A friend cheer me up to try few months ago. And in a very private betai achieve some people help. It gave humility and understanding of how hard writing is. I discovered that I wrote for myself because I love literature.
I still thinking is a good idea to develop a game that needs **interactivity ** . You as author has a a linear idea in mind a readet would be free of that. And ask questions that open your mind and add new plots.

14

When I first began testing for developers in the gaming world, there were nothing but private tests. Concept-testing, pre-Alpha builds and all sorts of testing all the way to post-release testing for release errors was done in private, often on-site and under non-disclosure agreements and other legal mumbo-jumbo that promised ruin on any that violated the secrets.

The industry has evolved to the point, where as @Lys says, AAA developers are throwing “pre-release” events out into the wild and substituting rote testing and automated checks as a legitimate alternative to the labor and cost intensive tasks of testing. An example just announced the other day by Activision - their franchise IP will be having “pre-release beta playing for pre-order customers” - so this practice is being done industry-wide from the multi-billion dollar companies to the niche developers/publishers like CoG here.

The way that the culture here in the CoG/Hosted community has evolved is a lot healthier and much more conductive to produce actionable feedback and critique that I am more confident in the process here then I am in others.

This doesn’t mean that I advocate the pure open development that @Havenstone has shown can work - I prefer a multi-phased process that includes both small and larger private testing to evolve my project to the point where I’d submit it as you have - at the same time I am not afraid of the mandatory “Hosted” label requirement because I’ve seen the working of that process.

If you follow the link @Havenstone provided and peruse the Diamant Rose project, you will see how an experimental work by an accomplished author here was able to be sampled for a limited time and then given the community’s buy in. The author did not leave the work open for long and the publisher got assurances that there was a viable market for this if they went ahead and published it.

I am sorry to double post in your thread - yet I felt this was important to you so I should address it.

2 Likes
15

First of all, people seem to have missed a critical detail for you. If the game is written in choice script it can only be published under Hosted Games. All other “elsewhere” options must be free of charge unless you come to an arrangement with Choice of Games.

On point 3: There are some atrociously bad games on the hosted game brand - try out some of the earlier games on the hosted games page of the main site and you’ll see why public testing is a hard requirement. That said if you can afford professional proof-reading/editing knock yourself out. I think you’d have to be spectacularly lucky to recoup your costs on that though.

1 Like
17

You know how much think Translation or edit cost? Cost more of the profit I could make in ten years if lucky my friends. Around 80,000 words and I have already more is 6,000 words edition. A translation services could be double that and not a good one. You are saying 18,000 dollars in something that not going produce more than 2,000 in my end. I am broke. And no one work in base of a percentage of an author who never produced anything. And I don’t want writing in Spanish. Why I should? I hardly doubt I will learning English writing on other language. My problems with beta is not being shy at all. It was the way trolls and no one else try it.

3 Likes
18

Thanks @Eiwynn, I’ll check out Diamant Rose. (And I appreciate your perspective on the beta testing side of things.)

Don’t worry, @LordOfLA I know about that requirement. It’s extremely unlikely that I’d end up taking the game elsewhere (though ultimately up to the community and CoG, I suppose), but if I were to do so, I’d port the game to another language. I have yet to find any atrocious games, but I’ll admit that I’ve stayed on the established side of things (Broadsides, Robots, Everything Max Gladstone). I’d be curious to look at the earlier hosted gamebooks as you suggest to get some perspective.

2 Likes
19

I’m surprised they teach that to be honest. (That’s not what I’ve been taught which is more along the lines of have your own ideas of how the story is to progress but listen to feedback and decide whether to act on it. It’s not to say you shouldn’t have your own work and style, but there’s nothing wrong with listening to what people want. You’re talking about making it into a commercial product after all rather than just a personal piece. If the advice doesn’t fit your story line, you’re under no compulsion to fix it in that exact way, but you still know it needs to be changed. That’s especially true if you have multiple people telling you the same thing, you should definitely listen. If that’s the case you need to justify why you can ignore the advice rather than the other way around. It can sometimes also give you some great ideas for story lines and character interactions. Sometimes people will see my characters or the MC in quite a different but still very valid light to me.

I find IF to be quite different to standard novels where you can get away with less feedback in my opinion because they’re your characters, acting how you decide the story is going to play out. IF is interactive, people want to be able to have a particular amount of freedom which testers can pick up for you. For example you may have decided the story needs to go in a particular direction but a lot of readers hate that words are being put into mouth so to speak. A few extra choices or a bit of rewording to show why something has to be that way will result in a much happier audience long term.

Anyway, good luck with it :slight_smile:

7 Likes
20

TL;DR @Doug
Listen to any comments, suggestions, and criticisms. But you as the writer/developer still hold the wheel to decide how you want your story goes.

1 Like
21

Just wanted to affirm that your concerns are commonsense ones, and this isn’t the first time they’ve been raised. Which is probably why so many people are quick to whip out the things we’ve said before, especially me, Eric, and Eiwynn. :slight_smile:

We had a similar conversation a couple of years ago, when the requirement was still new and the open/closed issue more negotiable:

And since then:

There’s some on it here:

This thread had some derails, but here’s a post from an established HG author who doesn’t like open beta tests:

And finally, regarding CoG “crowdsourcing” HG proofreading and editing, here’s a recent post from Jason that I believe @Lys may have had in mind above:

8 Likes