On Beta Testing

Game Development
1

Several people (@Mirabella, @FairyGodfeather, and others) mentioned an interest in a thread to discuss beta testing, beta testers, and authors’ thoughts on the process. So here we are!

First things first: CoG’s post on beta testing has lots of helpful guidelines.

And now, to answer @HalfOfLancelot’s questions to me taken from this thread (and sorry for the wait!), and also some from @Shockbolt.

Some authors do, some don’t! One author in particular - just the one - responded to my feedback point-by-point. That was so much fun that I’ll happily test for that author till the sun rises in the west. Not everyone has the ability or desire to do that though, of course!

Other authors have occasionally sent me a little thank-you note. Some authors I haven’t heard from at all. In that last category, sometimes I’ve noticed the change log includes every single suggestion I’ve made (that was kind of amusing). Other times I haven’t noticed any of my suggestions being implemented, outside maybe the typos, across several drafts.

I do have to be careful because I’m an editor, and editing-brain is not the same thing as beta-brain. I try to focus on typos, anything I found immersion-breaking, and suggestions that make it easier to role-play the character I envision.

As for my perspective as a (read: developing, hopeful) author - I appreciate all of the positive feedback, because it encourages me to keep going on a very long game. But as to what was most helpful? So far, several wonderful people have spotted typos and continuity errors, and that was amazing and brilliant! That’d be the low-level feedback mentioned in the CoG post above.

Then there are two specific pieces of high-level feedback - plot, content, characters - that were extremely helpful to me. One was a suggestion to add an extra level of mechanics to one scene (letting people pick extra ice cream flavours, of all things, and it’s turned out to be a very popular addition!). The other was when someone outright told me one scene was very boring. That person, after some thought, told me why the scene felt boring - not enough plot, too much fussy automaton, and not enough action by the MC. I’m slowly working through it now, addressing those issues, and I feel like the scene is much stronger for the additions.

That said, I’m really relatively recent to Choice of Games compared to many on the forums. And I’ve yet to publish a piece of interactive fiction. I’d love to hear the perspectives of some other beta testers and authors!

19 Likes
3

Someone should make a Beta Tester Fill Out Form for CoG stories

1 Like
4

Would that really make a difference it seems like more hassle then it’s worth.

5

Standardization is only a bad thing with tests. Of course, there’s always the option to ignore it but it would make being new to testing easier as they would know what is expected.

6

Jason basically tells anyone who wants to beta test what their looking for before it starts.

7

To add structure, you mean? It might be really difficult to get a form to apply to the different genres and coding styles of games except in the most general way. A slightly art-nouveau mafia game with a a lot of violence and a touch of sci-fi and the occult will be very different to test than a spooky estate agent game, for instance, or a middle-school aimed game.

The link in my first post to CoG’s beta test instructions does have a general description of what kinds of feedback is useful, though! Did you mean something like that?

And then there are the different styles of beta testing. I hadn’t thought to mention it, but when I beta test I always use four categories, in this order: 1. Things I Liked; 2. Typos and Continuity [i.e., low-level feedback]; 3. Plot and Characters [i.e., high-level feedback]; 4. General Thoughts.

I go through and make a numbered list under each category, adding points as I notice things. Then at the end I give a few general reactions under that last header.

Depending on the length of the game, one play-through with that style of feedback can take me anywhere from 30 minutes for a very short game to seven hours for a long and complex one. (That’s my record so far . . . long game, lots of comments.) So I usually play through once per build, but I pay a lot of attention to the details. Next build, I make a new and different character, so I won’t necessarily see changes that only applied to my previous character build.

I just sort of made that up though so I’d have some structure. I have no idea what the other beta testers do, but I imagine we’ve each very different ways of approaching the beta process.

4 Likes
8

They’re* And like I said, optional. It’d be useful to both new testers and new writers.

9

What is some advice you have for striking the right tone in giving feedback?

I find that I sometimes come off a lot harsher then I mean when I give anything more than casual feedback.

When I interned for online news if I didn’t literally shout and threaten people they wouldn’t do anything outside of my mandatory AP style edits, and that’s bleed over into how I tend to give serious feedback.

It’s less that I intent to be terse with people so much as “short, direct, and to the point” are what is generally considered the standard in my area of the news business.

I don’t think that’s generally what the norm is in literary editing, and so I’m curious how people find a good balance between “professional editor” voice and “this is a friendly suggestion” voice.

For example, I can think of several times when I’ve basically told writers go completely overhaul news stories because of stuctural problems – changes that while they aren’t factual grammar edits they aren’t actually just my opinion either, they’re changes that the stylebook literally requires.

They weren’t “suggestions” they were errors, and whenever I ‘asked’ people to change them instead of ‘telling’ them the changes didn’t get made even when they weren’t optional.

While CoG doesn’t have too many absolute rules, there are times when there are things that aren’t a matter of opinion that need to be pointed out.

So I mean, I think what I’m trying to get at is does anybody have a good way to convey the difference between say “I think you should change the labels on the stat screen because this is my opinion” and “you need to change the labels the stats screen because you’re using a word incorrectly” with out basically stating every block of feedback if it’s opinion or you’re pointing out a factual error.

Edit: @Fiogan There’s a whole field of study dedicated to examining how internal biases effect the reviewing process. I can PM you some links if you’re intrested.

6 Likes
10

Piggybacking off of @Shockbolt’s post, but in the other direction.

I feel like I have the same problem with the concept of “tone”.

I mean, spelling and grammar errors are easy enough to correct because if they used the wrong “there/they’re/their” than its a matter of fact, but sometimes I feel like I’m over stepping my boundaries as a beta tester when I notice something is off context wise and comment on it.

How do you properly (and politely) draw attention to something that really needs closer examination, without sounding: “This is wrong. You did wrong” about it?

4 Likes
11

I didn’t know what CoG looks for in beta testers. So thank you for posting that.

Personally, I’ve never done closed beta testing, only open betas here on the forum. I search for bugs and errors first, but after that, my approach is to see how engaging, entertaining, and professional the game is. Basically, what the overall first impression is. If it’s lacking in any of those ares, I try to isolate what is negatively influencing the impression. If the first impression is good, then start playing different routes.

4 Likes
12

I really do not beta test much anymore (well except for the open WIP betas) for the reason I work better with a little feedback, this way you can work in tandem with author know what they are focusing on, and it makes it easier for me to notice other things while doing the normal low-high feedback.

One of the best authors I had the privileged to work with with was @Lucid he welcomes feedback and you can see the immediate responses in his work and comments. One that I more thank liked mucked up is Sergi’s and prolly made poor @jasonstevanhill pull out his hair and to beat me like a red-headed step-child. So I would suggest viewing this forum first for all prospective beta testers:

also

Look at the requirements for high and low level feedback this is what will be required of you, mid level is the what editors like @Fiogan get hired for, so know what you are getting into before you say pick me pick me! Also know being a beta tester for CoG is a privilege and a job your input makes all these games better. We all have seen a game or two in release that was buggy as all get go, that fault lies with the beta testers, bugs can not get squashed without being reported.

5 Likes
13

Well, for Slammed we did a beta that was very helpful. @FairyGodfeather can attest to the changes I put in based on feedback (and good to see you around again FG ^_^) We did one for a currently tabled game as well, and I am going to badly need one for Love and Lasers just because of how different it is from the usual CoG. What would you like to know from a game author’s perspective?

4 Likes
14

…what I’m gathering from this thread is that I probably shouldn’t volunteer to beta test a CoG because I can’t not jump on “mid level stuff.”

Have any of you found an effective way of… repressing, I suppose, that sort of inclination?

Probably a silly question, but is there such a thing as too much feedback?

2 Likes
15

Only if it means you don’t pass on the feedback by the deadline. Even if you write reams and reams of comments, no one is going to force the author to read through all of them. Some of them will, some will pick and choose.

That being said, being a good beta reader means trying to give proper weight to things. If the author is struggling with pacing, focus on that, rather than, I don’t know, writing a treatise on British vs American spelling conventions. :slight_smile:

9 Likes
16

When I beta I usually start with my general feeling what emotions give me the game.
Second If I found the flow and the plot with sense.
Then I explain the type of character I play, and if the game left room from customitation and some role playing stuff. if story felt railroad or not, characters…
Finally I give my ideas on how improve the game or just adding minor stuff to give flavor.
I can’t give grammar advice or edition stuff so I focused on the story itself and stats if are balanced or not.
Authors normally are very nice and easy to talk with.

3 Likes
17

Very good to know! I appreciate this. I’ve been wanting to help and do a bit of this, but life and kids tend to keep me bogged down. My forte is definitely more editing. I always wondered what sort of feedback authors found most helpful in their work.

2 Likes
18

Both Editing and plot feedback are important. Each of us should focus on what we are best. I being not native is futile attempt to edit something, it would end worse lol.

2 Likes
19

I’ve beta tested a few of the CoG, and I think they have internal readers to thoroughly edit for tone and theme. What I think they need when they ask the forum seems to be a larger number of people to comb through the game and try to break it with regard to continuity from choices and flush out remaining pesky typos. Whenever you release something to the public, the glitches will become immediately apparent (Like the recent Darksouls having a glitch at the very first checkpoint) so that is what they are trying to simulate by having 20-50 (?) extra people test it. They don’t need quality or artistic feedback at that point unless it’s something completely wrong…such as a low possibility branch that mentions a character has three eyes where it’s not been mentioned anywhere else.

3 Likes
20

Oh, no, I get that. I’m curious as to how people who have the same tendency as I to jump on everything are able to ignore that urge and focus solely on the low/high level stuff that CoG is looking for.

21

Usually I get wrapped up like I’m just reading the story and don’t have many comments - until the thing crashes or does something completely wrong. The CoG ones are usually quite polished by the time they get to forum beta testers. I actually never made it to the end of Hollywood Visionary because I couldn’t get past a certain junction.