On Beta Testing

Game Development
22

When I have the time, I’ll try to play through WIPs. I’ll point out a few specific things, but mainly just be encouraging and move on. Beta testing is different - I’ll send in documents full of typos, errors, suggestions, etc., with only the occasional compliment to remind the potentially-harried author I wouldn’t be sinking in so much time if I didn’t believe in the project.

I haven’t gotten to a full beta yet for my WIP, but the play testing as I write has been hugely helpful. It’s a story with a lot of choices and a focus on immersion, so if a section is boring, impossible to navigate, or not realistic, I can get feedback before it’s too late to change it. Plus it helps with the really bizarre bugs that quicktest doesn’t get, like the one with Unknown potentially showing up in your bathtub.

One thing I don’t really like during testing feedback is using persuasive arguments. If something makes you uncomfortable, or you love/hate a character, or a choice isn’t working for you, tell the author. You should trust that they’re listening, and you don’t have to write a speech to get their attention. In particular, debating to convince other testers to support an opinion of how the game should be changed seems counter-productive.

9 Likes
23

@Sashira

I think your first point is what I find myself doing.

When I read WiPs I’m not generally looking to nitpick people, so I comment on a few small or personal issues (Like wanting to pick six favorite flavors of ice cream!) rather than writing up huge long lists of typos and errors.

But i dunno if that’s actually what people want from readers. I feel like I see more authors saying “please point out typos” than I do anything else. But, given my news background, I also know that for many people there is nothing more demoralizing than a list of technically errors that scrolls on for pages and pages.

4 Likes
24

@Shockbolt As to striking a good tone in giving feedback - I may not be the best person to answer this, but I try to keep in mind that I’m making suggestions. My typo list doesn’t have much of a tone, I don’t think. It just looks like this (and much like a lot of others’ typo lists on forum WiPs):

(5). The sparkling pink vampire ponies leapt across teh* fast-moving crystal river.
- *the

[N.B. not a real line, nor from an actual beta test.]

When making a high-level suggestion about plot, characters, or mechanics, I try to use a question format when I can: “Do you think a ‘maybe’ option would be possible here, instead of just shades of ‘yes’ and ‘no’? I would have liked to take a more neutral position.”

I can only suggest what I think, and sometimes I do include a brief reason why. “I would have liked an option to tell Captain Thatfellow that I don’t mind if he sticks around, because I want to maintain a relationship without either alienating him or jumping him.”

With editing, I do address grammar of course, which is not looked for in Choice of Games beta tests (WiPs on the forums vary in this regard, I gather?). @Samuel_H_Young and @Nocturnal_Stillness, the only two ChoiceScript authors whose works I have edited, could probably answer questions about niceness levels with editing vs. better than I could.

Even with editing, though, I try to make a point of saying, ‘I would recommend this but style guides vary’ instead of just basic corrections without notation, if an issue could really go either way.

And any narrative or character choice is going to be a suggestion, to my mind. If I feel strongly about an issue when editing, I’ll say so. If I feel strongly about it when beta testing, I’ll just say something like, ‘This section broke my immersion because of Y and Z reasons’. Then the author can decide what to do about it, or whether to do anything at all. Does that sort of answer your question?

@WaterOracle Even the ‘this is wrong’ issues are still at the discretion of the author, though. Beta testers do not control the game. An editor or the publisher would be the one to take a stand on things, to my mind, and not beta testers. I liked @Sashira’s comment about not trying to convince the author with a paragraph of arguments as to why one is right.

@P_Chikiamco I really appreciate your point about weighting issues properly; that’s helpful. Thanks! Could you share an item or two of especially helpful feedback from Slammed!, if you don’t mind? And why was it exceptionally helpful - what changed, how it improved the game overall, anything like that?

@OScott That’s hard for me too because I have to not-edit! I always look at issues I spot and think, "Is this an ACTUAL TYPO that even my word processing program would spit back out, or something like swapped genders? *if (yes), *goto make a note. *if (no - there’s any amount of room for debate), *goto skipit. I think twice I’ve given actual grammar feedback, and only because I found it terribly distracting and I wasn’t sure it was intentional.

Sometimes ‘bad’ grammar IS intentional, too, or a result of house preferences (don’t get me started on ellipses). So reporting grammar errors isn’t helpful because beta testers would be talking over each other, and then the editor, house, and author are making the style decisions anyhow.

Besides, editors don’t have trouble catching poor grammar. Certain typos or continuity errors, though . . . I don’t know how other ChoiceScript editors work, but I look at the entire manuscript, code and all. I’m not playing the game, so I might not notice if a sentence fragment in line 120 needs the word ‘and’ to properly meld with the other half of the sentence down in line 258.

Another issue is that the writer and the editor sees ALL of the plot and character information. I’ve seen a few games critiqued because readers felt like the plot had holes, or the characters were shallow. The information was there in the story, somewhere, but the reader never saw it, and wasn’t inclined to do a replay after an unsatisfying game. The editor and the writer have already seen every single line, so it’s harder to tell if key information is missing from certain routes. Beta testers would notice, though!

(@Shockbolt if I DO have a huge list of typos and errors lurking in my prosey halls, please at least tell me they’re there so I can try to hunt them down . . .)

I also don’t think it’s any one beta tester’s job, as @P_Tigras pointed out on another thread, to ‘fix’ games. Lots of beta testers noticing different small things is the way to go - especially in a game, where the beta testers may only be seeing 30% of the game per run, and all shuffled together in different ways.

11 Likes
25

With beta testing I want readers to be honest tell me what works and what doesn’t. Which characters you like or dislike and tell me why. Critiques are important to help improve a writers ability.

11 Likes
26

Hm, it’s been a while. One thing I do remember changing because of @FairyGodfeather was allowing the player to tell Ecstasy to wait when Ecstasy confesses. FG pointed out that in some play throughs Ecstasy would have done the exact same thing to the PC, and it would make sense from both a game standpoint and a real life stand point if the PC had the option to do that as well.

6 Likes
27

It’s been a long time since I’ve tested any games so I’m trying my best to remember my process.

Firstly, I think it’s vitally important to tell the author what I do enjoy about a game, and why it works for me, as well as what doesn’t work. I try to be as constructive as I can. I beta-test games for the author’s benefit. I don’t expect them to change anything about the game, and I’m often surprised when they do. Everything is suggestions, the author can take them or leave them. In fact if they’re not going to follow my suggestions, I’d rather just not hear, I don’t need them to explain why. It’s their story after all.

I get responses from some authors and not from others. I do like it when authors respond, although there has been a couple of times I’ve asked them not to. Generally when I’m uncertain of the feedback, or when it feels very personal, and when my anxiety’s running wild.

I try to be extremely thorough in my feedback. I’ll generally play through a game a couple of times, depending on the length of the game, and then run through the code and add in any comments that arise from that.

I try to note down the time I start playing a game, and write down when I finish, just because I think the author might be curious as to how long it takes me to complete a playthrough. Mind you I don’t always remember to do this, and I’ll often be slower in playing a game when I’m testing it.

I do a screenshot of my stats at the finish, or just copy and paste them into my testing feedback, since I think maybe the author’s going to be curious as to how I played.

If I’m testing the game on my tablet, I’m generally a lot faster. I’ll screenshot at points I want to take note of, I’ll run through the game and then write down my impressions of it. Then I’ll likely go over everything again in more depth.

If I’m testing on my desk-top I’ll often split the screen, so I have a notepad window on one side, and the game on the other. That way I can easily take notes as I play.

Sometimes I write stream of consciousness things while I play, giving my impressions on the game, just so the author can get the idea of what a player might think.

I’ll generally make note if there’s a section with choices where I’d like an additional choice. Or if nothing seems to fit my stats. Or if something unusual or unexpected happens as a result of my choices. I comment on what works for me with the story, and what doesn’t. I’ll point out any continuity errors. I’ll mention if anything about my stats seems wonky, if something seems too difficult. I keep an eye out for pronoun problems, since that’s often a code glitch. Also typos, although I don’t focus on those, if I catch them I’ll note them down though. If paragraphs seem to run together in weird ways I’ll mention that too, since that’s often a code issue too.

If I end up dying, especially unexpectantly, I will often rant off paragraphs to the poor author complaining. :slight_smile: But that usually means I’m at least invested in the game.

I often let them know what I think of various NPCs, especially love interests.

To use Slammed! as an example. In Slammed, there’s a point where Ecstasy, who has been flirting with you, rejects your advances and says it’s not the right time. Then, the night before one of your super-important fights, when a lot is going on, Ecstasy tries to start things up with you again. If you chose to tell them no, Ecstasy disappeared and went and got engaged to someone else. I found that really, really frustrating, especially after everything we’d been through. I pointed out Ecstasy was being a hypocrite, and Paolo went and added in another scene, making it so that you could ask Ecstasy to wait, and it is one of my favourites of the game and the way I prefer playing the Ecstasy romance. I also had a lot of comments to make about the JJ romance, which also made it into the game.

I’m likely missing out things. Mostly I add in as much as I think seems relevant, and likely an awful lot that isn’t. I don’t expect the author to change anything based on my feedback, I just want to give them some insight.

17 Likes
28

I actually disagree that’s what’s most demoralizing. I think people would rather get that, than have no feedback on a WIP. Nothing more likely to kill off a WIP than thinking no-one is reading it IMO anyway :slight_smile: Getting pages of feedback shows someone cared enough about the story to put the time into writing it down.

Guess it depends on what each author is looking for though. Some will helpfully state what it is (ie mainly typos, pacing, anything goes). I can see if it’s not stated why you’d tread carefully with the amount of crit though. I often end up doing the same myself.

6 Likes
29

Personally what I really want to get is feedback on the story itself, characters, flow of the story, the coherence if it still makes any sense, interactions of the MC to the NPCs and the pacing. Not just my occasional spelling or grammar mistakes. (I really can’t avoid them, but I think I’ll find some way to get use to using English.)

I think the hard part about beta testing is really committing to the testing part. Some just, well, I don’t know, just play the game once and then they are done. No comment or suggestions or questions. Some even go AWOL. That is kinda sad and frustrating for the writer since how can the writer even know if the story is good or not. I think giving the tester a list on what the writer wants can help? Maybe?

4 Likes
30

I’d like to add to this conversation by commenting on this.

A writer should give general direction on “focused testing” and let the testers provide generally unsupervised feedback on general testing. What I mean by this is that there are two basic tiers of testing - tier one is to gauge interest and to get a general sense of what the WiP will bring to the table. Tier two is the focused testing and the reason it is focused is because the author is looking for specific things.

What those specific things are is unknown to the tester unless it is communicated to them. When I test Zombie Exodus, I talk with @JimD and he tells me, please test a, b, c and d. Then when my feedback is given it is focused on those subjects and he can take it from there.

I was really off-put by a recent testing I was part of because the lack of communication made me feel like I was doing nothing but previewing an author’s game. Low-level feedback was not wanted but how can I give high level feedback when I do not know the author’s direction, or focus for the testing?

An editor will bridge the two with mid-level feedback but at the stage where most WiP are the editor is a non-realistic option and expense.

I love testing, I’ve been doing it since I was 9 years old and have tested software, hardware and products for all sorts of companies ranging from Microsoft to software companies that publish AAA graphic types of games - each coder/programmer/author has their individual needs and desires - but I don’t know those until actually interacting with them.

One final piece of advise: Whoever you are testing for needs to hear honesty and straight-forward feedback. Even if they chose to ignore you or to even disagree with you. If your feedback later proves correct, then they will listen to you in the future and they will trust you to tell them - even if you disagree with them, they will learn to love you (even if they hate you)

Testing can be very trying and sometimes is not fun but if done right you will find it very rewarding.

10 Likes
31

O.K. I will make an additional comment here. When testing, I generally will play the same character build on every submitted update. An author needs both types of testing; @Fiogan will catch things I will not but I will be able to communicate with the author if their changes are what they were looking for.

When the author specifies a change in character, then I will do so. Again, this really is dependent on what the author’s needs and desires are - if you need to find out if a romance option is working, if no one tests that option how are you going to get the feedback you need? By directing your tester.

ok. Back to life’s needs. ttyl.

3 Likes
32

Something I just remembered.

If a game author doesn’t want a player to look through the code, I’d really appreciate being told so. Looking at the code provides a completely different insight into games, and a different experience from not looking at the code. It makes it far more obvious what are false choices, for instance, and where there’s railroads.

I also really appreciate a change-log between game drafts. That way I know what’s been changed and where to focus my attention and where not to. I find that the more times I play a game, the less effective my feedback becomes as I get muddled between drafts.

11 Likes
33

This! I recently helped to test a CoG for the first time, and if I’d have known how to review the code i absolutely would have. Recently learnt how to view a CoG game’s code though, and wondered if it would have been ethical / in line with CoG’s expectations if i’d have done so during testing.

2 Likes
34

[quote=“FairyGodfeather, post:32, topic:16584, full:true”]
Something I just remembered.

If a game author doesn’t want a player to look through the code, I’d really appreciate being told so. Looking at the code provides a completely different insight into games, and a different experience from not looking at the code. It makes it far more obvious what are false choices, for instance, and where there’s railroads. [/quote]

I have yet to encounter an author who doesn’t appreciate being shown the exact line on which a game bug is located along with a brief explanation. That level of specificity can save an author a great deal of time, but is impossible without reading through the code.

This. After doing a very thorough read-through and review of a game, I don’t generally have the time or energy to do another when I don’t know what’s been changed. I’m much more inclined to put additional time into going through newer revisions when the author tells me what’s changed and/or where to focus my attention. Otherwise things tend to get muddled in our minds as @FairyGodfeather has said.

9 Likes
35

I’ve tested a couple of games where the author changed something only on a specific branch, but didn’t tell us what branch it was. They told the testers after we asked, but it would be more helpful if they just provided a change log up front. It’s also useful if the changes they made involve subtle things like hidden stats and variables that aren’t immediately noticeable.

5 Likes
36

I’ve surprised authors in the past by providing feedback on the code of their games since they didn’t know it was even possible. There’s one instance that I recall, several years ago now, where I really upset the author by looking at the code and it made me feel awful. I’ve also known authors who’ve put things into their code files, and tried to make it more difficult to read. So, I think it’s at least worth a mention.

4 Likes
37

Given that even published games can be opened up and looked at without difficulty, and guides for looking at code can be found in the threads of this CoG-owned forum, including a guide written by one of the mods, I think authors should be aware that if they put their game out there, either as a WIP or a published game, there will be people who will read their code. If there is a problem, I believe it’s that some of the less computer-savvy authors aren’t aware of this, mistakenly thinking their code isn’t viewable, and as a result are caught by surprise when someone freely mentions that they took a look.

2 Likes
38

While every fact you lay forth is 100% correct, I have run into coders/programmers/authors who become very possessive of their work. Even if technically allowed viewing and even using their code become something personal. It is one reason I always ask people’s permission here to view their code and why I ask permission, if I find something useful to me, to use what they did in my work.

I am a very novice choice-script author, so anyone looking at my code will know more than I - and a younger me would have been insecure in that knowledge… now I know even the best can learn new tricks (both on a sub-conscious and conscious level) but back then I might have reacted more poorly.

Since we are talking from the tester point of view - to successfully get your feedback through it may take finesse - and that is where @FairyGodfeather is approaching all this from I feel.

8 Likes
39

It’s why I phrased things the way I did. I’m testing to help an author. If the author does, or doesn’t want me to do a specific thing, I’d rather they said. If the idea of me looking at their code makes them uncomfortable, I’d rather they said so.

You can look at the code and cheat at the game, for instance, always knowing what are the right choices to make. (Not that there should be right choices mind you). I do sometimes do that when playing, particularly in games where failure comes at a high price, or where you can die, or if I get stuck at a point, or just when I get anxious.

I generally mention I do this though, usually after complaining about a section being tricky. I do find looking at code, even if it’s a quick glance, changes how I play a game, which in turn changes the sort of feedback I give, so I generally don’t do it on the first playthrough. I do like looking at code though.

6 Likes
40

I sometimes do and sometimes don’t when playing a game. When testing, I will only do so if directed.

In the situations where there is unusually tough complexity or if there are lose-lose choices only available, I will usually phrase my feedback as: “in this section I felt very backed into a corner with my choices, may I suggest you look at the available choices again or if you prefer, perhaps I can look at your code to determine what is bothering me” …

It is amazing how very alike in generalities veteran testers are, yet nuances make us all different.

2 Likes
41

Well, I am very different in my way of beta testing. I am looking it from the perspective of a buyer. I imagine like It is released an I just bought it. How Do I feel? Is the flow good? Do I feel railroad? Could I customize my character? Are good romances? Are some plot hole in the game from my point of view?
And Then What would I like to see . Like date X character or just talk about y with a friend. Or maybe just select our pet name instead of Bogard. I never look code , because I role-playing look at it would kill game forever for me. I look more the emotional side than the professional and cold facts.

4 Likes