A little note on emails, in Jason’s request for runt of the litter the email address was Jason at choiceofgames . It might be just me but should there not be something at the end like .com ?
Me being from the UK would naturally add .co.uk
Or am I missing something?
Yes. It’s jason at choiceofgames dot com.
Note I have to spell the period and the at sign because we don’t want spam crawlers picking up that address.
I did wonder, thank you.
When I did my closed beta last summer, I got emails from the CoG editors full of comments. I don’t remember if they had the beta tester’s name and email attached–they may well have been, but I’m not sure. At any rate–and this was my failing–I understood the beta test information as an information flow from the CoG editors to me, and then me back to them with the fixes. I didn’t know whether trying to contact a beta tester was frowned upon, or desired.
I wonder, if, for example, someone sent feedback that said “this love interest is annoying, not at all attractive” and I wrote back, “I totally see what you mean, but it’s too much to fix so close to release” if that would be good, or even more annoying. I think thanks and appreciation are the key issues. I know this for sure: I am not going to think about the closed beta the same way next time around. Of course, now I internet-know you, and I have for months! So if I know something is coming from you, it’s different from an email coming out of the void.
I think your point about not wanting to provide feedback in a group setting is a really important one. I am the same way. I think a group setting for beta could run the risk of groupthink, and you wouldn’t be able to stop conversation at any specific point. I guess the danger for me as a writer is that I wouldn’t like to feel like my game was in constant beta. Then again, that seems to work really well for lots of Hosted Games, as far as I can tell.
Do you think CoG would consider involving testers in an earlier stage of the process, perhaps at the end of the first draft stage or perhaps even as the first draft is devloped? Perhaps it would be more conducive to incorporating more “high level” feedback at that earlier stage?
I realize CoG doesn’t want to overwhelm its official writers with tons of feedback from tons of people, but there are only what, six folks on staff? It seems to me as the company scales with more and more releases, you might need more volunteer help?
Perhaps a dozen or so testers with strong track records could be recruited to help test a story as its developed, thus giving the testers a stronger attachment to the story?
I don’t really use beta testers (as I test my games in the forum through WIP, which is a very rewarding experience to myself, and I hope I haven’t attracted the “not listen badge yet”… I hope I explain all my decisions, and try always as best as possible to incorporate suggestions, even though in some instances it might be impossible due to “structural” elements), so my suggestion/proposal might be out of place:
Maybe it would be possible for CoG to recruit a small number of “reliable beta testers” at the start or half way through the writing of the game, who provide feedback to the author as he/she is writing it (similar to how the WIPs work on the forum, though privately). In this way the author might be able to realize any major structural problems/issues with stats, etc. This can be complementary to having a full beta test at the end. Having a small number of committed beta testers from the start could help improve communication/satisfaction, though I can also appreciate that not all authors like to work this way (I’ve come to think that for my style its better writing about half the game before showing it to others, but I know many authors feel that showing something before its finished affects their writing).
Anyway, just something to consider, that could be offered by CoG to “external” authors who are not so familiar with the philosophy around CoG and Hosted Games.
DAMN, @Eric_Moser, you beat me by a few seconds to post this!!! 
Damn, @adrao, great minds!! (I suppose that’s subjective). 
Not really, no. The complaint is that no one is communicating with them. Not the author, not other testers. Their efforts exist in a vacuum and it feels unfun and pointless. And if they’re not getting anything out of the experience except free access to a game they’re going to buy anyway, then why bother?
I was worried my last post was too negative.
I absolutely love beta-testing and I’m so sad that my concentration and anxiety do not allow me to do it anymore. I miss it.
I loved being part of the creative process and getting a glimpse at how games are made. I loved being able to help others, to provide feedback that others might find useful.
I loved being able to discuss how to make games more inclusive and offer suggestions.
I love what Choice of Games stand for and I really enjoyed being able to help, even if it was in such a small way.
And it was always nice to hear back from an author, if just to say thanks for the feedback.
That’s why all us are here @FairyGodfeather If not we just move on and go elsewhere. there are many similar companies out there with forums and asking for beta testers. But Cog is special, or at least it was. I think sometimes Cog staff forget with the pressure of projects and survive being a small fish in a big pond How special and important there are for many people, hom important are for me. That is why I am so passionate about all and try hard helping and say what is wrong, I do it to helping fix stuff, to try to return something for the place I love. Sadly, normally people misunderstanding me and think I am just a troll.
A multi-phase testing period with different goals during each phase is important. Choosing a testing team is hard as well. That is why developing relationship with testers is a very important but very underappreciated task that authors and developers need to do.
I’ve tested games for a very long time. I’ve known, as friends, developers for years. When a developer starts a project they know me, know what I am good at testing and what I’m not very good at testing. So, when someone I know says: “Hey, I have this project I’m working on,” both of us know what the other is bringing to the table.
This may not be viable at this time, but the WiP forum is a very valuable screening tool -if it is utilized by the author. You’re absolutely correct, time, effort and energy is going to be expended by everyone to get that done - but if you want a product tested right - it is going to need to be done.
I know some people are gun-shy about public or group feedback and if you provide an email or PM source where they can give the private feedback then you can have them in your group testing as well.
As far as wanting the same people - maybe. It depends on everyone’s strengths. Someone might be very good at picking up continuity errors and logical break-downs and so you want them in on the testing early. Another may be really good at misspellings and grammar and so bringing them in later might be a good idea.
Also, if your story is a historical setting and deep in politics or military jumk, then observe the forum users in @Cataphrak or @Goshman’s threads. Both of those authors have a hard-core following that would be wonderful for your project… if you want more fanciful, bright 18th century type of story telling, look at @jeantown or @Fiogan’s threads… their fans love the good story set in mystery and mystic.
Sending blind feedback to an email addy not knowing anything about the entire process - schedule, testing needs, goals and wants is not a very productive way of conducting testing. I’m sorry, it is not.
I’d be happy to share my views and experience but I don’t need to be lectured or preached at either.
@H2O’s post put me on the defensive (hence why I did not respond to it) - because they seem to believe I want a golden lollypop and the keys to the Chocolate Factory in order to participate in testing.
Not at all.
Your posting is rarely negative and if it is, that is because a negative needs to be discussed.
I am sorry your not able to give feedback as you once were because [quote=“FairyGodfeather, post:30, topic:22357”]
I loved being part of the creative process and getting a glimpse at how games are made. I loved being able to help others, to provide feedback that others might find useful.
I loved being able to discuss how to make games more inclusive and offer suggestions.
[/quote]
All of that comes through in everything you do. One of the best gifts I got on my Helvetti project was your feedback and I will never forget the kind words and the feedback you gave which helped me a lot.
Anyone getting feedback from @FairyGodfeather will understand - it is on point, positive and encouraging. Even the “negative stuff”.
What are CoGs offical requirements for authors incorporating beta testing feedback?
So basically, making the closed beta more “open” in a way?
I think it would be ideal to get an author’s perspective on these matters as well, since in order to make a significant change in the beta testing process, opinions on both sides needs to be considered.
I apologize if I came off as accusatory or abrasive in my opinion, but I don’t believe there is anything wrong with wanting a reward for your efforts. Most beta-testers are volunteers, so providing some kind of incentive is a good way to keep them as beta-testers in the future.
Nobody does anything without expecting some kind of return, even if the return is just a simple “thank you”.
So I have nothing but respect for beta-testers, because as it is now, I will most likely stay just as a consumer.
In fact I have beta tested a lot before the rule only one beta at time that is impossible know when a beta ended so I ended not presenting acess to 90% of them because no clue if the beta I had applied months ago has ended And I don’t want being banned for going against rules.
However I have to say that I have seen many of my feedback used so That’s not my problem at all. In fact, when I had issues with games I didn’t beta testing but had in my opinion disgusting content as suggested rape, they were tremendously supportive and changed the scene right away.
It is a communication problem, If you’re testing something you want be useful, that is the point. But you don’t feel useful if you don’t know anything.
If they just send a email saying update we want feedback on regard X character. Or the new content is in that route. I would play that and try to focus in that content. My insight could be used or not but probably they would receive info more focused in what they want. Not let us blindness and in silence.
Like is now I just send 3 mails right away with feedback and wait if I win the lottery and someone says something. If not I would play the rest of beta in silence except bugs or something BIG BAD like rape incest or something like that … The rest even if I think that x new scene is terrible and is sexist or provide plot holes. I stay mute they DON’T WANT KNOW MY FEEDBACK so why shouldI spamming a mute cold mail.?
If they do a little forum closed beta … Only with the update info and that within even real contact with author I will posted 100% more feedback. It could be used or not but I really think more is better for them. I don’t know they hate my ideas and myself so just my two cents.
A peak behind the curtain, as it were.
It entirely possible that a number of the issues we’re having are a result of bad habits that I’ve fallen into. But perhaps I can give some insight into circumstances, as well as competing interests, and then find a solution that more people will be happy with.
In the past, there would sometimes be as many as forty-two beta-testers (Pendragon Rising, as a random example). Testers apply, I send them the link. They send feedback. I skim it, make sure there isn’t anything that I substantively disagree with, give it a “grade” based on its “depth,” and then send it to the author. The author sends back updates to the files, which I post to the beta. I post an “updated draft” on the beta thread. This continues until we move into Continuity Testing.
Continuity Testing is where I use the Randomtest tool to produce randomly generated playthroughs of the game. I send “seeds” out to readers (who read the playthrough like it was a novel) and look for continuity errors and other problems. Then send the seeds back to me, I send them on to the author. The author sends me an updated draft. I run new seeds, and send them out to the readers.
Once the seeds come back “clean,” we send the game to copyedit. Then, if someone applies to the beta after the game has gone to copyedit, I go and lock the beta thread; otherwise, I just let it drop into oblivion.
The beta doesn’t officially end, I suppose, until the game goes to copyedit. The beta thread’s “pin” expires after a week, but each time I post “new draft updated,” it gets pushed to the top, where it will soon drop again considering the amount of activity we have on the forums. But, it’s notable that, if someone sends in more beta feedback during the continuity phase, it gets passed on to the author all the same.
So, to Mara’s point about “you don’t tell us when the beta is over,” you’re right, I don’t. Because I didn’t know that it mattered. To me, when I’ve stopped posting the “new draft” posts, I’m going to mostly stop getting access emails. And that’s the functional end of the beta. Again, though, if someone sends in feedback, it’ll still get sent on. It’s only when someone applies after the game has gone to copyedit that I officially lock the thread, and that may be a more definitive “the beta is over” moment. Or, more importantly, that the feedback no longer gets passed to the author.
That said, sure, I could make a change where I go and post a “the game has been sent to copyedit; the beta is over” post in the thread. I just don’t see/haven’t seen how that’s useful helpful, but if it is, I’m happy to do it.
The next point very much worth addressing is the question of “acknowledgement.”
You’re absolutely right, I don’t engage with the beta testers regarding their feedback or thank them for it. I also don’t require authors to respond to the testers. To begin, I’m not micro-managing the beta process. I don’t look at the notes that the beta testers send and then go review the authors’ new drafts afterwards to make sure they’ve incorporated it. So I don’t specifically know what they incorporate and what they disregard. But if I see similar things called out in multiple beta feedbacks–especially when those things agree with my own opinion of a work–then I have a conversation with the author about the aggregate feedback, and work with them on that.
What that means though, is that in large part, I can’t/couldn’t give anything more than a blanket “thank you for your feedback email. We’ll take it into consideration.”
Me being me, I feel like that’s a waste of both of our time. But if it would help to, for example, send an email when the game goes to copyedit saying that the beta is officially over and thank you for your participation, that could be arranged. (It’s also worth noting that, in real life, I’m terrible at thank-you notes.)
As for the authors responding to feedback: we could encourage authors to send personalized feedback to beta testers. My only problem is that we already ask so much of our authors that it makes me hesitant to ask for yet more. But if we stressed to them that “they’ll get more/better feedback” if they engage with the testers, then maybe they will.
That said, I hope you see how I don’t know how to immediately address your concerns: I’m extremely hesitant to require authors to engage with the testers, and I myself am not really in a position to do so other than in a rote way.
As for asking the authors to come onto the forums: again, that’s something I feel even less comfortable doing. Not everyone is a @cataphrak or @JimD who has the time and willingness to engage with our rather raucus community. Again, I could create a hidden thread for the testers and the author, but some (many?) of the authors just won’t be willing/interested in doing that.
I also want to respond to @Eiwynn’s point that not knowing “testing needs or goals is not a productive way of doing beta testing.” There are two ways of doing beta testing: volunteer and paid. If you’re volunteer (like the people on this forum) I can’t (legally) give you directions about how to play the game, because then I’m directing you. “Direction” is one of those ticky-boxes that the Federal and State governments go to to determine employment status; us giving testers “direction” (or worse, a “schedule”!) is taking away work from a potential/hypothetical employee. Thus, it is my understanding that we can’t give direction or schedules to volunteer beta testers.
As to other peoples’ suggestion that we start the beta process earlier: this is something that I would be amenable to, but it’s not as easy as it might sound. In particular, as editors, we work from reading the code; we reference line numbers. If the beta was happening in parallel, and the author was making changes to the code, the line numbers would no longer correspond. And we want the testers’ efforts to have the maximum impact; inviting the testers in before the rest of the editorial team has had a chance to give feedback would probably end up doing more harm than good, as they duplicate efforts.
Now, to return to the issue of bad habits. Because (historically) we’ve had lots of beta volunteers, my focus has been on speed and efficiency. (Then again, that’s often my focus.) Right now, I could write (for example) a personalized feedback email to the single person who’s submitted beta feedback on Runt of the Litter. (And quite effusive, I might add, as it has prompted the author to rewrite the last chapter.) But that doesn’t scale. And it certainly doesn’t scale when I’m running two or three (or four!) betas at the same time, as happens in Q3 and early Q4.
I hope this has given you some insight into the reasons for certain things in our process. And I am glad that we’re having this discussion. So, please, interrogate what I’ve said here, and I’m open to further suggestions. In particular, if we can convert my bad habits into something positive for the community, so much the better. We want both the community and our beta testers to be happy both with the production process and the finished product.
Lastly, I’ve tried to address the issues that were expressed in this thread. If there’s something you feel I haven’t responded to, please reiterate your concern.
Thank you so much for your very detailed response.
I don’t think thank yous are a waste of time. Admittedly form thank yous aren’t the best, but they’re better than nothing. Yes, ideally a response from the author would be better.
While you do demand a lot from your authors, they are at least getting paid. I think asking the author to send out thank yous to the beta-testers, even if it’s just a generic thank you, might increase good will? You could leave it to the author how much they want to engage?
That said I have sometimes said “please don’t respond to my feedback” and I would rather they didn’t. Sometimes I just don’t want to hear back. (Silly anxiety.)
Does the author need to participate in the hidden thread? Could they just be sent a link to all of the feedback (or have it mailed to them) without needing to engage with it? I know responding to every piece of feedback can be quite a time sink.
I would find this very helpful, personally. There have been a few times when I’ve thought of sending in another round of feedback (and I’ve occasionally asked whether it would be helpful), but if it’s been a few weeks and I’m not sure, I hesitate.
It takes me at least three to five hours to play a game and type up feedback, typically; sometimes a bit more or less for a very long or short game. So I’m less likely to make the effort if I’m uncertain as to whether further feedback would be truly helpful, or whether it’d be impractical for an author to do more than sort bugs and typos by that point in the game (so to speak).
I think this would be very helpful. Even though it is a form letter, something of the sort would have encouraged me, at least, to jump into beta testing a bit more quickly than I did. It’s courteous, and also lets testers know that their feedback has at least been acknowledged.
On another note:
There have been a few games that I didn’t particularly like personally, but I tested them several times because I had, in some way or another, been told it was helpful. In one author’s case, it was a very short thank-you note; in another, it was a regular change log where I could see several of the beta suggestions implemented. (And in one case, there was a game I actually hated, not because it was bad but because I’m me—but I was sent an email saying ‘hey those continuity errors were really helpful, can you find some more?’ So I played through twice more and sent on some more errors.)
That said, I’ve been swamped this year and haven’t been able to beta test hardly at all, but I look forward to beta-ing again in the coming months. I really appreciate CoG, its goals, the staff, the authors, and the games. So I’m happy to make a small contribution through beta testing.
And as far as I’m concerned (I bet @Cataphrak is in the same boat), my time is really becoming limited as I write more and more. It’s a balancing act of time vs. effort vs. more feedback.