A peak behind the curtain, as it were.
It entirely possible that a number of the issues we’re having are a result of bad habits that I’ve fallen into. But perhaps I can give some insight into circumstances, as well as competing interests, and then find a solution that more people will be happy with.
In the past, there would sometimes be as many as forty-two beta-testers (Pendragon Rising, as a random example). Testers apply, I send them the link. They send feedback. I skim it, make sure there isn’t anything that I substantively disagree with, give it a “grade” based on its “depth,” and then send it to the author. The author sends back updates to the files, which I post to the beta. I post an “updated draft” on the beta thread. This continues until we move into Continuity Testing.
Continuity Testing is where I use the Randomtest tool to produce randomly generated playthroughs of the game. I send “seeds” out to readers (who read the playthrough like it was a novel) and look for continuity errors and other problems. Then send the seeds back to me, I send them on to the author. The author sends me an updated draft. I run new seeds, and send them out to the readers.
Once the seeds come back “clean,” we send the game to copyedit. Then, if someone applies to the beta after the game has gone to copyedit, I go and lock the beta thread; otherwise, I just let it drop into oblivion.
The beta doesn’t officially end, I suppose, until the game goes to copyedit. The beta thread’s “pin” expires after a week, but each time I post “new draft updated,” it gets pushed to the top, where it will soon drop again considering the amount of activity we have on the forums. But, it’s notable that, if someone sends in more beta feedback during the continuity phase, it gets passed on to the author all the same.
So, to Mara’s point about “you don’t tell us when the beta is over,” you’re right, I don’t. Because I didn’t know that it mattered. To me, when I’ve stopped posting the “new draft” posts, I’m going to mostly stop getting access emails. And that’s the functional end of the beta. Again, though, if someone sends in feedback, it’ll still get sent on. It’s only when someone applies after the game has gone to copyedit that I officially lock the thread, and that may be a more definitive “the beta is over” moment. Or, more importantly, that the feedback no longer gets passed to the author.
That said, sure, I could make a change where I go and post a “the game has been sent to copyedit; the beta is over” post in the thread. I just don’t see/haven’t seen how that’s useful helpful, but if it is, I’m happy to do it.
The next point very much worth addressing is the question of “acknowledgement.”
You’re absolutely right, I don’t engage with the beta testers regarding their feedback or thank them for it. I also don’t require authors to respond to the testers. To begin, I’m not micro-managing the beta process. I don’t look at the notes that the beta testers send and then go review the authors’ new drafts afterwards to make sure they’ve incorporated it. So I don’t specifically know what they incorporate and what they disregard. But if I see similar things called out in multiple beta feedbacks–especially when those things agree with my own opinion of a work–then I have a conversation with the author about the aggregate feedback, and work with them on that.
What that means though, is that in large part, I can’t/couldn’t give anything more than a blanket “thank you for your feedback email. We’ll take it into consideration.”
Me being me, I feel like that’s a waste of both of our time. But if it would help to, for example, send an email when the game goes to copyedit saying that the beta is officially over and thank you for your participation, that could be arranged. (It’s also worth noting that, in real life, I’m terrible at thank-you notes.)
As for the authors responding to feedback: we could encourage authors to send personalized feedback to beta testers. My only problem is that we already ask so much of our authors that it makes me hesitant to ask for yet more. But if we stressed to them that “they’ll get more/better feedback” if they engage with the testers, then maybe they will.
That said, I hope you see how I don’t know how to immediately address your concerns: I’m extremely hesitant to require authors to engage with the testers, and I myself am not really in a position to do so other than in a rote way.
As for asking the authors to come onto the forums: again, that’s something I feel even less comfortable doing. Not everyone is a @cataphrak or @JimD who has the time and willingness to engage with our rather raucus community. Again, I could create a hidden thread for the testers and the author, but some (many?) of the authors just won’t be willing/interested in doing that.
I also want to respond to @Eiwynn’s point that not knowing “testing needs or goals is not a productive way of doing beta testing.” There are two ways of doing beta testing: volunteer and paid. If you’re volunteer (like the people on this forum) I can’t (legally) give you directions about how to play the game, because then I’m directing you. “Direction” is one of those ticky-boxes that the Federal and State governments go to to determine employment status; us giving testers “direction” (or worse, a “schedule”!) is taking away work from a potential/hypothetical employee. Thus, it is my understanding that we can’t give direction or schedules to volunteer beta testers.
As to other peoples’ suggestion that we start the beta process earlier: this is something that I would be amenable to, but it’s not as easy as it might sound. In particular, as editors, we work from reading the code; we reference line numbers. If the beta was happening in parallel, and the author was making changes to the code, the line numbers would no longer correspond. And we want the testers’ efforts to have the maximum impact; inviting the testers in before the rest of the editorial team has had a chance to give feedback would probably end up doing more harm than good, as they duplicate efforts.
Now, to return to the issue of bad habits. Because (historically) we’ve had lots of beta volunteers, my focus has been on speed and efficiency. (Then again, that’s often my focus.) Right now, I could write (for example) a personalized feedback email to the single person who’s submitted beta feedback on Runt of the Litter. (And quite effusive, I might add, as it has prompted the author to rewrite the last chapter.) But that doesn’t scale. And it certainly doesn’t scale when I’m running two or three (or four!) betas at the same time, as happens in Q3 and early Q4.
I hope this has given you some insight into the reasons for certain things in our process. And I am glad that we’re having this discussion. So, please, interrogate what I’ve said here, and I’m open to further suggestions. In particular, if we can convert my bad habits into something positive for the community, so much the better. We want both the community and our beta testers to be happy both with the production process and the finished product.
Lastly, I’ve tried to address the issues that were expressed in this thread. If there’s something you feel I haven’t responded to, please reiterate your concern.