Year-by-year analysis of Omnibus data (CoG and HG)

Sometime back, I heard a remark that expectations are increasing for writers. That got me thinking, and I decided to crunch some numbers.

The CoG/HG Omnibus provides us three sets of quantitative data (Rating, Rating Count and Word Count). I decided to have a look and see what changed over the years. The data begins from the first game released following the release of the iOS Omnibus (My memories of that era are hazy).

Hosted Games

Choice of Games

X

Average values by year in table

Hosted Games

Column 1 Column 2 Column 3 Column 4 E F G H
Year No. of Books Word Count Word Count (>99k) Rating Rating (<4.9) No. of Ratings No. of Ratings (<2000)
2018 22* 192,318 224,444 4.450 4.429 2,422 433
2019 21 275,429 326,471 4.471 4.426 2,806 491
2020 21 213,714 297,857 4.324 4.295 2,203 480
2021 18 198,722 267,500 4.333 4.333 982 619
2022 28 248,679 357,222 4.343 4.300 859 472
2023 19 446,368 554,667 4.321 4.167 1,596 291
2024 7** 258,429 346,000 3.943 3.943 156 156

X

Choice of Games

Year No. of Books Word Count Word Count (>99k) Rating Rating (<4.8) No. of Ratings No. of Ratings (<1500)
2018 17* 273,529 285,000 4.529 4.493 1,214 698
2019 17 300,000 300,000 4.671 4.631 1,753 683
2020 14 342,857 392,500 4.493 4.409 1,727 760
2021 12 321,667 321,667 4.567 4.520 1,061 784
2022 13 364,615 364,615 4.500 4.410 1,073 706
2023 9 470,000 470,000 4.389 4.338 699 394
2024 3** X X X X X X

*Count does not include games released prior to omnibus
** Averages were not calculated for CoG as 3 games is too small a sample size. Use data in 2024 with caution as we are only halfway through the year. Life As a Lich was not included as it is a new release.

Let’s break down the data.

Word Count
I think everyone predicted this. Word Counts are largely trending up for both labels. Even if we filter out all the smaller games (<100k words), the trend remains the same. Word Counts are (rightfully or otherwise) a strong indicator of value and quality to our audience. The audience is demanding more and more, and writers are continously pushing the envelope in this regard. Million-word games are still rare, but I suspect they’ll be much more common in a few years.

The scatterplots in the image do seem to indicate that high wordcounts guarantee high ratings, but this effect is more pronounced for the HG label versus the CoG label.

Another interesting (and unexpected) thing I noticed is that average word counts in the CoG label are higher than the HG label, for all six years. Even if we took out the short games (<100k words), the CoG label still beats out the HG label in 4 out of 6 years.

I am also concerned that this could make the HG label increasingly inaccessible to first-time writers, unless they wish to make a long game their first project, or unless they’re fine with releasing a flop. Note that releasing a flop could weigh down on your reputation as a writer, so I would advise new writers not to treat your first game as a test case if you’re planning to be in this for the long haul.

Ok, so quality (read: word counts) are trending up. Which brings me to the next point.

Ratings
I’ll be the first to admit - I did not predict this. Ratings are clearly trending downwards, even as indicators of quality like word counts are trending upwards.

On the HG side, last year saw the release of three blockbuster games, yet the average rating of games released last year was at an all time low. Even with three big guns to pull up the average, an average game last year would have had a 4.3 rating, placing it behind roughly 60% of the existing full catalogue. That same game might have been a 4.5 or 4.6 if released in 2018, especially considering word count trends.

If we removed the 4.9 scoring games (to get a better sense of performance for the less popular writers), the drop is even more pronounced, Ratings would have dropped by 0.3 from 2018 to 2023, and that’s also assuming the game keeps pace with increasing expectations, including those in word count.

On the CoG side, average ratings have also trended downwards over six years. An average game last year would have had a 4.3-4.4 rating, which might have been a 4.5 or 4.6 in 2018, especially considering word count trends.

Still, the audience is grading games across both labels more harshly, even as more work (including word counts) goes into making these games. I’m not sure how much more this trend can be maintained, but 2024 and beyond could be pretty brutal for writers under both labels, since their work would likely be met with increasingly poor ratings compared to older releases.

The average rating of HG releases is at an unusually low 3.9 this year (as of this writing). Subsequent releases might pull up the average, but even then, my prediction is that this year’s average at the end of it will be a substantial drop from 2023.

Rating Counts
This is probably the most tricky to handle. Rating counts can range from single digits to 20k. A single game with 20k ratings can wreak havoc on the year’s average. As an imperfect solution, each table has one column which filters out the games with more ratings.

Also, I’m not sure how accurate this is, especially if the early rollout of the Omnibus was on the rocky side. Additionally, I don’t see any useful trends. But I did notice something.

On the HG side, if we look at the average games (by excluding games with over 2000 ratings), the average rating count was at an all time low of 291 last year. The average games had suddenly received a drop in attention. I’m not sure if there’s an explanation for this, but I thought I’d point this out.

On the CoG side, the average rating count was also at an all time low last year, well below the previous years. Unlike the HG label, the CoG label didn’t have a trio of AAA blockbuster releases last year, so this observation applies even without excluding the more popular games (> 1500 ratings). If we did exclude the popular games, the average rating count sits at an all time low of 394 (Better than the HG label, but still very low by the standards of the previous years). The last four (of nine) CoG games released in 2023 were all below 300 ratings.

Number of books
Interestingly, the CoG label has been releasing less games every year. The downward trend is quite obvious here. The CoG website indicates that there is limited capacity for taking on new writers and projects, so I’m guessing that this is not due to a lack of qualified writers.

On the other hand, the HG label (which accepts most submissions) has published a largely consistent number of books every year (minus a big jump in 2022).

In conclusion:
It’s getting harder for everyone. I have no plans to do a similar analysis for HC.

I’m not perfect, and this analysis is not perfect. If there’s anything you disagree with or feel is incorrect, feel free to point it out.

40 Likes

Am I reading the scatterplots wrong? Sure there are numerous <150,000 with lower scores but there’s quite a few above 4.0. Vs. CoG where there’s only 5/6 above 4.0 with less than 150,000 words.

2 Likes

Shouldn’t this be private?

Edit: i mean only authors could see it, 'cause it’s for their growth.

1 Like

If you check the HG scatterplot, games with low wordcounts can range from high to low ratings, but as the wordcount increases, games start moving towards only having higher ratings.

This isn’t obvious with the COG scatterplot, since there are numerous outliers, but the trend in the HG scatterplot is still sort of visible in the COG scatterplot if you disregard the outliers.

This is all publicly available information in the CoG/HG omnibus. Anyone with the Omnibus can build these spreadsheets and tables on their own.

If there is any confidential data here which anyone wants me to remove, please let me know. I don’t think there is, but just in case.

9 Likes

Huh. This is pretty neat to note and I must say the large number of ratings correlating so strongly to a high word count is not one that surprises me but it is still a bit wild seeing just how clear the correlation is. Seems the trend of not checking out a game if its not atleast above a certain number of words is quite widespread. And I can’t say I’m not part of that, I’ve certainly overlooked a fair share of the books in my newsletter when I saw a low wordcount and got giddy when I saw a high one.

Although I wonder if this may not be fully complete set of data as I’ve never really used the omnibus. I preferred APKs myself. But I guess those numbers are skewed by the negative reviews of people who thought it was entirely free and not a demo.

4 Likes

I think the trend of works being rated more harshly as time goes by is entirely predictable. In the beginning IF fans were starved for content and when there is little to choose from you look on anything you can get more favourably, regardless of its objective quality. As you get more choice and more examples of just what can be achieved in IF it makes you look on works that don’t meet the mark more harshly.

15 Likes

I think that the mass of works coming out in 2022 might have been because of Covid, people got more time to write during 2020 and 2021…

The reason why the word count jumped up so much, and the ratings went down during 2023 might be because of the release of the big three. All long, all highly anticipated by the fans. It is only natural that people might judge other things they bought against their favorites. Not fair at all, but I think it might have affected people’s judgment. A bit like… a game that the year before was seen as a 5 might be seen as a 4 now, because the favorite game was obviously a 5, and that game wasn’t as good, so it must be a 4.

18 Likes

Nice job, it was very interesting! I really like this type of posts, thank you for your time and your effort!
I’d like to make a couple of observations that occurred to me while reading.

Does anyone know if the ratings in the Android version of the Omnibus app are merged together with the ones from the iOS one? Because it could be that they are completely indipendent apps and that they count ratings indipendently. Just a doubt that can be easily verified but I thought worth brigning up, since it may increase the statistics noticeably. And it’s always good to increase the statistics!

Another thing that I think shouldn’t be overlooked while reading these data is the fact that in the omnibus app (and the standalones too) it’s possible to rate a game only if you’ve finished it. I think that there are for a certainty some people who didn’t like some games and didn’t finish them. My guess is that the real ratings are a little lower than the ones collected by @ChanceOfFire (not your fault obviously, but of the rating system!). I don’t know and I can’t guess how much weight this bias has (since we have no access to that type of data), but it’s there nonetheless.

Regarding ratings vs wordcounts, from the graph we can say that CoG label suffers less from this than HG. I believe it’s like that because their games tend to be 150k+ words (many people dislike or completely skip short games by default and that happens more with HG titles) and they have a more consistent quality (I know, there’s a quite a discussion regarding this, but I think that more or less we all agree about the consistent quality of CoG vs HG, and I think it plays an important role in this analysis).

I believe the rating vs rating count plot is quite interesting for its outlayers. It shows that there are some exceptionally well liked games played by a lot of people. Meanwhile, the majority of games is in the right part of the graph (<1500-2000 rating counts approx), where the distribution is slightly shifted towards higher ratings more than lower ones, showing that more or less the majority of games is still appreciated. Please keep in mind that although both graphs for CoG and HG are similar the ratings ranges in the two labels are different: HG is between 3.2 and 5 while CoG is between 3.6 and 5, hinting again the hit-or-miss nature of HG titles.

Regarding the general trending of the ratings in the years, I think @Lance_Heyen makes an excellent point, as well as @malinryden about word count and ratings in years when there are the releases of highly anticipated titles.

Even though the data are incomplete because of their nature, since we have access only to publicly available resources, I believe we can still deduce quite a lot of interesting trends of observations from them. Thank you again for your time @ChanceOfFire, we truly need more topics like this. It’s always super interesting to get a glimpse of the state of IF world (CoG and HG in this case) more in its entirety than just from its games, that are scattered everywhere.

3 Likes

As far as I know, the scores on the omnibuses are an aggregate of the rating players give at the end of the game (“how likely are you to recommend this”) and are the same for both apps.

2 Likes

Review scores and aggregate stats are not a good measure to base your game making decisions on.

There are a few reasons why this is. For example:

This might be a direct result of having full back-end support from the very beginning of the project’s life, including an editor and access to experienced feedback readers/testers sooner and more often.

This type of support doesn’t guarantee better success (witness the many CoG projects languishing or stalled) but it certainly can stack the deck in an author’s favor.

In addition, some HG or CoG authors can build up additional resources that really affect their ability to complete games. An author’s experiences and ability are much more important in completing a game project.

Developing these skills and providing authors with support should be the focus of the HG community.

It is true that an author should take individual reviews in and use them as actionable feedback, for sure!

Making gaming decisions based on an aggregate of reviews will likely be a strategy that leads to burn-out at best or an eventual exit from game making at worse.

4 Likes

Agreed. I just thought it might be interesting for the community to see this data and figure out some useful insights from it.

Still, high ratings are always a nice morale booster (and I need morale boosters to keep me going)

Good point. But counterpoint- statistics/data-wise, this leads to some kind of consistency in the ratings, since everyone who left a rating experienced the game in full, versus an inconsistent distribution of players who may or may not have finished the game.

5 Likes

Positive reinforcement is critical indeed! Even the most experienced writer can learn from all the different sources of data and feedback available to them.

I am also grateful that you, @ChanceOfFire, brought this into the general forums, instead of placing it in published author only places.

My worry is that new authors and new game makers either take too much from something they may, or may not fully understand, or, they get confused and become lost and discouraged from such “industry trends” or analysis.

Every month, I get questions from new authors asking about why “this” or “that” is important.

3 Likes

Something I said on another thread on the subject of longer HGs often resulting in higher sales:

And:

It’s also maybe worth noting that if authors are planning to write a continuous series with the same PC, each instalment will end up larger so it can become unwieldy very quickly. So, with all that in mind I would recommend being cautious about aiming to make larger games if that’s on anyone’s mind. (Mostly, from my own experience and talking to others, larger games tend to just happen without being aimed for.)

It’s out of the scope of this dataset, but I think genre and tone, are major factors that contribute to sales, on top of wordcount, as well as existing fanbases… but maybe even more than any of those, I believe player feedback plays a hugely important role, both in providing attention to the game and in helping authors polish their work.

11 Likes

True, with the available data this is the more consistent approach. However, I still think they have some weight, but we can’t say how much. I’d just like to say to consider that.

1 Like

Is it really that clear? I’m no statistician, but the ratings data look pretty noisy to me; the trend varies a lot depending on what you pick as your first and last year, especially when you remove the HG outliers.

We’re talking about a pretty short stretch of years regardless. My gut feeling is that the data so far are consistent with 2023 being a one-off dip, rather than the culmination of a longer-term trend, and that we should see how 2024 looks before getting too alarmed.

Thanks for sharing this, btw – and for the time you put into the analysis. Definitely think it’s worthwhile having the conversation.

8 Likes

Fair. I do have a tendency to see trends where there are none, and we don’t have a big sample size to work with.

In any case, for the more visual fiolks…

The ratings for Hosted Games in 2024 seem uh… alarmingly low. I’m a bit surprised at Tale of Two Cranes getting a 4.5 rating, considering that it checks many boxes (850k word length, fantasy genre, romance, gender-choice), but I haven’t played it. Also, we have two games scoring in the 2s, which is entirely unprecedented (the lowest rating was 3.3 prior to 2024), and I don’t think those games were exceptionally bad. Happy to be proven wrong, but 2024 isn’t off to a promising start.

It’s a bit harder to say for CoG, since we have only three releases so far, and two of the three were very heavy hitters.

I’m not sure if this is damaging to this year’s releases, however, as it’s possible that customers might see a 4.3 rated game this year, adjust their expectations and say 'ratings this year were, lower, so maybe it ain’t that bad." Or not.

7 Likes

I think it’s also worth noting that we’re talking about variation in a pretty narrow band. 4.4s and 4.3s for unadjusted HGs, 4.6s and 4.5s for CoGs before the 2023 dip. That feels to me like the scale of variation that you can expect from the normal vicissitudes of taste, and not necessarily something to draw big conclusions from.

If the two 2s drag down the average for 2024, that’s in significant part a consequence of new writers trying to do new, much more linear, and ultimately unpopular things with the medium. I don’t think we should lament that because it drags down an average that most readers will never see.

The big picture over these years is also one of significant overall growth in the audience for these games, and blockbusters (both HG and COG, but especially HG) reaching sales levels that would have been impossible in earlier years. I don’t think overall HG authors should be afraid or discouraged. This is still, by far, the best place to find an audience for your interactive fiction.

8 Likes

Sidenote, thanks for teaching me a new word. Didn’t know this one before.

3 Likes

I don’t know if this was already mentioned, and please take my hypothesis with all the grains of salt in the multi-verse, but I wonder if some of the lower ratings are the natural result of healthy competition - I.e. there’s now twine games to compete with CS games. While technically there have always been competitors to CS most amateur game makers probably are going to have trouble learning Java or C#, while both CS and Twine are far easier to learn. Added to that Twine games can do multiple choices per page and then have the choices immediately have an effect via an “enchantment” where CS games can’t.

Now don’t take that as a lack of love for CS games from myself - I adore these games and personally find CS far less likely to be a bug ridden mess than a Twine game, but some players may like the added functionalities I’ve mentioned.

Anyway, interesting topic and thank you for sharing. It’s nice to have a seat at the table, so to state.

3 Likes

I don’t want to rip into any individual story, since that’s not the point of this thread, but… I don’t really think a game like EXODUS: Climate Activist getting low scores should be any cause for concern. And since it’s a part of a very small sample, and thus has a significant impact on the average, I wouldn’t really worry about overall ratings either.

And yeah, “vicissitudes” is a pretty cool word.

3 Likes