Generate Statistics About Your Stats

To help me balance my stats in my work-in-progress, I modified randomtest to take snapshots of my stats and put them in a .csv file, which Excel and other spreadsheet programs can open. I put the changed code on Github for anyone who wants to give it a try. You should be able to use it just like you would ChoiceScript from the main repository.

To try this out, in your startup.txt file, put a special comment that sets up what variables to save:

*comment savestatsetup [stat1] [stat2] [stat3]

Replace [stat1] and the rest with a list of the variables you want to save. Put a space between each variable’s name.

Then, wherever you want to take a snapshot of the variables’ values, put the following comment:

*comment savestats

If you run randomtest from a terminal window, the program will ask you if you want to save stats to a file. Say “yes” to save the variables’ values to randomtest-stats.csv.

I included an example of how to use it in the sample game that comes with ChoiceScript.

Once you’ve got your variables saved to a .csv file, you can open that file in Excel and get an idea of how much your stats are varying by over a bunch of runs. For example, I ran randomtest a thousand times to see what the stats looked like after the first chapter of my work-in-progress. Then I made histograms to see how the stat varies. Here’s one skill’s value:

labwork skill

From this I can see that the most common set of values is around 31, but that a really determined player could push it all the way up to nearly 50. If that’s too high for me, then I need to make changes to the code to make that high value less possible.

I also looked at an opposed stat value that starts at 50.

driven trait

This tells me that it’s easier to push the stat in one direction than in the other. That may be okay! Or I may want to tweak things to give a more even distribution so that there’s a peak around 50 and it falls off more-or-less equally to either side.

Finally, if you really want to get fancy, you can see how different stats influence each other by looking at their correlation.

A positive value means that when one stat goes up, in general so does the other. A negative value means that when one stat goes up, in general the other goes down. A value near zero means they don’t influence each other at all. From this, I can see that emphasizing any one skill comes at the expense of another, except (weirdly) for labwork and political insight. I can also see that, since the value for rhetoric and reasoning is almost zero (-0.447), they don’t influence each other much.

44 Likes

This sounds amazing. Thanks for putting it out there!

3 Likes

That seems really similar to what I’ve been doing but mine is a bit more simple just throwing the values on a spreadsheet without graphics. The only problem I have is that I can’t run the game more than 3000 times or the program crashes. I’ll analyze your option and see if it’ll work for me. Thanks for sharing!

1 Like

This is so cool - thank you so much for developing this method and sharing!

2 Likes

To be fair, the graphics are me making graphs in Excel after the stats are collected. The code’s not doing anything fancy beyond capturing the values.

3 Likes

The way I do it is just placing {stat1}, {stat2}, ${stat3} and so on at the place I want to test the values and then run the game with the randomtest with the option to save the output showing all the text. Does your way shows all the text from the story or just the values of the variables?. I think you could have solved my problem unintentionally already…

Edit: I’ve just tested it, it works just like I wanted. man, you just have solved my problem almost at the same time I posted it here to ask for help. Thanks a lot! I don’t have to waist time going around giant .txt filtering the values anymore. Thanks!

Edit2: Yes totally confirmed, 25k iterations on one run without any issues. Thanks again

2 Likes

Ooh, I’ll definitely get some use out of this. Thanks Stephen!

Can we get some statistics about how many people are going to generate statistics about their statistics?

6 Likes

Okay this just blown my mind! This is a serious way to evaluate and test game mechanics in a less maddening way.

Bless u buddy!

1 Like

I just wanted to make a question that I’m pretty sure you can answer.
When I use the randomtest the console shows " *****Seed ### " (### is the seed number)
I guess that it does that to show the progress, though, if that part of the program is disabled somehow, could that make the program more efficient and faster? or instead of disabling it completely only show every 10.000 seed or something like that?

You could comment out that line in randomtest.js (the console.log("*****Seed " + (i+randomSeed)); line around line 793) and you’d save a little bit of time, but I don’t know that it would be a lot of time.

I think I did that (with // at the beginning of the line if I’m not wrong) but it keeps showing the seed number, I don’t know what I did wrong, I tried commenting the whole block too (line 793 to 803) but didn’t work either. Thanks for the reply! sorry to bother.

One of the counterintuitive aspects of these sorts of systems is that unless a certain operation is very time-consuming (say, reading from or writing to a physical medium), checking a condition so you only perform that operation occasionally is actually slightly less efficient than not checking at all and just doing the thing every time.

1 Like

Aha, I gave you the line numbers for the asynchronous randomtest function. Try adding a // in front of line 812, where the other console.log() function is.

2 Likes

Yes, that one worked, Thanks a lot, I’ll have to test the efficiency of both methods now. I’ll post my findings, maybe someone find it kind of relevant. I’m just trying to save time, it might not seem much but if you want to test your game a lot and lot of times a simple check like that could ad a considerable amount of time to the process, just a thought. Thanks again for the improvement of this tool.

1 Like

Well, although I only did 1 test of each I was expecting to find more difference between them, anyway, here are the results:

700.000 iterations
starting seed [0]
avoid used options [y]
show full text [n]
Show selected choices [n]
How many times line used [n]
Save stats to .cvs [y]
Write output to file [n]

The test that shows the seed on the window took 19092,196 seconds (5:18 hours) and the one that doesn’t show the seed took 18971,372 seconds (5:16 hours).
The difference resulted to be 120 seconds over a test of 700.000 that gives a 1% speed difference (about 0,000172606 seconds per iteration) which I thinks is completely negligible due to possible differences in cpu load, os hiccup, etc…
Both tests were performed when the computer was not being used, same programs on background (none of them of a significant demand, antivirus, firewall, dropbox already sync, and regular stuff) and the game was the same also, so there was always a constant enough ram and processing power available, which leads again to the conclusion that it doesn’t matter much whether if the randomtest shows the seed on screen or not.
I really appreciate the contribution to modify the program and the help to make this test possible, it is a really powerful tool once your branches begin to spread all over the place.
If anyone have another idea that could make this software even more efficient I’ll be happy to test it, but I think that it doesn’t get much better than this.

Thanks and may your test debug your code.

4 Likes