The script analyze-transcripts-curves.sh draws cumulative error curves for human players sharing a particular experience. It pulls the data for a group of players from the data store (the SQL database and transcript files) and produces SVG images displaying the individual players' cumulative errors curves and their median curve.
This script is to be used in much the same way as analyze-transcripts-mwh.sh. It takes the same arguments related to choosing the players (e.g. by specifying an experiment plan or supplying a list of players) and identifying distinct experiences (-precMode) as that script.
Support for the -export and -import options has the same form as well, although the exported CSV file now has an additional column, containing the summary of the each series' transcript (in terms of the sequence of "e[j]:p0[j]" pairs for all moves in the series). Additionally, several options related to drawing curves can be provided.
The option -curveMode yVar:xVar specifies what you actully want to plot. The possible values are as follows.
In all formulas, m is the move count (more precisely, the number of successful and failed moves and failed picks) since the beginning of the series. (Successful picks are entirely ignored, as if we deem them to be merely "slips of the fingers", or actions of a player who does not understand what he needs to do). q is the number of game pieces removed so far in the series, i.e. the number of successful moves.
yVar specifies the variable plotted on the vertical axis. The possible values are:
AAID (proposed by Paul on 2025-11-19): Can there be an "normalized to start of good runs" curve for each different player that is on the C vs. W plot?
sumW=0;
sumE=0;
sumC=0;
C=0;
plot(0,0);
at each move that is either correct or wrong (not ignored)
if wrong: sumW+=1
sumE+=(1-p)
no point on plot
if correct: sumC +=1
if preceding move was correct
plot (sumC,y)
if preceding move was wrong
y=(sumW/sumE)*sumC
plot(sumC,y)
if this is the first move, plot (1,0).
I think if the player moves randomly, this plot will be around the 45 degree line, whether the rule is hard or easy.
If the player starts a run of correct moves, the y-value will become flat, and will stay flat until a wrong moves.
if the player makes a wrong move, the trace of errors will be smoothed, by averaging against the whole run, from the start. I think but am not sure, that this approach will not produce strange dips at the end of a run of correct moves.
AAIE is very similar to AAID, except that sumE includes (1-p) for all moves, not just the incorrect ones. Thus,
sumW=0;
sumE=0;
sumC=0;
C=0;
plot(0,0);
at each move that is either correct or wrong (not ignored)
sumE+=(1-p)
if wrong: sumW+=1
no point on plot
if correct: sumC +=1
if preceding move was correct
plot (sumC,y)
if preceding move was wrong
y=(sumW/sumE)*sumC
plot(sumC,y)
if this is the first move, plot (1,0).
xVar specifies the variable plotted on the horizontal axis. The possible values are:
The default value of curveMode is AAIC:C, meaning plotting AAIC as a function of C.
An advantage of using C instead of M as the variable on the horizontal axis is that in a game where most players clear all boards presented to them (i.e. no incentive scheme, and no "give up" option), all curves will end at roughly the same value of C, so a median would be much better defined than it would in a plot where the variable being plotted is plotted with respect to M.
If -curveMode has been specified as ALL.ALL (or just as all, for short), the tool will plot the curves for all supported yVar.xVar combinations, placing each family of plots in an appropriately named subdirectory of the out directory. E.g. the plots of AAI vs. C will be in the subdirectory AAI_C.
The tool creates directory out; in it, subdirectories corresponding to display modes (W_M etc). Within each of them, plots are written as individual SVG files, each file name corresponding to the identifier of the "experience" reflected therein. SDepedning on the -precMode, that may be just the name of the rule set, or a combination of strings including the preceding rule sets. In the latter case, the components of the file names are separated by dots (because using semicolons would not be appropriate in file names).
The plotting tool uses the same criterion as the MWH tool (-targetR or -targetStreak) to identify "learners", i.e. players who appear to have "mastered the rules" and have demonstrated this mastery by making a sequence of correct moves. The two options (-targetR or -targetStreak) correspond to Paul's two incentive schemes (LIKELIHOOD or DOUBLING, respectively); if the data you analyze have come from an experiment in which one of these two incentive schemes was employed, you may want to use the corresponding option, with the appropriate value (as per the experiment's parameter sets), so that the tool's identification of the players are learners or non-learners matched the identification of them as such made in real time by the Game Server.
In W and AAIB plots, the curves for the identified "learners" can be easily identified, because on the graph the curve is extrapolated, by a solid horizontal line, beyond the point where the player stopped playing.
The solid line shows how the curve would hypothetically continued if the player kept playing, without making any more errors than Wstar, the number of errors he had made by the point of reaching mastery. The position of the solid line can be theoretically computed as W=Wstar, or (for a game with a constant p for all moves), AAIB=Wstar/(1-p).
In the AAI plots, there is no extrapolations, because I did not bother drawing hyperboles (for AAI(m)=const/m).
In AAID and AAIE plots, the curve for a learner becomes a horizontal line after the last error has been made, because of the design of these metrics: the "freeze" the metric value during an error-free stretch. (Of course, should even a single error happen later, the metric will make a huge jump forward, more or less toward the hyperbole of AAIC). Thus on the plots we also draw horizontal extrapolation lines.
For AAIC, the AAIC_C curve of a learner, after the player has mastered the rule, is either a hyperbole (for a game with constant p for all moves), or an approximation of such a hyperbole. Consider a game with a constant p, e.g. quadNearby, where the probability of a good move for a COMPLETELY_RANDOM player is p=0.25. One can show that for a perfect learner the post-mastery curve is
For a game where p varies from one move to the next, the "future behavior" of a curve for a perfect learner (after he has stopped actually playing) is, in general, not perfectly predictable, since for some more complicated games their randomly generated boards may come with different distribution of p. Still, it stands to reason that if the player kept playing (without making more errors), his AAIC curve would
The median curve so constructed typically has discontinuities at the points where some of the participating players' curves end. (For example, if we records of 10 players up to m=30, but of only 9 players at m=31, then the median curve will likely have a discontinuity at m=30, since to the left of that point the median is constructed as the median of 10 functions, and to the right of that point, as the median of 9 functions).
You probably only want to use the -median Extra mode when you're working with a data set that includes only learners, i.e. one where all curves are extrapolated. Otherwise, you'll see that the median curve changes in an "unnatural" way once the "non-learners'" curves end and only extrapolated "learners'" curves remain.
This example can be fined in the script /home/vmenkov/curves/curves-ignore.sh
#!/bin/csh rm -rf out #-- Extract all records for players who played plan "FDCL/basic". Save them to a CSV file, and also produce the default set of curves /home/vmenkov/w2020/game/scripts/analyze-transcripts-curves.sh -precMode Ignore -targetR 1000000 -export all-ignore.csv FDCL/basic > tmp.log mv out out-all-ignore #--select players who have "learned" head -1 all-ignore.csv > learned-ignore.csv awk -F ',' '$7 == "true"' all-ignore.csv >> learned-ignore.csv #--draw curves for learners, /home/vmenkov/w2020/game/scripts/analyze-transcripts-curves.sh -precMode Ignore -import learned-ignore.csv -median Extra > tmp-2.log mv out out-learned-ignore #-- list the directories with SVG files du out-all-ignore du out-learned-ignore
I have copied the output directories (for all players, and for learners only) to http://action.rutgers.edu/tmp/out-learned-ignore/ http://action.rutgers.edu/tmp/out-all-ignore/
See also curves-every.sh in the same directory. It's output has been copied to http://action.rutgers.edu/tmp/out-learned-every/ http://action.rutgers.edu/tmp/out-all-every/
As Paul requested,
"(d) it would be good to show the purely random line in some uniform way because scales may change."
The "purely random line" was interpreted in a slightly broader sense, as the one that would describe the median curve of a large population of players described by the same random player model that is used to compute p0 in this particular series of plots. (At present, the Game Server and its analysis tools support two such models, COMPLETELY_RANDOM and MCP1. The former is, I think, a fine candidate for producing what you describe as a " purely random line").
Naturally, the behavior of the "purely random line" will depend, in general, not only on the rule set, but also on the mix of the initial boards being played. (The rule set ordL1 is a good example here: http://action.rutgers.edu/tmp/out-all-ignore-COMPLETELY_RANDOM/AAIB_C/FDCL/basic/ordL1.svg ; the random line is the blue short-dashed one). Therefore, an effort was made for the "purely random" line to be based on the same boards on which the actual players played. For every "series" (rule set + player) combination in the analysis, several (3) random players were created, and each one was made to play the same sequence of boards that the real player did. That created a sufficiently large population of random players whose curves can be meaningfully averaged (or, actually, "medianized").
One can see such sample curves for the FDCL/basic rule sets in
http://action.rutgers.edu/tmp/out-all-ignore-COMPLETELY_RANDOM/ http://action.rutgers.edu/tmp/out-all-ignore-MCP1/ http://action.rutgers.edu/tmp/out-learned-ignore-COMPLETELY_RANDOM/ http://action.rutgers.edu/tmp/out-learned-ignore-MCP1/(In these runs, precMode=Ignore, i.e. the experiences of all players who played a particular rule set are put together, regardless of what rule sets they may have played previously).