Rate Your Gate - Cyto2011 Results
This was our first experience playing the game, and our first few participants had a pretty rough time of it. Our bad - we didn't have the right game directions and setup. So we fixed that, and the rest of the folks did much better.
Methods & Materials
We generated a dataset of 73 synthesized-data files, each having 3 correlated parameters and 3 levels of expression for each parameter. 50,000 events were contained in each file. 17 participants analyzed 22 of the data files; several participants analyzed more than one file. Each participant was shown an example file with example regions that provided reasonably good estimates of the correct answers. A new file was then randomly assigned to the participant and opened in WinList. Participants were actively coached if we realized they were making egregious mistakes such as leaving a large number of events out of a region. A Reduced Chi-Squared score was created for each analysis.
The optimal score is near 1.0, and the score of the example shown to participants was 22.55. The best participant score was 17.94. The highest score was 250.49. The median for all scores was 29.57.
Participants used a broad variety of strategies for drawing regions to segment the 3 levels of expression in each parameter. It was not unusual for one region to be drawn as a rectangle or ellipse, and another to be drawn freehand. Some drew all regions on 2P histograms; some only used 1P histograms.
In order for participants to do reasonably well (score < 30), there were two key factors. Firstly, they had to properly identify the 3 levels of expression and capture events in a region. Secondly, they had to define a reasonable demarkation boundary between the peaks and the transitions. This second factor was anticipated to be a key element, since this is the point of overlap in populations. Where overlap occurs, a gating boundary arbitrarily divides cells that are really in a continuum.
Different strategies for the same parameter
We expect to run the game again at the GLIIFCA meeting, Sept, 2011...