When we allow breakoff, we typically tell respondents that they will be paid a bonus commensurate with the amount and quality of work they do at the end of the study. We don't typically inform them of when the study will end, since we may need to re-run surveys and we'd like to keep respondents from trying to game our algorithms. The experiments we did in the fall that awarded bonuses used a tiered system. The ones listed here are using respondent's scores to identify the top 95% of scores and award those respondents two cents per question. I'd like to investigate more sophisticated pricing schemata with Sara in the future.

### Wage Survey

It was my intention to only have one wage survey running on AMT. However, as I've been porting the Python analyses into Clojure, I found that I actually had three instances running. Given the expiration dates on the latter two, I'm pretty sure they were posted accidentally. I should probably consider asking the user if their sure they want to continue, or have a safe mode that asks a million times "are you sure you want to do X?" so future users don't make this mistake. There's also small possibility that when I extended the original HIT, it somehow spawned two new HITs instead. This isn't documented anywhere, but it's something I probably want to double check on sandbox.

### Choose Randomly

I ran a survey that wasn't featured in our OOPSLA paper because it was one I made up entirely. The idea was to post a survey with two floating blocks, where both asked respondents to choose one of the responses randomly, but the one block had identical options, whereas the other had arbitrary categories of things. I wanted to see how random people could actually be. I also wanted to track timing information. One of the things I noticed about this survey, which I posted on a Wednesday morning, was that I was able to collect all of my responses within 90 minutes. The time per question here is clearly less than 10s. If it takes about 2 seconds per question, then the expected cost for a completed survey would be about $0.08. As a result, I decided to not award bonuses. The total cost was$16.50.

### Phonology

Finally we have our classic phonology survey. With an average path length of 99 questions, we would expect to pay $1.993750 per survey. With a target of 150 responses, our total cost, including commission, would be$328.96875.

We ran this survey 3 times (not counting our preliminary run). The details of each run can be found in an earlier post. We collected 395 responses total and had 311 unique respondents. 22 respondents accounted for the 84 duplicate responses. This initial run cost us $43.45. 182 responses were classified as valid responses. The total bonuses to be paid were calculated to be$327.96. Factoring in AMT commission, this comes to $360.756. In all, this survey will have cost us$404.206.

### Notes on timing

Some of the the timing data we returned was flawed, so we couldn't use that information to improve our payment scheme. One possible use of this information would be to use it as a proxy for the difficulty the question and vary the payment in accordance with this information.