There's this website I came across while investigating what other people are doing for online experiments. It's called SocialSci.com. They have a platform for writing experimental surveys. They also boast of a well-curated participant pool. Here's the claim on their front page:

We take a three-tiered approach to our participant pool. We first authenticate users to make sure they are human and not creating multiple accounts. We then send them through our vetting process, which ensures that our participants are honest by tracking every demographic question they answer across studies. If a participant claims to be 18-years-old one week and 55-years-old the next, our platform will notify you and deliver another quality participant free-of-charge. Finally, we compensate participants via a secure online transaction where personally identifiable information is never revealed.

Ummmmm, okay. That's not game-able at all.

If you were thinking about just using their survey design tools and instead posting your instrument on AMT, be warned! They point out that AMT is full of scammers! I'm not sure this is true anymore; my own research has lead me to believe that traditional survey issues such as fatigue and attention issues are a bigger threat to validity. In any case, I looked up quotes for using their participant pool. The costs for 150 respondents (assured to be valid, via some common sense heuristics and their more complete knowledge of the participant pool) and no restrictions on the demographics (e.g. can include non-native English speakers) are:

[table id=6 /]

Even with bad actors, I'm pretty sure AMT is cheaper. The results were the same when I submitted a request for country of origin==USA. There were also options for UK and Australia, but these are not yet available (buttons disabled). I'll leave any analysis of the pay to Sara.

Their filters include Country, Language, Age, Sex, Gender, Sexual Orientation, Relationship Status, Ethnicity, Income, Employment Status, Education, Occupation, Lifestyle, and Ailments. What I find potentially useful for survey writers here is the opportunity to target low-frequency groups. There's our now infamous NPR story on how teens respond mischievously to survey questions. The set of responses from these teens have a high number of low-frequency responses, which amplifies false correlations when researchers analyze low-frequency populations. A service like SocialSci could provide useful, curated pools that rely less on one-time self reporting. However, it doesn't look like they're there yet, since there are the options available (shown, but currently disabled options are listed in brackets):

Country: USA, [UK], [Australia]
Language: English, [Spanish]
Age: 13-17, 18-40, 41-59, [51+], [60+]
Sex: Male, Female, [Transgender], [Intersex]
Gender: Cis Male, Cis Female, [Trans Male], [Trans Female]
Sexual Orientation: Heterosexual, Homosexual, BiSexual (sic), Other
Relationship Status: Married, Single, Co-Habitating, Dating
Ethnicity: Caucasian, [Asian], [Hispanic], [Black], [Native American], [Multiracial]
Income: less than 25K, 25K-50K, 50K-75K, 75K-100K, [100K-125K], [125K or more]
Employment Status: Full Time, Unemployed, [Part Time], [Temporary Employee Or Independent Contractor]
Education: Some College, Associate's Degree, High School Diploma, Bachelors Degree, Some High School, [Masters Degree], [Doctoral Degree]
Occupation: Student, Professional, Technical, Teacher, Sales, Corporate
Lifestyle: Smoke, Used to Smoke, Have Children, Have Cell Phone
Ailments: Chronic Pain, Addiction (Smoking)

** sorry for the lack of formatting!! Also, what do you think of these demographic characteristics? Please leave comments!**

I want to note that the front page advertises "a global pool." In keeping with the spirit of the times, I've included a screenshot (things I've learned from shaming bigoted celebrities who express themselves a little too freely on twitter!)

Screen Shot 2014-07-31 at 7.04.57 PM

Now, it's quite possible that they have at least one person from every demographic listed, but not 150. Until they can compete with AMT in size (where 150 respondents ain't nothin), I think they should be careful about what they advertise to researchers.

It's not clear to me how SocialSci recruits their participants. Here's the page. I had never heard of them until I went looking for resources used in other experiments (via this page Emery recommended) I just did a search of craigslist to see if there's anything there. No dice. $10 says it's just their team plus friends:

[youtube]https://www.youtube.com/watch?v=N9qYF9DZPdw[/youtube]

Snark aside, I'd like to see what a well-curated pool looks like. Not sure these folks are up to the task, but just in case, I did them a solid and posted their signup page on craigslist. I'm not holding out on it being as effective as the Colbert bump, but a girl can dream.