Top 5 MTurk Survey Participant Recruitment Alternatives in '24

Recruiting participants for your consumer or market research can be challenging. Thanks to crowdsourcing platforms such as Amazon Mechanical Turk, you can find the right target for your survey, request the crowd to answer your questions, and get the results quickly to grasp their thoughts. 

However, even though it seems like a seamless process, there may be some drawbacks to recruiting participants for your surveys, such as the reliability of the responses or data quality, that directly affect the results.

In this article, we highlight

• Some statistics of Amazon Mechanical Turk
• Challenges of using MTurk to recruit survey participants
• The top 5 alternatives to MTurk in recruiting participants for surveys.

Top 5 alternatives to Amazon MTurk Survey Participants Recruitment Service

We conducted a research on the available survey participants recruitment services on the market. Here you can see the results of our study in the table below.

If interested, you can also check out our data-driven list of survey participant recruitment services and survey tools.

Challenges of using Amazon Mechnical Turk for recruiting participants

Much research has been conducted on the reliability of MTurk survey responses, and a study claims that 10% of respondents are responsible for 40% of the responses on the platform.¹ It has also been shown that the data quality of respondents has significantly decreased since 2018.²

Here, we review the challenges of Amazon MTurk and propose some solutions:

1- Speeders

Some respondents usually do not carefully read the questions and use little effort to provide the proper answer. New research reveals that Among MTurk workers who have a 98% of approval rate, almost 25% of them could not pass at least one of the attentional check questions in the surveys.³ While investigating the reason for the inattentiveness, the same study found that prior experience with the technology significantly decreases attention.

Solution/recommendation: Implement attentional check questions and remove those who fail to pass them. You can also conduct post-hoc statistical analysis to detect fraudulent respondents.

2- Cheaters

A study showed that 5% to 7% of participants intentionally provide wrong or misleading responses.⁴ Some respondents also claim to be in another country while, in reality, they are not. Studies also show that this may take up to 10% percent of respondents, and the same study showed that they change their locations using a VPN to meet the survey requirements.⁵

Solution/recommendation: Track the response time in seconds per item (SPI) using a tool as an important validity indicator and integrate tools into your survey to detect suspected VPN IP addresses to prevent taking the survey with those addresses. ⁶

3- Bot accounts

Contrary to MTurk’s mission and business model, which relies on human intelligence in solving tasks, sometimes bots are used to complete the surveys automatically, and creating distortions in the results. 

Solution/recommendation: Utilize reCAPTCHA at the beginning or at the end of the survey to detect bot accounts.

4- Participants who take the survey multiple times

Research on Amazon MTurk showed that almost 20% of respondents take the surveys multiple times, decreasing the reliability of the survey responses.

Solution/recommendation: Disable the same IP addresses filling the survey more than once by implementing third-party tools.

Transparency statement:

AIMultiple serves numerous emerging tech companies, including resonio by Clickworker.

FAQ

Further Reading

• 4 SurveyMonkey Audience Alternatives
• Top 4 Qualtrics Competitors
• Top 5 Pollfish Alternatives
• Market Survey Research: Benefits, 3 Use Cases & Tips
• Online Survey Analytics: How It Works & 5 Essential Tips

Here is also our data-driven list of market research tools.

Please contact us if you have any additional questions about conducting surveys.

External Links

• 1. Ehrich, K. Mechanical Turk: Potential Concerns and Their Solutions. Summit. Accessed: 05/May/2024.
• 2. Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464-473. Accessed: 10/May/2024.
• 3. Saravanos, A., Zervoudakis, S., Zheng, D., Stott, N., Hawryluk, B., & Delfino, D. (2021). The hidden cost of using Amazon Mechanical Turk for research. In HCI International 2021-Late Breaking Papers: Design and User Experience: 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings 23 (pp. 147-164). Springer International Publishing. Accessed: 10/May/2024.
• 4. Ahler, D. J., Roush, C. E., & Sood, G. (2019). The micro-task market for lemons: Data quality on Amazon’s Mechanical Turk. Political Science Research and Methods, 1-20. Accessed: 10/May/2024.
• 5. Cobanoglu, C., Cavusoglu, M., & Turktarhan, G. (2021). A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The case of Amazon Mechanical Turk (MTurk). Journal of Global Business Insights, 6(1), 92-97. Accessed: 10/May/2024.
• 6. Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464-473.Accessed: 10/May/2024.