Re: CAPTCHAs – Understanding CAPTCHA-Solving Services in an ...

Text-only Preview

Re: CAPTCHAs – Understanding CAPTCHA-Solving Services in an
Economic Context
Marti Motoyama, Kirill Levchenko, Chris Kanich, Damon McCoy,
Geoffrey M. Voelker and Stefan Savage
University of California, San Diego
{mmotoyam, klevchen, ckanich, dlmccoy, voelker, savage}
alphanumeric characters that are distorted in such a way
that available computer vision algorithms have difficulty
Reverse Turing tests, or CAPTCHAs, have become an
segmenting and recognizing the text. At the same time,
ubiquitous defense used to protect open Web resources
humans, with some effort, have the ability to decipher
from being exploited at scale. An effective CAPTCHA
the text and thus respond to the challenge correctly. To-
resists existing mechanistic software solving, yet can
day, CAPTCHAs of various kinds are ubiquitously de-
be solved with high probability by a human being. In
ployed for guarding account registration, comment post-
response, a robust solving ecosystem has emerged, re-
ing, and so on.
selling both automated solving technology and real-
This innovation has, in turn, attached value to the
time human labor to bypass these protections. Thus,
problem of solving CAPTCHAs and created an indus-
CAPTCHAs can increasingly be understood and evaluated
trial market. Such commercial CAPTCHA solving comes
in purely economic terms; the market price of a solution
in two varieties: automated solving and human labor.
vs the monetizable value of the asset being protected. We
The first approach defines a technical arms race between
examine the market-side of this question in depth, ana-
those developing solving algorithms and those who de-
lyzing the behavior and dynamics of CAPTCHA-solving
velop ever more obfuscated CAPTCHA challenges in re-
service providers, their price performance, and the un-
sponse. However, unlike similar arms races that revolve
derlying labor markets driving this economy.
around spam or malware, we will argue that the underly-
ing cost structure favors the defender, and consequently,
the conscientious defender has largely won the war.
The second approach has been transformative, since
Questions of Internet security frequently reflect under-
the use of human labor to solve CAPTCHAs effectively
lying economic forces that create both opportunities and
side-steps their design point. Moreover, the combination
incentives for exploitation. For example, much of today’s
of cheap Internet access and the commodity nature of
Internet economy revolves around advertising revenue,
today’s CAPTCHAs has globalized the solving market;
and consequently, a vast array of services—including e-
in fact, wholesale cost has dropped rapidly as providers
mail, social networking, blogging—are now available to
have recruited workers from the lowest cost labor mar-
new users on a basis that is both free and largely anony-
kets. Today, there are many service providers that can
mous. The implicit compact underlying this model is that
solve large numbers of CAPTCHAs via on-demand ser-
the users of these services are individuals and thus are
vices with retail prices as low as $1 per thousand.
effectively “paying” for services indirectly through their
In either case, we argue that the security of CAPTCHAs
unique exposure to ad content. Unsurprisingly, attack-
can now be considered in an economic light. This prop-
ers have sought to exploit this same freedom and acquire
erty pits the underlying cost of CAPTCHA solving, ei-
large numbers of resources under singular control, which
ther in amortized development time for software solvers
can in turn be monetized (e.g., via thousands of free Web
or piece-meal in the global labor market, against the
mail accounts for sourcing spam e-mail messages).
value of the asset it protects. While the very existence of
CAPTCHA-solving services tells us that the value of the
CAPTCHAs were developed as a means to limit the
ability of attackers to scale their activities using auto-
associated assets (e.g., an e-mail account) is worth more
mated means. In its most common implementation, a
to some attackers than the cost of solving the CAPTCHA,
the overall shape of the market is poorly understood. Ab-
CAPTCHA consists of a visual challenge in the form of

(a) Aol.
(c) phpBB 3.0
(d) Simple Machines Forum
(e) Yahoo!
(f) youku
Figure 1: Examples of CAPTCHAs from various Internet properties.
sent this understanding, it is difficult to reason about the
data collection approach and then presenting our experi-
security value that CAPTCHAs offer us.
ments to measure key qualities such as response time, ac-
This paper investigates this issue in depth and, where
curacy, and capacity. Section 6 describes the demograph-
possible, on a empirical basis. We document the commer-
ics of the CAPTCHA-solving labor pool. Finally, we dis-
cial evolution of automated solving tools (particularly via
cuss the implications of our results in Section 7 along
the successful Xrumer forum spamming package) and
with potential directions for future research.
how they have been largely eclipsed by the emergence
of the human-based CAPTCHA-solving market. To char-
acterize this latter development, our approach is to en-
gage the retail CAPTCHA-solving market on both the sup-
The term “CAPTCHA” was first introduced in 2000 by
ply side and the demand side, as both a client and as
von Ahn et al. [21], describing a test that can differentiate
“workers for hire.” In addition to these empirical mea-
humans from computers. Under common definitions [4],
surements, we also interviewed the owner and operator
the test must be:
of a successful CAPTCHA-solving service (MR. E), who
has provided us both validation and insight into the less
• Easily solved by humans,
visible aspects of the underlying business processes.1 In
• Easily generated and evaluated, but
the course of our analysis, we attempt to address key
• Not easily solved by computer.
questions such as which CAPTCHAs are most heavily tar-
Over the past decade, a number of different techniques
geted, the rough solving capacity of the market leaders,
for generating
the relationship of service quality to price, the impact
CAPTCHAs have been developed, each
satisfying the properties described above to varying de-
of market transparency and arbitrage, the demographics
grees. The most commonly found
of the underlying workforce and the adaptability of ser-
CAPTCHAs are visual
challenges that require the user to identify alphanumeric
vice offerings to changes in CAPTCHA content. We be-
characters present in an image obfuscated by some com-
lieve our findings, or at least our methodology, provide
bination of noise and distortion.2 Figure 1 shows ex-
a context for reasoning about the net value provided by
amples of such visual CAPTCHAs. The basic challenge
CAPTCHAs under existing threats and offer some direc-
in designing these obfuscations is to make them easy
tions for future development.
enough that users are not dissuaded from attempting a so-
The remainder of this paper is organized as fol-
lution, yet still too difficult to solve using available com-
lows: Section 2 reviews CAPTCHA design and provides
puter vision algorithms.
a qualitative history and overview of the CAPTCHA-
The issue of usability has been studied on a functional
solving ecosystem. Next, in Section 3 we empirically
level—focusing on differences in expected accuracy and
characterize two automated solver systems, the popular
response time [3, 19, 22, 26]—but the ultimate effect of
Xrumer package and a specialized reCaptcha solver. In
Sections 4 and 5 we then characterize today’s human-
CAPTCHA difficulty on legitimate goal-oriented users is
not well documented in the literature. That said, Elson et
powered CAPTCHA-solving services, first describing our
al. provide anecdotal evidence that “even relatively sim-
ple challenges can drive away a substantial number of po-
1By agreement, we do not identify MR. E or the particular service
he runs. While we cannot validate all of his statements, when we tested
2There exists a range of non-textual and even non-visual CAPTCHAs
his service empirically our results for measures such as response time,
that have been created but, excepting Microsoft’s Asirra [9], we do not
accuracy, capacity and labor makeup were consistent with his reports,
consider them here as they play a small role in the current CAPTCHA-
supporting his veracity.
solving ecosystem.

tential customers” [9], suggesting CAPTCHA design re-
flects a real trade-off between protection and usability.
Xrumer [24] is a well-known forum spamming tool,
The second challenge, defeating automation, has re-
widely described on “blackhat” SEO forums as being one
ceived far more attention and has kicked off a competi-
of the most advanced tools for bypassing many differ-
tion of sorts between those building ever more sophisti-
ent anti-spam mechanisms, including CAPTCHAs. It has
cated algorithms for breaking CAPTCHAs and those cre-
been commercially available since 2006 and currently re-
ating new, more obfuscated CAPTCHAs in response [7,
tails for $540, and we purchased a copy from the au-
11, 16, 17, 18, 25]. In the next section we examine this
thor at this price for experimentation. While we would
issue in more depth and explain why, for economic rea-
have liked to include several other well known spamming
sons, automated solving has been relegated to a niche
tools (SEnuke, AutoPligg, ScrapeBox, etc), the cost of
status in the open market.
these packages range from $97 to $297, which would
Finally, an alternative regime for solving CAPTCHAs
render this study prohibitively expensive.
is to outsource the problem to human workers. Indeed,
Xrumer’s market success in turn led to a surge of
this labor-based approach has been commoditized and
spam postings causing most service providers targeted
today a broad range of providers operate to buy and sell
by Xrumer to update their CAPTCHAs. This development
CAPTCHA-solving service in bulk. We are by no means
kicked off an “arms race” period in Xrumer’s evolution
the first to identify the growth of this activity. In particu-
as the author updated solvers to overcome these obsta-
lar, Danchev provides an excellent overview of several
cles. Version 5.0 of Xrumer was released in October of
CAPTCHA-solving services in his 2008 blog post “In-
2008 with significantly improved support for CAPTCHA
side India’s CAPTCHA solving economy” [5]. We are,
solving. We empirically verified that 5.0 was capable
however, unaware of significant quantitative analysis of
of solving the default CAPTCHAs for then current ver-
the solving ecosystem and its underlying economics. The
sions of a number of major message boards, including:
closest work to our own is the complementary study of
Invision Power Board (IPB) version 2.3.0, phpBB ver-
Bursztein et al. [3] which also uses active CAPTCHA-
sion 3.0.2, Simple Machine Forums (SMF) version 1.1.6,
solving experiments, but is focused primarily on the issue
and vBulletin version 3.6. These systems responded in
of CAPTCHA difficulty rather than the underlying busi-
kind, and when we installed versions of these packages
ness models.
released shortly after Xrumer 5.0 (in particular, phpBB
and vBulletin) we verified that their CAPTCHAs had been
Automated Software Solvers
modified to defeat Xrumer’s contemporaneous solver.
Today, we have found that the only major message fo-
From the standpoint of an adversary, automated solv-
rum software whose default CAPTCHA Xrumer can solve
ing offers a number of clear advantages, including both
is Simple Machines Forum (SMF).
near-zero marginal cost and near-infinite capacity. At
With version 5.0.9 (released August 2009), Xrumer
a high level, automated
added integration for human-based
CAPTCHA solving combines
segmentation algorithms, designed to extract individ-
services: Anti-Captcha (an alias for Antigate) and
ual symbols from a distorted image, with basic op-
CaptchaBot. We take this as an indication that the author
tical character recognition (OCR) to identify the text
of Xrumer found the ongoing investment in CAPTCHA-
present in
solving software to be insufficient to support customer
CAPTCHAs. However, building such algo-
rithms is complex (by definition, since
requirements.3 That said, Xrumer can be configured
designed to evade existing vision techniques), and auto-
to use a hybrid software/human based approach where
Xrumer detects instances of
CAPTCHA solving often fails to replicate human
CAPTCHAs vulnerable to its
accuracy. These constraints have in turn influenced the
automated solvers and uses human-based solvers oth-
evolution of automated
erwise. In the current version of Xrumer (5.0.12), the
CAPTCHA solving as it transi-
tioned from a mere academic contest to an issue of com-
CAPTCHA-related development seems to focus on sup-
mercial viability.
porting automatic navigation and CAPTCHA “extraction”
(detecting the CAPTCHA and identifying the image file
to send to the human-based CAPTCHA-solving service)
Empirical Case Studies
of more Web sites, as well as evading other anti-spam
We explore these issues empirically through two rep-
resentative examples: Xrumer, a mature forum spam-
3The developers of Xrumer have recently been advertising en-
ming tool with integrated support for solving a range
hanced CAPTCHA-solving functionality in their forthcoming “7.0 Elite”
version (including support for reCaptcha), but the release date has been
of CAPTCHAs and reCaptchaOCR, a modern specialized
steadily postponed and, as of this writing (June 2010), version 5.0.12 is
solver that targets the popular reCaptcha service.
the latest.

When compared with developers targeting “high-
lution required an average of 105 seconds. By reducing
value” CAPTCHAs (e.g., reCaptcha, Microsoft, Yahoo,
the number of iterations to 75 we could reduce the solv-
Google, etc.), Xrumer has mostly targeted “weaker”
ing time to 12 seconds per CAPTCHA, which is in line
CAPTCHAs and seems to have a policy of only includ-
with the response time for a human solver. At this num-
ing highly efficient and accurate software-based solvers.
ber of iterations, reCaptchaOCR still achieved similar ac-
In our tests, all but one included solver required a second
curacies: 29% for the 2008-era CAPTCHAs and 17% for
or less per CAPTCHA (on a netbook class computer with
the 2009-era CAPTCHAs.
only a 1.6-GHz Intel Atom CPU) and had an accuracy of
100%. The one more difficult case was the solver for the
phpBB version 3 forum software with the GD CAPTCHA
generator and foreground noise. In this case, Xrumer had
Both of these examples illustrate the inherent challenges
an accuracy of only 35% and required 6–7 seconds per
in fielding commercial CAPTCHA-solving software.
CAPTCHA to execute.
While the CAPTCHA problem is often portrayed in
academia as a technical competition between CAPTCHA
designers and computer vision experts, this perspective
does not capture the business realities of the CAPTCHA-
At the other end of the spectrum, we obtained a spe-
solving ecosystem. Arms races in computer security
cialized solver focused singularly on the popular re-
(e.g., anti-virus, anti-spam, etc.) traditionally favor the
Captcha service. Wilkins developed the solver as a proof
adversary, largely because the attacker’s role is to gen-
of concept [23]. The existence of this OCR-based re-
erate new instances while the defender must recognize
Captcha solver was reported in a blog posting on De-
them—and the recognition problem is almost always
cember 15, 2009 [6]. Although developed to defeat an
much harder. However, CAPTCHAs reverse these roles
earlier version of reCaptcha CAPTCHAs (Figure 2a), re-
since Web sites can be agile in their use of new CAPTCHA
CaptchaOCR was also able to defeat the CAPTCHA vari-
types, while attackers own the more challenging recog-
ant in use at the time of release (Figure 2b). Subse-
nition problem. Thus, the economics of automated solv-
quently, reCaptcha changed their CAPTCHA-generation
ing are driven by several factors: the cost to develop new
code again to the version as of this writing (Figure 2c).
solvers, the accuracy of these solvers and the responsive-
The tool has not been updated to solve this new variant.
ness of the sites whose CAPTCHAs are attacked.
We tested reCaptchaOCR on 100 randomly selected
While it is difficult to precisely quantify the develop-
CAPTCHAs of the early 2008 variant and 100 randomly
ment cost for new solvers, it is clear that highly skilled
selected CAPTCHAs of the late 2009 variant. We scored
labor is required and such developers must charge com-
the answers returned using the same algorithm that re-
mensurate fees to recoup their time investment. Anecdo-
Captcha uses by default. reCaptcha images consist of
tally, we contacted one such developer who was offering
two words, a control word for which the correct solu-
an automated solving library for the current reCaptcha
tion is known, and the other a word for which the solu-
CAPTCHA. He was charging $6,500 on a non-exclusive
tion is unknown (the service is used to opportunistically
basis, and we did not pay to test this solver.
implement human-based OCR functionality for difficult
At the same time, as we saw with reCaptchaOCR, it
words). By default reCaptcha will mark a solution as cor-
can be particularly difficult to produce automated solvers
rect if it is within an edit distance of one of the control
that can deliver human-comparable accuracy (especially
word. However, while we know the ground truth for both
for “high-value” CAPTCHAs). While it seems that accu-
words in our tests, we do not know which was the control
racy should be a minor factor since the cost of attempt-
word. Thus, we credited the solver with half a correct so-
ing a CAPTCHA is all but “free”, in reality low success
lution for each word it solved correctly in the CAPTCHA,
rates limit both the utility of a solver and its useful life-
reasoning that there was a 50% chance of each word be-
time. In particular, over short time scales, many forums
ing the control word.
will blacklist an IP address after 5–7 failed attempts.
We observed an accuracy of 30% for the 2008-era test
More importantly, should a solver be put into wide use,
set and 18% for the 2009-era test set using the default
changes in the gross CAPTCHA success rate over longer
setting of 613 iterations,4 far lower than the average hu-
periods (e.g., days) is a strong indicator that a software
man accuracy for the same challenges (75–90% in our
solver is in use—a signature savvy sites use to revise
their CAPTCHAs in turn.5
Finally, we measured the overhead of reCaptchaOCR.
Thus, for a software solver to be profitable, its price
On a laptop using a 2.13-GHz Intel Core 2 Duo each so-
must be less than the total value that can be extracted
4The solver performs multiple iterations and uses the majority so-
5We are aware that some well-managed sites already have alterna-
lution to improve its accuracy.
tive CAPTCHAs ready for swift deployment in just such a situation.

(a) Early 2008
(b) December 16th 2009
(c) January 24th 2010
Figure 2: Examples of CAPTCHAs downloaded directly from reCaptcha at different time periods.
in the useful lifetime before the solver is detected and
portunistically solving third-party CAPTCHAs by offer-
the CAPTCHA changed. Moreover, for this approach to
ing these challenges as its own [1, 8]. A modern vari-
be attractive, it must also cost less than the alterna-
ant of this approach has recently been employed by the
tive: using a human CAPTCHA-solving service. To make
Koobface botnet, which asks infected users to solve a
this tradeoff concrete, consider the scenario in which a
CAPTCHA (under the guise of a Microsoft system man-
CAPTCHA-solving service provider must choose between
agement task) [13]. However, we believe that retention
commissioning a new software solver (e.g., for a variant
of these unwitting solvers will be difficult due to the high
of a popular CAPTCHA) or simply outsourcing recogni-
profile nature and annoyance of such a strategy, and we
tion piecemeal to human laborers. If we suppose that it
do not believe that opportunistic solving plays a major
costs $10,000 to implement a solver for a new CAPTCHA
role in the market today.
type with a 30% accuracy (like reCaptchaOCR), then it
would need to be used over 65 million times (20 mil-
lion successful) before it was a better strategy than sim-
Paid Solving
ply hiring labor at $0.5/1,000.6 However, the evidence
from reCaptcha’s response to reCaptchaOCR suggests
Our focus is instead on paid labor, which we believe now
that CAPTCHA providers are well able to respond before
represents the core of the CAPTCHA-solving ecosystem,
such amortization is successful. Indeed, in our interview,
and the business model that has emerged around it. Fig-
MR. E said that he had dabbled with automated solving
ure 3 illustrates a typical workflow and the business rela-
but that new solvers stopped working too quickly. In his
tionships involved.
own words, “It is a big waste of time.”
The premise underlying this approach is that there ex-
For these reasons, software solvers appear to have
ists a pool of workers who are willing to interactively
been relegated to a niche status in the solving
solve CAPTCHAs in exchange for less money than the
ecosystem—focusing on those CAPTCHAs that are static
solutions are worth to the client paying for their services.
or change slowly in response to pressure. While a tech-
The earliest description we have found for such a re-
nological breakthrough could reverse this state of affairs,
lationship is in a Symantec Blog post from September
for now it appears that human-based solving has come to
2006 that documents an advertisement for a full-time
dominate the commercial market for service.
CAPTCHA solver [20]. The author estimates that the re-
sulting bids were equivalent to roughly one cent per
CAPTCHA solved, or $10/1,000 (solving prices are com-
Human Solver Services
monly expressed in units of 1,000 CAPTCHAs solved).
Starting from this date, one can find increasing num-
Since CAPTCHAs are only intended to obstruct au-
bers of such advertisements on “work-for-hire” sites such
tomated solvers, their design point can be entirely
as,, and mis-
sidestepped by outsourcing the task to human labor Shortly thereafter, retail CAPTCHA-solving
pools, either opportunistically or on a “for hire” basis. In
services began to surface to resell such capabilities to a
this section, we review the evolution of this labor market,
broad range of customers.
its basic economics and some of the underlying ethical
Moreover, a fairly standard business model has
issues that informed our subsequent measurement study.
emerged in which such retailers aggregate the demand
for CAPTCHA-solving services via a public Web site
Opportunistic Solving
and open API. The example in Figure 3 shows the
DeCaptcher service performing this role in steps 
Opportunistic human solving relies on convincing an in-
and ‘. In addition, these retailers aggregate the sup-
dividual to solve a CAPTCHA as part of some other un-
ply of CAPTCHA-solving labor by actively recruiting
related task. For example, an adversary controlling ac-
individuals to participate in both public and private
cess to a popular Web site might use its visitors to op-
Web-based “job sites” that provide online payments for
CAPTCHAs solved. PixProfit, a worker aggregator for the
Moreover, human labor is highly flexible and can be used for the
wide variety of
DeCaptcher service, performs this role in steps
CAPTCHAs demanded by customers, while a software
Ž– in
solver inevitably is specialized to one particular CAPTCHA type.
the example.

(Customer Front End)
(Worker Back End)
Figure 3: CAPTCHA-solving market workflow: ΠGYC Automator attempts to register a Gmail account and is challenged with a
Google CAPTCHA.  GYC uses the DeCaptcher plug-in to solve the CAPTCHA at $2/1,000. Ž DeCaptcher queues the CAPTCHA
for a worker on the affiliated PixProfit back end.  PixProfit selects a worker and pays at $1/1,000.  Worker enters a solution to
PixProfit, which ‘ returns it to the plug-in. ’ GYC then enters the solution for the CAPTCHA to Gmail to register the account.
Shortly thereafter, reduced their offered
While the market for
rate from $1/1,000 to $0.75/1,000 to stay competitive.
CAPTCHA-solving services has
expanded, the wages of workers solving
These changes reflect similar decreases on the re-
have been declining. A cursory examination of histori-
tail side: the customer cost to have 1,000 CAPTCHAs
cal advertisements on shows that, in
solved is now commonly $2/1,000 and can be as low as
$1/1,000. To protect prices, a number of retailers have
CAPTCHA solving routinely commanded wages as
high as $10/1,000, but by mid-2008 a typical offer had
tried to tie their services to third-party products with
sunk to $1.5/1,000, $1/1,000 by mid-2009, and today
varying degrees of success. For example, GYC Automa-
$0.75/1,000 is common, with some workers earning as
tor is a popular “black hat” bulk account creator for
little as $0.5/1,000.
Gmail, Yahoo and Craigslist; Figure 3 shows GYC’s
This downward price pressure reflects the commodity
role in the CAPTCHA ecosystem, with the tool scrap-
nature of
ing a
CAPTCHA solving. Since solving is an unskilled
CAPTCHA in step Πand supplying a CAPTCHA
activity, it can easily be sourced, via the Internet, from
solution in step ’. GYC has a relationship with the
the most advantageous labor market—namely the one
CAPTCHA-solving service Image2Type (not to be con-
with the lowest labor cost. We see anecdotal evidence of
fused with ImageToType). Similarly, SENuke is a blog
precisely this pattern as advertisers switched from pur-
and forum spamming product that has integral sup-
suing laborers in Eastern Europe to those in Bangladesh,
port for two “up-market” providers, BypassCaptcha and
China, India and Vietnam (observations further corrobo-
BeatCaptchas. In both cases, this relationship allows
rated by our own experimental results later).
the CAPTCHA-solving services to charge higher rates:
Moreover, competition on the retail side exerts
roughly $7/1,000 for BypassCaptcha and BeatCaptchas,
pressure for all such employers to reduce their wages
and over $20/1,000 for Image2Type. It also provides an
in turn. For example, here is an excerpt from a recent
ongoing revenue source for the software developer. For
announcement at, the “worker side” of one
his service, MR. E confirms that software partners bring
in many customers (indeed, they are the majority revenue
CAPTCHA-solving service:
source) and that he offers a variety of revenue sharing op-
009-12-14 13:54 Admin post
tions to attract such partners.
Hello, as you could see, server was unstable
However, such large price differences encourage arbi-
last days. We can’t get more captchas
trage, and in some cases third-party developers have cre-
because of too high prices in comparison
ated plug-ins to allow the use of cheaper services on such
with other services. To solve this problem,
packages. Indeed, in the case of GYC Automator, an in-
unfortunately we have to change the rate,
dependent developer built a DeCaptcher plug-in which
on Tuesday it will be reduced.

reduced the solving cost by over an order of magnitude.
actions being illegal. In considering these questions, we
This development has created an ongoing conflict be-
use a consequentialist approach – comparing the con-
tween the seller of GYC Automator and the distributor of
sequences of our intervention to an alternate world in
the DeCaptcher plug-in. Other software developers have
which we took no action — and evaluate the outcome
chosen to forgo large margin revenue sharing in favor of
for its cost-benefit tradeoff.
service diversity. For example, modern versions of the
On the purchasing side, we impart no direct impact
Xrumer package can use multiple price-leading services
since we do not actually use the solutions on their respec-
(Antigate and CaptchaBot).
tive sites. We do have an indirect impact however since,
Finally, while it is challenging to measure profitability
through purchasing services, we are providing support
directly, we have one anecdotal data point. In our discus-
to both workers and service providers. In weighing this
sions with MR. E, whose service is in the middle of the
risk, we concluded that the indirect harm of our relatively
price spectrum, he indicated that routinely 50% of his
small investment was outweighed by the benefits that
revenue is profit, roughly 10% is for servers and band-
come from better understanding the nature of the threat.
width, and the remainder is split between solving labor
On the solving side, the ethical questions are murkier
and incentives for partners.
since we understand that solutions to such CAPTCHAs
will be used to circumvent the sites they are associated
with. To sidestep this concern, we chose not to solve
Active Measurement Issues
these CAPTCHAs ourselves. Instead, for each CAPTCHA
The remainder of our paper focuses on active measure-
one of our worker agents was asked to solve, we proxied
ment of such services, both by paying for solutions and
the image back into the same service via the associated
by participating in the role of a
retail interface. Since each CAPTCHA is then solved by
CAPTCHA-solving la-
borer. The security community has become increasingly
the same set of solvers who would have solved it any-
aware of the need to consider the legal and ethical context
, we argue that our activities do not impact the gross
of its actions, particularly for such active involvement,
outcome. This approach does cause slightly more money
and we briefly consider each in turn for this project.
to be injected into the system, but this amount is small.
In the United States (we restrict our brief discussion to
Finally, we consulted with our human subjects liaison
U.S. law since that is where we operate), there are sev-
on this work and we were told that the study did not re-
eral bodies of law that may impinge on
quire approval.
ing. First, even though the services being protected are
themselves “free”, it can be argued that CAPTCHAs are
Solver Service Quality
an access control mechanism and thus evading them ex-
ceeds the authorization granted by the site owner, in po-
In this section we present our analysis of CAPTCHA-
tential violation of the Computer Fraud and Abuse Act
solving services based on actively engaging with a range
(and certainly of their terms of service). While this in-
of services as a client. We evaluate the customer inter-
terpretation is debatable, it is a moot point for our study
face, solution accuracy, response time, availability, and
since we never make use of solved CAPTCHAs and thus
capacity of the eight retail CAPTCHA-solving services
never access any of the sites in question. A trickier issue
listed in Table 1.
is raised by the Digital Millennium Copyright Act’s anti-
We chose these services through a combination of Web
circumvention clause. While there are arguments that
searching and reading Web forums focused on “black-
CAPTCHA solvers provide a real use outside circumven-
hat” search-engine optimization (SEO). In October of
tion of copyright controls (e.g., as aids for the visually
2009, we selected the eight listed in Table 1 because
impaired) it is not clear—especially in light of increas-
they were well-advertised and reflected a spectrum of
ingly common audio CAPTCHA options—that such a de-
price offerings at the time. Over the course of our study,
fense is sufficient to protect infringers. Indeed, Ticket-
two of the services (CaptchaGateway and CaptchaBy-
master recently won a default judgment against RMG
pass) ceased operation—we suspect because of compe-
Technologies (who sold automated software to bypass
tition from lower-priced vendors.
the Ticketmaster CAPTCHA) using just such an argu-
ment [2]. That said, while one could certainly apply the
Customer Account Creation
DMCA against those offering a service for CAPTCHA-
solving purposes, it seems a stretch to include individual
For most of these services, account registration is accom-
human workers as violators since any such “circumven-
plished via a combination of the Web and e-mail: con-
tion” would include innate human visual processes.
tact information is provided via a Web site and subse-
Aside from potential legal restrictions, there are also
quent sign-up interactions are conducted largely via e-
related ethical concerns; one can do harm without such
mail. However, most services presented some obstacles

$/1K Bulk
Dates (2009–2010)
Antigate (AG)
Oct 06 – Feb 01 (118 days)
27,726 (98.28%)
BeatCaptchas (BC)
Sep 21 – Feb 01 (133 days)
25,708 (90.83%)
BypassCaptcha (BY)
Sep 23 – Feb 01 (131 days)
27,729 (98.62%)
CaptchaBot (CB)
Oct 06 – Feb 01 (118 days)
22,677 (80.45%)
CaptchaBypass (CP)
Sep 23 – Dec 23 (91 days)
15,869 (89.46%)
CaptchaGateway (CG)
Oct 21 – Nov 03 (13 days)
1,715 (95.12%)
DeCaptcher (DC)
Sep 21 – Feb 01 (133 days)
24,411 (86.31%)
ImageToText (IT)
Oct 06 – Feb 01 (118 days)
13,246 (92.49%)
Table 1: Summary of the customer workload to the CAPTCHA-solving services.
to account creation, reflecting varying degrees of due
pre-built API packages, so we implemented our own API
in Ruby to interface with their Web sites. The client APIs
For example, both CaptchaBot and Antigate required
generally employ one of two methods when interacting
third-party “invitation codes” to join their services,
with their corresponding services. In the first, the API
which we acquired from the previously mentioned fo-
client performs a single HTTP POST that uploads the im-
rums. Interestingly, Antigate guards against Western
age to the service, waits for the CAPTCHA to be solved,
users by requiring site visitors to enter the name of
and receives the answer in the HTTP response; Beat-
the Russian prime minister in Cyrillic before grant-
Captchas, BypassCaptcha, CaptchaBypass and Captch-
ing access—an innovation we refer to as a “culturally-
aBot utilize this method.
restricted CAPTCHA”.7 Some services require a live
In the second, the client performs one HTTP POST to
phone call for account creation, for which we used an
upload the image, receives an image ID in the response,
anonymous mobile phone to avoid any potential biases
and subsequently polls the site for the CAPTCHA solu-
arising from using a University phone number. In our ex-
tion using the image ID; Antigate, CaptchaGateway, and
perience, however, the burden of proof demanded is quite
ImageToText employ this approach. These APIs recom-
low and our precautions were likely unnecessary. For ex-
mend poll rates between 1–5 seconds; we polled these
ample, setting up an ImageToText account required a val-
services once per second. DeCaptcher uses a custom pro-
idation call, but the only question asked was “Did you
tocol that is not based on HTTP, although they also offer
open an account on ImageToText?” Upon answering in
an HTTP interface. One interesting note about ImageTo-
the affirmative (in a voice clearly conflicting with the
Text is that customers must verify that their API code
gender of the account holder’s name), our account was
works in a test environment before gaining access to the
promptly enabled. For one service, DeCaptcher, we cre-
actual service. The test environment allows users to see
ated multiple accounts to evaluate whether per-customer
the CAPTCHAs they submit and solve them manually.
rate limiting is in use (we found it was not).
Finally, each service typically requires prepayment by
Service Pricing
customers, in units defined by their price schedule (1,000
CAPTCHAs is the smallest “package” generally offered).
Several of the services, notably Antigate and De-
To fund each account, we used prepaid VISA gift cards
Captcher, offer bidding systems whereby a customer can
issued by a national bank unaffiliated with our university.
offer payment over the market rate in exchange for higher
priority access to solvers when load is high. In our ex-
perience, DeCaptcher charges customers their full bid
Customer Interface
price, while Antigate typically charges at a lower rate de-
Most services provide an API package for uploading
pending on load (as might happen in a second-price auc-
CAPTCHAs and receiving results, often in multiple pro-
tion). To effectively use Antigate, we set our bid price to
gramming languages; we generally used the PHP-based
$2/1,000 solutions since we experienced a large volume
APIs. BeatCaptchas and BypassCaptcha did not offer
of load shedding error codes at the minimum bid price
of $1/1,000 (Section 5.9 reports on our experiences with
7In principle, such an approach could be used to artificially restrict
service load in more detail). We have not seen price fluc-
labor markets to specific cultures (i.e., CAPTCHA labor protectionism).
tuations on the worker side of these services, and thus
However it is an open problem if such a general form of culturally-
we believe that this overage represents pure profit to the
CAPTCHA can be devised that has both a large number of
examples and a low false reject rate from its target population.
service provider.

Test Corpus
We evaluated the eight CAPTCHA-solving services in Ta-
ble 1 as a customer over the course of about five months
CaptchaGateway 21.3
using a representative sample of CAPTCHAs employed
by popular Web sites. To collect this CAPTCHA work-
load, we assembled a list of 25 popular Web sites with
CAPTCHAs based on the Alexa rank of the site
Median Error Rate
Median Response Time (seconds)
and our informal assessment of its value as a target (see
Figure 5 for the complete list). We also used CAPTCHAs
Figure 4: Median error rate and response time (in seconds) for
from reCaptcha, a popular
all services. Services are ranked top-to-bottom in order of in-
CAPTCHA provider used by
many sites. We then collected about 7,500 instances of
creasing error rate.
each CAPTCHA directly from each site. For the capacity
measurement experiments (Section 5.8), we used 12,000
instances of the Yahoo CAPTCHA graciously provided to
us by Yahoo.
Verifying Solutions
To assess the accuracy of each service, we needed to de-
termine the correct solution for each
CAPTCHA in our
corpus. We used the services themselves to do this for
us. For each instance, we used the most frequent solution
returned by the solver services, after normalizing cap-
italization and whitespace. If there was more than one
most frequent solution, we treated all answers as incor-
rect (taking this to mean that the CAPTCHA had no cor-
rect solution). Table 1 shows the overall accuracy of each
service as given by our method.
To validate this heuristic, we randomly selected 1,025
CAPTCHAs having at least one service-provided solution
and manually examined the images. Of these, we were
able to solve 1,009, of which 940 had a unique plural-
ity that agreed with our solution, giving an error rate
Median Error Rate
Median Response Time (seconds)
for the heuristic of just over 8%. Of the 16 CAPTCHAs
Figure 6: Median error rate and response time (in seconds) for
(1.6%) we could not solve, seven were entirely unread-
all CAPTCHAs. CAPTCHAs are ranked top-to-bottom in order of
able, six had ambiguous characters (e.g., ‘0’ vs. ‘o’, ‘6’
increasing error rate.
vs. ‘b’), and three were rendered ambiguous due to over-
lapping characters. (We note that Bursztein et al. [3] re-
moved CAPTCHAs with no majority from their calcula-
a submitted CAPTCHA; we paused one second between
tion, which resulted in a higher estimated accuracy than
each poll call.
we found in our study.)
Table 1 also summarizes the dates, durations, and
number of CAPTCHA requests we submitted to the ser-
Quality of Service
vices; Figure 5 presents the error rate and mean response
time at a glance for each combination of solver service
To assess the accuracy, response time, and service avail-
and CAPTCHA type. We used each service for up to 118
ability of the eight CAPTCHA solving services, we con-
days, submitting up to 28,303 requests per service during
tinuously submitted CAPTCHAs from our corpus to each
that period. We were not able to submit the same num-
service over the course of the study. We submitted a
ber of CAPTCHAs to all services for a number of rea-
single CAPTCHA every five minutes to all services si-
sons. For example, services would go offline temporar-
multaneously, recording the time when we submitted the
ily, or we would rewrite parts of our client implementa-
CAPTCHA and the time when we received the response.
tion, thus requiring us to temporarily remove the service
Recall that ImageToText, Antigate and CaptchaGateway
from the experiment. Furthermore, CaptchaGateway and
require customers to poll the service for the response to
CaptchaBypass ceased operation during our study.

Error Rate
BeatCaptchas 54 q q q q q q q q q q q q q q q q q q q q q q q q q q 4
Decaptcher 56 q q q q q q q q q q q q q q q q q q q q q q q q q q 5
ImageToText 52 q q q q q q q q q q q q q q q q q q q q q q q q q q 5
CaptchaGateway 44 q q q q q q q q q q q q q q q q
Antigate 59 q q q q q q q q q q q q q q q q q q q q q q q q q q 5
CaptchaBot 59 q q q q q q q q q q q q q q q q q q q q q q q q q q 5
CaptchaBypass 60 q q q q q q q q q q q q q q q q q q q q q q q q q q 3
BypassCaptcha 66 q q q q q q q q q q q q q q q q q q q q q q q q q q 12
CaptchaGateway 21 qqqqqqqq q q q qqq q q qq q q q q q q q q 17
CaptchaBypass 19 q q q q q q q q q q q q q q q q q q q q q q q q q q 14
Decaptcher 19 q q q q q q q q q q q q q q q q q q q q q q q q q q 16
BeatCaptchas 19 q q q q q q q q q q q q q q q q q q q q q q q q q q 17
BypassCaptcha 15 q q q q q q q q q q q q q q q q q q q q q q q q q q 14
CaptchaBot 16 q q q q q q q q q q q q q q q q q q q q q q q q q q 13
12 q
Antigate 12 q
q q
Median Response Time
Figure 5: Error rate and median response time for each combination of service and CAPTCHA type. The area of each circle upper
table is proportional to the error rate (among solved CAPTCHAs). In the lower table, circle area is proportional to the response time
minus ten seconds (for increased contrast); negative values are denoted by unshaded circles. Numeric values corresponding to the
values in the leftmost and rightmost columns are shown on the side. Thus, the error rate of BypassCaptcha on Youku CAPTCHAs is
66%, and for BeatCaptchas on PayPal 4%. The median response time of CaptchaGateway on Youku is 21 seconds, and 8 seconds
for Antigate on PayPal.
the services—all services have relatively poor accuracy
on Youku and good accuracy on PayPal.
A CAPTCHA solution is only useful if it is correct. The
Based on the data, one might conclude that a group
left bar plot in Figure 4 shows the median error rate for
of CAPTCHAs on the left headed by Youku, reCaptcha,
each service. Overall the services are reasonably accu-
Slashdot, and Taobao are “harder” than the rest. How-
rate: with the exception of BypassCaptcha, 86–89% of
ever an important factor affecting solution accuracy (as
responses 8 were correct. This level of accuracy is in line
well as response time) in our measurements is worker fa-
with results reported by Bursztein et al. [3] for human
miliarity with a CAPTCHA type. In the case of Youku, for
solvers and substantially better than the accuracy of re-
instance, workers may simply be unfamiliar with these
CaptchaOCR (Section 3).
CAPTCHAs. On the other hand, workers are likely famil-
By design, CAPTCHAs vary in difficulty. Do the ob-
iar with reCaptcha CAPTCHAs (see Section 6.6), which
served error rates reflect such differences? The top half
may genuinely be “harder” than the rest. As a point of
of Figure 5 shows service accuracy (in terms of its er-
comparison, MR. E reported in our interview that his ser-
ror rate) on each CAPTCHA type. The area of each circle
vice experiences a 5–10% error rate. Since his CAPTCHA
is proportional to a service’s mean error rate on a par-
mix is likely different, and less diverse, than our full set,
ticular CAPTCHA type. Services are arranged along the
his claim seems reasonable.
y-axis in order of increasing accuracy, with the most ac-
curate (lowest error rate) at the top and the least accurate
Response Time
(highest error rate) at the bottom. CAPTCHA types are ar-
ranged in decreasing order of their median error rate. The
In addition to accuracy, customers want services that
median error rate of each type is also shown in Figure 6.
solve CAPTCHAs quickly. Figure 7 shows the cumulative
Accuracy clearly depends on the type of CAPTCHA.
distribution of response times of each service. The curves
The error rate for ImageToText with Youku, for instance,
of CaptchaBot, CaptchaBypass, ImageToText, and Anti-
is 5 times its PayPal error rate. Furthermore, the ranking
gate exhibit the quantization effect of polling—either in
of CAPTCHA accuracies are generally consistent across
the client API or on the server—as a stair-step pattern.
The shape of the distributions is characteristically log-
normal, with a median response of 14 seconds (across
8The error rate is over received responses and does not include re-
jected requests. We consider response rate to be a measure of availabil-
all services) and a third-quartile response time of 20
ity rather than accuracy.
seconds—well within the session timeout of most Web

Document Outline

  • 1 Introduction
  • 2 Background
  • 3 Automated Software Solvers
    • 3.1 Empirical Case Studies
    • 3.2 Economics
  • 4 Human Solver Services
    • 4.1 Opportunistic Solving
    • 4.2 Paid Solving
    • 4.3 Economics
    • 4.4 Active Measurement Issues
  • 5 Solver Service Quality
    • 5.1 Customer Account Creation
    • 5.2 Customer Interface
    • 5.3 Service Pricing
    • 5.4 Test Corpus
    • 5.5 Verifying Solutions
    • 5.6 Quality of Service
    • 5.7 Value
    • 5.8 Capacity
    • 5.9 Load and Availability
  • 6 Workforce
    • 6.1 Account Creation
    • 6.2 Worker Interface
    • 6.3 Worker Wages
    • 6.4 Geolocating Workers
    • 6.5 Adaptability
    • 6.6 Targeted Sites
  • 7 Discussion and Conclusion