← Back to blog
GeneralAbout 7 min read

Scrapling Got 99,000 Stars by Being Boring

Published Nov 8, 2026
Scrapling Got 99,000 Stars by Being Boring

Scrapling has 99,627 stars on GitHub as of this week. That is a strange number for a web scraping library, a category where the honest work is unglamorous and the popular projects tend to be either abandoned or rewritten every eighteen months.

Its README makes a large claim in the first two lines — "the BEST web scraping framework that handles everything from a single request to a full-scale crawl" — and then, unusually, spends the rest of the document earning it with specifics. Install commands that break if you skip a step. A Cloudflare challenge being solved in four lines of Python. A note that the parser-only install will raise ModuleNotFoundError the moment you import a fetcher, which is exactly the kind of thing a README usually hides.

That last detail is why I trust the rest of it.

The install is the tutorial

You need Python 3.10 or better. Then:

pip install scrapling

That gets you the parsing engine and nothing else. No fetchers, no CLI, no browsers. If you try to import from scrapling.fetchers, it fails — and the README says so in a warning box instead of letting you find out.

To actually fetch pages:

pip install "scrapling[fetchers]"
scrapling install

The second command downloads browsers, their system dependencies, and the fingerprint-manipulation layer. There are further extras if you need them: [ai] for the MCP server, [rag] for the RAG helpers, [shell] for the interactive scraping shell and the extract command, [all] for everything. There is a Docker image too, which is what most people will end up using in production.

The structure tells you who this is for. A data analyst who wants to parse one HTML file pays for nothing extra. Someone running a crawl pays the browser download once.

Four lines that used to be forty

The basic pattern is a session with a browser fingerprint attached:

with FetcherSession(impersonate='chrome') as session:
    page = session.get(url, stealthy_headers=True)
    quotes = page.css('.quote .text::text').getall()

That impersonate argument is doing real work. It makes the request look like the current version of Chrome at the TLS level, not just at the user-agent string. Anyone who has watched a scraper get blocked for no visible reason knows that the giveaway is usually the handshake, not the headers.

When a page needs JavaScript, the same selectors work against a real browser:

with StealthySession(headless=True, solve_cloudflare=True) as session:
    page = session.fetch(target, google_search=False)

And Scrapling keeps one API across all of it. page.css(), page.xpath(), text search, regex search — the same calls whether the page came from a raw HTTP request or a headless Chromium that just cleared an interstitial. That consistency is most of the value. Most scraping stacks force you to write two versions of every parser and then maintain both.

A mechanical keyboard in a dark room, teal and amber reflections on the keycaps

The feature that actually justifies the stars

Web scraping breaks constantly, and it almost never breaks loudly. A site changes a class name, your selector returns an empty list, and your pipeline keeps running and writing nulls into a database until someone notices a week later.

Scrapling's answer is adaptive element tracking: it stores enough information about a matched element that it can relocate it after the page structure changes, using similarity rather than an exact selector. There is also a "find similar elements" call that locates siblings of a matched element automatically — useful when a listing page adds a badge or wraps items in a new div.

This is not magic, and it will not save you from a full redesign. But the failure mode it addresses — a selector that silently stops matching — is the single most expensive one in this line of work. Anything that turns a silent failure into a fuzzy match is worth a dependency.

Why it is trending now, and not two years ago

The timing is not about scraping. It is about who is doing the scraping.

The repository ships an agent skill — listed on Clawhub, documented alongside the library — so an AI agent can call Scrapling as a tool rather than generating a one-off requests plus BeautifulSoup script every time it needs a page. There is also an MCP server behind the [ai] extra.

That is a different product category from "a library humans import." When an agent writes its own scraper, it tends to produce code that works once: no retry logic, no fingerprint handling, no idea what to do with a Cloudflare page. Handing it a tool that already handles all of that is cheaper than hoping the generated script is good.

So the star count is partly a bet on a workflow: not "I will write a scraper," but "my agent will need to read a page, and I want that to be one call."

A ten-minute evaluation

Do not judge it on a friendly page. Pick the site that broke your last scraper — the one with lazy-loaded listings and a bot check that appears after three requests.

Fetch it once with Fetcher and note whether you get the real HTML or a shell. Then fetch it with StealthyFetcher and see how long the browser takes to come up and clear the challenge. That number is your per-page budget; anything above a second or two changes how you architect a crawl.

Then deliberately break the selector. Change a class name in a saved copy of the page and re-run the parse. If adaptive tracking relocates the element, you have found the feature that will save you the most time over a year. If it returns nothing, you know the limit.

Last, point it at a page that returns a 404 and a page that times out. The error handling is where libraries usually reveal whether anyone has run them in production.

Where it fits next to a browser framework

Scrapling is not trying to replace Playwright, and the comparison is worth being clear about. Playwright is a browser automation library: it drives a browser and exposes everything the browser can do, including clicking, typing, waiting on network conditions, and taking screenshots. It is the right tool for testing a UI.

Scrapling is a scraping library that will use a browser when a page requires one. Its API is built around extracting data, not around reproducing user behaviour. If your job is "read this table from forty thousand product pages," that difference is the whole point; if your job is "log in, add to cart, verify the receipt," you want the automation tool.

The overlap is real, and most teams end up with both: Playwright for the flows that must be scripted, Scrapling for the pages that must be read.

What to watch out for

Solving Cloudflare challenges is a legitimate technical capability and a legally grey one depending on where you are and what you are collecting. The library will not tell you that. Terms of service, robots.txt, rate limits, and personal data rules still apply whether or not your client renders JavaScript.

Browser automation at scale is also heavy. Each stealthy session is a real browser. Ten thousand pages through StealthyFetcher is a compute bill, not a script. Start with plain Fetcher and only escalate the pages that actually need it.

And the README's confidence is a personality, not a guarantee. The project compares itself favorably to alternatives; those comparisons are the author's, run on the author's benchmarks. Treat them as a reason to test, not a result.

The takeaway

The interesting thing about Scrapling is not that it beats other scrapers on a benchmark. It is that a library whose main job is to be boring got 99,000 stars by being honest about the boring parts — the fingerprints, the browser install, the selectors that stop matching, the page that needs a challenge solved before it will talk to you.

Most of the web is still held together with divs and hope. Tooling that respects that is worth keeping.

Related articles