cyberivy
Crawl4AIWeb ScrapingRAGAI AgentsOpen Source AIDeveloper ToolsMCPData Extraction

Crawl4AI turns web pages into clean Markdown for AI agents and RAG

November 18, 2026

Hellbeiges Banner mit dem grünen Crawl4AI Cloud Logo oben links, großer dunkler Schrift links und rechts einer weißen Karte mit der Zeile POST /scrape, grauen und grünen Balken als Platzhalter für Text und der Statuszeile 200 OK Markdown

The open source library Crawl4AI delivers web content as Markdown or JSON for language models. Version 0.9.5 from 14 October 2026 closes six flaws in the Docker server.

What this is about

Crawl4AI is an open source Python library that turns web pages into clean Markdown or structured JSON so that language models, retrieval augmented generation (RAG) systems and AI agents can work with them. The project lives on GitHub at unclecode/crawl4ai, was created in May 2024, is licensed under Apache 2.0 and had about 87,500 stars on 17 November 2026.

Why now: on 14 October 2026 version 0.9.5 shipped, which the changelog explicitly calls a security release for the self hosted Docker server. At the same time the team offers a hosted variant, Crawl4AI Cloud. Anyone using or considering the tool therefore has two questions to answer again: how current is my own server, and is the cloud worth it?

What Crawl4AI actually does

Crawl4AI drives a real browser through Playwright (Chromium, Firefox or WebKit), loads a page including JavaScript and returns the content as Markdown with headings, tables and code blocks. Filters such as BM25 or a pruning filter remove menus, footers and other clutter so less text goes to the language model. Links can optionally end up as a numbered reference list at the bottom.

For structured data there are two routes: schemas with CSS or XPath selectors that run without any language model, or extraction through a model of your choice, including a local one. On top come deep crawls (breadth first, depth first or by relevance) that can resume after a crash, an adaptive crawler that stops once it has gathered enough for a question, screenshots, PDFs, proxies, sessions with saved logins and a command line tool called crwl.

As a Docker server, Crawl4AI provides a REST API, a dashboard, a playground and an MCP endpoint. Since version 0.9.0 every endpoint requires a token. The cloud offers the same engine behind one API key and adds web search, Google results pages as data (beta) and an experimental answer function.

Why it matters

When teams feed language models with current web content, the model is rarely the problem. Dirty input is: cookie banners, navigation, duplicated paragraphs. Crawl4AI targets exactly that and can run entirely on your own infrastructure without sending pages through a third party service. For teams with data protection requirements that is a real difference.

The independent test by Geekflare (updated on 19 October 2026, version 0.9.5) calls it the most complete free option for Markdown and JSON from web pages. In the test the server was ready after 74 seconds, used 538 MB of memory at idle and crawled 50 pages in 13 seconds with a peak of 644 MB. The documentation still recommends 4 GB of RAM and 1 GB of shared memory for the container.

According to the changelog, version 0.9.5 closes six coordinated disclosure vulnerabilities in the Docker server, including an SSRF path, JavaScript injection through css_selector and wait_for, ReDoS through caller supplied regular expressions and Chromium being launched without certificate checks. Anyone running the server themselves should upgrade.

In plain language

Imagine cutting recipes out of magazines. Between the ingredients there are ads, prize draws and letters to the editor. Crawl4AI is the pair of scissors that cuts out only the recipe and glues it neatly onto an index card, so your cooking assistant does not accidentally read the detergent ad as well.

A practical example

An online shop with 12 employees wants to feed its support chatbot with its own help center: about 400 articles that change every week. The team starts the Docker server on an existing machine with 8 GB of RAM and runs a deep crawl with a maximum depth of 3 every Sunday. With Fit Markdown each page shrinks noticeably because navigation and footer are dropped. The results go into the chatbot's vector database.

For competitor price lists the team instead uses a CSS schema that outputs product name and price as JSON without paying for a language model. Only when a page is so nested that selectors break does it switch to extraction with a local model. A sensible first test: send ten of your own pages through crwl URL -o markdown and compare the result with the original.

Scope and limits

  • Law and fairness: Crawl4AI ships a stealth mode and tools against bot detection. Whether a crawl is allowed depends on terms of use, copyright and, for personal data, the GDPR. The tool does not take that assessment off anyone's hands.
  • Operations and security: The Docker server is a service that loads third party pages and processes input. Six security advisories in one release show that it needs maintenance. According to Geekflare the image is 8.2 GB; it should never be reachable without a token.
  • Maturity and cost: The package is classified as beta on PyPI. When self hosting you bring your own proxies and language model keys. According to its own documentation the cloud is in a soft launch, so prices and the free allowance may still change. According to Geekflare the LICENSE file adds an attribution request to Apache 2.0; check that before commercial use.

SEO & GEO keywords

Crawl4AI, Crawl4AI Cloud, unclecode, web crawler for LLMs, Markdown for RAG, web scraping Python, Playwright, MCP server, structured data extraction, open source crawler, Docker self hosting, AI agent web access

💡 In plain English

Crawl4AI loads web pages like a browser and returns only the actual content as tidy text or as a table. That saves language models work and cost. You can run it yourself for free or use it as a cloud service paid with credits.

Key Takeaways

  • →Crawl4AI is a Python library under Apache 2.0 that converts web pages into Markdown or JSON for language models.
  • →Version 0.9.5 from 14 October 2026 closes six security vulnerabilities in the self hosted Docker server.
  • →Structured data can be extracted with CSS or XPath schemas without any language model.
  • →The Docker server offers a REST API, dashboard and MCP endpoint and has required a token since version 0.9.0.
  • →Crawl4AI Cloud starts with 1,000 free credits per month; according to the documentation paid plans begin at 10 US dollars.
  • →Legal questions about terms of use, copyright and GDPR remain with the user.

FAQ

Is Crawl4AI free?

The library and Docker server are free under Apache 2.0; you pay for infrastructure, proxies and model keys yourself. The cloud offers 1,000 free credits a month and paid plans start at 10 US dollars; the provider still describes pricing as a soft launch.

Do I need a language model to extract data?

No. Extraction works without a model using CSS or XPath schemas. A language model, including a local one, is only needed for pages where fixed selectors are not enough.

Do I need to do anything after version 0.9.5?

If you run the Docker server yourself, upgrade to 0.9.5 or later because this release closes six reported vulnerabilities. Users of the Python library alone should check the changelog against their setup.

What hardware does the Docker server need?

The documentation recommends at least 4 GB of RAM and 1 GB of shared memory. In Geekflare's test idle usage was 538 MB.

May I read any website with Crawl4AI?

Not automatically. Terms of use, copyright and, for personal data, the GDPR still apply; the tool does not check that for you.

Sources & Context