vLLM本地部署教程：高性能大模型推理引擎怎么搭

什么是Crawl4AI

Crawl4AI是专为AI和大语言模型设计的开源爬虫工具。传统爬虫拿到的是满屏广告的脏HTML，Crawl4AI直接输出干净Markdown，可以直接喂给AI做RAG、训练、分析。GitHub 72.8K Star。

核心优势

一、安装

pip install crawl4ai

# 安装浏览器引擎（用于JS渲染）
crawl4ai install

二、抓取网页

from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)  # 干净的Markdown内容

import asyncio
asyncio.run(main())

三、批量抓取

urls = [
    "https://site1.com/article1",
    "https://site2.com/article2",
    "https://site3.com/article3"
]

async with AsyncWebCrawler() as crawler:
    results = await crawler.arun_many(urls)
    for r in results:
        print(f"{r.url}: {len(r.markdown)} chars")

四、配合RAG使用

# 抓取 → 切片 → 存入向量数据库
from crawl4ai import AsyncWebCrawler
from langchain.text_splitter import RecursiveCharacterTextSplitter

async def build_knowledge_base(urls):
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun_many(urls)
    
    splitter = RecursiveCharacterTextSplitter(chunk_size=500)
    chunks = []
    for r in results:
        chunks.extend(splitter.split_text(r.markdown))
    
    return chunks  # 存入向量数据库

提示：Crawl4AI抓取的数据可以直接存入 Dify RAG知识库 或 LangChain RAG 系统。

五、结构化提取

# 用CSS选择器提取特定字段
result = await crawler.arun(
    url="https://news-site.com",
    extraction_strategy={
        "type": "css",
        "selectors": {
            "title": "h1.article-title",
            "content": "div.article-body",
            "date": "span.publish-date"
        }
    }
)
print(result.extracted_data)