---

截图

---

细节

---

---

Cockroach Crawler 主要特点

• Open source
  Its code is available for free and can be checked or modified.
• Policy limits
  Operators can set time and resource limits for crawls.
• Version 0.2.0
  It is stable and ready for use.
• Bounded crawl jobs
  It performs controlled crawling with clear limits and output formats.
• Inspect the source
  You can see and review the code behind it.
• Node.js compatible
  It works with Node.js version 20.18 or higher.
• Three entry points
  You can run it using CLI, JavaScript API, or an agent adapter.
• Crawl the public web
  It can scan and collect data from the public web within set boundaries.
• Start in five minutes
  It is quick to set up and start using.
• Default network options
  It connects safely by default with redirects validated at every step.
• Explicit crawl boundary
  You can set a clear start and end point for each crawl.
• Chromium rendering options
  It can optionally load pages with a browser to see dynamic content, within resource limits.
• Local CLI and JavaScript API
  You can use it with a command line tool or program it with JavaScript.
• Outputs in Markdown and JSONL
  It provides data in easy-to-read formats.
• Denied routes do not reach extractor
  Blocked pages are not processed to save resources.
• Requests cross one reviewed boundary
  It only fetches pages within approved boundaries, ensuring safety.
• Convert pages into MD, JSON, or JSONL
  It turns web pages into Markdown, JSON, or JSONL files for easy use.
• Security is built into the crawl path
  It checks for unsafe content and rejects problematic URLs automatically.
• JavaScript API with options and callbacks
  You can customize the crawler's behavior and handle events in your code.
• Uses a tested provider and serverless contracts
  It works reliably with tested services and serverless setups.
• Enforces robots, network policy, and resource limits
  It follows rules for web crawling and manages network and memory use.

---

徽章

---

Cockroach Crawler 替代品

---

使用的技术

查看所有技术