DocumentationKNet Crawlers
KNet Crawlers
Web crawlers that collect sites into structured data and searchable indexes.
KNet Crawlers crawls websites and turns them into data you can search or store: pages, documents and extracted fields, indexed for hybrid (keyword and semantic) search.
Saved Crawlers & Execution
- Crawler Cards: Inspect saved crawlers (such as
TestContinueAsNewandMy DeftHub) with tags (Standalone), trigger types (Manualor scheduled), and creation history. - + New crawler: Create crawl configurations with custom start URLs, depth constraints, domain filters, and extractors.
- Search & Filter: Quickly find crawlers across large enterprise workspaces.
Pages
| Page | What it is for |
|---|---|
| Crawlers | Manage your configured crawler jobs and trigger manual runs |
| Marketplace | Crawler agents to start a new crawler from, including the Native Web Crawler and published crawler K-Nets |
| Runs | Real-time crawl run logs, extracted pages, dataset downloads, and error diagnostics |
| Settings | Rate limits, proxy configurations, and storage destinations for this workspace |
Typical tasks
- Try the Native Web Crawler in its Playground, then New Crawler to save it with a start URL and limits, and run it.
- Send its pages to a Knowledge Base to answer from them.
See also KNet Crawler.