KNet Crawler
Crawl websites and turn them into indexed, searchable knowledge for your agents.
Placeholder. The overview, links and concepts below are accurate; the detailed guides are still being written. Anything not yet covered is listed under What is not documented yet.
KNet Crawler fetches web content on a schedule and indexes it so your agents and knowledge bases can search it. Use it when the source of truth lives on a website — documentation, a product catalogue, a public register — rather than in a file you can upload.
Open it: flow.knetai.com/krawlers
Where it fits
Crawler produces indexed content. It does not answer questions on its own.
- Crawler collects and indexes pages.
- Knowledge Bases stores and retrieves that content for retrieval-augmented generation.
- Workflow Builder consumes it in an agent.
- KNet Traces shows you what a retrieval actually returned when an answer looks wrong.
If you are trying to give an agent knowledge of a website, you will use Crawler and Knowledge Bases together.
Concepts
Crawl — one configured job: a starting URL, rules about how far to follow links, and a schedule.
Scope — the boundary of a crawl. Usually a domain or a path prefix. Getting this wrong is the most common problem: too narrow and you miss pages, too wide and you index the entire internet.
Index — the processed, searchable output of a crawl, consumed by knowledge bases.
What is not documented yet
- Configuring crawl scope and depth
- Scheduling and re-crawl behaviour
- Authentication for pages behind a login
- Handling JavaScript-rendered pages
- API and MCP endpoints
- Quotas and rate limiting
Getting help
Something here wrong or missing? Get in touch — a person reads every support request, and gaps in these pages are worth reporting.