Our integrations
Web Sites
Crawl public or private websites and convert pages into query-ready knowledge.
What does Web Sites integration does?
NovaLuna crawls URLs, extracts readable text, tables, and metadata, then keeps the dataset fresh via configurable re-crawl intervals.
Integration features available
- Auth via Basic, Cookie, or OAuth
- Robots.txt & rate-limit respect built in
- HTML → Markdown conversion
- Auto-snapshot & diff tracking
- Tag pages by sitemap section
What does Web Sites integration does?
NovaLuna crawls URLs, extracts readable text, tables, and metadata, then keeps the dataset fresh via configurable re-crawl intervals.
Integration features available
- Auth via Basic, Cookie, or OAuth
- Robots.txt & rate-limit respect built in
- HTML → Markdown conversion
- Auto-snapshot & diff tracking
- Tag pages by sitemap section
What does Web Sites integration does?
NovaLuna crawls URLs, extracts readable text, tables, and metadata, then keeps the dataset fresh via configurable re-crawl intervals.
Integration features available
- Auth via Basic, Cookie, or OAuth
- Robots.txt & rate-limit respect built in
- HTML → Markdown conversion
- Auto-snapshot & diff tracking
- Tag pages by sitemap section
