Web Crawl
Scrape any page or crawl a whole site into clean markdown
What it does
Fetches one or more web pages — or spiders an entire site — and returns each page as clean markdown, plus metadata, links, and harvested contact details (emails, phone numbers, social profiles).
How it works
Web Crawl is a native connector. No account connection is required. AINexLayer loads pages in a real browser, so JavaScript-rendered and lazy-loaded content is captured.
- Chat — describe a URL or site you want read or crawled. The agent runs the web crawler automatically.
- API Playground — open API Playground → Web → Crawl, fill in the form, and click Run.
Use in API Playground
- Open API Playground → Web → Crawl
- Add Start URLs — one or more seed URLs
- For a single page: leave Max crawl depth at
0 - For a small site crawl: set Max crawl depth to
1and Max crawl pages to10 - Optional: set Include URL patterns or Exclude URL patterns to control which links are followed
- Click Run
Contact details often live on about, contact, or privacy pages. Use Max crawl depth of at least 1 if you need emails or social links from a whole site.
Useful options
| Option | What it does |
|---|---|
| Max crawl depth | Link-hops from each seed (0 = seeds only; up to 5) |
| Max crawl pages | Total pages fetched per run, seeds included (max 200) |
| Max length | Max characters of markdown kept per page |
| Include URL patterns | Regexes a link must match to be followed |
| Exclude URL patterns | Regexes that block links from being followed |
What you get back
One item per fetched page with markdown content, metadata (title, description), crawl depth, links classified as internal/external/social/email/tel, and page-level contacts. The response also includes a site-wide contacts summary that deduplicates every email, phone, and social profile found across all pages.
Runs are billed per successfully fetched page. The price per page is shown on the Web Crawl card in API Playground.