See what AI crawlers and agents can read on your site, and what they miss.
Find the pages and settings that get in the way. See the evidence, then give your team or coding agent a clear brief to fix it.
Free: 500 credits every month for Crawl, Scrape and Map. Visibility needs a paid plan or bought credits. No credit card required.
Check your website’s crawlability and content, then see what to fix.
A clear path for crawlers and agents
Map lists every URL, Scrape turns pages into readable Markdown, Crawl checks pages and returns findings with evidence, and AI visibility tracks how models describe the brand, so people and AI tools find the same story.
How it works
1. Enter your URL
Enter your website. We check the pages we find, up to the free Crawl allowance.
2. See findings
Filter the findings and inspect the page evidence behind each one.
3. Fix and export
Copy a Markdown fix brief for each finding, or fetch findings as JSON and fix briefs as JSON or Markdown from the API. Scrape also downloads Markdown and JSON.
Map
Find the URLs before you choose the pages.
Map lists URLs from sitemaps and links. This Map of www.novagentum.ai returned 29 URLs, ready to filter and choose for a Crawl or Scrape. No page content is stored.
Learn about Map- 29 URLs
- 2 sitemaps
- 27 links
- /
- /solutions
- /company
- /pricing
- /login?next=%2Fdashboard
- /signup
- /crawl
- /scrape
- /visibility
- /docs/api
- /solutions/business-owners
- /solutions/marketing-teams
- /solutions/seo-teams
- /solutions/agencies
- /our-approach
- /examples
- /contact
- /tools/llms-txt-checker
- /tools/structured-data-checker
- /tools/robots-txt-checker
- /tools
- /compare/chatgpt
- /compare/firecrawl
- /benchmarks
- /imprint
- /privacy-policy
- /terms
- /examples/novagentum-ai
- /examples/hubql
Scrape
Any page, as clean Markdown.
Turn one page or a batch of up to 10 into Markdown, cleaned HTML, links and metadata for your agent or content workflow. PDFs become Markdown too.
Learn about Scrape- 277 words
- HTTP 200
HTML chrome
- <nav> · <footer>
- utility classes
- scripts and banners
Markdown
# Help the right people find your business.
Opening line from the /company scrape.
Crawl
Every finding points to a page.
Crawl follows your sitemap and links across up to 100 pages and renders pages that need JavaScript. Each finding includes the affected pages and a fix brief you can copy for your team or coding agent.
Learn about Crawl- 10 pages
- 3 findings
/docs/api · medium
Needs a clear primary heading
2 H1 elements detected.
/solutions · medium
Has no structured data
No parseable JSON-LD was detected on this page.
/company · medium
Needs headings that answer buyer questions
0 question-led headings detected.
AI visibility
See how AI models describe you.
Visibility asks ChatGPT, Claude, Perplexity and Gemini the questions your buyers ask, then keeps every answer and cited source as evidence. Answers come from each model's API, so they can differ from what people see in the apps.
Learn about AI visibility- Named by 3 of 4
- 4 answers kept
Buyer question · example
Which tools check whether AI crawlers can read my website?
- ChatGPT2nd of 5 tools · 3 sources citedNamed
- Claude4th of 6 tools · 2 sources citedNamed
- PerplexityNot named · 7 sources citedMissing
- Gemini1st of 4 tools · 4 sources citedNamed
MCP and API
Works where you already work.
Connect Claude, ChatGPT, Cursor or VS Code to the Novagentum MCP server and ask for a Crawl, Scrape or Map in plain words. Each connection gets its own spending limit. Your own code can call the REST API. Every route uses the same credits and prices as the dashboard.
Connect an AI toolNovagentum MCP
Server URL
https://www.novagentum.ai/api/mcp
Crawl www.novagentum.ai and give me a fix brief for the top finding.
- novagentum_crawlwww.novagentum.ai · 10 pages
- novagentum_crawl_findings3 findings
- novagentum_fix_brief/docs/api · primary heading
The top finding: /docs/api has two H1 headings. Here is the brief to give your team or coding agent…
What our research found
26% of readable files (131 of 507) block an AI crawler by name.
Source: Novagentum robots.txt research, Tranco top 1,000, 10 November 2026. 507 readable robots.txt files of 1,000.
On hubql.com, Markdown used 94.3% fewer tokens than the same pages as HTML (17.6 times fewer). On novagentum.ai, Markdown used 86.7% fewer tokens (7.5 times fewer).
Source: Novagentum measurements of hubql.com and novagentum.ai on 10 November 2026. Token counts use OpenAI’s o200k_base encoding.
Start free. Keep going with Starter.
Free: 500 credits every month for Crawl, Scrape and Map. Visibility needs a paid plan or bought credits. Plan members can buy extra credits: 5,000 credits for $10.
Compare plans| Plan | Price | Credits each month |
|---|---|---|
| Starter | $29/month | 10,000 |
| Pro | $79/month | 30,000 |
| AgencyShared team workspace | $199/month | 75,000 |
Frequently asked questions
- Is it free?
- Free: 500 credits every month for Crawl, Scrape and Map. Visibility needs a paid plan or bought credits. Free credits renew on the first of each month (UTC) and don't roll over. One allowance per confirmed email address, so email aliases and extra accounts don't add more. They stop while you're on a paid plan. Sign up and confirm your email to get your free credits. No credit card required.
- What counts as a credit?
- 500 credits = $1 of usage. Crawl: 250 credits for the first page, then 5 credits per 5 additional pages, rounded up. For a paid Crawl, choose up to 100 pages and review the maximum credits before starting. Scrape costs 1 credit per page. Visibility is quoted in credits before you run it.
- Do you respect robots.txt?
- Yes. Crawls respect robots.txt. We do not fetch pages robots.txt disallows.
- What data do you store?
- We store findings and the page evidence needed to show results, plus page content for Scrape and content crawls so a run can be reopened. If you sign in, we also store your account email. You can delete a run; account deletion removes associated stored runs on a best-effort basis. We do not sell that data.
Check your site free.
Enter your website. You'll get the findings, the pages behind them and a fix brief for each one.
Start with your website.
Check your website’s crawlability and content, then see what to fix.