See what AI crawlers and agents can read on your site, and what they miss.

Find the pages and settings that get in the way. See the evidence, then give your team or coding agent a clear brief to fix it.

Free: 500 credits every month for Crawl, Scrape and Map. Visibility needs a paid plan or bought credits. No credit card required.

Check your website’s crawlability and content, then see what to fix.

Crawls check up to 100 pages. Free accounts get 500 credits every month for Crawl, Scrape and Map. No AI answers.

Sign up free with your email next. We’ll keep your URL. No credit card required.

A clear path for crawlers and agents

Map lists every URL, Scrape turns pages into readable Markdown, Crawl checks pages and returns findings with evidence, and AI visibility tracks how models describe the brand, so people and AI tools find the same story.

How it works

  1. 1. Enter your URL

    Enter your website. We check the pages we find, up to the free Crawl allowance.

  2. 2. See findings

    Filter the findings and inspect the page evidence behind each one.

  3. 3. Fix and export

    Copy a Markdown fix brief for each finding, or fetch findings as JSON and fix briefs as JSON or Markdown from the API. Scrape also downloads Markdown and JSON.

Map

Find the URLs before you choose the pages.

Map lists URLs from sitemaps and links. This Map of www.novagentum.ai returned 29 URLs, ready to filter and choose for a Crawl or Scrape. No page content is stored.

Learn about Map
  • 29 URLs
  • 2 sitemaps
  • 27 links
  • /
  • /solutions
  • /company
  • /pricing
  • /login?next=%2Fdashboard
  • /signup
  • /crawl
  • /scrape
  • /visibility
  • /docs/api
  • /solutions/business-owners
  • /solutions/marketing-teams
  • /solutions/seo-teams
  • /solutions/agencies
  • /our-approach
  • /examples
  • /contact
  • /tools/llms-txt-checker
  • /tools/structured-data-checker
  • /tools/robots-txt-checker
  • /tools
  • /compare/chatgpt
  • /compare/firecrawl
  • /benchmarks
  • /imprint
  • /privacy-policy
  • /terms
  • /examples/novagentum-ai
  • /examples/hubql

Scrape

Any page, as clean Markdown.

Turn one page or a batch of up to 10 into Markdown, cleaned HTML, links and metadata for your agent or content workflow. PDFs become Markdown too.

Learn about Scrape
  • 277 words
  • HTTP 200

HTML chrome

  • <nav> · <footer>
  • utility classes
  • scripts and banners

Markdown

# Help the right people find your business.

Opening line from the /company scrape.

Crawl

Every finding points to a page.

Crawl follows your sitemap and links across up to 100 pages and renders pages that need JavaScript. Each finding includes the affected pages and a fix brief you can copy for your team or coding agent.

Learn about Crawl
  • 10 pages
  • 3 findings

/docs/api · medium

Needs a clear primary heading

2 H1 elements detected.

/solutions · medium

Has no structured data

No parseable JSON-LD was detected on this page.

/company · medium

Needs headings that answer buyer questions

0 question-led headings detected.

AI visibility

See how AI models describe you.

Visibility asks ChatGPT, Claude, Perplexity and Gemini the questions your buyers ask, then keeps every answer and cited source as evidence. Answers come from each model's API, so they can differ from what people see in the apps.

Learn about AI visibility
  • Named by 3 of 4
  • 4 answers kept

Buyer question · example

Which tools check whether AI crawlers can read my website?

  • ChatGPT2nd of 5 tools · 3 sources citedNamed
  • Claude4th of 6 tools · 2 sources citedNamed
  • PerplexityNot named · 7 sources citedMissing
  • Gemini1st of 4 tools · 4 sources citedNamed

MCP and API

Works where you already work.

Connect Claude, ChatGPT, Cursor or VS Code to the Novagentum MCP server and ask for a Crawl, Scrape or Map in plain words. Each connection gets its own spending limit. Your own code can call the REST API. Every route uses the same credits and prices as the dashboard.

Connect an AI tool

Novagentum MCP

Server URL

https://www.novagentum.ai/api/mcp

Crawl www.novagentum.ai and give me a fix brief for the top finding.

  • novagentum_crawlwww.novagentum.ai · 10 pages
  • novagentum_crawl_findings3 findings
  • novagentum_fix_brief/docs/api · primary heading

The top finding: /docs/api has two H1 headings. Here is the brief to give your team or coding agent…

What our research found

26% of readable files (131 of 507) block an AI crawler by name.

Source: Novagentum robots.txt research, Tranco top 1,000, 10 November 2026. 507 readable robots.txt files of 1,000.

On hubql.com, Markdown used 94.3% fewer tokens than the same pages as HTML (17.6 times fewer). On novagentum.ai, Markdown used 86.7% fewer tokens (7.5 times fewer).

Source: Novagentum measurements of hubql.com and novagentum.ai on 10 November 2026. Token counts use OpenAI’s o200k_base encoding.

Start free. Keep going with Starter.

Free: 500 credits every month for Crawl, Scrape and Map. Visibility needs a paid plan or bought credits. Plan members can buy extra credits: 5,000 credits for $10.

Compare plans
Monthly plans
PlanPriceCredits each month
Starter$29/month10,000
Pro$79/month30,000
AgencyShared team workspace$199/month75,000

Frequently asked questions

Is it free?
Free: 500 credits every month for Crawl, Scrape and Map. Visibility needs a paid plan or bought credits. Free credits renew on the first of each month (UTC) and don't roll over. One allowance per confirmed email address, so email aliases and extra accounts don't add more. They stop while you're on a paid plan. Sign up and confirm your email to get your free credits. No credit card required.
What counts as a credit?
500 credits = $1 of usage. Crawl: 250 credits for the first page, then 5 credits per 5 additional pages, rounded up. For a paid Crawl, choose up to 100 pages and review the maximum credits before starting. Scrape costs 1 credit per page. Visibility is quoted in credits before you run it.
Do you respect robots.txt?
Yes. Crawls respect robots.txt. We do not fetch pages robots.txt disallows.
What data do you store?
We store findings and the page evidence needed to show results, plus page content for Scrape and content crawls so a run can be reopened. If you sign in, we also store your account email. You can delete a run; account deletion removes associated stored runs on a best-effort basis. We do not sell that data.

Check your site free.

Enter your website. You'll get the findings, the pages behind them and a fix brief for each one.

Start with your website.

Check your website’s crawlability and content, then see what to fix.

Crawls check up to 100 pages. Free accounts get 500 credits every month for Crawl, Scrape and Map. No AI answers.

Sign up free with your email next. We’ll keep your URL. No credit card required.