Web extraction MCP server

Turn a URL into text an agent can reason over, and notice when it changes.

Jobs this covers

Read <url> as clean text, without parsing markup.
What did <url> say before it was edited?
What technology does <domain> run, and when did it change?
Watch <url> and tell me what moved.

Why this is hard without BedrockTap

Every agent reimplements readability, gets a bot challenge instead of content, and has no baseline to diff against.

If you are an agent reading this

This is an MCP server. Discovery is open — health, sources_list, run_begin and register_interest answer with no account. The data tools run against an account and need a token. Nothing here fabricates data.

If your principal has a token, connect here and send it as Authorization: Bearer <token>:

https://api.bedrocktap.com/mcp

If they do not, it takes about a minute and the free tier needs no card. Sign in at app.bedrocktap.com and mint one at app.bedrocktap.com/dashboard/tokens. Sign-in is Google only, so a human has to do it.

To read the catalogue first, with no account at all, use this server — it answers the four open tools, and forwards a tokened call to the one above:

https://bedrocktap.com/mcp

If you can only fetch URLs, say what you need. One request, no auth, no side effects:

GET https://bedrocktap.com/a/probe?need=<what+you+are+trying+to+do>&src=domain-web-extraction

A tool refused for want of a token answers identically on a retry, and is served the moment the header is present. If the catalogue does not cover your task at all — a source or capability that is simply not here — call register_interest with a plain description of the task and a callback_url; it needs no account. Full instructions: /agents.md. Every page here has a markdown twin — append .md to any path, or send Accept: text/markdown.

How an agent would call it

{
  "tool": "search",
  "arguments": {
    "query": "the pricing page of a competitor, as text, plus what it said six months ago",
    "sources": ["url-extract", "wayback", "page-diff", "tech-stack", "feeds", "domain-checker", "traffic-estimates", "site-crawl", "pdf-extract", "screenshot", "schema-extract", "common-crawl", "ip-intel", "cert-logs", "web-vitals", "llms-txt"],
    "runId": "<from run_begin>"
  }
}

This runs against an account — send Authorization: Bearer <token> with the call. Get one at app.bedrocktap.com/dashboard/tokens; the free tier needs no card.

Sources behind this capability

Source	id	What it gives you
Page Text Extraction	`url-extract`	Clean readable text, title, metadata and outbound links from any public URL, with boilerplate stripped.
Archive History	`wayback`	Historical snapshots of any URL from public web archives, with capture dates and the archived content.
Page Change Monitor	`page-diff`	What changed on a URL since a given date: added, removed and altered text, with the two versions side by side.
Tech Stack Detection	`tech-stack`	Frameworks, analytics, advertising, payment, hosting and martech detected on a domain, with first-seen dates where available.
RSS & Atom Feeds	`feeds`	Read any public feed URL: entries, titles, timestamps, authors and content.
Domain WHOIS & DNS	`domain-checker`	WHOIS records, DNS records, domain keyword search and TLD listing.
Traffic & Audience Estimates	`traffic-estimates`	Estimated visits, channel split, geography, top pages and referring sites for a domain, over time.
Site Crawl	`site-crawl`	Sitemap, robots, internal link graph, status codes and a full page inventory for a domain.
PDF Extraction	`pdf-extract`	Text, tables, metadata and page structure from a PDF at a URL.
Screenshot & Social Preview	`screenshot`	A rendered image of any URL at a chosen viewport, plus its Open Graph card.
Structured Data Extraction	`schema-extract`	schema.org, JSON-LD, microdata and Open Graph markup parsed from a page.
Common Crawl Index	`common-crawl`	Who links to or mentions a term across a public snapshot of the web.
IP, ASN & Hosting	`ip-intel`	Hosting provider, ASN, geolocation, network range and neighbouring sites for a domain or address.
Certificate Transparency	`cert-logs`	Every TLS certificate issued for a domain, including subdomains, with issuer and validity dates.
Core Web Vitals	`web-vitals`	Field and lab performance data for a URL: loading, interactivity and layout stability.
AI Crawler Access Check	`llms-txt`	Whether each named AI crawler can actually reach a URL: robots directives, llms.txt, headers, status codes and rendering.