Crawl and Map

While Search finds needles in the haystack and Extract parses specific needles, Crawl and Map allow you to systematically explore the entire haystack. These endpoints are designed to traverse web infrastructure starting from a base URL.

⚠️ Note on Crawl Accessibility: The Crawl endpoint is highly resource-intensive and is currently available on an invite-only basis or specific enterprise tiers. For access information, please visit crawl.tavily.com.

Tavily Crawl

The Crawl feature lets you traverse a website's content, extracting pages as it navigates. What makes Tavily's crawler unique is the instructions parameter—you can provide natural language instructions, and an AI agent will actively decide which links to click to satisfy your prompt, avoiding irrelevant pages.

Example: Targeted AI Crawling

Suppose you want to scrape documentation for a specific feature, but you don't know the exact URLs.

from tavily import TavilyClient

client = TavilyClient()

# Crawl Wikipedia starting at 'Lemon', but instruct the AI to only 
# surface and extract pages specifically about citrus fruits.
response = client.crawl(
    url="https://wikipedia.org/wiki/Lemon",
    max_depth=3,
    max_breadth=5,
    limit=20,
    instructions="Find all pages discussing the cultivation of citrus fruits.",
    format="markdown"
)

for result in response["results"]:
    print(f"Extracted URL: {result['url']}")
    # result['raw_content'] contains the markdown of the page

Crawl Parameters

Navigating the web requires strict boundaries to prevent infinite loops. Use these parameters to control the crawler's behavior:

  • max_depth (int): The maximum number of clicks away from the starting URL. (e.g., Depth 1 is the start page, Depth 2 are pages linked from the start page).
  • max_breadth (int): The maximum number of links to follow per page at a given depth.
  • limit (int): A hard cap on the total number of pages the API will extract, regardless of depth/breadth. This protects your API credit balance.
  • instructions (str): Natural language instructions to guide the crawler's decision-making process on which links are "relevant" to follow.
  • select_paths / exclude_paths (Sequence[str]): strict URL path filtering.
    • Example: select_paths=["/docs/", "/blog/"] ensures it only crawls URLs containing those substrings.
  • select_domains / exclude_domains (Sequence[str]): Restrict or expand the crawler's domain scope.
  • allow_external (bool): By default, the crawler stays on the domain of the starting URL. Set to True to allow it to follow links to other websites.
  • extract_depth / format: Same as the Extract API, determining how the final pages are parsed.

Tavily Map

Map operates very similarly to Crawl, but with one critical difference: Map does not extract page content.

Instead, Map navigates the site and returns a structural blueprint—a simple list of discovered URLs. Because it skips the heavy content extraction phase, Map is significantly faster and cheaper to run.

When to use Map

  • Site Audits: You want to get a list of all URLs on a website to build a sitemap.
  • Pre-Scraping Strategy: You want to discover all URLs first, filter the list locally in your Python script, and then selectively pass the final list to the Extract API.

Example: Discovering a Site Structure

response = client.map(
    url="https://docs.tavily.com",
    max_depth=2,
    limit=50,
    # Only find pages related to the Python SDK
    select_paths=["/python"]
)

print("Discovered Site Structure:")
# The response is just a list of URL strings
for url_string in response["results"]:
    print(f"- {url_string}")

Map accepts all the same navigation parameters as Crawl (max_depth, select_paths, allow_external, etc.), but omits content-specific parameters like format.