Tavily Extract

The Extract API is a powerful utility within the Tavily SDK. While the Search API helps you find information, the Extract API is used when you already know the URLs you want to process.

Extract takes one or more web addresses and returns the raw, cleaned content of those pages. It handles the heavy lifting of bypassing bot protections, rendering Javascript (if needed), stripping out navigation menus and ads, and returning clean Markdown or plaintext.

When to use Extract

  • Data Pipelines: You have a database of URLs that you need to ingest into a Vector Database.
  • Agentic Workflows: Your AI agent found an interesting link on Twitter and needs to read the full article.
  • Web Scraping: You want to quickly parse the content of a specific blog post or documentation page without writing custom BeautifulSoup selectors.

Extracting Content

You can pass a single URL string or a list of URL strings. Tavily can process up to 20 URLs concurrently in a single API call, making it highly efficient for batch operations.

from tavily import TavilyClient

client = TavilyClient()

target_urls = [
    "https://en.wikipedia.org/wiki/Artificial_intelligence",
    "https://en.wikipedia.org/wiki/Machine_learning",
    "https://example.com/some-broken-link" # Intentional bad link
]

# Execute the extract request
response = client.extract(
    urls=target_urls,
    extract_depth="advanced",
    format="markdown"
)

Understanding the Response

Web scraping is inherently volatile. Websites go down, block IPs, or take too long to respond. The extract method gracefully handles this by splitting the response into results (successes) and failed_results (failures).

# 1. Process successful extractions
for result in response.get("results", []):
    print(f"\n✅ Successfully Extracted: {result['url']}")
    # 'raw_content' contains the parsed markdown
    print(f"Content snippet: {result['raw_content'][:150]}...")

# 2. Handle failures gracefully
if response.get("failed_results"):
    print("\n❌ Failed to extract the following URLs:")
    for failed in response["failed_results"]:
        print(f"- {failed['url']}: {failed['error']}")

Extract Parameters

Fine-tune the extraction process with the following arguments:

  • urls (str | List[str]): The URL or list of URLs to target.
  • extract_depth ("basic", "advanced"):
    • "basic": Fast standard HTTP GET parsing.
    • "advanced": Employs headless browsers to execute Javascript, wait for dynamic content to load, and bypass aggressive anti-bot measures. Slower, but much more reliable for modern web apps.
  • format ("markdown", "text"): The formatting style of the output raw_content. Markdown is generally preferred for LLMs as it preserves structural semantics (headers, lists, code blocks).
  • include_images (bool): If true, returns an array of image URLs found within the main content block of the page.
  • include_favicon (bool): Returns the URL of the site's favicon.
  • chunks_per_source (int): Optional. If provided, Tavily will pre-chunk the extracted text into roughly equal segments, which is highly useful if you intend to immediately embed the text for a Vector DB.
  • timeout (float): Custom timeout for the HTTP request (default is 30 seconds). If you are using extract_depth="advanced" on slow sites, you may need to increase this.