Ground RAG in {fresh content}

Collect web pages and documents, keep a source with every chunk, and refresh the content that changes. Build your retrieval pipeline with Scrape, Crawl, Parse, and Monitors.

Get API KeyNo card required

Book a Call

Collect, index, and keep your content current

Collect web content

Scrape a page or crawl a site. Use Batches when you need to collect a large URL list in the background.

const page = await
  client.web.scrape({
    url: 'https://example.com',
    formats: { markdown: true },
  });

const text = page.markdown.data;

Include your documents

Parse uploaded files into Markdown, then chunk and index them alongside web content with source references.

const document = await
  client.parse.handle(
    createReadStream('report.pdf'),
    { extension: 'pdf' },
  );

const text = document.markdown;

Refresh changed sources

Use Monitor change events to schedule a new scrape and replace the affected chunks in your retrieval index.

const monitor = await
  client.monitors.create({
    name: 'Knowledge base source',
    target: {
      type: 'page',
      url: 'https://example.com',
    },
    change_detection: { type: 'exact' },
    schedule: {
      type: 'interval',
      frequency: 6,
      unit: 'hours',
    },
    webhook: {
      url: 'https://app.example.com/hooks',
      events: ['change.detected'],
    },
  });