> ## Documentation Index
> Fetch the complete documentation index at: https://docs.opencompany.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Document Processing

> RAG pipeline nodes for document ingestion, processing, and vector storage

# Document Processing

MachinaOs provides a complete RAG (Retrieval-Augmented Generation) pipeline for processing documents, generating embeddings, and storing vectors for semantic search.

## Pipeline Overview

```
[HTTP Scraper] --> [File Downloader] --> [Document Parser] --> [Text Chunker] --> [Embedding Generator] --> [Vector Store]
```

| Stage       | Node                | Purpose                            |
| ----------- | ------------------- | ---------------------------------- |
| 1. Collect  | HTTP Scraper        | Scrape URLs from web pages         |
| 2. Download | File Downloader     | Download files in parallel         |
| 3. Parse    | Document Parser     | Extract text from documents        |
| 4. Chunk    | Text Chunker        | Split text into overlapping chunks |
| 5. Embed    | Embedding Generator | Generate vector embeddings         |
| 6. Store    | Vector Store        | Store and query vectors            |

***

## HTTP Scraper

Scrapes links from web pages with support for pagination and date ranges.

### Modes

| Mode   | Description                  | Use Case                     |
| ------ | ---------------------------- | ---------------------------- |
| single | One request to the URL       | Simple page scraping         |
| date   | Iterate through a date range | News archives, dated content |
| page   | Iterate through page numbers | Multi-page listings          |

### Parameters

<ParamField path="url" type="string" required>
  URL template. Use `{date}` (date mode) or `{page}` (page mode) placeholders for iteration.
</ParamField>

<ParamField path="iteration_mode" type="select" default="single">
  Iteration mode: single, date, or page
</ParamField>

<ParamField path="link_selector" type="string" default="a[href$=&#x22;.pdf&#x22;]">
  CSS selector for links to extract from each fetched page
</ParamField>

<ParamField path="headers" type="string" default="{}">
  JSON string of request headers
</ParamField>

<ParamField path="start_date" type="string">
  Start date for date mode (YYYY-MM-DD)
</ParamField>

<ParamField path="end_date" type="string">
  End date for date mode (YYYY-MM-DD)
</ParamField>

<ParamField path="date_placeholder" type="string" default="{date}">
  Token in the URL replaced with each date (date mode)
</ParamField>

<ParamField path="start_page" type="number" default="1">
  First page number (page mode)
</ParamField>

<ParamField path="end_page" type="number" default="10">
  Last page number, inclusive (page mode)
</ParamField>

<ParamField path="max_pages" type="number" default="10">
  Safety cap on the number of pages fetched (1-1000)
</ParamField>

<ParamField path="use_proxy" type="boolean" default="false">
  Route requests through a residential proxy provider. When enabled, exposes `proxy_provider`, `proxy_country`, and `session_type` (rotating/sticky).
</ParamField>

### Output

```json theme={null}
{
  "items": [
    {"url": "https://example.com/article1.pdf", "text": "Article 1", "source_url": "https://news.site/archive/2025-01-01"},
    {"url": "https://example.com/article2.pdf", "text": "Article 2", "source_url": "https://news.site/archive/2025-01-01"}
  ],
  "item_count": 2,
  "errors": []
}
```

### Example: Scrape News Archive

```
URL: https://news.site/archive/{date}
Iteration Mode: date
Start Date: 2025-01-01
End Date: 2025-01-31
Link Selector: a.article-link
```

***

## File Downloader

Downloads files from URLs in parallel using semaphore-based concurrency.

### Parameters

<ParamField path="output_dir" type="string" default="downloads">
  Directory to save downloaded files. Defaults to a `downloads/` folder inside the workflow workspace.
</ParamField>

<ParamField path="max_workers" type="number" default="4">
  Maximum parallel downloads (1-32)
</ParamField>

<ParamField path="skip_existing" type="boolean" default="true">
  Skip files that already exist
</ParamField>

<ParamField path="timeout" type="number" default="60">
  Download timeout in seconds (1-600)
</ParamField>

### Input

Accepts an array of `items` with a `url` field (from HTTP Scraper), or a plain `urls` list:

```json theme={null}
{
  "items": [
    {"url": "https://example.com/file1.pdf"},
    {"url": "https://example.com/file2.pdf"}
  ]
}
```

### Output

Counts plus a `files` array describing the successful downloads:

```json theme={null}
{
  "downloaded": 2,
  "skipped": 0,
  "failed": 0,
  "files": [
    {"status": "downloaded", "url": "...", "path": "/output/file1.pdf", "filename": "file1.pdf", "size": 1024000},
    {"status": "downloaded", "url": "...", "path": "/output/file2.pdf", "filename": "file2.pdf", "size": 2048000}
  ],
  "output_dir": "/output"
}
```

***

## Document Parser

Parses documents to extract text content using configurable parsers.

### Parsers

| Parser        | Description          | Supported Formats        |
| ------------- | -------------------- | ------------------------ |
| PyPDF         | Fast PDF parsing     | PDF                      |
| Marker        | GPU-accelerated OCR  | PDF (scanned documents)  |
| Unstructured  | Multi-format parsing | PDF, DOCX, HTML, TXT, MD |
| BeautifulSoup | HTML parsing         | HTML                     |

### Parameters

<ParamField path="parser" type="select" default="pypdf">
  Parser to use: pypdf, marker, unstructured, beautifulsoup
</ParamField>

<ParamField path="file_path" type="string">
  Single file path to parse (takes precedence over `input_dir`)
</ParamField>

<ParamField path="input_dir" type="string">
  Directory to scan for files matching `file_pattern`
</ParamField>

<ParamField path="file_pattern" type="string" default="*.pdf">
  Glob pattern used when scanning `input_dir` (e.g. `*.pdf`, `*.html`)
</ParamField>

### Input

Set `file_path` for a single file, or `input_dir` + `file_pattern` for a directory scan.

### Output

```json theme={null}
{
  "documents": [
    {
      "source": "/output/file1.pdf",
      "filename": "file1.pdf",
      "content": "Extracted text content...",
      "length": 24500,
      "parser": "pypdf"
    }
  ],
  "parsed_count": 2,
  "failed": []
}
```

### Parser Comparison

| Feature      | PyPDF | Marker    | Unstructured | BeautifulSoup |
| ------------ | ----- | --------- | ------------ | ------------- |
| Speed        | Fast  | Slow      | Medium       | Fast          |
| Accuracy     | Good  | Excellent | Good         | Good          |
| OCR          | No    | Yes (GPU) | Optional     | No            |
| Formats      | PDF   | PDF       | Many         | HTML          |
| GPU Required | No    | Yes       | No           | No            |

***

## Text Chunker

Splits text into overlapping chunks for embedding generation.

### Strategies

| Strategy  | Description                           | Best For               |
| --------- | ------------------------------------- | ---------------------- |
| recursive | Split by paragraphs, sentences, words | Most documents         |
| markdown  | Preserve markdown structure           | Markdown files         |
| token     | Falls back to recursive splitting     | Consistent chunk sizes |

### Parameters

<ParamField path="strategy" type="select" default="recursive">
  Chunking strategy: recursive, markdown, or token
</ParamField>

<ParamField path="chunk_size" type="number" default="1000">
  Target chunk size in characters (100-8000)
</ParamField>

<ParamField path="overlap" type="number" default="200">
  Overlap between chunks (0-1000)
</ParamField>

### Input

Accepts a `documents` array (with a `content` field, from Document Parser) or a plain `text` string:

```json theme={null}
{
  "documents": [
    {"source": "...", "content": "Long document text..."}
  ]
}
```

### Output

```json theme={null}
{
  "chunks": [
    {
      "source": "/output/file1.pdf",
      "chunk_index": 0,
      "content": "Chunk 1 text...",
      "length": 1000
    }
  ],
  "chunk_count": 50
}
```

### Choosing Chunk Size

| Document Type  | Recommended Size | Overlap |
| -------------- | ---------------- | ------- |
| Technical docs | 500-1000         | 100-200 |
| Articles       | 1000-1500        | 200-300 |
| Books          | 1500-2000        | 300-400 |
| Code           | 500-800          | 100-150 |

***

## Embedding Generator

Generates vector embeddings from text chunks using various providers.

### Providers

| Provider    | Model                  | Dimensions | Cost         |
| ----------- | ---------------------- | ---------- | ------------ |
| HuggingFace | BAAI/bge-small-en-v1.5 | 384        | Free (local) |
| HuggingFace | BAAI/bge-large-en-v1.5 | 1024       | Free (local) |
| OpenAI      | text-embedding-3-small | 1536       | Paid         |
| OpenAI      | text-embedding-3-large | 3072       | Paid         |
| Ollama      | nomic-embed-text       | 768        | Free (local) |

### Parameters

<ParamField path="provider" type="select" default="huggingface">
  Embedding provider: huggingface (local), openai, or ollama (local server)
</ParamField>

<ParamField path="model" type="string" default="BAAI/bge-small-en-v1.5">
  Model name for the chosen provider. Override when switching provider (e.g. `text-embedding-3-small` for OpenAI).
</ParamField>

<ParamField path="batch_size" type="number" default="32">
  Batch size for embedding generation (1-256)
</ParamField>

<ParamField path="api_key" type="string">
  OpenAI API key. Passed as a node parameter (only used when provider is openai).
</ParamField>

### Input

Accepts a `chunks` array (from Text Chunker), each with a `content` field:

```json theme={null}
{
  "chunks": [
    {"content": "Chunk text...", "source": "...", "chunk_index": 0}
  ]
}
```

### Output

Parallel arrays: `embeddings` (one vector per chunk) plus the original `chunks` echoed back for pairing.

```json theme={null}
{
  "embeddings": [[0.123, -0.456, ...]],
  "embedding_count": 50,
  "dimensions": 384,
  "chunks": [{"content": "Chunk text...", "source": "...", "chunk_index": 0}],
  "provider": "huggingface",
  "model": "BAAI/bge-small-en-v1.5"
}
```

***

## Vector Store

Stores and queries vector embeddings using various backends.

### Backends

| Backend  | Description        | Best For                    |
| -------- | ------------------ | --------------------------- |
| ChromaDB | Local SQLite-based | Development, small datasets |
| Qdrant   | High-performance   | Production, large datasets  |
| Pinecone | Cloud-hosted       | Serverless, managed service |

### Operations

| Operation | Description                      |
| --------- | -------------------------------- |
| store     | Add embeddings to the store      |
| query     | Search for similar documents     |
| delete    | Remove documents by ID or filter |

### Parameters

<ParamField path="backend" type="select" default="chroma">
  Vector store backend: chroma, qdrant, or pinecone
</ParamField>

<ParamField path="operation" type="select" default="store">
  Operation: store, query, or delete
</ParamField>

<ParamField path="collection_name" type="string" default="documents">
  Collection/index name
</ParamField>

<ParamField path="top_k" type="number" default="5">
  Number of results for the query operation (1-100)
</ParamField>

<ParamField path="query_embedding" type="array">
  Single query vector for the query operation (from Embedding Generator)
</ParamField>

<ParamField path="ids" type="array">
  Vector IDs to remove for the delete operation
</ParamField>

<ParamField path="persist_dir" type="string" default="./data/vectors">
  ChromaDB persistence directory (chroma backend)
</ParamField>

<ParamField path="qdrant_url" type="string" default="http://localhost:6333">
  Qdrant server URL (qdrant backend)
</ParamField>

<ParamField path="pinecone_api_key" type="string">
  Pinecone API key, passed as a node parameter (pinecone backend)
</ParamField>

### Store Operation

Input (from Embedding Generator) — pass both `embeddings` and `chunks`:

```json theme={null}
{
  "embeddings": [[0.123, -0.456, ...]],
  "chunks": [{"content": "...", "source": "file1.pdf", "chunk_index": 0}]
}
```

Output:

```json theme={null}
{
  "stored_count": 50,
  "collection_count": 50,
  "collection_name": "documents",
  "backend": "chroma",
  "operation": "store"
}
```

### Query Operation

Query with a precomputed embedding vector (embed your question first, then feed it here):

```json theme={null}
{
  "query_embedding": [0.123, -0.456, ...],
  "top_k": 5
}
```

Output — the match shape differs by backend (`distance` for Chroma, `score` for Qdrant/Pinecone):

```json theme={null}
{
  "matches": [
    {
      "id": "a1b2c3...",
      "document": "Relevant chunk text...",
      "metadata": {"source": "file1.pdf", "chunk_index": 5},
      "distance": 0.11
    }
  ],
  "collection_name": "documents",
  "backend": "chroma",
  "operation": "query"
}
```

***

## Complete RAG Pipeline Example

### Workflow

```
[HTTP Scraper] --> [File Downloader] --> [Document Parser] --> [Text Chunker] --> [Embedding Generator] --> [Vector Store]
```

### Configuration

1. **HTTP Scraper**
   * URL: `https://docs.example.com/api/{page}`
   * Iteration Mode: page
   * Link Selector: `a.pdf-link`

2. **File Downloader**
   * Output Dir: `./downloads`
   * Max Workers: 10

3. **Document Parser**
   * Parser: pypdf

4. **Text Chunker**
   * Strategy: recursive
   * Chunk Size: 1000
   * Overlap: 200

5. **Embedding Generator**
   * Provider: huggingface
   * Model: BAAI/bge-small-en-v1.5

6. **Vector Store**
   * Backend: chroma
   * Operation: store
   * Collection: api-docs

### Querying the Pipeline

Create a separate workflow for querying:

```
[Webhook Trigger] --> [Embedding Generator] --> [Vector Store (query)] --> [AI Agent] --> [Webhook Response]
```

The AI Agent receives relevant context from the vector store to answer questions.

***

## Tips

<Tip>
  Start with **ChromaDB** for development - no setup required.
</Tip>

<Tip>
  Use **HuggingFace** embeddings for free local processing without API keys.
</Tip>

<Tip>
  Adjust **chunk size** based on your embedding model's context window.
</Tip>

<Tip>
  Include meaningful **metadata** (source, page, section) for better retrieval context.
</Tip>

<Warning>
  GPU is required for the **Marker** parser (OCR). Use **PyPDF** for non-scanned documents.
</Warning>

***

## Related

<CardGroup cols={2}>
  <Card title="AI Agents" icon="robot" href="/nodes/ai-agent">
    Use RAG context with AI agents
  </Card>

  <Card title="Webhooks" icon="webhook" href="/nodes/webhooks">
    Trigger and respond over HTTP
  </Card>

  <Card title="AI Models" icon="brain" href="/nodes/ai-models">
    AI providers for generation
  </Card>

  <Card title="Schedulers & Triggers" icon="clock" href="/nodes/schedulers">
    Run the RAG pipeline on a schedule
  </Card>
</CardGroup>