Document Processing

MachinaOs provides a complete RAG (Retrieval-Augmented Generation) pipeline for processing documents, generating embeddings, and storing vectors for semantic search.

Pipeline Overview

[HTTP Scraper] --> [File Downloader] --> [Document Parser] --> [Text Chunker] --> [Embedding Generator] --> [Vector Store]

Stage	Node	Purpose
1. Collect	HTTP Scraper	Scrape URLs from web pages
2. Download	File Downloader	Download files in parallel
3. Parse	Document Parser	Extract text from documents
4. Chunk	Text Chunker	Split text into overlapping chunks
5. Embed	Embedding Generator	Generate vector embeddings
6. Store	Vector Store	Store and query vectors

HTTP Scraper

Scrapes links from web pages with support for pagination and date ranges.

Modes

Mode	Description	Use Case
single	One request to the URL	Simple page scraping
date	Iterate through a date range	News archives, dated content
page	Iterate through page numbers	Multi-page listings

Parameters

url

string

required

URL template. Use {date} (date mode) or {page} (page mode) placeholders for iteration.

iteration_mode

select

default:"single"

Iteration mode: single, date, or page

link_selector

string

default:"a[href$=\".pdf\"]"

CSS selector for links to extract from each fetched page

headers

string

default:"{}"

JSON string of request headers

start_date

string

Start date for date mode (YYYY-MM-DD)

end_date

string

End date for date mode (YYYY-MM-DD)

date_placeholder

string

default:"{date}"

Token in the URL replaced with each date (date mode)

start_page

number

default:"1"

First page number (page mode)

end_page

number

default:"10"

Last page number, inclusive (page mode)

max_pages

number

default:"10"

Safety cap on the number of pages fetched (1-1000)

use_proxy

boolean

default:"false"

Route requests through a residential proxy provider. When enabled, exposes proxy_provider, proxy_country, and session_type (rotating/sticky).

Output

{
  "items": [
    {"url": "https://example.com/article1.pdf", "text": "Article 1", "source_url": "https://news.site/archive/2025-01-01"},
    {"url": "https://example.com/article2.pdf", "text": "Article 2", "source_url": "https://news.site/archive/2025-01-01"}
  ],
  "item_count": 2,
  "errors": []
}

Example: Scrape News Archive

URL: https://news.site/archive/{date}
Iteration Mode: date
Start Date: 2025-01-01
End Date: 2025-01-31
Link Selector: a.article-link

File Downloader

Downloads files from URLs in parallel using semaphore-based concurrency.

Parameters

output_dir

string

default:"downloads"

Directory to save downloaded files. Defaults to a downloads/ folder inside the workflow workspace.

max_workers

number

default:"4"

Maximum parallel downloads (1-32)

skip_existing

boolean

default:"true"

Skip files that already exist

timeout

number

default:"60"

Download timeout in seconds (1-600)

Input

Accepts an array of items with a url field (from HTTP Scraper), or a plain urls list:

{
  "items": [
    {"url": "https://example.com/file1.pdf"},
    {"url": "https://example.com/file2.pdf"}
  ]
}

Output

Counts plus a files array describing the successful downloads:

{
  "downloaded": 2,
  "skipped": 0,
  "failed": 0,
  "files": [
    {"status": "downloaded", "url": "...", "path": "/output/file1.pdf", "filename": "file1.pdf", "size": 1024000},
    {"status": "downloaded", "url": "...", "path": "/output/file2.pdf", "filename": "file2.pdf", "size": 2048000}
  ],
  "output_dir": "/output"
}

Document Parser

Parses documents to extract text content using configurable parsers.

Parsers

Parser	Description	Supported Formats
PyPDF	Fast PDF parsing	PDF
Marker	GPU-accelerated OCR	PDF (scanned documents)
Unstructured	Multi-format parsing	PDF, DOCX, HTML, TXT, MD
BeautifulSoup	HTML parsing	HTML

Parameters

parser

select

default:"pypdf"

Parser to use: pypdf, marker, unstructured, beautifulsoup

file_path

string

Single file path to parse (takes precedence over input_dir)

input_dir

string

Directory to scan for files matching file_pattern

file_pattern

string

default:"*.pdf"

Glob pattern used when scanning input_dir (e.g. *.pdf, *.html)

Input

Set file_path for a single file, or input_dir + file_pattern for a directory scan.

Output

{
  "documents": [
    {
      "source": "/output/file1.pdf",
      "filename": "file1.pdf",
      "content": "Extracted text content...",
      "length": 24500,
      "parser": "pypdf"
    }
  ],
  "parsed_count": 2,
  "failed": []
}

Parser Comparison

Feature	PyPDF	Marker	Unstructured	BeautifulSoup
Speed	Fast	Slow	Medium	Fast
Accuracy	Good	Excellent	Good	Good
OCR	No	Yes (GPU)	Optional	No
Formats	PDF	PDF	Many	HTML
GPU Required	No	Yes	No	No

Text Chunker

Splits text into overlapping chunks for embedding generation.

Strategies

Strategy	Description	Best For
recursive	Split by paragraphs, sentences, words	Most documents
markdown	Preserve markdown structure	Markdown files
token	Falls back to recursive splitting	Consistent chunk sizes

Parameters

strategy

select

default:"recursive"

Chunking strategy: recursive, markdown, or token

chunk_size

number

default:"1000"

Target chunk size in characters (100-8000)

overlap

number

default:"200"

Overlap between chunks (0-1000)

Input

Accepts a documents array (with a content field, from Document Parser) or a plain text string:

{
  "documents": [
    {"source": "...", "content": "Long document text..."}
  ]
}

Output

{
  "chunks": [
    {
      "source": "/output/file1.pdf",
      "chunk_index": 0,
      "content": "Chunk 1 text...",
      "length": 1000
    }
  ],
  "chunk_count": 50
}

Choosing Chunk Size

Document Type	Recommended Size	Overlap
Technical docs	500-1000	100-200
Articles	1000-1500	200-300
Books	1500-2000	300-400
Code	500-800	100-150

Embedding Generator

Generates vector embeddings from text chunks using various providers.

Providers

Provider	Model	Dimensions	Cost
HuggingFace	BAAI/bge-small-en-v1.5	384	Free (local)
HuggingFace	BAAI/bge-large-en-v1.5	1024	Free (local)
OpenAI	text-embedding-3-small	1536	Paid
OpenAI	text-embedding-3-large	3072	Paid
Ollama	nomic-embed-text	768	Free (local)

Parameters

provider

select

default:"huggingface"

Embedding provider: huggingface (local), openai, or ollama (local server)

model

string

default:"BAAI/bge-small-en-v1.5"

Model name for the chosen provider. Override when switching provider (e.g. text-embedding-3-small for OpenAI).

batch_size

number

default:"32"

Batch size for embedding generation (1-256)

api_key

string

OpenAI API key. Passed as a node parameter (only used when provider is openai).

Input

Accepts a chunks array (from Text Chunker), each with a content field:

{
  "chunks": [
    {"content": "Chunk text...", "source": "...", "chunk_index": 0}
  ]
}

Output

Parallel arrays: embeddings (one vector per chunk) plus the original chunks echoed back for pairing.

{
  "embeddings": [[0.123, -0.456, ...]],
  "embedding_count": 50,
  "dimensions": 384,
  "chunks": [{"content": "Chunk text...", "source": "...", "chunk_index": 0}],
  "provider": "huggingface",
  "model": "BAAI/bge-small-en-v1.5"
}

Vector Store

Stores and queries vector embeddings using various backends.

Backends

Backend	Description	Best For
ChromaDB	Local SQLite-based	Development, small datasets
Qdrant	High-performance	Production, large datasets
Pinecone	Cloud-hosted	Serverless, managed service

Operations

Operation	Description
store	Add embeddings to the store
query	Search for similar documents
delete	Remove documents by ID or filter

Parameters

backend

select

default:"chroma"

Vector store backend: chroma, qdrant, or pinecone

operation

select

default:"store"

Operation: store, query, or delete

collection_name

string

default:"documents"

Collection/index name

top_k

number

default:"5"

Number of results for the query operation (1-100)

query_embedding

array

Single query vector for the query operation (from Embedding Generator)

ids

array

Vector IDs to remove for the delete operation

persist_dir

string

default:"./data/vectors"

ChromaDB persistence directory (chroma backend)

qdrant_url

string

default:"http://localhost:6333"

Qdrant server URL (qdrant backend)

pinecone_api_key

string

Pinecone API key, passed as a node parameter (pinecone backend)

Store Operation

Input (from Embedding Generator) — pass both embeddings and chunks:

{
  "embeddings": [[0.123, -0.456, ...]],
  "chunks": [{"content": "...", "source": "file1.pdf", "chunk_index": 0}]
}

Output:

{
  "stored_count": 50,
  "collection_count": 50,
  "collection_name": "documents",
  "backend": "chroma",
  "operation": "store"
}

Query Operation

Query with a precomputed embedding vector (embed your question first, then feed it here):

{
  "query_embedding": [0.123, -0.456, ...],
  "top_k": 5
}

Output — the match shape differs by backend (distance for Chroma, score for Qdrant/Pinecone):

{
  "matches": [
    {
      "id": "a1b2c3...",
      "document": "Relevant chunk text...",
      "metadata": {"source": "file1.pdf", "chunk_index": 5},
      "distance": 0.11
    }
  ],
  "collection_name": "documents",
  "backend": "chroma",
  "operation": "query"
}

Complete RAG Pipeline Example

Workflow

[HTTP Scraper] --> [File Downloader] --> [Document Parser] --> [Text Chunker] --> [Embedding Generator] --> [Vector Store]

Configuration

HTTP Scraper
- URL: https://docs.example.com/api/{page}
- Iteration Mode: page
- Link Selector: a.pdf-link
File Downloader
- Output Dir: ./downloads
- Max Workers: 10
Document Parser
- Parser: pypdf
Text Chunker
- Strategy: recursive
- Chunk Size: 1000
- Overlap: 200
Embedding Generator
- Provider: huggingface
- Model: BAAI/bge-small-en-v1.5
Vector Store
- Backend: chroma
- Operation: store
- Collection: api-docs

Querying the Pipeline

Create a separate workflow for querying:

[Webhook Trigger] --> [Embedding Generator] --> [Vector Store (query)] --> [AI Agent] --> [Webhook Response]

The AI Agent receives relevant context from the vector store to answer questions.

Tips

Start with ChromaDB for development - no setup required.

Use HuggingFace embeddings for free local processing without API keys.

Adjust chunk size based on your embedding model’s context window.

Include meaningful metadata (source, page, section) for better retrieval context.

GPU is required for the Marker parser (OCR). Use PyPDF for non-scanned documents.

AI Agents

Use RAG context with AI agents

Webhooks

Trigger and respond over HTTP

AI Models

AI providers for generation

Schedulers & Triggers

Run the RAG pipeline on a schedule

​Document Processing

​Pipeline Overview

​HTTP Scraper

​Modes

​Parameters

​Output

​Example: Scrape News Archive

​File Downloader

​Parameters

​Input

​Output

​Document Parser

​Parsers

​Parameters

​Input

​Output

​Parser Comparison

​Text Chunker

​Strategies

​Parameters

​Input

​Output

​Choosing Chunk Size

​Embedding Generator

​Providers

​Parameters

​Input

​Output

​Vector Store

​Backends

​Operations

​Parameters

​Store Operation

​Query Operation

​Complete RAG Pipeline Example

​Workflow

​Configuration

​Querying the Pipeline

​Tips

​Related

AI Agents

Webhooks

AI Models

Schedulers & Triggers

Document Processing

Pipeline Overview

HTTP Scraper

Modes

Parameters

Output

Example: Scrape News Archive

File Downloader

Parameters

Input

Output

Document Parser

Parsers

Parameters

Input

Output

Parser Comparison

Text Chunker

Strategies

Parameters

Input

Output

Choosing Chunk Size

Embedding Generator

Providers

Parameters

Input

Output

Vector Store

Backends

Operations

Parameters

Store Operation

Query Operation

Complete RAG Pipeline Example

Workflow

Configuration

Querying the Pipeline

Tips

Related