Scraping & Proxies

Scraping nodes fetch and extract content from pages you already know about. If that is not quite your problem, a neighbouring page probably is:

You want to…	Use
Find pages / get search snippets	Web Search
Read content from known pages at scale	This page (Crawlee, Apify)
Click, type, and interact with a live page	Browser Automation

This page also covers the residential proxy system — provider configuration, proxied requests, and health monitoring — which the scraping nodes can route through.

Crawlee Scraper

Python-based scraping on top of the crawlee library. Crawlee manages concurrency, retry back-off, and request queues internally — you feed in configuration and get back a list of pages. Also usable as an AI agent tool.

Engines

Engine	How it fetches	Use for
`beautifulsoup`	Plain HTTP + HTML parsing	Fast scraping of static pages
`playwright`	Full headless browser	JS-rendered pages that need a real render
`adaptive`	Currently aliased to `beautifulsoup`	(No automatic escalation today)

Modes

single — scrape only the seed URL (page and depth caps are forced to 1 and 0).
crawl — follow links matching Link Selector (defaults to a[href]) and an optional URL Pattern, up to Max Pages / Max Depth.

Parameters

string

required

Seed URL

select

default:"beautifulsoup"

beautifulsoup, playwright, or adaptive

select

default:"single"

single or crawl

string

CSS selector for text/HTML extraction. Empty extracts the whole page.

select

default:"text"

text, html, or markdown (markdown falls back to text if the html2text package is not installed)

boolean

default:"false"

Include a links[] array on each page

number

default:"10"

Page cap in crawl mode (1-1000)

number

default:"2"

Link-following depth in crawl mode (0-50)

string

Glob-style pattern (e.g. */blog/*) restricting which discovered links are followed (crawl mode)

number

default:"60"

Timeout in seconds — applied per request handler and to the whole run (1-3600)

Playwright-only knobs: browser_type (chromium / firefox / webkit), wait_for_selector + wait_timeout (await an element before extracting), take_screenshot (base64 PNG per page). Proxy knobs: use_proxy plus proxy_provider (auto picks by health score), proxy_country, session_type (rotating / sticky), and sticky_duration — see Proxies below.

Output

{
  "pages": [
    {
      "url": "https://example.com/article",
      "title": "Article title",
      "content": "Extracted text...",
      "links": ["https://example.com/next"]
    }
  ],
  "page_count": 1,
  "crawler_type": "beautifulsoup",
  "mode": "single",
  "proxied": false
}

Per-page content is capped at 100,000 characters.

Example: crawl a blog

URL: https://blog.example.com
Crawler Type: beautifulsoup
Mode: crawl
Link Selector: a.post-link
URL Pattern: */blog/*
Max Pages: 25
Max Depth: 2

Proxy failures degrade gracefully: if the proxy service cannot supply a URL, the crawl proceeds without a proxy and reports proxied: false — which means the target site sees your real IP. Check proxied in the output if that matters.

Apify Actor

Run an Apify actor — a pre-built scraper or automation hosted on the Apify platform — wait for it to finish, and return its dataset items. This is the escape hatch for platforms without a first-class OpenCompany integration: Instagram, TikTok, Twitter/X, LinkedIn, Facebook, YouTube, Google Search and Maps, and general website crawling. Also usable as an AI agent tool. To connect your account, open API Credentials, choose Apify, paste your key (format apify_api_...) into the API Key field, and hit Validate.

Apify is a paid platform. Actors consume Apify usage on your account as they run (many actors also have their own per-result pricing), and heavier memory settings cost more per second. The compute_units field in the output actually reports the run’s total USD usage. Check an actor’s pricing on Apify before running it at scale.

Parameters

select

default:"apify/instagram-scraper"

required

One of 10 curated presets — apify/instagram-scraper, clockworks/tiktok-scraper, apidojo/tweet-scraper, apify/linkedin-scraper, apify/facebook-pages-scraper, streamers/youtube-scraper, apify/google-search-scraper, compass/crawler-google-places, apify/website-content-crawler, curious_coder/web-scraper — or custom

string

Any actor id from the Apify store (when Actor is custom)

string

Raw JSON run-input for the actor. Invalid JSON is silently treated as {}.

number

default:"100"

Maximum dataset items to fetch (1-10000)

number

default:"300"

Run timeout in seconds (1-3600)

select

default:"1024"

Actor memory in MB: 128 / 256 / 512 / 1024 / 2048 / 4096 / 8192

Five presets also expose “quick helper” fields that fill the actor input for you — e.g. instagram_urls (Instagram), tiktok_profiles / tiktok_hashtags (TikTok), twitter_search_terms / twitter_handles (tweet scraper), google_search_query / google_search_pages (Google Search), and crawler_start_urls / crawler_max_depth / crawler_max_pages (website content crawler). Quick helpers overwrite matching keys in your raw Actor Input JSON.

Output

{
  "run_id": "aBcD1234",
  "actor_id": "apify/website-content-crawler",
  "status": "SUCCEEDED",
  "items": [{"url": "https://example.com", "text": "Page content..."}],
  "item_count": 42,
  "dataset_id": "xYz987",
  "compute_units": 0.12,
  "started_at": "2026-07-15T10:00:00Z",
  "finished_at": "2026-07-15T10:03:12Z"
}

Runs that end FAILED, TIMED-OUT, or ABORTED surface as node errors with the actor’s error message.

Proxies

OpenCompany includes a residential proxy service: you register one or more proxy providers (gateway host/port plus a URL template), store their credentials, and the service picks the best provider per request using health scores, routing rules (domain glob -> preferred providers), geo-targeting, and rotating or sticky sessions. The scraping and HTTP nodes opt in transparently — Crawlee Scraper, HTTP Request, and HTTP Scraper all have a use_proxy flag; when set, the node asks the proxy service for a URL and routes traffic through it. Three dedicated nodes manage and use the service directly.

Proxy Config

Admin node (and AI tool) for managing providers, credentials, and routing rules. Dispatches on Operation:

Operation	Required params	What it does
`list_providers`	none	List configured providers
`add_provider`	`name`, `gateway_host`, `gateway_port`, `url_template`	Register a provider (JSON URL template)
`update_provider`	`name` + changed fields	Update an existing provider
`remove_provider`	`name`	Delete a provider
`set_credentials`	`name`, `username`, `password`	Store the provider’s credentials (encrypted)
`test_provider`	`name`	Live test via httpbin.org/ip — returns IP and latency
`get_stats`	none	Aggregate usage and health statistics
`add_routing_rule`	`domain_pattern`	Route matching domains (glob like `*.example.com`) to preferred providers
`list_routing_rules`	none	List routing rules
`remove_routing_rule`	`rule_id`	Delete a routing rule

Additional add_provider / update_provider fields: cost_per_gb (USD), priority (0-100, higher ranks first), enabled. Additional add_routing_rule fields: preferred_providers (JSON array), required_country, session_type.

Proxy Request

A proxy-first HTTP client with explicit retry and failover. Unlike the use_proxy flag on other nodes (best-effort), Proxy Request requires the proxy service to be enabled and errors out if no provider is available. Each attempt reports latency and success back into the provider’s health score, and successful requests record byte-based cost.

string

required

Target URL

select

default:"GET"

GET / POST / PUT / DELETE / PATCH

object

Request headers

string

Request body for POST/PUT/PATCH — JSON object, or a string (parsed as JSON when possible, sent raw otherwise)

string

default:"auto"

Specific provider name, or auto for health-based selection

string

ISO country code for geo-targeting

select

default:"rotating"

rotating (new IP per request) or sticky (keep an IP for sticky_duration seconds, default 600)

number

default:"3"

Retry attempts with provider failover (0-10)

number

default:"30"

Request timeout in seconds (1-600)

Output:

{
  "status": 200,
  "data": {"ip": "203.0.113.7"},
  "headers": {"content-type": "application/json"},
  "url": "https://httpbin.org/ip",
  "method": "GET",
  "proxy_provider": "auto",
  "latency_ms": 412,
  "bytes_transferred": 33,
  "attempt": 1
}

An HTTP status of 400 or above counts as a failed attempt and triggers retry/failover; when all attempts fail the node errors with the last error message.

Proxy Status

Read-only snapshot of the proxy runtime — useful on dashboards or as an agent tool so the agent can reason about which provider to target.

{
  "enabled": true,
  "providers": [
    {
      "name": "provider-a",
      "enabled": true,
      "priority": 50,
      "score": 0.97,
      "success_rate": 0.99,
      "avg_latency_ms": 380,
      "total_requests": 1204,
      "total_bytes": 8123456
    }
  ],
  "stats": {}
}

When the proxy service is disabled the node still succeeds and returns enabled: false with empty collections — check enabled, not the node’s success state. The provider_name parameter is currently ignored; the node always returns all providers.

Example: scrape a JS-heavy page and summarize

[Cron Scheduler] --> [Crawlee Scraper] --> [AI Agent] --> [Console]

Crawlee Scraper — Crawler Type: playwright, Mode: single, URL: the page to watch, Wait For Selector: the element that signals the page has rendered (e.g. .article-body), Output Format: text.
AI Agent — prompt: “Summarize the key points of this page in five bullets.” The scraped pages[0].content flows in as input.
Console — shows the summary on each scheduled run.

The playwright engine renders JavaScript before extraction, so client-side apps and infinite-scroll shells produce real content instead of an empty HTML skeleton.

The RAG pipeline’s HTTP Scraper on Document Processing is a different node: it extracts links from listing pages to feed the download/parse/chunk/embed pipeline, while the nodes on this page extract content.

Tips

Start with beautifulsoup — it is much faster and cheaper than a browser. Switch to playwright only when the page genuinely needs JavaScript to render its content.

In crawl mode, always set URL Pattern and a sensible Max Pages. An unconstrained crawl of a large site burns your whole timeout on pages you do not need.

Use Proxy Config’s test_provider operation right after set_credentials — it does a live round-trip and returns the exit IP and latency, so a typo in credentials surfaces immediately.

Web Search

Find URLs before you scrape them

Browser Automation

Interact with pages instead of just reading them

Document Processing

Feed scraped documents into the RAG pipeline

AI Agents

Give agents scraping tools

Concepts

Node catalog

Crawlee Scraper

Engines

Modes

Parameters

Output

Example: crawl a blog

Apify Actor

Parameters

Output

Proxies

Proxy Config

Proxy Request

Proxy Status

Example: scrape a JS-heavy page and summarize

Tips

Web Search

Browser Automation

Document Processing

AI Agents

​Crawlee Scraper

​Engines

​Modes

​Parameters

​Output

​Example: crawl a blog

​Apify Actor

​Parameters

​Output

​Proxies

​Proxy Config

​Proxy Request

​Proxy Status

​Example: scrape a JS-heavy page and summarize

​Tips

​Related

Web Search

Browser Automation

Document Processing

AI Agents

Crawlee Scraper

Engines

Modes

Parameters

Output

Example: crawl a blog

Apify Actor

Parameters

Output

Proxies

Proxy Config

Proxy Request

Proxy Status

Example: scrape a JS-heavy page and summarize

Tips

Related