> ## Documentation Index > Fetch the complete documentation index at: https://docs.opencompany.sh/llms.txt > Use this file to discover all available pages before exploring further. # Architecture > Deep dive into MachinaOs system architecture, execution engine, AI agents, and cache/security layers (plugin-first, WebSocket-first, Temporal execution) # Architecture A detailed look at how MachinaOs works under the hood. For contributors, system designers, and the curious. For a navigable tour with hyperlinks, see the [DeepWiki page](https://deepwiki.com/trohitg/MachinaOS). ## System Overview MachinaOs is three loosely-coupled tiers talking over a single persistent WebSocket connection: ``` +------------------------------------------------------------------------+ | Frontend tier (client/) | | React 19 + TypeScript + Vite + React Flow + Zustand | | Tailwind v4 + shadcn/ui + Radix + TanStack Query | | Themed canvas, parameter panel, credentials modal | +------------------------------------------------------------------------+ | WebSocket | /ws/status (single long-lived connection) v +------------------------------------------------------------------------+ | Backend tier (server/) | | FastAPI + WebSocket handlers + dependency injection container | | Plugin-first nodes (BaseNode folders) -> backend NodeSpec is SSOT | | WorkflowService (facade) -> NodeExecutor -> per-node execute() | | ParameterResolver -> template {{node.field}} substitution | | AuthService -> encrypted credentials (Fernet + PBKDF2) | +------------------------------------------------------------------------+ | v +------------------------------------------------------------------------+ | Execution tier | | Temporal workers (distributed) or local asyncio decide loop | | Redis (cache, locks, streams) SQLite (workflow.db, credentials.db)| | Node.js server (JS/TS code exec, port 3020) WhatsApp RPC (9400) | | Android relay WebSocket Temporal server (ports 7233/8080) | +------------------------------------------------------------------------+ | v External services: OpenAI, Anthropic, Gemini, OpenRouter, DeepSeek, Kimi, Mistral, Groq, Cerebras, plus local Ollama / LM Studio servers, Google Workspace, WhatsApp, Telegram, Twitter/X, Brave, Serper, Perplexity, Apify, Stripe, residential proxies, webhooks. ``` AI, agents, social, Android, documents, Google Workspace, code, browser, proxies, utilities OpenAI, Anthropic, Gemini, OpenRouter, DeepSeek, Kimi, Mistral, Groq, Cerebras, plus local Ollama and LM Studio Android, Coding, Web, Task, Social, Travel, Tool, Productivity, Payments, Consumer, Autonomous, Orchestrator, AI Employee, RLM, Claude Code, Codex A single persistent connection carries all frontend-backend RPC and push updates Dozens of skills across many categories, DB-backed with SKILL.md defaults Temporal distributed, Redis parallel, sequential fallback ## Execution Engine ### Conductor's Decide Pattern Workflow orchestration is a single function with fork/join parallelism: ``` _workflow_decide(ctx): 1. Find ready nodes (all dependencies satisfied) 2. asyncio.gather() the ready layer -> run in parallel 3. Checkpoint state to Redis/SQLite 4. Recurse until every node terminal (completed / skipped / failed) ``` Each workflow run has its own isolated `ExecutionContext`. No shared global state between concurrent runs. Decide loops serialize per execution via Redis `SETNX` distributed locks. ### Three Execution Modes ``` workflow.execute(workflow_id, workflow_data) | v TEMPORAL_ENABLED and server reachable? | yes -> _execute_temporal() -- per-node activities, retries, horizontal scaling (primary production mode) no -> Redis available? | yes -> _execute_parallel() -- decide loop + Kahn layers + Prefect-style input hash cache + DLQ + heartbeat recovery no -> _execute_sequential() -- topological walk (fallback for minimal env) ``` ### Layer Computation via Kahn's Algorithm Before execution, the DAG is sorted into layers. Layer 0 is the set of nodes with no dependencies; each subsequent layer depends only on earlier layers. Parallel execution runs each layer with `asyncio.gather()`: ``` Layer 0: [start, cronScheduler] Layer 1: [httpRequest, whatsappReceive] Layer 2: [aiAgent] Layer 3: [whatsappSend, console] ``` Toolkit sub-nodes (e.g., Android service nodes connected to `androidTool`) are detected and excluded from execution layers -- they run only when the parent toolkit invokes them via tool calling. ### Result Caching, Recovery, and DLQ * **Prefect-style caching**: every node result is stored in Redis/SQLite keyed by `hash_inputs(inputs)`. Re-running an identical node returns the cached result with status `TaskStatus.CACHED`. * **Heartbeat recovery**: `RecoverySweeper` scans `executions:active` every 60s; nodes with stale heartbeats (> 5 min) are marked stuck and recovered on next startup. * **Dead Letter Queue**: failed nodes (after retry exhaustion) are quarantined with full input snapshot. Inspect, replay, or purge via the `get_dlq_*` / `replay_dlq_entry` / `purge_dlq` WebSocket handlers. ### Edge Conditions Edges carry optional conditions for runtime branching with 20+ operators (`eq`, `neq`, `gt`, `lt`, `contains`, `exists`, `matches`, `in`, ...). Unmatched branches are marked `TaskStatus.SKIPPED`. ## Event-Driven Deployment Deployments are event-driven: each trigger event spawns an **independent concurrent execution run**. There is no iteration loop. ``` deploy_workflow(workflow_id) | v Set up triggers, return | +-> cronScheduler fires -> ExecutionRun 1 (isolated context) +-> cronScheduler fires -> ExecutionRun 2 (isolated context) +-> whatsappReceive fires -> ExecutionRun 3 (isolated context) +-> webhookTrigger fires -> ExecutionRun 4 (isolated context) +-> telegramReceive fires -> ExecutionRun 5 (isolated context) +-> taskTrigger fires -> ExecutionRun 6 (from delegated agent) ``` Multiple runs execute simultaneously with no interference. The firing trigger is marked complete before downstream execution starts; every other trigger node in the same run is auto-marked `_pre_executed` with `{not_triggered: True}` so they never block as event waiters. ### Push vs Polling Triggers ``` Push triggers (fire on an inbound event): whatsappReceive, webhookTrigger, chatTrigger, taskTrigger, telegramReceive, emailReceive Polling triggers (poll an API on an interval): googleGmailReceive (Gmail push requires paid Google Cloud setup) twitterReceive (X API has no webhook on the free tier) Scheduler: cronScheduler (cron-expression schedule) ``` Under the default Temporal execution path, each deployed trigger runs as a listener workflow that spawns an isolated child run per event. On the legacy local path, push triggers register a waiter with a filter closure and suspend until an external service dispatches a matching event (in-memory `asyncio.Future`, or Redis Streams for multi-worker deployments). ## AI Agent System ### 11 Chat-Model Providers Each provider below has its own chat-model node and appears in every agent's provider dropdown. (xAI is reachable on the native chat path through the shared OpenAI-compatible client, but it is not a standalone chat-model node.) | Provider | Native SDK Path | LangChain Path | Notes | | ---------- | --------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------- | | OpenAI | Yes | Yes | GPT-5.5/5.4/5.2, GPT-4.1 family, o3/o4-mini reasoning, GPT-4o/4o-mini | | Anthropic | Yes | Yes | Claude Fable 5, Opus 4.8/4.7, Sonnet 4.6, Haiku 4.5 (extended thinking via `budget_tokens`) | | Gemini | Yes | Yes (fallback) | Gemini 3.5-flash, 3.1-pro/flash-lite, 3-flash, 2.5-pro/flash/flash-lite; native path bypasses a LangChain Windows hang | | OpenRouter | Yes | Yes | 200+ models through one API | | DeepSeek | Yes (shared OpenAIProvider) | Yes | deepseek-v4-flash, deepseek-v4-pro (chat/reasoner legacy) | | Kimi | Yes (shared OpenAIProvider) | Yes | Moonshot kimi-k2.6, kimi-k2.5, kimi-k2.7-code | | Mistral | Yes (shared OpenAIProvider) | Yes | Large / Medium / Small / Codestral (latest) | | Groq | No (LangChain only) | Yes | Llama 3.3-70b, Llama 3.1-8b, GPT-OSS-120b/20b, Qwen3-32b | | Cerebras | No (LangChain only) | Yes | GPT-OSS-120b, Qwen-3-235b, GLM-4.7 | | Ollama | Yes (shared OpenAIProvider) | Yes | Local server; whatever models the user has pulled | | LM Studio | Yes (shared OpenAIProvider) | Yes | Local server; whatever models the user has loaded | ### Dual-Path Architecture ``` execute_chat(parameters) execute_agent(parameters) direct chat completions tool-calling agent loop | | v v is_native_provider(provider)? create_model(provider, ...) | | yes -+-- create_provider() -> LLMResponse v | LangChain ChatOpenAI / no -+-- create_model() -> chat_model ChatAnthropic / .invoke() -> LLMResponse ChatGoogleGenerativeAI | v bind_tools(...) | v _run_agent_loop (LLM step <-> tool exec loop) ``` The native path returns a normalized `LLMResponse` dataclass across all providers. Agent tool-calling still uses the LangChain chat-model wrappers (`bind_tools`) driven by the plain-async `_run_agent_loop`; the native SDK path powers direct chat completions and model discovery. Groq and Cerebras remain LangChain-only for chat as well. ### 16 Specialized Agents All specialized agents share the same handle architecture (`input-main`, `input-memory`, `input-skill`, `input-tools`, `input-task`) and dispatch through the same `BaseNode.execute()` plugin path. They mostly differ in icon, title, theme color, and default prompt. ``` android_agent coding_agent web_agent task_agent social_agent travel_agent tool_agent productivity_agent payments_agent consumer_agent autonomous_agent orchestrator_agent ai_employee rlm_agent claude_code_agent codex_agent ``` Routing: * Most agents run the shared plain-async agent loop (`_run_agent_loop`) for tool calling and delegation. * **`rlm_agent`** routes to `RLMService` (REPL-based recursive LM with `llm_query` / `rlm_query` / `FINAL`). * **`claude_code_agent`** and **`codex_agent`** run through the multi-provider CLI agent framework (Claude Code / Codex CLIs) instead of the standard LLM loop. ### Agent Teams Topology `orchestrator_agent` and `ai_employee` have an extra `input-teammates` handle. Connected agents become `delegate_to_` tools automatically: ``` +-------------------------+ | AI Employee | | (orchestrator_agent) | +-----+------+------+-----+ | | | input-teammates +-------------+ | +------------+ | | | +------v------+ +-------v-----+ +-------v-----+ | coding_agent| | web_agent | | task_agent | +-------------+ +-------------+ +-------------+ delegate_ delegate_ delegate_ to_coding_ to_web_ to_task_ agent tool agent tool agent tool ``` The team lead's LLM decides when to delegate based on task context. Under the default Temporal execution path, delegation spawns the teammate as a child workflow whose progress mirrors onto the parent's canvas badge and whose result flows back into the parent's loop. On the legacy local path, delegation is fire-and-forget: the child spawns as `asyncio.create_task()`, the parent continues, and the child broadcasts its own status independently. Either way, child completion can be consumed by `taskTrigger` nodes elsewhere in the workflow. ### Agent Loop Flow ``` +---------+ START ---> | agent | | (LLM) | +----+----+ | any tool_calls? / \ tools / \ end v v +---------+ END | tools | | (exec) | +----+----+ | +---------> loop back to agent ``` The loop routes purely on the model's `tool_calls`. A recursion limit guards against infinite tool loops, and a token-based compaction threshold summarizes the transcript before it overflows the model's context. Tools are built as Pydantic-schema `StructuredTool` instances with the node's parameter schema. ## Memory, Skills, Tokens, and Cost ### Markdown-Based Memory The `simpleMemory` node stores conversation history as editable Markdown in the parameter panel. The AI Agent handler reads, parses to LangChain messages, executes, appends the new exchange, trims to a window, and archives removed messages to an `InMemoryVectorStore` (optional) using HuggingFace `BAAI/bge-small-en-v1.5` embeddings. ``` AI Agent reads memoryContent (Markdown) | v _parse_memory_markdown() -> LangChain Messages | v (optional) vector store similarity_search(prompt, k=retrievalCount) | v Execute LLM with full history + retrieved context | v Append human + ai messages to Markdown | v _trim_markdown_window(windowSize) -> (kept, removed) | v If longTermEnabled: store.add_texts(removed) | v Save updated Markdown back to node parameters ``` ### Skill System Dozens of built-in skills organized into folders under `server/skills/`. Each top-level folder appears as an option in the Master Skill node's folder dropdown: ``` assistant/ android_agent/ autonomous/ coding_agent/ productivity_agent/ rlm_agent/ social_agent/ task_agent/ travel_agent/ web_agent/ terminal/ payments_agent/ ``` Each skill is a folder with a `SKILL.md` file containing YAML frontmatter (name, description, allowed-tools, metadata) and Markdown instructions. First load seeds the database; after that the database is source of truth so users can edit skill instructions in the UI. "Reset to Default" reloads the original `SKILL.md`. The `masterSkill` node aggregates multiple skills with enable/disable toggles. A split-panel editor shows the skill list on the left and the selected skill's Markdown on the right. When connected to an agent, the backend expands the `skillsConfig` parameter into individual skill entries injected into the agent's system message. ### Token Tracking and Compaction Every AI execution stores a `TokenUsageMetric` row with input/output/cache/reasoning token counts and calculated costs (USD) based on `server/config/pricing.json`. Cumulative state per session lives in `SessionTokenState`. Compaction threshold priority (highest to lowest): 1. Per-session `custom_threshold` (user-set) 2. Per-user `UserSettings.compaction_ratio` x the model's context window 3. Environment `COMPACTION_RATIO` (default **0.8**) x the model's context window (e.g., \~838K for Claude Sonnet 4.6 with 1M context) 4. JSON fallback `llm_defaults.json` ratio x context window When the cumulative session tokens cross the threshold, `CompactionService.compact_context()` generates a 5-section summary (Task Overview, Current State, Important Discoveries, Next Steps, Context to Preserve) following the Claude Code pattern and replaces the memory content. Anthropic and OpenAI also have native compaction APIs (`context_management` edits, `compact_threshold`) that are configured transparently. ## Communication Layer A single persistent WebSocket at `/ws/status` handles most frontend-backend communication. Handlers live in the `MESSAGE_HANDLERS` map in `server/routers/websocket.py` plus plugin-registered handlers that self-wire through `services.ws_handler_registry` (each self-contained plugin folder can own its own WebSocket commands). Together they cover: node parameters, tool schemas, node execution, triggers, dead letter queue, deployment, AI operations, API keys, OAuth flows (Claude, Twitter, Google), Android, WhatsApp, Telegram, workflow storage, chat messages, console logs, skills, memory, user settings, pricing, agent teams, and model registry. ### Request/Response Pattern ``` Frontend Backend | | |-- {type: "execute_node", data: ...}-->| | |-- dispatch to handle_execute_node() | |-- run node handler | | |<-- {type: "node_status", ... }---------| (broadcast) |<-- {type: "node_output", ... }---------| (broadcast) |<-- {type: "execute_node_response", ...}| (direct reply, request_id matched) ``` ### Broadcast Message Types ``` node_status workflow_status api_key_status node_output android_status token_usage_update variable_update compaction_starting node_parameters_updated variables_update compaction_completed ``` ### Auto-Reconnect and Keepalive Frontend sends `{"type": "ping"}` every 30 seconds. Reconnection uses PartySocket with jittered exponential backoff and message replay, so a dropped socket recovers automatically and queued requests replay on the new connection. Connection is gated on `isAuthenticated` so logged-out users never connect. ## Cache, Persistence, and Security ### Cache Fallback Hierarchy MachinaOs follows the n8n cache pattern with automatic environment-based fallback: ``` Production: Redis -> SQLite (cache_entries) -> in-memory dict Local dev: SQLite (cache_entries) -> in-memory dict (Redis disabled via REDIS_ENABLED=false) ``` `CacheService` in `server/core/cache.py` checks each backend in order. TTL expiration is supported across all three. A background `cleanup_expired_cache()` task removes expired SQLite rows. ### Encrypted Credentials API keys and OAuth tokens live in a **separate SQLite database** (`credentials.db`) isolated from the main workflow database (`workflow.db`). Encryption uses Fernet (AES-128-CBC + HMAC-SHA256) with keys derived from a server-scoped config key via PBKDF2HMAC (600K iterations, OWASP 2024 recommendation). ``` API_KEY_ENCRYPTION_KEY (.env) + salt (from credentials.db, 256 bits) | v PBKDF2HMAC-SHA256 (600,000 iterations) | v urlsafe_b64encode -> Fernet key | v held in memory only | v encrypt() / decrypt() inside EncryptionService ``` Two separate credential systems inside `credentials.db`: * **API key system** (`EncryptedAPIKey` table): provider keys the user enters manually * **OAuth token system** (`EncryptedOAuthToken` table): tokens from OAuth flows (Google, Twitter, Claude.ai) All routers access credentials through `AuthService` only. Direct database access is forbidden. Decrypted values are cached in `AuthService` memory dicts and never written to disk or Redis. For cloud deployments, `CREDENTIAL_BACKEND` can be switched to `keyring` (OS-native) or `aws` (Secrets Manager) via the `CredentialBackend` abstraction. ### Authentication JWT in HttpOnly cookies following the n8n pattern. Two modes: * `AUTH_MODE=single` - first user becomes owner, registration disabled after * `AUTH_MODE=multi` - open registration for cloud deployments * `VITE_AUTH_ENABLED=false` - bypass login entirely for local development WebSocket connections check the JWT cookie before accepting. Auth context has exponential-backoff retry (5 attempts) to survive race conditions where the frontend starts before the backend is ready. ## Related Browse all workflow nodes by category 11 chat-model providers with native SDK and LangChain paths 16 specialized agents with teams and delegation Source code and in-repo `docs-internal/` deep dives