# AutoAir > AutoAir is a visual DAG builder for Apache Airflow 3. DAGs are drawn on a canvas or written in code mode, validated, and saved as readable Python that Airflow loads like any hand-written DAG. It runs inside the Airflow API server as a plugin or as a standalone server, deploys to the DAGs folder, Git (pull requests or direct pushes) or object storage, and authorizes every request through Airflow. # What is AutoAir Source: https://ctolon.github.io/autoair/docs/intro AutoAir is a DAG builder for Apache Airflow 3. You design a DAG on a canvas (or in a code editor, or through an AI assistant or MCP agent), and AutoAir stores it as a versioned **DAG specification**, generates a readable Python DAG file from that specification, validates it with Airflow's own `DagBag`, and deploys it to the place Airflow reads DAGs from: a local folder, object storage, or a Git repository (directly or as a pull request). AutoAir is one Python package (`autoair`, Apache-2.0) with a React frontend shipped inside the wheel. It supports Airflow 3.0 or later, below 4, on Python 3.10 to 3.14. CI covers Airflow 3.0.6, 3.1.8, 3.2.2 and 3.3.2 plus the 3.4.0b2 pre-release, and generated imports follow the target Airflow version. Releases are published to PyPI as `autoair` and as the container image `ghcr.io/ctolon/autoair` (the official Airflow image with AutoAir installed). [Screenshot: The AutoAir builder: operator palette on the left, a DAG of connected tasks on the canvas, the inspector on the right and the Problems and Code tabs in the bottom panel (The builder: palette, canvas, inspector and bottom panel.)] ## The spec-centric model Every DAG built with AutoAir is a JSON document, the **DagSpec** (`spec_version: 3`), saved as `.autoair.json` next to the generated file. Everything else is derived from it: ```mermaid flowchart LR canvas[Canvas] --> spec[(DagSpec)] code[Code mode] --> spec ai[AI / MCP agents] --> spec import[Import: Python, YAML, Airflow] --> spec spec --> validate[Validation: graph, catalog, Jinja, policy, DagBag] spec --> codegen[Code generation: Python, Airflow YAML, loader] codegen --> deploy[Deploy target: local, object, git] deploy --> airflow[Airflow parses the DAG file] ``` - **One source of truth.** The canvas, the code preview, the validator, the version history, the diff view, reviews and deployment all read the same spec. There is no second model that can drift. - **Readable output.** The default output is an ordinary Python DAG file that uses `airflow.sdk` and the operator classes you picked. Generated imports follow the target Airflow version. The file carries a header saying it is generated; AutoAir refuses to overwrite a generated file that someone edited by hand unless the deploy is forced. - **Validation before deploy.** Graph rules (cycles, duplicate ids), operator parameters against the installed operator catalog, Jinja templates, organisation policy and an import check in a real `DagBag` run before anything is written. - **Deployment is a write to the place Airflow already reads.** AutoAir does not run DAGs itself; Airflow's dag-processor picks up the file. Runs, logs and XComs are read back from Airflow's REST API. The [concepts](https://ctolon.github.io/autoair/docs/getting-started/concepts) page explains each of these parts in more detail. ## Three run modes The same package runs in three ways. They share the UI, the API and the settings; they differ in where the server runs and how users authenticate. | Mode | Command / entry point | Authentication | Needs an Airflow server | Typical use | | --- | --- | --- | --- | --- | | Airflow plugin | `airflow.plugins` entry point, loaded by the Airflow API server | Airflow's auth manager (the Airflow UI session) | Yes, AutoAir runs inside it | Teams that already run Airflow 3 and want the builder in the Airflow UI | | Standalone | `autoair serve` | Airflow token verified against Airflow: the Airflow session cookie, username/password exchanged for a token, or a pasted token | Yes, reached over its REST API | Running AutoAir as its own service, scaled and upgraded separately from Airflow | | Local | `autoair local` | None on loopback; optional static token | No | Trying AutoAir or authoring DAG files on a workstation | [Installation overview](https://ctolon.github.io/autoair/docs/installation/overview) compares the modes in detail and links to the setup page of each. ## What AutoAir is not - **Not a scheduler or executor.** AutoAir writes DAG files; Airflow schedules and runs them. Triggering, backfill, clearing and the run overlay call Airflow's REST API with the caller's own token. - **Not a replacement for Airflow's permissions.** In plugin and standalone mode every request is authorized by Airflow (per-DAG access stays with Airflow). AutoAir adds its own roles on top for builder actions such as deploy, review and administration; see [IAM](https://ctolon.github.io/autoair/docs/admin/iam). - **Not a closed format.** Generated Python is plain Airflow code. You can export a DAG as Python, Airflow YAML, dag-factory YAML or spec JSON at any time and stop using AutoAir. - **Not a sandbox for untrusted code.** Python task bodies, inline functions and script files end up in DAG files that Airflow executes. Restrict them with `allow_python_code`, `features.python_code` and the policy settings; see [policy](https://ctolon.github.io/autoair/docs/admin/policy). ## Feature summary | Area | What you get | Read more | | --- | --- | --- | | Canvas | Operator palette for every installed provider and custom operator, task groups, notes, command palette, copy/paste, auto layout, undo/redo | [Canvas](https://ctolon.github.io/autoair/docs/user-guide/canvas), [Shortcuts](https://ctolon.github.io/autoair/docs/user-guide/shortcuts) | | Code mode | Edit the generated file and apply it back into the spec | [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode) | | Editors | Jinja-aware template fields, XCom drag-to-map, Python and shell completion, Markdown docs | [Editors](https://ctolon.github.io/autoair/docs/user-guide/editors), [Jinja, XCom and Python](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python) | | TaskFlow | `@task` and its variants, dynamic task mapping, task loops (Airflow 3.4) | [TaskFlow and loops](https://ctolon.github.io/autoair/docs/user-guide/taskflow-and-loops) | | Scheduling | Presets, cron, intervals, events, asset and asset-or-time schedules, partitions, with a next-runs preview | [Scheduling and assets](https://ctolon.github.io/autoair/docs/user-guide/scheduling-and-assets) | | Validation | Live problems that jump to the field, import check in Airflow's `DagBag` | [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation) | | Deploy | Local folder, object storage or Git (push or pull request on GitHub, GitLab, Bitbucket, Azure DevOps, Gitea/Forgejo or any remote through the git CLI), ETag locking and three-way merge | [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy) | | Governance | Reviews and four-eyes approvals, organisation policy, audit trail | [Reviews and approvals](https://ctolon.github.io/autoair/docs/user-guide/reviews-and-approvals), [Audit](https://ctolon.github.io/autoair/docs/admin/audit) | | Runs | Trigger, backfill, clear; run states, logs and XComs on the canvas | [Runs](https://ctolon.github.io/autoair/docs/user-guide/runs) | | Environments | dev/staging/prod overlays and promotion | [Environments](https://ctolon.github.io/autoair/docs/user-guide/environments) | | Collaboration | Live co-editing with presence and per-user undo | [Collaboration](https://ctolon.github.io/autoair/docs/user-guide/collaboration) | | Reuse | Templates, fragments, operator templates, signed marketplaces | [Templates and fragments](https://ctolon.github.io/autoair/docs/user-guide/templates-and-fragments), [Marketplace](https://ctolon.github.io/autoair/docs/user-guide/marketplace) | | AI | Generate and edit DAGs, explain problems, write docs; MCP server for agents | [AI introduction](https://ctolon.github.io/autoair/docs/ai/introduction), [MCP](https://ctolon.github.io/autoair/docs/ai/mcp) | | Operations | Health, readiness and Prometheus endpoints, OpenTelemetry, PostgreSQL state for several replicas | [Observability](https://ctolon.github.io/autoair/docs/admin/observability), [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling) | | Extending | Custom operators, catalog sources, node SDK, REST API | [Extending](https://ctolon.github.io/autoair/docs/extending/overview), [REST API](https://ctolon.github.io/autoair/docs/extending/rest-api) | ## Who it is for - **Data and analytics engineers** who know what their pipeline should do but do not want to hand-write Airflow boilerplate, and who still want reviewable Python in Git. - **Platform teams** that run Airflow 3 for many users and need guard rails: policy rules, operator allow and deny lists, reviews before deploy, and an audit trail. - **Airflow developers** who want fast feedback (parameter types, Jinja completion, live `DagBag` validation) and can switch to code mode at any time. - **Agent builders** who want an AI assistant or an MCP client to author DAGs through the same validated path as humans. ## Next steps 1. [Quick start](https://ctolon.github.io/autoair/docs/getting-started/quick-start): run AutoAir in a few minutes. 2. [Build your first DAG](https://ctolon.github.io/autoair/docs/getting-started/first-dag): a step-by-step tutorial. 3. [Concepts](https://ctolon.github.io/autoair/docs/getting-started/concepts): the model behind the UI. 4. [Installation](https://ctolon.github.io/autoair/docs/installation/overview): choose a mode for a real deployment. # Quick start Source: https://ctolon.github.io/autoair/docs/getting-started/quick-start Pick the path that matches what you have. Local mode needs no Airflow server; the other three connect to Airflow 3. | You have | Path | Time | | --- | --- | --- | | A laptop with Python | [Local mode](#local-mode-no-airflow-server) | minutes | | An Airflow 3 installation you can add packages to | [Plugin](#plugin-in-an-existing-airflow) | one restart of the API server | | Docker with Compose v2 | [Compose demo](#docker-compose-demo) | first build takes several minutes | | An Airflow 3 reachable over HTTP and an Airflow login | [Standalone server](#standalone-server-next-to-airflow) | minutes | ## Prerequisites - **Python 3.10 to 3.14** for the `autoair` package (CI runs 3.10 to 3.14 against Airflow 3.3.2). - **Apache Airflow 3.0 or later, below 4,** for the plugin, standalone and Compose paths. CI tests 3.0.6, 3.1.8, 3.2.2 and 3.3.2, plus a 3.4 pre-release. - **Node.js 22** only when you build the frontend from a source checkout. Released wheels contain the built UI. - [uv](https://docs.astral.sh/uv/) is used in the examples; `pip` and `pipx` work the same way. ## Local mode (no Airflow server) `autoair local` starts the builder on `http://127.0.0.1:8765/ui/` with a settings preset for a workstation: the operator catalog comes from the Python packages installed next to AutoAir, validation is static (plus the `DagBag` import check when Airflow is installed), deploys write to `./dags`, and everything that needs an Airflow server (runs, test runs, connection/pool/variable pickers) is switched off. Details: [Local mode](https://ctolon.github.io/autoair/docs/installation/local). ### From a source checkout This works today and is what the project's own development uses: ```bash git clone https://github.com/ctolon/autoair.git cd autoair uv sync # includes apache-airflow and uvicorn (dev group) (cd frontend && npm ci && npm run build) # writes the UI into src/autoair/static/ uv run autoair local --open --dags-folder ./dags ``` If you skip the frontend build, `/ui/` answers `503` with "Frontend bundle not built. Run `npm run build` in `frontend/`." ### From a published package `autoair local` needs two things the core package does not install: 1. **uvicorn**, the HTTP server, which comes with the `standalone` extra. Without it the command stops with `ModuleNotFoundError: No module named 'uvicorn'`. 2. **Operators to put in the palette.** The catalog is built from installed packages. With no Airflow and no providers installed, the palette has no operators (only the built-in Python task, task group and note nodes). The `airflow` extra installs `apache-airflow` and `apache-airflow-providers-standard` (its operators and sensors, among them `BashOperator`, `PythonOperator` and `EmptyOperator`) and enables the `DagBag` import check. So the command to use is: ```bash uvx --from 'autoair[standalone,airflow]' autoair local --open ``` Add the providers your DAGs use with `--with`: ```bash uvx --from 'autoair[standalone,airflow]' --with apache-airflow-providers-amazon autoair local --open ``` The package is published to PyPI as `autoair` by the release workflow on every `v*` tag. To try an unreleased commit, run from a source checkout instead (see [Local mode](https://ctolon.github.io/autoair/docs/installation/local)). **Note:** The plain `uvx autoair local` shown in some older notes does not start a server: `uvicorn` is only in the `standalone` extra. The command then exits with a message naming the extras to install (`uvx --from 'autoair[standalone,airflow]' autoair local --open`). Use the command above. To require a token (for example when you bind another address), pass `--token` or set `AUTOAIR_LOCAL__TOKEN`; the sign-in page then shows an **Access token** field. Binding a non-loopback address without a token is refused: ```text refusing to serve 0.0.0.0 without --token: local mode has no other authentication ``` ## Plugin in an existing Airflow AutoAir registers itself through the `airflow.plugins` entry point, so installing the package into the environment of the Airflow **API server** is enough. Install it wherever the API server runs (for the official image, in a derived image), then restart the API server. ```bash pip install 'autoair[format,git,mcp]' # or the wheel: pip install dist/autoair-*.whl airflow api-server # restart it ``` - Airflow 3.1 and newer: open **Browse → DAG Builder**. Every DAG page also gets an **AutoAir** tab. - Airflow 3.0: open `http:///autoair/ui/`. The DAG files are written to Airflow's `[core] dags_folder` under the `autoair/` sub-folder by default. For anything beyond a single machine (several API servers, Git or object storage bundles), read [Plugin installation](https://ctolon.github.io/autoair/docs/installation/plugin) and [Deployment](https://ctolon.github.io/autoair/docs/deployment/overview). ## Docker Compose demo The repository contains a self-contained demo in `deploy/compose`: PostgreSQL, Airflow 3.3.2 split into its components (API server with the AutoAir plugin, scheduler with the LocalExecutor, dag-processor, triggerer), and two standalone AutoAir replicas sharing state in PostgreSQL. It builds the frontend and the wheel from the working tree, so no local Node or Python is needed. ```bash git clone https://github.com/ctolon/autoair.git cd autoair/deploy/compose cp .env.example .env # optional: every value has a default docker compose -f docker-compose.demo.yml up --build -d --wait ``` The first build takes a few minutes; later starts take about a minute. When `--wait` returns: | Service | URL | | --- | --- | | Airflow, then **Browse → DAG Builder** (the plugin) | `http://localhost:8080` | | AutoAir replica A | `http://localhost:8765/ui/` | | AutoAir replica B | `http://localhost:8766/ui/` | | AutoAir API docs (replica A) | `http://localhost:8765/api/docs` | Two demo DAGs (`demo_hello`, `demo_pipeline`) are deployed on first start, and a signed template marketplace is available in the template gallery. Opt-in profiles add a load balancer (`lb`), Redis (`redis`), the separate AI service (`ai`), a local LLM (`ollama`), a Gitea deploy target (`git`), TLS (`tls`) and tracing (`observability`). The default start runs no AI provider. The demo makes every Airflow visitor an administrator (SimpleAuthManager `all_admins`) and uses public default secrets; it is for a workstation only. See [Docker Compose](https://ctolon.github.io/autoair/docs/installation/docker-compose) for the profiles, configuration and the workflows to try. ## Standalone server next to Airflow `autoair serve` runs AutoAir as its own service. Every request carries an Airflow token, which AutoAir verifies against Airflow and forwards, so Airflow's permissions apply. ```bash pip install 'autoair[standalone,format]' autoair serve \ --airflow-url http://airflow.internal:8080 \ --dags-folder /shared/dags ``` Open `http://127.0.0.1:8765/ui/`. The sign-in page offers, depending on the setup: - **Sign in with Airflow**: reuses the Airflow UI session when the browser can send Airflow's cookie to AutoAir (same host, or a shared parent domain). - **Username and password**: exchanged once for an Airflow token through Airflow's `/auth/token` endpoint (`iam.login_enabled`, on by default). - **Access token**: paste an Airflow API token. To obtain one: ```bash curl -s -X POST http://airflow.internal:8080/auth/token \ -H 'Content-Type: application/json' \ -d '{"username": "admin", "password": "..."}' ``` `--dags-folder` must be the folder Airflow's dag-processor reads (a shared volume), or configure a Git or object-storage deploy target instead. `autoair serve` binds `127.0.0.1:8765` by default; use `--host 0.0.0.0` behind a reverse proxy. Production settings (TLS, proxy headers, several workers): [Standalone installation](https://ctolon.github.io/autoair/docs/installation/standalone). ## Next steps - [Build your first DAG](https://ctolon.github.io/autoair/docs/getting-started/first-dag) - [Concepts](https://ctolon.github.io/autoair/docs/getting-started/concepts) - [Troubleshooting FAQ](https://ctolon.github.io/autoair/docs/troubleshooting/faq) # Build your first DAG Source: https://ctolon.github.io/autoair/docs/getting-started/first-dag This tutorial builds a three-task pipeline named `orders_daily`: ```mermaid flowchart LR extract["extract
BashOperator"] --> transform["transform
@task (Python)"] --> load["load
SQLExecuteQueryOperator"] ``` - `extract` is a `BashOperator` that downloads the day's orders as a JSON file. - `transform` is a Python task (TaskFlow `@task`) that reads the file and returns the day's order total. The returned value goes to XCom. - `load` is a `SQLExecuteQueryOperator` that inserts the total into a warehouse table, reading it from XCom with Jinja. The DAG runs daily. Along the way you fix a problem that live validation finds, preview the generated Python and deploy it. The screenshots below were taken while building exactly this DAG. [Video: Building a DAG on the canvas] ## Before you start You need a running AutoAir with `BashOperator` and `SQLExecuteQueryOperator` in its palette. `BashOperator` comes with `apache-airflow-providers-standard`; `SQLExecuteQueryOperator` with `apache-airflow-providers-common-sql`. Both must be installed next to AutoAir (and in Airflow, to run the DAG): - **Local mode** is enough for every step except running the DAG: follow [Quick start: local mode](https://ctolon.github.io/autoair/docs/getting-started/quick-start#local-mode-no-airflow-server). Install the `airflow` extra (or Airflow itself) so that `apache-airflow-providers-standard` is present and **Validate in Airflow** can import the DAG. - **Plugin or standalone mode** additionally lets you trigger the deployed DAG and watch it run. In Airflow 3.1 and newer, open **Browse → DAG Builder**; in Airflow 3.0 open `/autoair/ui/`. To deploy you need the permission to save specs (the **Author** role or more; in local mode every user has full rights). See [IAM](https://ctolon.github.io/autoair/docs/admin/iam). ## 1. Start with an empty DAG A new builder opens on an empty canvas with **Start your DAG**: drag an operator from the palette, or use **Search operators…**, **Start from a template…**, **Generate with AI…** or **Write it as code**. This tutorial uses the canvas. If another DAG is open, use **File → New DAG**. [Screenshot: An empty canvas showing the Start your DAG panel with buttons to search operators, start from a template, generate with AI or write code] Click an empty spot of the canvas. With no node selected, the inspector on the right shows **DAG settings** with the tabs **General**, **Schedule**, **Params**, **Defaults**, **Alerts**, **Environments** and **Docs**. On **General**: 1. Set **DAG id** to `orders_daily`. The DAG id is unique in Airflow and is also the file name (`orders_daily.py`); it may contain letters, digits, `_`, `-` and `.`. A new DAG starts as `my_new_dag`. 2. Set **Description** to `Daily orders export into the warehouse`. 3. Keep the **Tags** entry `autoair`; add more with **Add tag** if you like. Leave **Output** at **Generated Python file (readable, reviewable)**. ## 2. Add the extract task The palette on the left lists every operator, sensor and transfer found in the installed packages, plus built-in nodes. Its tabs are **All**, **Operators**, **Sensors**, **Transfers**, **Built-in**, **Favorites** and **Recent**. 1. Type `bash` into **Search operators…**. 2. Drag `BashOperator` onto the canvas, or double-click it, or select it with the arrow keys and press Enter. [Screenshot: The palette filtered by the search term bash, listing BashOperator and the @task.bash decorator] The node appears with a generated task id and is selected, so the inspector shows its parameters. 1. Set **Task id** (at the top of the inspector) to `extract`. 2. In **bash_command**, type: ```text curl -sf -o /data/orders/{{ ds }}.json https://shop.example.com/api/orders?date={{ ds }} ``` `bash_command` is a template field, so the editor is Jinja-aware: typing `{{` offers the context variables (`ds`, `ds_nodash`, `params`, `var`, `conn`, macros, `ti.xcom_pull`). Each parameter shows its Python type, and a red `*` marks required ones. [Screenshot: The extract BashOperator node on the canvas with the inspector showing its Task id and the bash_command template editor containing the curl command] **Tip:** Ctrl/⌘+K opens the command palette, which can also insert nodes. The [shortcuts page](https://ctolon.github.io/autoair/docs/user-guide/shortcuts) lists the rest. ## 3. Add the transform and load tasks 1. In the palette, open the **Built-in** tab and add **Python task** ("Plain @task function with your own code"). For Python tasks the first inspector tab is **Code**. 2. Set **Task id** and **Function name** to `transform`. The function name must be a valid Python identifier. 3. Replace the **Function body** with: ```python import json with open(f"/data/orders/{ds}.json") as f: orders = json.load(f) return sum(o["total"] for o in orders) ``` `return` pushes the total to XCom. (This body has a deliberate mistake that step 5 fixes.) 4. Search the palette for `SQLExecuteQueryOperator`, add it and set its **Task id** to `load`. 5. In **sql**, type: ```sql INSERT INTO sales.daily_orders VALUES ('{{ ds }}', {{ ti.xcom_pull('transform') }}) ``` In a real DAG also choose the database connection in **conn_id**. The connection picker lists Airflow's connections, filtered by connection type, and can create one with **New connection**; in local mode, without an Airflow server, you type the connection id. See [Connections](https://ctolon.github.io/autoair/docs/user-guide/connections). Now connect the tasks: drag from the handle on the **right** edge of `extract` to the handle on the **left** edge of `transform`, then from `transform` to `load`. An edge means "runs after"; a connection that would create a cycle is refused. Press Shift+L (or **Auto layout** in the toolbar) to tidy the graph. [Screenshot: extract, transform and load connected left to right; transform is selected and shows 1 error, and the inspector lists the problem code: Undefined name ds] Select `transform`: its node already carries a red **1 error** badge, and the inspector lists the problem at the top. **Inputs from upstream tasks** shows `extract`'s `return_value`; you could drag it onto an argument to pass `extract`'s XCom to the function (see [Jinja, XCom and Python](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python)). ## 4. Schedule it daily Click an empty part of the canvas to return to **DAG settings** and open the **Schedule** tab. 1. Set **Schedule** to **Preset (@daily, …)** and pick `@daily` ("every day at midnight"). 2. Set **Start date**. A new DAG starts on 1 January of the current year (UTC). The scheduler never creates runs before it. 3. Leave **Catchup** off unless you want a run for every missed interval between the start date and now. 4. Optionally set **DAG timezone** to an IANA name such as `Europe/Istanbul`. Start and end dates and the `@daily` preset are then local to that zone (DST aware). An unknown zone is flagged ("Unknown timezone") and not applied. **Next runs** previews the upcoming run times, in your local time and in UTC, computed by Airflow's timetable engine. [Screenshot: The Schedule tab of DAG settings with the @daily preset selected, start date, catchup switch, DAG timezone and the Next runs preview] Other schedule types (cron, fixed interval, multiple crons, specific dates, assets, assets-or-time, partitioned schedules) are described in [Scheduling and assets](https://ctolon.github.io/autoair/docs/user-guide/scheduling-and-assets). ## 5. Fix the problems AutoAir validates the spec while you edit. The **Problems** tab of the bottom panel (Ctrl/⌘+J toggles the panel) lists every problem grouped by task, with filters **All**, **Errors**, **Warnings** and **Info**. Clicking a problem selects the node and focuses the exact field, including fields inside nested editors. [Screenshot: The Problems tab listing one error on transform: code: Undefined name ds, reported by ruff as F821] Here ruff reports `F821 Undefined name 'ds'`: inside a TaskFlow function, `ds` is not a global. Click the problem to jump to the function body and read the date from the task context instead: ```python import json from airflow.sdk import get_current_context ds = get_current_context()["ds"] with open(f"/data/orders/{ds}.json") as f: orders = json.load(f) return sum(o["total"] for o in orders) ``` The Problems tab then shows "No problems detected." Other typical problems are: - a required parameter that is empty, for example a `BashOperator` without `bash_command`; - a duplicate task id; - a Jinja error or an undeclared `params.` in a template; - a policy rule set by your administrator, such as a required owner or a minimum number of retries; - a hard-coded secret (a password or token typed into a field), with a hint to use Connections or Variables. While there are errors, the **Deploy** button explains "Fix 1 problem to deploy (click to see it)" and clicking it opens the Problems tab. Warnings and info items do not block a deploy. Details: [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation). ## 6. Preview the generated code Open the **Code** tab of the bottom panel. It shows the file AutoAir will deploy, regenerated as you edit. The selector switches between **Python**, **Airflow YAML**, **dag-factory** and **Spec JSON** (the document AutoAir stores). If the spec has errors, the tab says "Fix the problems to see the generated code." [Screenshot: The Code tab of the bottom panel showing the generated orders_daily Python DAG file] The DAG body of the generated file for this tutorial starts like this (the module docstring and imports are above it): ```python with DAG( dag_id="orders_daily", description="Daily orders export into the warehouse", schedule="@daily", start_date=datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc), catchup=False, tags=["autoair"], default_args={"owner": "airflow", "retries": 1, "retry_delay": datetime.timedelta(seconds=300)}, ) as dag: t_extract = BashOperator( task_id="extract", bash_command="curl -sf -o /data/orders/{{ ds }}.json https://shop.example.com/api/orders?date={{ ds }}", ) @task(task_id="transform") def transform(): import json from airflow.sdk import get_current_context ds = get_current_context()["ds"] with open(f"/data/orders/{ds}.json") as f: orders = json.load(f) return sum((o["total"] for o in orders)) t_transform = transform() t_load = SQLExecuteQueryOperator( task_id="load", sql="INSERT INTO sales.daily_orders VALUES ('{{ ds }}', {{ ti.xcom_pull('transform') }})", ) t_extract >> t_transform t_transform >> t_load ``` The default arguments (`owner`, `retries`, `retry_delay`) come from the **Defaults** tab of DAG settings; a new DAG starts with one retry after 300 seconds. The imports follow the target Airflow version shown in the toolbar. Airflow YAML output does not support Python tasks, so the **Airflow YAML** view reports that this DAG cannot be expressed in YAML. To edit the file directly instead, switch to code mode; applying it imports the code back into the spec. See [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode). ## 7. Validate in Airflow Live validation is static. To import the generated file in a real `DagBag`, choose **Run → Validate in Airflow**. A clean import reports **Valid**: "The DAG imports cleanly in Airflow (DagBag)." Import errors (a missing provider, a syntax error in a task body) are listed with their messages. The import runs in a subprocess on the AutoAir server, so the providers your DAG uses must be installed there. In plugin mode that is Airflow's own environment. Every deploy runs the same check again unless `import_check_on_deploy` is off. ## 8. Deploy Click **Deploy** in the toolbar, or press Ctrl/⌘+S. The arrow next to it opens **Deploy…**, a dialog that shows: - **Target**: where the files go, for example `local · /opt/airflow/dags/autoair` or a Git repository and branch; - **Environment**, when environments are configured (otherwise "No environments are configured; the base spec is deployed."); - **Message**: the version or commit message ("What changed and why?"); Ctrl+Enter deploys; - for Git targets, whether the deploy pushes or opens a pull request, and a preview of the commit and pull request text. [Screenshot: The Deploy orders_daily dialog with the target, environment and message fields and the Deploy button] On success the toolbar shows **Deployed** and the status chip changes from **new** to **deployed**. With the local target, the deploy writes two files into the DAG folder (`./dags` in local mode, `/autoair` by default otherwise): | File | Content | | --- | --- | | `orders_daily.autoair.json` | The spec. AutoAir reads it back when you reopen the DAG. | | `orders_daily.py` | The generated DAG file Airflow parses. | What can go differently: - **Someone else deployed the DAG since you opened it.** AutoAir uses ETags; instead of overwriting their work you get "Someone else changed this DAG" and a three-way merge dialog. - **The generated file was edited by hand.** The deploy stops and reports the file as ejected; AutoAir overwrites it only when you force the deploy. - **A file with the same name was not created by AutoAir.** It is never overwritten. - **Git target in pull-request mode.** The deploy opens a pull request ("Pull request opened"); the DAG goes live when the pull request is merged. - **Four-eyes approvals enabled.** The deploy is stored as a proposal until another reviewer approves this version. See [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy) and [Reviews and approvals](https://ctolon.github.io/autoair/docs/user-guide/reviews-and-approvals). ## 9. Run it (plugin and standalone) Airflow's dag-processor picks the file up on its next parse. New DAGs usually start paused (Airflow's `dags_are_paused_at_creation`); the toolbar chip shows the paused state and can unpause the DAG. Then: 1. Choose **Run → Trigger…** to start a run. 2. Open the **Runs** tab of the bottom panel and select the run. Task states are painted on the canvas. 3. Select `transform` and open the **Task** tab to read its **Logs** and **XCom**. The XCom holds the day's order total. In local mode runs are switched off (`features.runs`), so these items are not available; open the deployed file in an Airflow of your own instead. See [Runs](https://ctolon.github.io/autoair/docs/user-guide/runs). ## Where to go next - [Concepts](https://ctolon.github.io/autoair/docs/getting-started/concepts): what the spec, validation and deploy targets are. - [Operators and parameters](https://ctolon.github.io/autoair/docs/user-guide/operators-and-parameters) and [Jinja, XCom and Python](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python). - [TaskFlow and loops](https://ctolon.github.io/autoair/docs/user-guide/taskflow-and-loops): mapping over the transform output. - [Versions and diff](https://ctolon.github.io/autoair/docs/user-guide/versions-and-diff): compare what you just deployed with the next change. # Concepts Source: https://ctolon.github.io/autoair/docs/getting-started/concepts Each section below ends with a link to the detailed page. ## The DagSpec A DAG built with AutoAir is a JSON document called the **DagSpec**. It is validated by a Pydantic model on the server (`autoair.spec.DagSpec`) and carries a `spec_version` (currently `3`); older documents are migrated forward when they are read, and a document newer than the running AutoAir is rejected ("spec_version N is newer than this AutoAir supports (M); upgrade AutoAir"). The spec holds: | Part | Examples | | --- | --- | | DAG settings | `dag_id`, `description`, `tags`, `owner_links`, `schedule`, `start_date`, `end_date`, `catchup`, DAG timezone, `default_args`, `params`, callbacks, `max_active_runs` | | `nodes` | operators, Python tasks, task groups, extension nodes, notes | | `edges` | dependencies between nodes, with optional labels | | `output_mode` | how the DAG file is produced: `python`, `loader` or `yaml` | | `environments` | per-environment overlays (see [Environments](#environments)) | The spec is stored as `.autoair.json` in the deploy target, next to the generated file, and every deploy keeps a version of it in the history (`storage.history_limit`, 100 per DAG by default). The canvas, code mode, AI and MCP tools, the importers and the REST API all read and write this one document. You can work with specs outside the UI: ```bash autoair generate orders_daily.autoair.json -o orders_daily.py # spec -> Python autoair validate orders_daily.autoair.json # graph, catalog, policy, DagBag import autoair export-schema -o dag-spec.schema.json # JSON Schema of the spec ``` See [Spec format](https://ctolon.github.io/autoair/docs/extending/spec-format) and the [CLI reference](https://ctolon.github.io/autoair/docs/reference/cli). ## Nodes and edges Each node has a stable internal `id` (used by edges and XCom references) and, for tasks, a `task_id` that Airflow sees. Node kinds: | Kind | Canvas name | Becomes | | --- | --- | --- | | `operator` | the operator class, e.g. `BashOperator` | an operator instance; `params` hold its constructor arguments | | `python_task` | **Python task** and its decorator variants (`@task`, `@task.branch`, `@task.short_circuit`, `@task.virtualenv`, `@task.external_python`, `@task.branch_virtualenv`, `@task.branch_external_python`, `@task.sensor`, `@task.bash`, `@task.docker`, `@task.kubernetes`, `@task.stub`, and from `apache-airflow-providers-common-ai` `@task.llm`, `@task.agent`, `@task.llm_branch` and `@task.llm_sql`) | a TaskFlow function with the body you write | | `task_group` | **Task group** | a `TaskGroup` | | `custom` | **Extension** | code produced by a registered node handler (see [Node SDK](https://ctolon.github.io/autoair/docs/extending/node-sdk)) | | `note` | **Note** | nothing; canvas documentation only | A parameter value can be a literal, a Jinja template, the **output of an upstream task** (an XCom reference, rendered as `t_upstream.output` for an operator, the TaskFlow call result for a Python task, optionally subscripted with a key), a Python reference (an import path), a Python function written in place, or an embedded script file. Edges become `>>` dependencies. Trigger rules, retries, setup/teardown, mapping (`expand`) and the other task attributes live on the node. The operator palette comes from the **operator catalog**: AutoAir parses the installed Airflow providers, third-party operator packages and your custom operators statically (without importing them) and caches the result. Operators that are not installed where AutoAir runs do not appear. See [Operators and parameters](https://ctolon.github.io/autoair/docs/user-guide/operators-and-parameters) and [Catalog sources](https://ctolon.github.io/autoair/docs/extending/catalog-sources). ## Task groups A task group is a node that contains other nodes. On the canvas you group a selection, ungroup it, or collapse it to one box. In the generated code the group becomes a `TaskGroup` and its members' task ids are prefixed with the group id (the inspector shows the effective **Airflow task id**). Edges to and from a group apply to the whole group. Groups can be nested; a group cannot be placed inside itself. Reusable groups of tasks can be saved as **fragments**. See [Canvas](https://ctolon.github.io/autoair/docs/user-guide/canvas) and [Templates and fragments](https://ctolon.github.io/autoair/docs/user-guide/templates-and-fragments). ## Generated code The spec's `output_mode` decides what is deployed next to the spec: | Output | Setting in **DAG settings → General → Output** | Deployed file | | --- | --- | --- | | `python` (default) | Generated Python file (readable, reviewable) | `.py`, a normal Airflow DAG using `airflow.sdk`, formatted with ruff when it is installed | | `loader` | JSON spec + runtime loader | a small stub that builds the DAG from the spec at parse time | | `yaml` | Airflow YAML DAG | an Airflow 3.4 YAML DAG; task groups and Python tasks are not supported | Imports and constructs follow the **target Airflow version** (the installed one, or `target_airflow_version`); features that need a newer Airflow than the target are reported as problems. Besides the deployed output you can export a DAG as Python, Airflow YAML, dag-factory YAML or spec JSON, and import hand-written Python DAGs, Airflow YAML or DAGs that already exist in Airflow. Generated files start with a header that names the spec; AutoAir will not overwrite a generated file that was changed by hand ("ejected") unless the deploy is forced. See [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode) and [Import and export](https://ctolon.github.io/autoair/docs/user-guide/import-export). ## Validation Validation runs continuously while you edit and again on every deploy: 1. **Schema**: the spec must match the model (types, required fields, identifiers). 2. **Graph**: no cycles, unique task ids, valid group membership and XCom references. 3. **Catalog**: every operator exists in the catalog and its required parameters are set with values of the right type. 4. **Jinja**: templates parse; undeclared `params`, and time variables in asset-triggered DAGs, are flagged. 5. **Policy**: organisation rules such as `require_owner`, `min_retries`, `required_tags`, `forbidden_operators`, `max_tasks` and `dag_id_prefix`, plus the policy engine (see [Policy](https://ctolon.github.io/autoair/docs/admin/policy); every rule is listed under [`policy`](https://ctolon.github.io/autoair/docs/reference/configuration#policy) in the configuration reference). 6. **Version gates**: features newer than the target Airflow. 7. **Import check**: the generated file is imported in an Airflow `DagBag` in a subprocess (warm worker pool by default). It needs Airflow installed where AutoAir runs, and on deploy it runs only for callers holding `code:execute` (`iam.import_check_requires_code_execute`). Problems have a severity (error, warning, info); only errors block a deploy. See [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation). ## Deploy targets A deploy writes the spec and the generated file to a **deploy target** (`storage.mode`): | Target | Airflow reads DAGs through | Behaviour | | --- | --- | --- | | `local` | a `LocalDagBundle` / the DAG folder on a shared volume | atomic file writes to `/` (or `storage.local_path`) | | `object` | `S3DagBundle`, `GCSDagBundle` or another object-storage bundle | writes to an fsspec URL (`storage.object_url`) | | `git` | `GitDagBundle` or a git-sync sidecar | commits per DAG; `pr` mode opens a branch and pull request (default), `push` commits to the branch. GitHub, GitLab, Bitbucket, Azure DevOps, Gitea/Forgejo or any remote through the git CLI | Every write uses an ETag for optimistic locking, so two people deploying the same DAG get a three-way merge instead of a lost update. Administrators can also choose the target in the UI (**Versions & Git → Deploy target…**), which lists the DAG folder, configured GitDagBundles and a git-sync checkout. See [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy), [Git bundles](https://ctolon.github.io/autoair/docs/deployment/git-bundles) and [Object storage](https://ctolon.github.io/autoair/docs/deployment/object-storage). ## Environments `environments` (for example `["dev","staging","prod"]`) lets one spec be deployed differently per environment. An **environment overlay** in the spec can override the schedule, `is_paused_upon_creation`, tags, default arguments, parameter defaults, `max_active_runs`, and map logical connection ids and Variable keys to the environment's own. With a Git target, `storage.git.environments` maps each environment to a branch and **Promote environment…** moves a DAG from one to the next. Loader-mode DAGs apply the overlay named by `AUTOAIR_ENVIRONMENT` at parse time. See [Environments](https://ctolon.github.io/autoair/docs/user-guide/environments). ## IAM roles Authentication always goes through Airflow (the Airflow session in plugin mode; a verified Airflow token in standalone mode). Authorization combines Airflow's per-DAG permissions with AutoAir's own roles: | Role | Can | | --- | --- | | Viewer | read specs, the catalog, code preview, history and audit | | Author | create and edit specs, validate, test runs, use AI; proposes changes (pull requests) | | Reviewer | read everything, approve spec versions, AI review | | Deployer | deploy live, delete, promote between environments, trigger runs | | CodeDeployer | Deployer, plus deploying DAGs that keep their hand-written Python file | | Operator | trigger, backfill, clear, pause and unpause runs | | TemplateManager | curate templates, marketplace installs and icons | | Admin | everything, including the deploy target and configuration | How users get these roles is set by `iam.strategies`: from Airflow's DAG and configuration permissions (default), from Airflow role names (`role_mapping`), or from AutoAir permissions created in the Airflow auth manager (`custom_view`). Four-eyes approvals (`iam.require_approval`) and direct-push rights for Git are configured here too. `autoair iam roles` prints the effective table. See [IAM](https://ctolon.github.io/autoair/docs/admin/iam). ## Collaboration Several people can edit the same DAG at once. Live co-editing uses Yjs over a WebSocket at `/collab/`; the UI shows presence avatars, remote cursors and selections, and undo is per user. A WebSocket is opened with a short-lived ticket that requires edit permission on the DAG. It needs the `collab` extra and `features.collab`; with several replicas, rooms are shared through the database or Redis (`collab.sync`). Deploy conflicts are still caught by ETags. See [Collaboration](https://ctolon.github.io/autoair/docs/user-guide/collaboration). ## AI AI assistance is optional and off until a model provider is configured (`AUTOAIR_AI__PROVIDER`, `AUTOAIR_AI__MODEL` and an API key or Airflow connection). It can generate a DAG from a prompt, edit a DAG (shown as a reviewed diff), explain problems, review a DAG and write documentation. Every AI result goes through the same validation as a human edit. Rule-based review, leak scan and optimisation work without a model. AutoAir also serves an MCP endpoint (`/mcp`) so external agents can author DAGs with the caller's permissions. See [AI introduction](https://ctolon.github.io/autoair/docs/ai/introduction), [Guardrails](https://ctolon.github.io/autoair/docs/ai/guardrails) and [MCP](https://ctolon.github.io/autoair/docs/ai/mcp). # Installation overview Source: https://ctolon.github.io/autoair/docs/installation/overview AutoAir is one Python package, `autoair`. Its wheel contains the backend, the CLI and the built frontend. What you install around it, and where, depends on the run mode. ## Requirements | Component | Requirement | | --- | --- | | Python | 3.10 or newer (CI runs 3.10 and 3.12) | | Apache Airflow | 3.0 or newer, below 4. CI matrix: 3.0.6, 3.1.8, 3.2.2, 3.3.2, plus the 3.4.0b2 pre-release (allowed to fail). The `mcp` extra cannot be installed next to Airflow 3.1: see [Airflow compatibility](https://ctolon.github.io/autoair/docs/installation/compatibility) | | Airflow REST API | API v2 (`/api/v2/...`), reachable from AutoAir in standalone mode | | Node.js | 22, only to build the frontend from a source checkout | | Optional services | PostgreSQL for shared state, Redis for collaboration fan-out, a Git host or object storage as deploy target, an LLM provider for AI | Features that need a newer Airflow than the one installed (or the one named by `AUTOAIR_TARGET_AIRFLOW_VERSION`) are reported as validation problems rather than failing at parse time. ## Choosing a mode | | Plugin | Standalone | Local | | --- | --- | --- | --- | | Start | Loaded by the Airflow API server through the `airflow.plugins` entry point | `autoair serve` | `autoair local` | | UI location | Airflow ≥ 3.1: **Browse → DAG Builder** and an **AutoAir** tab on DAG pages; Airflow 3.0: `/autoair/ui/` | `http://:8765/ui/` | `http://127.0.0.1:8765/ui/` | | API base | `/autoair/api` | `/api` | `/api` | | Authentication | Airflow's auth manager (the UI session) | Airflow token verified against Airflow: Airflow session cookie, username/password, or pasted token | None on loopback; optional static token (`--token`) | | Installs into | Airflow's Python environment (API server, and the same packages everywhere DAGs are parsed) | Its own environment or container | Any environment, e.g. `uvx` | | Operator palette | Providers installed in Airflow's image | Providers installed next to AutoAir (install the same ones as Airflow) | Packages installed next to AutoAir | | Runs, logs, pickers | Yes | Yes, through Airflow's REST API | No (`features.runs`, `features.test_runs`, `features.airflow` off) | | Scaling | With the API server replicas | Independent workers/replicas | Single process | | Upgrade | Rebuild the Airflow image | Independent of Airflow | Reinstall | | Best for | Teams running Airflow 3 who want the builder inside the Airflow UI | Separate lifecycle, separate scaling, or an Airflow you cannot add packages to | Evaluation, offline authoring | Rules of thumb: - If you can change the Airflow image, start with the **plugin**: no second service, no second sign-in, and the operator catalog matches Airflow exactly. - Choose **standalone** when AutoAir must be upgraded, scaled or secured separately from Airflow, or when Airflow is managed by someone else and you only have API access. - Use **local** to try AutoAir or to author DAG files without an Airflow server. All modes read the same settings (`AUTOAIR_*` environment variables or configuration files); see [Configuration](https://ctolon.github.io/autoair/docs/admin/configuration). ## Installation methods | Method | Mode | Page | | --- | --- | --- | | `uvx` / `pip` on a workstation | Local | [Local mode](https://ctolon.github.io/autoair/docs/installation/local) | | `pip install` into the Airflow environment or a derived Airflow image | Plugin | [Airflow plugin](https://ctolon.github.io/autoair/docs/installation/plugin) | | `pip install` or a container running `autoair serve` | Standalone | [Standalone server](https://ctolon.github.io/autoair/docs/installation/standalone) | | Docker Compose (demo: Airflow + plugin + two standalone replicas + PostgreSQL) | Plugin and standalone | [Docker Compose](https://ctolon.github.io/autoair/docs/installation/docker-compose) | | Official Apache Airflow Helm chart with a derived image | Plugin | [Helm](https://ctolon.github.io/autoair/docs/installation/helm) | Pick optional features with [extras](https://ctolon.github.io/autoair/docs/installation/extras). For production topologies (Kubernetes, Git bundles, object storage, a reverse proxy in front of standalone, PostgreSQL, air-gapped installs) see [Deployment](https://ctolon.github.io/autoair/docs/deployment/overview). ## Getting the package - **Python package**: `pip install autoair` with the extras you need. The release workflow (`.github/workflows/release.yml`) publishes the wheel and sdist to PyPI as `autoair` when a `v*` tag matching the package version is pushed, and attaches them with `SHA256SUMS` to the GitHub release. - **Published container image**: the same workflow pushes `ghcr.io/ctolon/autoair` (linux/amd64 and linux/arm64, with SBOM and build provenance), built from `deploy/docker/Dockerfile` on the official `apache/airflow` image. Tags: ``, `-airflow` (currently Airflow 3.3.2), and for final releases `.` and `latest`. Check `autoair --version` (or `pip show autoair`) to see which version you run. - **From source**: build the frontend and the wheel with `make build` (runs `npm ci && npm run build` in `frontend/`, then `uv build`), which produces `dist/autoair--py3-none-any.whl`. A wheel built without the frontend still installs, but `/ui/` answers `503` ("Frontend bundle not built"). - **Building your own image**: `deploy/docker/Dockerfile` installs a prebuilt wheel on top of `apache/airflow` and precomputes the operator catalog; `deploy/docker/Dockerfile.demo` builds the frontend and wheel inside Docker. ## After installing 1. Open the UI and check the version and the target Airflow version in the toolbar. 2. Decide on the deploy target (local folder, object storage or Git) so that the files land where Airflow reads DAGs: [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy). 3. Configure access: [IAM](https://ctolon.github.io/autoair/docs/admin/iam). 4. For more than one replica, move state to PostgreSQL: [Database](https://ctolon.github.io/autoair/docs/admin/database), [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling). 5. Optionally enable AI: [AI providers](https://ctolon.github.io/autoair/docs/ai/providers). # Optional extras Source: https://ctolon.github.io/autoair/docs/installation/extras The core `autoair` package installs only what every mode needs (FastAPI, Pydantic, httpx2, fsspec, Jinja, jedi, Typer, PyYAML, cryptography, Pillow and the Prometheus client). Everything else is an **extra**. A feature whose extra is missing is either switched off with a log message or reports a clear error that names the extra to install. Not every extra can be installed next to every Airflow version: `mcp` does not resolve with Airflow 3.1. See [Airflow compatibility](https://ctolon.github.io/autoair/docs/installation/compatibility) for the matrix. ## Installing extras Extras go in square brackets. Quote the requirement so the shell does not expand the brackets. ```bash pip install 'autoair[standalone,format,git,mcp,collab,postgres]' # uv, in a project uv add 'autoair[standalone,format,git,mcp,collab,postgres]' # uv, one-off tool run uvx --from 'autoair[standalone,airflow]' autoair local # a local wheel pip install "dist/autoair-0.1.0-py3-none-any.whl[standalone,format]" ``` Inside an Airflow image, install AutoAir with the extras in the same `pip install` as your providers, and pin `apache-airflow` to the image's version so pip does not change it (the demo Dockerfile does `pip install "apache-airflow==${AIRFLOW_VERSION}" "[standalone,format,git,ai,ai-openai,ai-anthropic,collab,collab-redis,postgres,otel,yaml]" apache-airflow-providers-standard apache-airflow-providers-git`). ## Which extras for which mode | Mode | Recommended extras | | --- | --- | | Local mode | `standalone,airflow` (add `format` for formatted output) | | Airflow plugin | `format,git,mcp` plus `ai` and a provider extra such as `ai-anthropic` for AI (add `collab`, `postgres`, `otel`, `shell` as needed); never `airflow` (Airflow is already installed) and not `standalone` (Airflow's API server serves AutoAir). On Airflow 3.1 leave out `mcp` and [run the MCP server separately](https://ctolon.github.io/autoair/docs/installation/compatibility#mcp-next-to-airflow-31) | | Standalone server | `standalone,format,git,mcp,collab` plus `postgres` for several replicas and `ai,ai-` for AI | | Separate AI service | `standalone,mcp,ai` plus the `ai-*` provider extras you use (the main server then needs no `ai` extra) | ## Reference ### Runtime and server | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `airflow` | `apache-airflow>=3.0,<4`, `apache-airflow-providers-standard` | A standard operator palette (the operators and sensors of `apache-airflow-providers-standard`, such as `BashOperator`, `PythonOperator` and `EmptyOperator`) and the `DagBag` import check when AutoAir runs without an Airflow installation, mainly for `autoair local` | Installed packages are picked up automatically. In local mode `import_check_on_deploy` is turned on only when Airflow is importable. | | `standalone` | `uvicorn[standard]` | The HTTP server behind `autoair serve`, `autoair local`, `autoair mcp-serve --http` and `autoair ai-serve`. Without it these commands exit with code 1 and a message naming the extra to install (`autoair local` also suggests `airflow` while Airflow is not installed). | Running one of those commands. Not needed in plugin mode. | | `format` | `ruff` | `ruff format` of generated DAG files and live ruff lint of Python task code in the editor | Automatic when `ruff` is found on `PATH` or importable. Tune with `codegen.line_length`, `codegen.format_timeout`, `editor.python_lint_select`. Without ruff, generated code is emitted unformatted. | | `yaml` | `pyyaml` | YAML configuration files (`AUTOAIR_CONFIG_FILE=/etc/autoair/autoair.yaml`) | PyYAML is also a core dependency; the extra is a stable install name. A missing PyYAML is reported as "YAML configuration files need PyYAML: pip install 'autoair[yaml]'". | | `otel` | `opentelemetry-instrumentation-fastapi`, `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-http` | OpenTelemetry tracing of the API | `AUTOAIR_OTEL_ENABLED=true`. Exporter settings use the standard `OTEL_*` variables (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`); inside Airflow an existing tracer provider is reused. `server.otel_excluded_urls` skips health, ready and metrics. | ### Deploy targets and state | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `git` | nothing | Stable install name for the Git deploy target | `AUTOAIR_STORAGE__MODE=git`, `AUTOAIR_STORAGE__GIT__PROVIDER`, `AUTOAIR_STORAGE__GIT__REPO` and credentials (`AUTOAIR_STORAGE__GIT__CONN_ID` or `AUTOAIR_STORAGE__GIT__TOKEN`); every option is listed under [`storage.git`](https://ctolon.github.io/autoair/docs/reference/configuration#storagegit). The Git target talks to GitHub, GitLab, Bitbucket and Azure DevOps through their REST APIs; `storage.git.provider=local` (any remote) and `gitea` (Gitea or Forgejo) run the `git` command-line tool, which must be on `PATH`. | | `sqlite` | `sqlalchemy` (1.4.49 or newer; Airflow 3.0 installs 1.4) | AutoAir state (saved templates, the audit trail, reviews and deploy proposals, the deploy target chosen in the UI, custom icons, AI memory, the AI audit log, AI knowledge overrides, per-user AI model settings, collaboration room snapshots, users seen and replica heartbeats) in one SQLite file | `AUTOAIR_DATABASE__BACKEND=sqlite`, optionally `AUTOAIR_DATABASE__URL=sqlite:////var/lib/autoair/autoair.db` (default `/autoair.db`). Suits several workers on one host. | | `postgres` | `sqlalchemy`, `psycopg[binary]` | AutoAir state in PostgreSQL, shared by every replica | `AUTOAIR_DATABASE__BACKEND=postgresql`, `AUTOAIR_DATABASE__URL=postgresql+psycopg://autoair@db:5432/autoair`, `AUTOAIR_DATABASE__PASSWORD`. Missing driver: "cannot load the postgresql driver (...): pip install 'autoair[postgres]'". On Airflow 3.0 (SQLAlchemy 1.4) the driver is psycopg2: see [PostgreSQL on Airflow 3.0](https://ctolon.github.io/autoair/docs/installation/compatibility#postgresql-on-airflow-30). | **Note:** The `git` extra is empty: every Git provider uses either the host's REST API (over httpx2, a core dependency) or the `git` CLI, so AutoAir needs no Python Git library. The extra only keeps `autoair[git]` (and `autoair[all]`) installable under a stable name; a Git deploy target works without it as long as the `git` binary is installed where the `local` or `gitea` provider runs. (Airflow's `GitDagBundle` needs `apache-airflow-providers-git` on the Airflow side, which brings its own dependencies.) Object storage targets (`AUTOAIR_STORAGE__MODE=object`) use fsspec, which is a core dependency; install the fsspec implementation for your store yourself (`s3fs`, `gcsfs`, `adlfs`). See [Object storage](https://ctolon.github.io/autoair/docs/deployment/object-storage) and [Database](https://ctolon.github.io/autoair/docs/admin/database). ### Collaboration | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `collab` | `pycrdt`, `pycrdt-websocket` | Live co-editing (Yjs over WebSockets at `/collab/`) | `AUTOAIR_FEATURES__COLLAB=true` (default). Without the packages the `/collab` WebSocket endpoint is not mounted, so co-editing is unavailable. Several standalone replicas also need `AUTOAIR_COLLAB_SECRET` (or a shared database) and `collab.sync`. | | `collab-redis` | `redis` | Fan-out of collaboration rooms through Redis pub/sub | `AUTOAIR_COLLAB__SYNC=redis` and `AUTOAIR_COLLAB__REDIS_URL=redis://redis:6379/0`. `collab.sync=auto` (default) picks Redis when `collab.redis_url` is set. | ### Agents and AI | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `mcp` | `fastmcp`, `mcp` | The MCP endpoint at `/mcp` (plugin: `/autoair/mcp/`), `autoair mcp` (stdio) and `autoair mcp-serve` | `AUTOAIR_FEATURES__MCP=true` (default). Without the packages the server logs "MCP endpoint disabled: ... (install autoair[mcp])", `/api/meta` reports `mcp_unavailable_reason` and the admin *Features* tab shows `mcp` as *Not installed*. Writes are controlled by `ai.mcp.allow_writes`. Cannot be installed next to Airflow 3.1 ([why](https://ctolon.github.io/autoair/docs/installation/compatibility)). | | `ai` | `pydantic-ai-slim` | The AI runtime ([pydantic-ai](https://ai.pydantic.dev), as used by Airflow's common.ai provider; compatible with it in one environment). Without it AI is off: `/api/meta` reports `ai_unavailable_reason`, the admin panel shows the hint, only `GET /api/ai/status` answers, the AI MCP tools are hidden and the admin *Features* tab shows `ai` as *Not installed* and `autoair ai-serve` exits with the extra to install. | Install it with a provider extra, then configure `AUTOAIR_AI__PROVIDER`, `AUTOAIR_AI__MODEL`, `AUTOAIR_AI__API_KEY` (or `AUTOAIR_AI__LLM_CONN_ID`, a `pydanticai` connection). | | `ai-anthropic` | `pydantic-ai-slim[anthropic]` (the `anthropic` SDK) | Claude | `AUTOAIR_AI__PROVIDER=anthropic` | | `ai-openai` | `pydantic-ai-slim[openai]` (the `openai` SDK) | OpenAI, Azure OpenAI and OpenAI-compatible servers (vLLM, LM Studio, LiteLLM) | `AUTOAIR_AI__PROVIDER=openai` / `azure`; `AUTOAIR_AI__BASE_URL` for compatible servers | | `ai-google` | `pydantic-ai-slim[google]` (`google-genai`) | Gemini and Vertex AI | `AUTOAIR_AI__PROVIDER=gemini` / `google-cloud` | | `ai-ollama` | `pydantic-ai-slim[openai]` | Ollama (its OpenAI-compatible API) | `AUTOAIR_AI__PROVIDER=ollama`, `AUTOAIR_AI__BASE_URL=http://ollama:11434` | | `ai-bedrock`, `ai-groq`, `ai-mistral`, `ai-cohere` | `pydantic-ai-slim[bedrock]` / `[groq]` / `[mistral]` / `[cohere]` | Amazon Bedrock, Groq, Mistral, Cohere | `AUTOAIR_AI__PROVIDER=bedrock` / `groq` / `mistral` / `cohere` and `AUTOAIR_AI__MODEL` | | `ai-all` | every provider extra above | All providers | | | `ai-rerank` | `sentence-transformers` (pulls PyTorch; large) | Local cross-encoder reranking for AI retrieval | `AUTOAIR_AI__RETRIEVAL__RERANKER=cross_encoder`, optional `AUTOAIR_AI__RETRIEVAL__RERANK_MODEL`. Without it the reranker is skipped with a warning. | | `ai-guardrails` | `llm-guard` | The LLM Guard input scanners as an extra guardrail | `AUTOAIR_AI__GUARDRAILS__EXTERNAL='["llm_guard"]'`. NeMo Guardrails (`nemo:`) and Guardrails AI (`guardrails_ai`) need `nemoguardrails` or `guardrails-ai` installed separately. | See [AI providers](https://ctolon.github.io/autoair/docs/ai/providers), [Guardrails](https://ctolon.github.io/autoair/docs/ai/guardrails), [AI service](https://ctolon.github.io/autoair/docs/ai/ai-service) and [MCP](https://ctolon.github.io/autoair/docs/ai/mcp). ### Documentation knowledge base The knowledge base works without these extras on its default `local` store (SQLite, no dependencies). | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `kb-pgvector` | `sqlalchemy` (1.4.49 or newer), `psycopg[binary]` (psycopg2 on SQLAlchemy 1.4) | The `pgvector` knowledge-base store in PostgreSQL | `AUTOAIR_KB__STORE=pgvector`, optionally `AUTOAIR_KB__PGVECTOR__URL` (default: AutoAir's own database when `database.backend=postgresql`) | | `kb-elasticsearch` | nothing | Stable install name for the `elasticsearch` store, which uses the REST API over httpx2 | `AUTOAIR_KB__STORE=elasticsearch`, `AUTOAIR_KB__ELASTICSEARCH__URL` | See [Knowledge base](https://ctolon.github.io/autoair/docs/ai/knowledge-base). ### Editors | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `shell` | `shellcheck-py` (the ShellCheck binary) | ShellCheck diagnostics for `bash_command`, `@task.bash` and `.sh` script files, in the editor and as validation warnings | Automatic when installed. `editor.shellcheck_in_validation`, `editor.shellcheck_exclude`, `editor.shellcheck_exclude_extra`, `editor.shellcheck_timeout`. | | `lsp` | `python-lsp-server`, `python-lsp-ruff` | IDE-grade Python IntelliSense through pylsp | `AUTOAIR_FEATURES__IDE_LANGUAGE_SERVER=true` and `AUTOAIR_LSP__PYTHON_SERVER=pylsp` | | `lsp-pyright` | `basedpyright` (bundles Node through `nodejs-wheel`) | IDE-grade Python IntelliSense through basedpyright | `AUTOAIR_FEATURES__IDE_LANGUAGE_SERVER=true` (`lsp.python_server` defaults to `basedpyright`) | Without a language server the editors keep the built-in jedi completion, hover and signature help. Bash IntelliSense through a language server needs `npm i -g bash-language-server` (`lsp.bash_command`); without it shell fields keep the built-in completion and ShellCheck. Limits: `lsp.max_sessions` (16), `lsp.max_sessions_per_user` (4), `lsp.idle_timeout` (600 s), `lsp.start_timeout` (30 s), `lsp.memory_limit_mb` (2048), `lsp.cpu_limit_seconds` (3600) and `lsp.max_message_bytes` (8 MiB); see [`lsp`](https://ctolon.github.io/autoair/docs/reference/configuration#lsp) in the configuration reference. ### Astronomer Blueprint | Extra | Installs | Enables | Turned on by | | --- | --- | --- | --- | | `blueprint` | `airflow-blueprint>=0.3,<1` | Config schemas, task previews and config validation of [Blueprint steps](https://ctolon.github.io/autoair/docs/user-guide/blueprints) (templates are imported in a child process). Airflow needs the package too, to parse the generated DAGs | `features.blueprints` (default on) and templates from entry points, `blueprints.template_dirs` or `blueprints.modules` | ### `all` `autoair[all]` is a shortcut for `standalone,format,git,ai,ai-anthropic,ai-openai,otel,collab,shell,yaml`: everything that installs next to every supported Airflow. It does **not** include `mcp` (it cannot be installed with Airflow 3.1), `airflow`, `sqlite`, `postgres`, `collab-redis`, `kb-pgvector`, `kb-elasticsearch`, the other `ai-*` extras (`ai-google`, `ai-bedrock`, ..., `ai-rerank`, `ai-guardrails`), `lsp` or `lsp-pyright`; add those explicitly. ```bash pip install 'autoair[all,mcp,postgres]' # Airflow 3.0, 3.2 and newer, or standalone pip install 'autoair[all,postgres]' # Airflow 3.1 ``` ## Checking what is active - `autoair config show --changed` prints the settings that differ from the defaults and where each came from. - The admin panel shows the AutoAir version, the state backend and its schema version, and the operator catalog statistics. See [Admin panel](https://ctolon.github.io/autoair/docs/admin/admin-panel). - Server logs name each optional feature that was switched off because its extra is missing. # Airflow compatibility Source: https://ctolon.github.io/autoair/docs/installation/compatibility AutoAir supports Apache Airflow 3.0 or newer, below 4, on Python 3.10 to 3.14. Airflow 3.4 itself requires Python 3.11 or newer, so on Python 3.10 pip installs Airflow 3.3 at most; AutoAir keeps supporting Python 3.10 with Airflow 3.0 to 3.3. In plugin mode AutoAir runs inside Airflow's Python environment, so every extra has to resolve against the versions Airflow pins. Most extras install everywhere; two gaps come from Airflow itself. ## Extras per Airflow version | Extra | 3.0 | 3.1 | 3.2 | 3.3 | 3.4 (pre-release) | | --- | --- | --- | --- | --- | --- | | `all` (see [Extras](https://ctolon.github.io/autoair/docs/installation/extras#all)) | yes | yes | yes | yes | yes | | `mcp` | yes | **no** | yes | yes | yes | | `sqlite` | yes (SQLAlchemy 1.4) | yes | yes | yes | yes | | `postgres`, `kb-pgvector` | yes (SQLAlchemy 1.4, psycopg2) | yes | yes | yes | yes | | `ai`, `ai-*` | yes | yes | yes | yes | yes | | `otel`, `collab`, `collab-redis`, `standalone`, `format`, `shell`, `yaml`, `git` | yes | yes | yes | yes | yes | CI installs exactly these combinations: the Airflow 3.1 job runs without `mcp` and its MCP tests are skipped with the reason; every other job installs `all,mcp,sqlite,postgres,kb-pgvector`. The versions tested are 3.0.6, 3.1.8, 3.2.2, 3.3.2 and 3.4.0b2. The required test jobs hold Airflow's core dependencies to the versions in Airflow's own constraints file for that release. [`scripts/airflow_constraints.py`](https://github.com/ctolon/autoair/blob/main/scripts/airflow_constraints.py) derives those pins: it keeps the pins of the packages `apache-airflow-core` and `apache-airflow-task-sdk` depend on directly, drops the ones that conflict with AutoAir's requirements together with the pins that depend on a dropped package, and lists both in the job summary. A separate, non-blocking job (`nox -s "fresh(3.x)"`) installs without them, as a plain `pip install` does, and reports an upstream release that breaks Airflow or AutoAir. ## The pins behind it What `apache-airflow-core` requires, and what a fresh resolver picks today: | Airflow | SQLAlchemy | FastAPI | Starlette (resolved) | Pydantic | | --- | --- | --- | --- | --- | | 3.0.6 | `>=1.4.49,<2.0` (1.4.54) | `>=0.115` | 1.x | `>=2.11` | | 3.1.8 | `>=1.4.54` (2.x) | `>=0.116,<0.118` | `<0.49` (0.48) | `>=2.11` | | 3.2.2 | `>=2.0.48` | `>=0.129` | `>=1.0.1` | `>=2.11` | | 3.3.2 | `>=2.0.50` | `>=0.129,<0.137` | `>=1.0.1` | `>=2.11` | | 3.4.0b2 | `>=2.0.50,!=2.1.0` | `>=0.129,<0.137` | `>=1.3.1` | `>=2.12` | - **MCP on Airflow 3.1.** Every FastMCP 4 release requires Starlette 1.0.1 or newer. Airflow 3.1 caps FastAPI below 0.118, and those FastAPI releases cap Starlette below 0.49, so `autoair[mcp]` cannot be resolved next to Airflow 3.1. Older FastMCP lines are not an option: FastMCP 3 and 2 are built on the `mcp` SDK 1.x and `httpx` (AutoAir uses `mcp` 2.x and `httpx2`), and the newest FastMCP 3 releases need Starlette 1.0.1 as well. - **FastAPI on Airflow 3.2.** Airflow 3.2 sets no upper bound on FastAPI, but its execution API does not build with FastAPI 0.156.1 or newer (`PydanticUserError: Model 'TITerminalStatePayload' needs a discriminator field`): every task call fails, including AutoAir's DB-mode test runs. Airflow's 3.2 constraints file pins FastAPI 0.136. When you resolve Airflow 3.2 without that file, add `"fastapi<0.137"` (the cap Airflow 3.3 declares) to the install command. AutoAir itself needs only `fastapi>=0.115` and does not upgrade an installed FastAPI. - **SQLAlchemy on Airflow 3.0.** Airflow 3.0 keeps SQLAlchemy below 2.0. AutoAir's SQL state backends and the pgvector store use only APIs that SQLAlchemy 1.4 and 2.x share, so the `sqlite`, `postgres` and `kb-pgvector` extras accept SQLAlchemy 1.4.49 or newer and use whatever Airflow has installed. ## Airflow 3.4 Airflow 3.4.0 is not final yet; CI tests the newest pre-release (3.4.0b2) in a job that may fail without blocking. What changes for AutoAir on 3.4: | Change in Airflow 3.4 | What AutoAir does | | --- | --- | | Plugin views can be scoped with `applies_to` | `plugin.dag_tab_tags` (for example `["autoair"]`) shows the **AutoAir** tab only on DAGs with one of those tags. Empty (the default) keeps the tab on every DAG page, where a hand-written DAG can be imported. Before 3.4 the setting is ignored and logged. | | `PUT /parseDagFile/{file_token}` answers `403` for a file with no registered DAG unless the caller has the `REPARSE_ALL` view (admins by default) | A deploy to a local folder still succeeds when Airflow refuses the reparse request (verified against 3.4.0b2: `403` for a user without that view, also for a viewer on a registered DAG; a second request while one is queued gets `409`, which AutoAir reports as "reparse already requested"). The deploy message names the refusal and says that Airflow picks the file up on its next bundle refresh, instead of reporting that Airflow could not be contacted. | | An `up_for_retry` task instance carries the upcoming try number; the failed try is `try_number - 1` | The run panel's log view, the AI troubleshooter, the AI fix action and the MCP `get_task_log` tool open the failed try's log, not the upcoming try that has no log yet (`retry_try_is_next` in `GET /api/airflow/capabilities`). | | Task groups that depend on each other in a cycle are deprecated (`TaskGroupCycleDeprecationWarning`); Airflow 3.5 is planned to reject them | Validation reports a `task_group_cycle` warning on every Airflow version, naming the groups and tasks involved. | | Python 3.10 support removed | See the first paragraph: Python 3.10 users stay on Airflow 3.3. | `ui_translations`, also new in 3.4, overrides or adds strings of Airflow's own UI namespaces. Airflow shows plugin menu and tab names as they are configured, without a translation lookup, so it cannot translate the **DAG Builder** and **AutoAir** labels; set `plugin.nav_name` and `plugin.dag_tab_name` instead. AutoAir's own screens follow the language chosen in Airflow's UI. ## MCP next to Airflow 3.1 Without `autoair[mcp]` the plugin keeps working; only MCP is off: - `GET /autoair/api/meta` reports `mcp_enabled: false` and `mcp_unavailable_reason`, which names the Starlette conflict; - the admin panel's **Features** tab shows `mcp` as not installed with the same reason, and it cannot be switched on; - `/autoair/mcp/` is not mounted. To give agents MCP anyway, run the MCP server from a separate environment or container that does not contain Airflow 3.1: - **Standalone MCP server.** Install `autoair[standalone,mcp]` (plus the providers your DAGs use, or point `AUTOAIR_CATALOG_FILE` at a catalog built with `autoair catalog build` in the Airflow image) and run `autoair mcp-serve --http --host 0.0.0.0 --port 8766` with `AUTOAIR_AIRFLOW_URL` set to the Airflow API server. Clients authenticate with their Airflow token. See [MCP](https://ctolon.github.io/autoair/docs/ai/mcp). - **Through the AI service.** Run [`autoair ai-serve`](https://ctolon.github.io/autoair/docs/deployment/ai-service) in its own environment with `autoair[standalone,ai,mcp]` and set `AUTOAIR_AI__SERVICE__PROXY_MCP=true` on the plugin: `/autoair/mcp/` is then proxied to the service, the plugin needs no FastMCP, and clients keep the plugin URL. Either way the MCP tools validate and generate code for the Airflow installed in the MCP server's own environment (the newest version AutoAir knows when none is installed), not for the 3.1 API server. A DAG that uses a feature newer than 3.1 is therefore not flagged by MCP validation; the plugin's validation flags it when the DAG is validated or deployed there, and Airflow 3.1 reports an import error for generated code that uses a symbol it lacks. ## PostgreSQL on Airflow 3.0 SQLAlchemy 1.4 has no psycopg 3 dialect, so on Airflow 3.0 AutoAir drives PostgreSQL through psycopg2. A `postgresql://`, `postgres://` or `postgresql+psycopg://` URL in `database.url`, `kb.pgvector.url` or a `kb.pgvector.conn_id` connection is switched to `postgresql+psycopg2://` automatically. psycopg2 comes with `apache-airflow-providers-postgres`, which an Airflow running on a PostgreSQL metadata database already has; otherwise install `psycopg2-binary`. Without a driver the error reads *cannot load the postgresql driver (...): SQLAlchemy 1.4 (Airflow 3.0) drives PostgreSQL through psycopg2: pip install psycopg2-binary (or apache-airflow-providers-postgres)*. On SQLAlchemy 2 (Airflow 3.1 and newer, and standalone installs) PostgreSQL uses psycopg 3 as before. ## Airflow constraints files Airflow's [constraints files](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) pin every package to the versions an Airflow release was tested with. They are meant for installing Airflow and do not leave room for AutoAir: the 3.0 to 3.2 files pin packages below AutoAir's core requirements (for example `idna` below what `httpx2` needs), the 3.0 and 3.1 files pin OpenTelemetry 1.27 below the `otel` extra, the 3.2 file pins `pydantic-ai-slim` 1.x while `ai` needs 2.43 or newer, and the 3.3 file pins an `openai` SDK older than `ai-openai` needs. Install Airflow with its constraints file if you use one, then install AutoAir in a second `pip install` without `--constraint`, keeping `apache-airflow==` on that command line so pip does not change Airflow. # Local mode Source: https://ctolon.github.io/autoair/docs/installation/local `autoair local` runs the standalone server with a settings preset for a single user on a workstation. It needs no Airflow server: you design, validate, preview and deploy DAG files, then run them in an Airflow of your choice. ## Install and start `autoair local` needs the `standalone` extra (uvicorn) and something to put in the operator palette. The `airflow` extra installs Apache Airflow and the standard provider, which gives you `BashOperator`, `PythonOperator`, `EmptyOperator`, the standard sensors and the `DagBag` import check. ```bash uvx --from 'autoair[standalone,airflow]' autoair local --open # or in a virtual environment python -m venv .venv && . .venv/bin/activate pip install 'autoair[standalone,airflow,format]' autoair local --open ``` Released versions come from PyPI (published by the release workflow on `v*` tags). For an unreleased commit, install the wheel built by `make build` (`pip install 'dist/autoair--py3-none-any.whl[standalone,airflow]'`) or run from a source checkout. From a source checkout, `uv sync` installs the development group (which already contains Airflow and uvicorn); build the frontend once: ```bash uv sync (cd frontend && npm ci && npm run build) uv run autoair local --open ``` The command prints where it writes and a sign-in link with a random access token created for this run: ```text AutoAir (local mode): DAGs in dags Open http://127.0.0.1:8765/ui/#token=autoair-local-... (this per-run link signs you in; keep it to yourself) ``` `--open` opens that link. The UI keeps the token for the browser tab and removes it from the address bar; the token sits in the URL fragment, which the browser never sends to a server. The holder of the token gets `local.roles` (default `["Admin"]`: edit, validate, deploy). Other programs and web pages on your machine cannot use the API without it. After a restart the old link no longer works: open the new one. **Warning:** `uvx autoair local` without extras does not start: the core package has no HTTP server. The command exits with code 1 and says which extras to install: "`autoair local` needs an HTTP server, which comes with the 'standalone' extra (uvicorn). Install it with: pip install 'autoair[standalone,airflow]'", followed by the `uvx --from 'autoair[standalone,airflow]' autoair local --open` form. Without Airflow or providers installed, the palette shows no operators. ## Options | Option | Environment variable | Default | Effect | | --- | --- | --- | --- | | `--dags-folder PATH` | (sets `AUTOAIR_DAGS_FOLDER`) | `./dags` | Folder the spec and generated files are written to. Local mode writes directly into it (no `autoair/` sub-folder). | | `--host HOST` | | `127.0.0.1` | Bind address. Keep it on loopback. | | `--port PORT` | | `8765` | Port. | | `--token TOKEN` | `AUTOAIR_LOCAL__TOKEN` | a random token per run | Require this access token instead of a generated one. The sign-in page shows "This local server requires an access token (autoair local --token)." API clients send `Authorization: Bearer `. | | `--no-token` | | off | Serve without authentication (loopback only). Visitors get `iam.anonymous_roles`, default `["Viewer"]` (read-only); set `AUTOAIR_IAM__ANONYMOUS_ROLES='["Admin"]'` to let them edit and deploy. Requests whose `Host` or `Origin` is not a loopback address (or the host of `server.public_url`) get `403`. | | (config only) | `AUTOAIR_LOCAL__ROLES` | `["Admin"]` | AutoAir roles of the token holder; never `deploy:direct_push`. | | `--open` / `--no-open` | `AUTOAIR_LOCAL__OPEN_BROWSER` | off | Open the UI in the default browser once `/api/health` answers. | | `--config FILE`, `-c FILE` | `AUTOAIR_CONFIG_FILE` | none | Configuration file (YAML, JSON, INI, Python); repeatable. | Binding anything other than a loopback address requires a token you set yourself: ```bash autoair local --host 0.0.0.0 --token "$(openssl rand -hex 16)" ``` Without a token the command exits with code 2: `refusing to serve 0.0.0.0 without --token: local mode has no other authentication`. ## What the preset changes Local mode applies these settings on top of your configuration: | Setting | Value in local mode | Consequence | | --- | --- | --- | | `mode` | `local` | | | `dags_folder` / `subfolder` | `--dags-folder` (default `dags`) / empty | Files go straight into the folder; AutoAir state goes into `/.autoair` | | `features.runs`, `features.test_runs`, `features.airflow` | `false` | No trigger, backfill, clear, test runs, run overlay, task logs, connection/pool/variable pickers, DAG status, or import from Airflow | | `catalog_strategy` | `static` | Operators are found by parsing installed packages, nothing is imported | | `import_check_on_deploy` | `true` only when Airflow is importable | Deploys run the `DagBag` import check if Airflow is installed | | `dagbag_pool` | `false` | One import check at a time, no warm worker pool | | `sso.enabled` | `false` | No Airflow sign-in | Everything else (policy rules, environments, templates, AI, a Git deploy target) is configured as usual. For example, to deploy to a Git repository instead of `./dags`: ```bash export AUTOAIR_STORAGE__MODE=git export AUTOAIR_STORAGE__GIT__PROVIDER=local export AUTOAIR_STORAGE__GIT__REPO=git@github.com:acme/airflow-dags.git export AUTOAIR_STORAGE__GIT__SUBDIR=dags export AUTOAIR_STORAGE__GIT__MODE=push autoair local ``` `provider=local` drives the `git` command-line tool against any remote and pushes the pending branch without opening a pull request. `github`, `gitlab`, `bitbucket` and `azure` use the host's REST API, and `gitea` (Gitea or Forgejo) pushes with the `git` CLI and opens pull requests through the Gitea API; these five open pull requests in `storage.git.mode=pr` (the default) with `AUTOAIR_STORAGE__GIT__TOKEN`. See [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy) and [Git bundles](https://ctolon.github.io/autoair/docs/deployment/git-bundles). ## Adding operators The palette lists what is installed next to AutoAir. Add providers to the same environment: ```bash uvx --from 'autoair[standalone,airflow]' \ --with apache-airflow-providers-amazon \ --with apache-airflow-providers-common-sql \ autoair local ``` Custom operators in a folder are found with `AUTOAIR_CUSTOM_OPERATOR_PATHS='["/path/to/include"]'`, and internal operator packages with `AUTOAIR_EXTRA_OPERATOR_PACKAGES` or `AUTOAIR_EXTRA_OPERATOR_DISTRIBUTIONS`. The catalog is cached on disk (`AUTOAIR_CACHE_DIR`, default `$XDG_CACHE_HOME/autoair`) and rebuilt when the installed packages change. See [Catalog sources](https://ctolon.github.io/autoair/docs/extending/catalog-sources). ## Running the DAGs Local mode does not talk to Airflow. To run what you built, point an Airflow at the same folder (for example `AIRFLOW__CORE__DAGS_FOLDER=$PWD/dags airflow standalone`) or push to the repository your Airflow reads. When you need runs, logs and pickers inside AutoAir, switch to the [plugin](https://ctolon.github.io/autoair/docs/installation/plugin) or [standalone](https://ctolon.github.io/autoair/docs/installation/standalone) mode; specs written in local mode are read unchanged. ## Security notes - Anyone holding the token has `local.roles` (default Admin). Do not share the sign-in link; with `--token` keep the token secret. - With `--no-token`, anyone who can reach the loopback port has `iam.anonymous_roles` (default Viewer). Other users of a shared machine can reach it. - Python task bodies and inline functions are imported by the `DagBag` check in a subprocess on your machine. Do not deploy specs from untrusted sources without reviewing them. ## Related - [Quick start](https://ctolon.github.io/autoair/docs/getting-started/quick-start) - [Build your first DAG](https://ctolon.github.io/autoair/docs/getting-started/first-dag) - [Extras](https://ctolon.github.io/autoair/docs/installation/extras) # Airflow plugin Source: https://ctolon.github.io/autoair/docs/installation/plugin In plugin mode AutoAir runs inside the Airflow **API server**. Users open it from the Airflow UI with their Airflow session; there is no second service and no second sign-in. ## How the plugin is loaded The package declares an Airflow plugin entry point: ```toml [project.entry-points."airflow.plugins"] autoair = "autoair.plugin:AutoAirPlugin" ``` Airflow discovers it as soon as `autoair` is installed in the Python environment of the API server. No file has to be copied into `plugins/`. The plugin contributes: - `fastapi_apps`: the AutoAir application mounted under `/autoair` (API, UI assets, MCP, collaboration WebSocket); - `react_apps` (Airflow 3.1 and newer only): the navigation entry and the DAG page tab. The plugin never takes the API server down. If the configuration is invalid it logs `AutoAir: invalid configuration` with the reason and registers nothing; if the application fails to build it logs `AutoAir plugin failed to initialise`. In both cases Airflow keeps running without AutoAir. ## Install Install AutoAir in the same environment as Airflow, together with the providers your DAGs use. The providers also feed the operator palette. ### Into a derived image This is the recommended way and what `deploy/docker/Dockerfile` does: ```dockerfile ARG AIRFLOW_VERSION=3.3.2 FROM apache/airflow:${AIRFLOW_VERSION} COPY dist/autoair-*.whl /tmp/ RUN pip install --no-cache-dir "$(ls /tmp/autoair-*.whl)[git,mcp,ai,ai-anthropic,format]" \ apache-airflow-providers-git apache-airflow-providers-common-messaging \ && rm /tmp/autoair-*.whl # Precompute the operator catalog for the providers baked into this image, so every # api-server replica starts warm and serves an identical palette. RUN autoair catalog build --output /opt/airflow/autoair-catalog.json ENV AUTOAIR_CATALOG_FILE=/opt/airflow/autoair-catalog.json \ AUTOAIR_CACHE_DIR=/tmp/autoair-cache ``` On Airflow 3.1 leave `mcp` out of the extras: it cannot be installed next to Airflow 3.1 ([Airflow compatibility](https://ctolon.github.io/autoair/docs/installation/compatibility)). Build the wheel first (`make build` produces `dist/autoair-*.whl` with the frontend included), then: ```bash docker build -f deploy/docker/Dockerfile --build-arg AIRFLOW_VERSION=3.3.2 \ -t registry.example.com/my-airflow-autoair:3.3.2 . ``` Use this image for every Airflow component so that the API server, dag-processor and workers have the same providers. For Kubernetes, continue with [Helm](https://ctolon.github.io/autoair/docs/installation/helm). ### Into an existing virtual environment ```bash pip install 'autoair[format,git,mcp]' "apache-airflow==$(airflow version)" airflow plugins # the list now contains "autoair" ``` Restart the API server (`airflow api-server`, or `airflow standalone` in development). Pinning `apache-airflow` to the installed version prevents pip from upgrading Airflow while resolving AutoAir's requirements. Do not install the `airflow` extra (Airflow is already there) or `standalone` (the API server serves AutoAir). Choose the others with [Extras](https://ctolon.github.io/autoair/docs/installation/extras). ### Which Airflow components need AutoAir | Component | Needs `autoair` installed | Why | | --- | --- | --- | | API server | Yes | Runs the plugin | | dag-processor, workers, triggerer | Only for DAGs in **loader** output mode | A loader stub does `from autoair.loader import load_dag`. The default **Python** output imports only Airflow and providers. | | Scheduler | No | | Using one image everywhere avoids the question. ## UI integration ### Airflow 3.1 and newer Airflow 3.1 added `react_apps` to plugins. AutoAir uses it to render its UI natively inside Airflow's: | Entry | Default | Setting | | --- | --- | --- | | Navigation entry | **DAG Builder** under **Browse**, at `/plugin/dag-builder` | `plugin.nav_name` (`AUTOAIR_PLUGIN__NAV_NAME`), `plugin.nav_category` (`AUTOAIR_PLUGIN__NAV_CATEGORY`: `browse`, `admin`, `docs` or `user`; any other value puts the entry in Airflow's top-level navigation) | | DAG page tab | **AutoAir** on every DAG page; opens that DAG, or offers to import it when it was not built with AutoAir | `plugin.dag_tab_name` (`AUTOAIR_PLUGIN__DAG_TAB_NAME`), `plugin.dag_tab` (`AUTOAIR_PLUGIN__DAG_TAB=false` hides it), `plugin.dag_tab_tags` (Airflow 3.4 and newer: only on DAGs with one of these tags) | On Airflow 3.4 and newer the tab can be limited to the DAGs you manage with AutoAir. New DAGs created in the builder carry the tag `autoair` (**DAG settings → Tags**); give imported or older DAGs the same tag and set: ```bash AUTOAIR_PLUGIN__DAG_TAB_TAGS='["autoair"]' ``` Airflow's UI then offers the tab only on DAGs that carry at least one of the tags. Other DAGs no longer show it, so they cannot be imported from their DAG page; open **DAG Builder** and import them from there. The scoping only hides the tab: the builder route stays reachable, and AutoAir's own permission checks still apply. Before Airflow 3.4 the setting is ignored (the API server logs that) and the tab is on every DAG page. The bundle is served from `/autoair/static/main.js`. The full-page UI at `/autoair/ui/` is also available. ### Airflow 3.0 Airflow 3.0 has no `react_apps`, so there is no menu entry or DAG tab. Open the builder as a full page at: ```text https://airflow.example.com/autoair/ui/ ``` Everything else (API, validation, deploy, runs) behaves the same. ## URLs All paths are relative to Airflow's base URL. | Path | Purpose | Authentication | | --- | --- | --- | | `/autoair/ui/` | Full-page UI | Airflow session | | `/autoair/api/...` | REST API used by the UI (reference: [REST API](https://ctolon.github.io/autoair/api)) | Airflow session or bearer token | | `/autoair/api/health` | Liveness | None | | `/autoair/api/ready` | Readiness (checks the deploy target; Airflow too with `server.ready_check_airflow`) | None | | `/autoair/api/metrics` | Prometheus metrics (`server.metrics_enabled`) | None | | `/autoair/mcp/` | MCP endpoint (streamable HTTP), with the `mcp` extra and `features.mcp` | Airflow token | | `/autoair/collab/` | Collaboration WebSocket, with the `collab` extra and `features.collab` | Short-lived ticket | The health, ready and metrics routes are unauthenticated so that probes and Prometheus can reach them; restrict them at the network level if needed. ## Authentication and permissions In plugin mode Airflow's auth manager authenticates every request; AutoAir reads the user from the Airflow session and calls Airflow's REST API with the caller's token, so per-DAG permissions are Airflow's. AutoAir's own roles (Viewer, Author, Reviewer, Deployer, CodeDeployer, Operator, TemplateManager, Admin) are derived according to `iam.strategies`: - `airflow_permissions` (default): from the user's Airflow DAG and configuration permissions. - `role_mapping`: from Airflow role names, for example an Airflow role `AutoAir Deployer`. - `custom_view`: from AutoAir-specific permissions created in the auth manager. With the FAB auth manager (`apache-airflow-providers-fab`), the `custom_view` permissions must exist in FAB's database. Run once, where Airflow runs: ```bash autoair iam sync-fab --create-roles ``` This creates the AutoAir permissions and, with `--create-roles`, roles named `AutoAir ` (prefix `iam.role_prefix`) holding them. With Keycloak, the client's authorization decision strategy must be **Affirmative**. Details for each auth manager: [IAM](https://ctolon.github.io/autoair/docs/admin/iam). ## Configuration The plugin reads the same settings as every other mode: - environment variables `AUTOAIR_*` on the API server (in Helm, the chart's `env` and `secret` lists); - configuration files named by `AUTOAIR_CONFIG_FILE`, or by `config_file` in an `[autoair]` section of `airflow.cfg` (`AIRFLOW__AUTOAIR__CONFIG_FILE`). Defaults that come from Airflow in plugin mode: | Setting | Default in plugin mode | | --- | --- | | `dags_folder` | Airflow's `[core] dags_folder` | | `subfolder` | `autoair`, so files are written to `/autoair/` | | `airflow_url` | Airflow's `[api] base_url` | | `collab_secret` | derived from Airflow's `[api_auth] jwt_secret` | A minimal production configuration that deploys through Git instead of the local folder: ```bash AUTOAIR_STORAGE__MODE=git AUTOAIR_STORAGE__GIT__USE_AIRFLOW_BUNDLE=true # repo, branch, subdir, credentials from the GitDagBundle AUTOAIR_STORAGE__GIT__SUBDIR=dags/autoair AUTOAIR_STORAGE__GIT__MODE=pr AUTOAIR_EXPOSE_CONFIG=non-sensitive-only ``` All options: [Configuration](https://ctolon.github.io/autoair/docs/admin/configuration) and the [configuration reference](https://ctolon.github.io/autoair/docs/reference/configuration). ## Several API server replicas Each API server replica runs its own copy of AutoAir. Replicas need shared state for templates, audit, proposals and the deploy target chosen in the UI, and the DAG files must reach every dag-processor. Use a Git or object-storage deploy target and PostgreSQL state (`AUTOAIR_DATABASE__BACKEND=postgresql`), and run `autoair db upgrade` once before the replicas start. See [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling), [Database](https://ctolon.github.io/autoair/docs/admin/database) and the [Kubernetes deployment](https://ctolon.github.io/autoair/docs/deployment/kubernetes). ## Verify the installation 1. `airflow plugins` lists `autoair`. 2. `curl -s https://airflow.example.com/autoair/api/health` answers `200`. 3. In the Airflow UI (3.1+), **Browse** contains **DAG Builder**; on 3.0, `/autoair/ui/` loads the builder. 4. The palette lists the providers installed in the image. If it is empty or stale, see [Troubleshooting](https://ctolon.github.io/autoair/docs/troubleshooting/faq). # Standalone server Source: https://ctolon.github.io/autoair/docs/installation/standalone `autoair serve` runs AutoAir as a separate web service. It talks to Airflow only through Airflow's REST API, so it can be installed, upgraded and scaled independently, and it works with an Airflow you cannot add packages to. ```mermaid flowchart LR browser[Browser] -->|"/ui, /api (Airflow token)"| autoair[autoair serve] autoair -->|"REST API v2, caller's token"| airflow[Airflow API server] autoair -->|"spec + DAG file"| target[(Deploy target: shared folder, object storage or Git)] target --> processor[Airflow dag-processor] ``` ## Install ```bash python -m venv /opt/autoair && . /opt/autoair/bin/activate pip install 'autoair[standalone,format,git,mcp,collab]' pip install apache-airflow-providers-standard apache-airflow-providers-amazon ``` - `standalone` provides uvicorn; without it `autoair serve` cannot start. - The operator palette is built from the packages installed **next to AutoAir**. Install the same providers as your Airflow image, or the palette and validation will not match what Airflow can import. - The `DagBag` import check (Validate in Airflow, and on every deploy) needs `apache-airflow` in this environment. Install the same Airflow version as the cluster (`pip install "apache-airflow==3.3.2"`) or set `import_check_on_deploy=false` and rely on static validation. - Add `postgres` when you run more than one replica. See [Extras](https://ctolon.github.io/autoair/docs/installation/extras) for every optional group. ## Run ```bash autoair serve \ --airflow-url https://airflow.example.com \ --dags-folder /shared/airflow/dags \ --host 0.0.0.0 --port 8765 ``` The command prints `AutoAir: writing DAGs to /shared/airflow/dags/autoair; Airflow at https://airflow.example.com` and serves: | Path | Purpose | | --- | --- | | `/ui/` | The builder (`/` redirects here) | | `/api/...` | REST API ([reference](https://ctolon.github.io/autoair/api)) | | `/api/health`, `/api/ready`, `/api/metrics` | Liveness, readiness, Prometheus (unauthenticated) | | `/mcp/` | MCP endpoint (with the `mcp` extra) | | `/collab/` | Collaboration WebSocket (with the `collab` extra) | ### Command-line options | Option | Setting | Default | | --- | --- | --- | | `--host` | `server.host` (`AUTOAIR_SERVER__HOST`) | `127.0.0.1` | | `--port` | `server.port` (`AUTOAIR_SERVER__PORT`) | `8765` | | `--airflow-url` | `airflow_url` (`AUTOAIR_AIRFLOW_URL`) | Airflow's `[api] base_url` when Airflow is installed, else `http://localhost:8080` | | `--dags-folder` | `dags_folder` (`AUTOAIR_DAGS_FOLDER`) | Airflow's `[core] dags_folder`, else `./dags` | | `--workers` | `server.workers` | `1` | | `--ssl-certfile`, `--ssl-keyfile` | `server.ssl_certfile`, `server.ssl_keyfile` | unset (HTTP) | | `--ssl-ca-certs`, `--ssl-cert-reqs` | `server.ssl_ca_certs`, `server.ssl_cert_reqs` | unset, `none` (mutual TLS) | | `--forwarded-allow-ips` | `server.forwarded_allow_ips` (comma separated) | `127.0.0.1` | | `--no-auth` | | off | | `--config`, `-c` | `AUTOAIR_CONFIG_FILE` | none | Command-line options win over environment variables, which win over configuration files. Every other option is a setting: [Configuration](https://ctolon.github.io/autoair/docs/admin/configuration). ## Connecting to Airflow `airflow_url` is the base URL of Airflow's API server **as AutoAir reaches it** (for example the in-cluster service). AutoAir calls `/api/v2/...` for token checks, pickers, DAG status, runs, logs and XComs, and `/auth/token` for password sign-in. If browsers reach Airflow under another URL, set `AUTOAIR_SSO__AIRFLOW_PUBLIC_URL` (used for "Sign in with Airflow" and links to Airflow), and `AUTOAIR_SERVER__PUBLIC_URL` to AutoAir's own public URL (return URL after sign-in, links in commits and pull requests). Airflow's TLS certificate is verified against the default trust store. For a private CA set `AUTOAIR_OUTBOUND__CA_BUNDLE=/etc/ssl/certs/corp-root.pem`; proxies are `AUTOAIR_OUTBOUND__HTTPS_PROXY` and `AUTOAIR_OUTBOUND__NO_PROXY`. Each Airflow call times out after `airflow_timeout` (15 s). ## Authentication Every API request must carry an Airflow access token. AutoAir verifies it by calling `GET /api/v2/dags?limit=1` with it, caches the result for `auth_cache_ttl` seconds (60; never past the token's expiry), and forwards the same token on every call to Airflow. Airflow's permissions therefore apply to everything AutoAir does on the user's behalf; AutoAir's own roles are layered on top ([IAM](https://ctolon.github.io/autoair/docs/admin/iam)). The sign-in page offers up to three ways to obtain a token: | Method | When it is offered | Setting | | --- | --- | --- | | **Sign in with Airflow** | The browser can send Airflow's session cookie (`_token`) to AutoAir: AutoAir on the same host as Airflow (any port), under the same proxy, or on a sibling sub-domain when the proxy widens the cookie's domain | `sso.enabled` (default `true`), `sso.cookie_name` | | Username and password | Always in standalone mode, unless disabled. AutoAir forwards the credentials once to Airflow's `/auth/token` and never stores them. Failed attempts are rate limited (5 per 300 s per IP and per user, then HTTP 429). | `iam.login_enabled`, `iam.login_max_failures`, `iam.login_window_seconds` | | **Access token** | Always. Paste a token issued by Airflow. | | To get a token for scripts or for pasting: ```bash curl -s -X POST https://airflow.example.com/auth/token \ -H 'Content-Type: application/json' \ -d '{"username": "alice", "password": "..."}' # {"access_token": "eyJ...", ...} ``` API clients send it as `Authorization: Bearer `. Tokens expire with Airflow's `[api_auth] jwt_expiration_time`. The session-sharing setups, CSRF protection for cookie sessions and logout propagation are described in [SSO](https://ctolon.github.io/autoair/docs/admin/sso). **Danger:** `autoair serve --no-auth` accepts requests without a token and gives every caller the roles in `iam.anonymous_roles` (default `["Viewer"]`; set `AUTOAIR_IAM__ANONYMOUS_ROLES='["Admin"]'` to edit and deploy). It serves only on a loopback address (`127.0.0.1`, `::1`) and answers only requests whose `Host` and `Origin` are loopback or `server.public_url`, so a web page cannot reach it through DNS rebinding. Calls to Airflow are made without credentials. Use it only on a development machine; in a container, check health from inside it. ## Getting the DAG files to Airflow AutoAir writes the spec and the generated file to its deploy target. In standalone mode that target must be something Airflow's dag-processor also reads: | Setup | Configuration | | --- | --- | | Shared volume mounted by AutoAir and the dag-processor | `--dags-folder` pointing at the mounted DAG folder (files go to `/autoair/`) | | Git repository read by a `GitDagBundle` or git-sync | `AUTOAIR_STORAGE__MODE=git`, `AUTOAIR_STORAGE__GIT__PROVIDER`, `AUTOAIR_STORAGE__GIT__REPO` and credentials (`AUTOAIR_STORAGE__GIT__CONN_ID` or `AUTOAIR_STORAGE__GIT__TOKEN`); all options: [`storage.git`](https://ctolon.github.io/autoair/docs/reference/configuration#storagegit) | | Object storage read by an S3/GCS bundle | `AUTOAIR_STORAGE__MODE=object`, `AUTOAIR_STORAGE__OBJECT_URL` | Administrators can also pick the target in the UI (**Versions & Git → Deploy target…**). See [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy), [Git bundles](https://ctolon.github.io/autoair/docs/deployment/git-bundles) and [Object storage](https://ctolon.github.io/autoair/docs/deployment/object-storage). ## TLS, proxies and headers Either terminate TLS in AutoAir: ```bash autoair serve --ssl-certfile /etc/autoair/tls.pem --ssl-keyfile /etc/autoair/tls.key # encrypted key: AUTOAIR_SERVER__SSL_KEYFILE_PASSWORD (environment only) ``` or behind a reverse proxy, trusting its forwarded headers: ```bash AUTOAIR_SERVER__PUBLIC_URL=https://autoair.example.com AUTOAIR_SERVER__FORWARDED_ALLOW_IPS='["10.0.0.10"]' AUTOAIR_SERVER__HSTS_SECONDS=31536000 # only when every client uses HTTPS ``` Standalone mode sends a Content-Security-Policy and related security headers by default (`server.security_headers`). Details: [TLS and headers](https://ctolon.github.io/autoair/docs/admin/tls-and-headers) and [Reverse proxy](https://ctolon.github.io/autoair/docs/deployment/standalone-proxy). ## Workers and replicas `--workers N` starts several worker processes. With more than one worker or replica: - set `AUTOAIR_COLLAB_SECRET` to the same value everywhere (collaboration tickets are HMAC-signed with it; otherwise it is derived from Airflow's `[api_auth] jwt_secret`, which a standalone server usually does not know); - move state to PostgreSQL (`AUTOAIR_DATABASE__BACKEND=postgresql`) and run `autoair db upgrade` once before starting, with `AUTOAIR_DATABASE__MIGRATE_ON_START=false`; - use a Git or object-storage deploy target when replicas run on different hosts. See [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling) and [PostgreSQL](https://ctolon.github.io/autoair/docs/deployment/postgres). ## Container example `deploy/docker/Dockerfile.demo` builds the frontend and the wheel inside Docker and installs them, with `standalone,format,git,collab,postgres`, into the official Airflow image (which also gives the import check a real Airflow): ```bash docker build -f deploy/docker/Dockerfile.demo \ --build-arg AIRFLOW_VERSION=3.3.2 -t autoair:local . docker run -d --name autoair -p 8765:8765 \ -e AUTOAIR_AIRFLOW_URL=http://airflow-api-server:8080 \ -e AUTOAIR_DAGS_FOLDER=/opt/airflow/dags \ -e AUTOAIR_COLLAB_SECRET="$(openssl rand -hex 32)" \ -e AUTOAIR_LOGGING__FORMAT=json \ -v airflow-dags:/opt/airflow/dags \ --entrypoint autoair \ autoair:local serve --host 0.0.0.0 --port 8765 ``` Health check for the orchestrator: ```bash python -c "import urllib.request; urllib.request.urlopen('http://localhost:8765/api/health', timeout=5)" ``` The Compose demo runs two such replicas against PostgreSQL: [Docker Compose](https://ctolon.github.io/autoair/docs/installation/docker-compose). ## Verify 1. `curl -s http://localhost:8765/api/health` answers `200`. 2. `curl -s http://localhost:8765/api/ready` answers `200` when the deploy target is reachable (`server.ready_check_storage`; add `server.ready_check_airflow=true` to include Airflow). 3. Sign in at `/ui/`. "Cannot reach the AutoAir API" or `502 cannot reach Airflow` means `airflow_url` is wrong or blocked: see [Troubleshooting](https://ctolon.github.io/autoair/docs/troubleshooting/faq). Production topology, Kubernetes manifests and ingress: [Deployment](https://ctolon.github.io/autoair/docs/deployment/overview). # Docker Compose Source: https://ctolon.github.io/autoair/docs/installation/docker-compose `deploy/compose/` contains two Compose files for workstations: | File | What it runs | Use it to | | --- | --- | --- | | `docker-compose.demo.yml` | PostgreSQL, Airflow 3 split into API server, scheduler, dag-processor and triggerer with the AutoAir plugin, two standalone AutoAir replicas, seed jobs, and seven opt-in profiles | Evaluate plugin and standalone mode together, shared PostgreSQL state, collaboration across replicas, a Git deploy target with pull requests, the separate AI service, TLS and tracing | | `docker-compose.yaml` | One `airflow standalone` container with the plugin | The smallest plugin setup, built from a prebuilt wheel | **Danger: Workstation only** Every Airflow visitor is an administrator (SimpleAuthManager `all_admins`), every secret has a public default, and the demo ports bind to `127.0.0.1`. Before exposing anything, use a real auth manager, unique secrets, TLS and the IAM role mapping. For production see [Helm](https://ctolon.github.io/autoair/docs/installation/helm) and [Deployment](https://ctolon.github.io/autoair/docs/deployment/overview). The default start runs no AI provider, so it cannot incur provider charges. ## Prerequisites - Docker Engine with Compose v2 (`docker compose`). The demo uses `depends_on` with `required: false`, `env_file` with `required: false` and `--wait`, so use a current Compose release. - Free loopback ports 8080, 8765, 8766 and 5433, or the overrides in [Ports and credentials](#ports-and-credentials). - A clone of the repository. The demo builds everything from the working tree; no local Python or Node.js is required. ## Quick start ```bash git clone https://github.com/ctolon/autoair.git cd autoair/deploy/compose cp .env.example .env # optional: every value has a default docker compose -f docker-compose.demo.yml up --build -d --wait ``` The first build compiles both editor bundles and a wheel from the working tree and takes a few minutes; later starts take about a minute. When `--wait` returns: | Open | URL | | --- | --- | | Airflow, then **Browse → DAG Builder** (the plugin) | `http://localhost:8080` | | Plugin, direct | `http://localhost:8080/plugin/dag-builder` | | Standalone replica A | `http://localhost:8765/ui/` | | Standalone replica B | `http://localhost:8766/ui/` | | AutoAir API docs (replica A) | `http://localhost:8765/api/docs` | Two demo DAGs, `demo_hello` and `demo_pipeline`, are deployed through replica A's API on first start (`seed-dags`), and the signed template marketplace from `examples/marketplace` is available in the template gallery ([Marketplace](https://ctolon.github.io/autoair/docs/user-guide/marketplace)). Commands on this page assume the working directory is `deploy/compose`. From the repository root, add `-f deploy/compose/docker-compose.demo.yml` and prefix `--env-file` paths with `deploy/compose/`. ## Services ### Core services These start on every `up`: | Service | Image | Role | Health check | | --- | --- | --- | --- | | `postgres` | `postgres:17-alpine` | Two roles and databases created by `initdb/001-databases.sh`: `airflow` (Airflow metadata) and `autoair` (AutoAir state) | `pg_isready` against both databases | | `airflow-migrate` | demo image | One-shot `airflow db migrate` | exits 0 | | `autoair-migrate` | demo image | One-shot `autoair db upgrade`. All other AutoAir processes run with `database.migrate_on_start=false` | exits 0 | | `marketplace-seed` | demo image | One-shot: signs `examples/marketplace` with a generated Ed25519 key into the `marketplace` volume | exits 0 | | `airflow-apiserver` | demo image | Airflow UI and REST API, the execution API, and the AutoAir plugin (`/autoair/...`) | `/api/v2/monitor/health` | | `airflow-scheduler` | demo image | Scheduler with the `LocalExecutor` (runs the tasks) | scheduler health server `:8974/health` | | `airflow-dag-processor` | demo image | Parses the DAG bundles | `airflow jobs check --job-type DagProcessorJob` | | `airflow-triggerer` | demo image | Deferrable operators | `airflow jobs check --job-type TriggererJob` | | `autoair-a`, `autoair-b` | demo image | `autoair serve` replicas sharing the `autoair` database and the `dags` volume | `/api/health` (readiness: `/api/ready`, with `server.ready_check_airflow=true`) | | `seed-dags` | demo image | One-shot: deploys `seed/specs/*.json` through replica A, skipping DAGs that already exist. Disable with `DEMO_SEED_DAGS=false` | exits 0 | Start order is enforced with `depends_on` conditions: PostgreSQL healthy, then both migrations complete, then Airflow, then the replicas (after `marketplace-seed` and a healthy `airflow-apiserver`), then the seed. With profile `git`, `git-bootstrap` completes before any Airflow component or replica starts, because the `GitDagBundle` and AutoAir's Git target read the Airflow connection at startup. With profile `redis`, the replicas also wait for a healthy `redis`. ```mermaid flowchart TB pg[(postgres
databases airflow, autoair)] am[airflow-migrate] aam[autoair-migrate] ms[marketplace-seed] api[airflow-apiserver
+ AutoAir plugin] sch[airflow-scheduler] dp[airflow-dag-processor] tr[airflow-triggerer] a[autoair-a] b[autoair-b] seed[seed-dags] dags[(volume: dags)] pg --> am pg --> aam am --> api aam --> api api --> sch am --> dp am --> tr aam --> a aam --> b ms --> a ms --> b api --> a api --> b a --> seed dp --- dags a --- dags b --- dags api --- dags ``` ### The demo image Every Airflow component, replica and AutoAir job uses one image, `${DEMO_IMAGE:-autoair-demo:local}`, built from `deploy/docker/Dockerfile.demo`: `apache/airflow:${DEMO_AIRFLOW_VERSION:-3.3.2}` plus the AutoAir wheel from the working tree with the `standalone`, `format`, `git`, `collab`, `collab-redis`, `postgres`, `otel` and `yaml` extras, and the Airflow `standard` and `git` providers. Set `DEMO_AIRFLOW_VERSION` in `.env` to build against another Airflow 3.x release. The replicas start through `scripts/autoair-entrypoint.sh`, which reads `/opt/airflow/marketplace/sources.json` (written by `marketplace-seed`) into `AUTOAIR_MARKETPLACE_SOURCES` unless that variable is already set, then runs `autoair serve`. ### Shared AutoAir settings All AutoAir processes (the plugin inside `airflow-apiserver`, the replicas, `autoair-migrate` and `autoair-ai`) share the `x-autoair-env` block of the Compose file. The values that shape the demo: ```yaml AUTOAIR_CONFIG_FILE: /etc/autoair/autoair.yaml AUTOAIR_AIRFLOW_URL: http://airflow-apiserver:8080 AUTOAIR_DAGS_FOLDER: /opt/airflow/dags AUTOAIR_DATABASE__BACKEND: postgresql AUTOAIR_DATABASE__URL: postgresql+psycopg://autoair@postgres:5432/autoair AUTOAIR_DATABASE__PASSWORD: ${DEMO_AUTOAIR_DB_PASSWORD:-autoair-demo-only} AUTOAIR_DATABASE__MIGRATE_ON_START: "false" # autoair-migrate migrates once AUTOAIR_COLLAB__SYNC: ${DEMO_COLLAB_SYNC:-database} # PostgreSQL LISTEN/NOTIFY, or redis AUTOAIR_COLLAB_SECRET: ${DEMO_COLLAB_SECRET:-local-demo-collab-secret} AUTOAIR_IAM__LOGIN_ENABLED: "true" AUTOAIR_SSO__AIRFLOW_PUBLIC_URL: http://localhost:${DEMO_AIRFLOW_PORT:-8080} AUTOAIR_STORAGE__MODE: ${DEMO_STORAGE_MODE:-local} # git with env/git.env AUTOAIR_STORAGE__GIT__PROVIDER: gitea AUTOAIR_STORAGE__GIT__MODE: ${DEMO_GIT_MODE:-pr} AUTOAIR_STORAGE__GIT__USE_AIRFLOW_BUNDLE: "true" AUTOAIR_STORAGE__GIT__BUNDLE_NAME: gitea AUTOAIR_STORAGE__GIT__SUBDIR: autoair AUTOAIR_AI__ENABLED: ${DEMO_AI_ENABLED:-false} AUTOAIR_AI__SERVICE_URL: ${DEMO_AI_SERVICE_URL:-} AUTOAIR_AI__SERVICE__SHARED_SECRET: ${DEMO_AI_SERVICE_SECRET:-local-demo-ai-hmac-secret} AUTOAIR_OTEL_ENABLED: ${DEMO_OTEL_ENABLED:-false} OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318 AUTOAIR_FEATURES__IDE_LANGUAGE_SERVER: "false" ``` The replicas add `AUTOAIR_SERVER__HOST=0.0.0.0`, `AUTOAIR_SERVER__PORT=8765`, `AUTOAIR_SERVER__FORWARDED_ALLOW_IPS='["*"]'` (inside the Compose network only the load balancers forward to them) and `AUTOAIR_SERVER__READY_CHECK_AIRFLOW=true`. Language servers are disabled; see [Extras](https://ctolon.github.io/autoair/docs/installation/extras) to enable them in your own image. AutoAir's state (templates, audit, proposals, collaboration rooms) lives in the `autoair` database. DAG files are on the shared `dags` volume. Shared PostgreSQL state does not distribute DAG files; in a real multi-host deployment use a Git or object-storage deploy target ([Scaling](https://ctolon.github.io/autoair/docs/admin/scaling), [Git and DAG bundles](https://ctolon.github.io/autoair/docs/deployment/git-bundles)). ### Volumes | Volume | Content | | --- | --- | | `postgres` | Both databases | | `dags` | Airflow's `dags_folder`; the local deploy target writes to `dags/autoair/` | | `logs` | Task logs (the scheduler writes, the API server reads) | | `marketplace` | Signed example marketplace, its key pair and `sources.json` | | `gitea-data`, `gitea-config` | Profile `git` | | `certs` | Profile `tls`: demo CA and server certificate | | `ollama` | Profile `ollama`: downloaded models | ## Profiles Profiles add services. When a profile also needs the core services configured differently, a preset in `env/` sets the matching `DEMO_*` values. Pass the preset as a second `--env-file` after `.env`. | Profile | Adds | Preset | Purpose | | --- | --- | --- | --- | | `lb` | `lb` (nginx) | none | Round-robin over both replicas, WebSockets and SSE included, no sticky sessions | | `redis` | `redis` | `env/redis.env` | Collaboration fan-out through Redis pub/sub instead of PostgreSQL `LISTEN/NOTIFY` | | `ai` | `autoair-ai` | `env/ai.env` (hosted provider) or `env/ollama.env` | `autoair ai-serve` as a separate service. Replicas and the plugin proxy `/api/ai/*` to it, signing the caller with the HMAC secret | | `ollama` | `ollama`, `ollama-pull` | `env/ollama.env` | Local model server and a one-shot model pull. No provider charges | | `git` | `gitea`, `gitea-init`, `git-bootstrap` | `env/git.env` | Gitea as the deploy target in pull-request mode, read back by an Airflow `GitDagBundle` | | `tls` | `certs-init`, `tls-proxy`, `autoair-tls` | none | Self-signed CA, an HTTPS proxy, and a replica that serves HTTPS itself and reaches Airflow over verified HTTPS | | `observability` | `otel-collector`, `jaeger` | `env/observability.env` | OpenTelemetry traces from every AutoAir process, viewed in Jaeger | ```bash docker compose -f docker-compose.demo.yml --profile lb up -d --wait # profile + preset docker compose -f docker-compose.demo.yml --env-file .env --env-file env/git.env \ --profile git up -d --wait # several at once docker compose -f docker-compose.demo.yml --env-file .env \ --env-file env/git.env --env-file env/redis.env --env-file env/observability.env \ --profile git --profile redis --profile lb --profile observability up -d --wait ``` `.env` must exist when it is named with `--env-file`: run `cp .env.example .env` first (an all-comments `.env` is fine). Keep passing the same `--env-file` and `--profile` flags to later `ps`, `logs` and `down` commands, or set `COMPOSE_PROFILES=git,redis` in `.env`. The AI service and the tracing exporter are inert without their presets. Setting `DEMO_AI_SERVICE_URL` without starting profile `ai` makes `/api/ai/*` answer with proxy errors; everything else keeps working. What each preset sets: | Preset | Values | | --- | --- | | `env/redis.env` | `DEMO_COLLAB_SYNC=redis`, `DEMO_COLLAB_REDIS_URL=redis://:${DEMO_REDIS_PASSWORD:-redis-demo-only}@redis:6379/0` | | `env/ai.env` | `DEMO_AI_ENABLED=true`, `DEMO_AI_SERVICE_URL=http://autoair-ai:8790`. Put `DEMO_AI_PROVIDER` and `DEMO_AI_API_KEY` in `.env` | | `env/ollama.env` | `DEMO_AI_ENABLED=true`, `DEMO_AI_SERVICE_URL=http://autoair-ai:8790`, `DEMO_AI_PROVIDER=ollama`, `DEMO_AI_BASE_URL=http://ollama:11434`, `DEMO_AI_MODEL=${DEMO_OLLAMA_MODEL:-qwen2.5:1.5b}`, empty `DEMO_AI_API_KEY` | | `env/git.env` | `DEMO_STORAGE_MODE=git`, `DEMO_GIT_MODE=pr`, and a `DEMO_DAG_BUNDLE_CONFIG_LIST` with the local `dags-folder` bundle plus a `GitDagBundle` named `gitea` (`git_conn_id=gitea_dags`, `tracking_ref=main`, `refresh_interval=30`) | | `env/observability.env` | `DEMO_OTEL_ENABLED=true` | ### `lb`: load balancer ```bash docker compose -f docker-compose.demo.yml --profile lb up -d --wait curl -sI localhost:8780/api/health | grep X-Upstream ``` `lb` is `nginx:1.28-alpine` with `nginx/lb.conf`, listening on `DEMO_LB_PORT` (8780). It starts after both replicas are healthy. Use `http://localhost:8780/ui/`. No sticky sessions are configured; each response carries `X-Upstream` with the replica that answered. nginx balances per worker process, so a series of one-off `curl` calls can land on the same replica, while requests over one keep-alive connection alternate. This is the setup the [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling) page describes: any replica can serve any request because state and collaboration fan-out are shared. ### `redis`: collaboration over Redis ```bash docker compose -f docker-compose.demo.yml --env-file .env --env-file env/redis.env \ --profile redis up -d --wait ``` `redis` is `redis:7.4-alpine` with `--requirepass ${DEMO_REDIS_PASSWORD:-redis-demo-only}` and persistence off. The preset switches `collab.sync` to `redis` and points `collab.redis_url` at it, so collaboration rooms fan out through Redis pub/sub instead of PostgreSQL `LISTEN/NOTIFY` ([Collaboration](https://ctolon.github.io/autoair/docs/user-guide/collaboration)). ### `ai` and `ollama`: separate AI service Local model, no provider charges: ```bash docker compose -f docker-compose.demo.yml --env-file .env --env-file env/ollama.env \ --profile ai --profile ollama up -d --wait ``` `ollama` runs `ollama/ollama:${DEMO_OLLAMA_VERSION:-latest}` on `DEMO_OLLAMA_PORT` (11434). `ollama-pull` downloads `DEMO_OLLAMA_MODEL` (default `qwen2.5:1.5b`, about 1 GB) before `autoair-ai` starts. CPU inference is slow; expect tens of seconds per answer. Hosted provider (billed per token): set `DEMO_AI_PROVIDER` (`anthropic`, `openai`, `gemini` or `ollama`) and `DEMO_AI_API_KEY` in `.env`, optionally `DEMO_AI_MODEL` and `DEMO_AI_BASE_URL`, then: ```bash docker compose -f docker-compose.demo.yml --env-file .env --env-file env/ai.env \ --profile ai up -d --wait ``` `autoair-ai` runs `autoair ai-serve --host 0.0.0.0 --port 8790` with `AUTOAIR_AI__ENABLED=true`, published on `DEMO_AI_PORT` (8790). The replicas and the plugin set `AUTOAIR_AI__SERVICE_URL=http://autoair-ai:8790` and proxy `/api/ai/*` there. The proxy signs the caller's identity with `AUTOAIR_AI__SERVICE__SHARED_SECRET` (`DEMO_AI_SERVICE_SECRET`), and the service trusts only signed requests. Its health check is `http://localhost:8790/health`. See [AI service](https://ctolon.github.io/autoair/docs/ai/ai-service) and [Deploying the AI service](https://ctolon.github.io/autoair/docs/deployment/ai-service). ### `git`: Gitea deploy target with pull requests ```bash docker compose -f docker-compose.demo.yml --env-file .env --env-file env/git.env \ --profile git up -d --wait ``` What happens: 1. `gitea` runs `gitea/gitea:${DEMO_GITEA_VERSION:-1.24-rootless}` with SQLite, registration disabled, SSH disabled and `main` as the default branch, on `DEMO_GITEA_PORT` (3000). 2. `gitea-init` creates the admin user `DEMO_GITEA_USER` (`autoair`). 3. `git-bootstrap` creates the private repository `autoair/dags` (`DEMO_GITEA_REPO`) with a `main` branch and an access token. It stores URL, user and token in the Airflow connection `gitea_dags` (`DEMO_GIT_CONN_ID`) and never prints the token. On later `up` runs it keeps a token that still works. 4. Airflow's dag-processor reads two bundles: `dags-folder` (the local volume) and `gitea`, a `GitDagBundle` on `main` with `git_conn_id=gitea_dags`. 5. AutoAir runs with `storage.mode=git`, `storage.git.use_airflow_bundle=true` and `storage.git.bundle_name=gitea`. Repository, branch and credentials come from the bundle and its connection; AutoAir stores no credentials. AutoAir re-reads the connection after an authentication failure and every `storage.git.credentials_ttl` seconds, so a rotated token is picked up on the next deploy. 6. AutoAir uses `storage.git.provider=gitea`. In PR mode every deploy pushes the branch `autoair/` with git and opens a pull request into `main` through Gitea's REST API, with the connection's token. Each commit and pull request carries the governance context block, and each commit the `X-AutoAir-*` trailers. The seed DAGs arrive as two pull requests. Merge a deploy: 1. Open `http://localhost:3000/autoair/dags/pulls` and sign in as `autoair` / `autoair-demo-only`. The deploy result in AutoAir links to the pull request as well. 2. Open the pull request of `autoair/demo_hello` and merge it. 3. Within `refresh_interval` (30 s) the DAG shows up in Airflow with bundle `gitea`. Its runs are pinned to the merged commit SHA (`bundle_version`). `DEMO_GIT_MODE=push` commits straight to `main` instead. The `gitea` provider also works with Forgejo; GitHub, GitLab, Bitbucket and Azure DevOps have their own providers ([Git and DAG bundles](https://ctolon.github.io/autoair/docs/deployment/git-bundles)). The **Versions & Git → Deploy target…** dialog lists the `gitea` bundle as well, and you can switch between the local folder and the Git target there without a restart. If you change `DEMO_GIT_CONN_ID` in `.env`, change `git_conn_id` in `env/git.env` to match. ### `tls`: HTTPS in and out ```bash docker compose -f docker-compose.demo.yml --profile tls up -d --wait docker compose -f docker-compose.demo.yml --profile tls run --rm --no-deps --entrypoint cat \ certs-init /certs/ca.pem > demo-ca.pem curl --cacert demo-ca.pem https://localhost:8443/api/health # proxy -> replicas curl --cacert demo-ca.pem https://localhost:9443/api/v2/version # proxy -> Airflow curl --cacert demo-ca.pem -I https://localhost:8445/api/health # uvicorn TLS, HSTS header ``` - `certs-init` creates a 30-day demo CA and a server certificate for `localhost`, `tls-proxy`, `autoair-tls`, `autoair-a`, `autoair-b` and `127.0.0.1`, in the `certs` volume. The certificate includes the key identifiers that Python 3.13's strict verification needs. - `tls-proxy` (`nginx:1.28-alpine`, `nginx/tls.conf`) terminates HTTPS for the replicas on `DEMO_TLS_PORT` (8443) and for Airflow on `DEMO_TLS_AIRFLOW_PORT` (9443). - `autoair-tls` is a third replica that serves HTTPS itself on `DEMO_TLS_AUTOAIR_PORT` (8445): ```yaml AUTOAIR_AIRFLOW_URL: https://tls-proxy:9443 AUTOAIR_OUTBOUND__CA_BUNDLE: /certs/ca.pem AUTOAIR_OUTBOUND__VERIFY: "true" AUTOAIR_SERVER__SSL_CERTFILE: /certs/server.pem AUTOAIR_SERVER__SSL_KEYFILE: /certs/server.key AUTOAIR_SERVER__SSL_MIN_VERSION: TLSv1.2 AUTOAIR_SERVER__PUBLIC_URL: https://localhost:8445 AUTOAIR_SERVER__HSTS_SECONDS: "300" AUTOAIR_SERVER__COOKIE_SECURE: always ``` It trusts only the demo CA for outbound calls. Its `/api/ready` reporting `airflow: ok` proves the outbound TLS path. - Browsers warn about the demo CA. Import `demo-ca.pem` or accept the warning. See [TLS and security headers](https://ctolon.github.io/autoair/docs/admin/tls-and-headers) for the settings. ### `observability`: tracing ```bash docker compose -f docker-compose.demo.yml --env-file .env --env-file env/observability.env \ --profile observability up -d --wait ``` The preset sets `AUTOAIR_OTEL_ENABLED=true` on every AutoAir process. Spans go over OTLP/HTTP to `otel-collector:4318` (`otel/opentelemetry-collector-contrib:${DEMO_OTEL_COLLECTOR_VERSION:-0.116.1}`, configured by `otel/collector.yaml`), which forwards them to Jaeger (`jaegertracing/all-in-one:${DEMO_JAEGER_VERSION:-1.65.0}`) at `http://localhost:16686`. Service names are `autoair-a`, `autoair-b`, `autoair-tls`, `autoair` (the plugin inside Airflow) and `autoair-ai`. Health, readiness and metrics URLs are not traced (`server.otel_excluded_urls`). Prometheus metrics are always at `/api/metrics`. See [Observability](https://ctolon.github.io/autoair/docs/admin/observability). ## Ports and credentials All ports bind to `127.0.0.1`. Change any of them in `.env` to avoid collisions. | Variable | Default | Service | | --- | --- | --- | | `DEMO_AIRFLOW_PORT` | 8080 | Airflow + plugin | | `DEMO_AUTOAIR_A_PORT` | 8765 | replica A | | `DEMO_AUTOAIR_B_PORT` | 8766 | replica B | | `DEMO_POSTGRES_PORT` | 5433 | PostgreSQL | | `DEMO_LB_PORT` | 8780 | `lb` | | `DEMO_AI_PORT` | 8790 | `autoair-ai` | | `DEMO_OLLAMA_PORT` | 11434 | `ollama` | | `DEMO_GITEA_PORT` | 3000 | `gitea` | | `DEMO_TLS_PORT` | 8443 | `tls-proxy` → replicas (HTTPS) | | `DEMO_TLS_AIRFLOW_PORT` | 9443 | `tls-proxy` → Airflow (HTTPS) | | `DEMO_TLS_AUTOAIR_PORT` | 8445 | `autoair-tls` (HTTPS from uvicorn) | | `DEMO_JAEGER_PORT` | 16686 | Jaeger UI | | What | Credentials | | --- | --- | | Airflow UI | None: SimpleAuthManager `all_admins`, everybody is `Anonymous` with role `ADMIN` | | Airflow API token | `curl -s localhost:8080/auth/token` (works only in `all_admins` mode) | | AutoAir replicas | **Sign in with Airflow** (Airflow's session cookie is sent to every port on `localhost`), or paste the token above into **Access token** | | PostgreSQL | `postgres` / `postgres-demo-only`; `airflow` / `airflow-demo-only`; `autoair` / `autoair-demo-only` | | Gitea (profile `git`) | `autoair` / `autoair-demo-only` | | Redis (profile `redis`) | password `redis-demo-only` | | AI service HMAC | `DEMO_AI_SERVICE_SECRET` (`local-demo-ai-hmac-secret`) | | Gitea access token | Created by `git-bootstrap` and stored only in the Airflow connection `gitea_dags`, Fernet-encrypted with `DEMO_FERNET_KEY` | ## Configuration There are three layers, highest precedence first: 1. **`docker-compose.demo.yml` environment**, driven by `DEMO_*` variables from `.env` and the presets. `.env.example` lists every variable with its default and a comment. 2. **`autoair.env`** (optional, git-ignored, next to the Compose file): any extra `AUTOAIR_*` variable for every AutoAir container without editing the Compose file, for example `AUTOAIR_IAM__REQUIRE_APPROVAL=true` or `AUTOAIR_UI__BANNER_TEXT=Staging`. 3. **`config/autoair.yaml`**, mounted at `/etc/autoair/autoair.yaml` (`AUTOAIR_CONFIG_FILE`) in the plugin, the replicas and the AI service. It demonstrates: - feature flags: every `features` switch except `airflow` is written out, with `code_edit_inline` turned on and `ide_language_server` left off ([`features`](https://ctolon.github.io/autoair/docs/reference/configuration#features)); - IAM: strategies, Airflow role to AutoAir role mapping, the `AutoAir ` role prefix, a custom role, direct-push roles and the four-eyes switch; - policy rules: DAG and task id patterns, required description, maximum task count, privileged and secret-parameter patterns ([Policy](https://ctolon.github.io/autoair/docs/admin/policy)); - governance: commit, PR title and PR body templates, PR labels, commit author; - UI branding: the orange demo banner. The main `.env` variables besides the ports: | Variable | Default | Purpose | | --- | --- | --- | | `COMPOSE_PROJECT_NAME` | `autoair-demo` | Container and volume prefix | | `COMPOSE_PROFILES` | empty | Profiles to start, comma-separated | | `DEMO_IMAGE` | `autoair-demo:local` | Tag of the image built from the working tree | | `DEMO_AIRFLOW_VERSION` | `3.3.2` | `apache/airflow` base image (any 3.x) | | `DEMO_POSTGRES_PASSWORD`, `DEMO_AIRFLOW_DB_PASSWORD`, `DEMO_AUTOAIR_DB_PASSWORD` | `postgres-demo-only`, `airflow-demo-only`, `autoair-demo-only` | Applied only on the first start of the `postgres` volume. Use URL-safe characters | | `DEMO_FERNET_KEY` | public demo key | Encrypts Airflow connection passwords, including the Gitea token | | `DEMO_JWT_SECRET`, `DEMO_JWT_EXPIRATION_TIME` | public demo secret, `86400` | Airflow token signing key and token lifetime in seconds | | `DEMO_API_SECRET_KEY`, `DEMO_AIRFLOW_API_WORKERS`, `DEMO_AIRFLOW_LOG_LEVEL` | demo key, `2`, `INFO` | Airflow API server | | `DEMO_DAG_BUNDLE_CONFIG_LIST` | local `dags-folder` bundle | JSON bundle list for the dag-processor; `env/git.env` adds the Gitea bundle | | `DEMO_EXPOSE_CONFIG` | `non-sensitive-only` | `expose_config`: `false`, `true` or `non-sensitive-only` | | `DEMO_AUTOAIR_LOG_LEVEL`, `DEMO_AUTOAIR_LOG_FORMAT` | `INFO`, `text` | AutoAir logging (`text` or `json`) | | `DEMO_COLLAB_SECRET` | `local-demo-collab-secret` | HMAC of collaboration tickets, shared by all replicas | | `DEMO_SEED_DAGS` | `true` | Deploy `seed/specs/*.json` once the stack is up | | `DEMO_COLLAB_SYNC`, `DEMO_COLLAB_REDIS_URL`, `DEMO_REDIS_PASSWORD` | `database`, empty, `redis-demo-only` | Collaboration fan-out (`database`, `redis` or `memory`) | | `DEMO_STORAGE_MODE`, `DEMO_GIT_MODE`, `DEMO_GIT_CONN_ID` | `local`, `pr`, `gitea_dags` | Deploy target | | `DEMO_GITEA_USER`, `DEMO_GITEA_PASSWORD`, `DEMO_GITEA_REPO`, `DEMO_GITEA_VERSION` | `autoair`, `autoair-demo-only`, `dags`, `1.24-rootless` | Profile `git` | | `DEMO_AI_ENABLED`, `DEMO_AI_PROVIDER`, `DEMO_AI_MODEL`, `DEMO_AI_API_KEY`, `DEMO_AI_BASE_URL` | `false`, `openai`, empty, empty, empty | Central LLM | | `DEMO_AI_SERVICE_URL`, `DEMO_AI_SERVICE_SECRET` | empty, `local-demo-ai-hmac-secret` | Separate AI service | | `DEMO_OLLAMA_MODEL`, `DEMO_OLLAMA_VERSION` | `qwen2.5:1.5b`, `latest` | Profile `ollama` | | `DEMO_OTEL_ENABLED`, `DEMO_OTEL_COLLECTOR_VERSION`, `DEMO_JAEGER_VERSION` | `false`, `0.116.1`, `1.65.0` | Profile `observability` | Check the effective values and where each one comes from: ```bash docker compose -f docker-compose.demo.yml exec autoair-a autoair config show --changed ``` The demo sets `expose_config=non-sensitive-only`, so administrators can also read the configuration at `GET /api/config` and on the configuration page of the admin panel ([Configuration](https://ctolon.github.io/autoair/docs/admin/configuration), [all settings](https://ctolon.github.io/autoair/docs/reference/configuration)). Changes to `config/autoair.yaml` and `autoair.env` take effect on restart, for example `docker compose -f docker-compose.demo.yml restart autoair-a autoair-b airflow-apiserver`. Changes to `.env` need `up -d` so that Compose recreates the affected containers. ## Walkthroughs ### Build, validate and deploy in the plugin 1. Open `http://localhost:8080/plugin/dag-builder`. The banner comes from `config/autoair.yaml`. 2. Drag an **EmptyOperator** and a **BashOperator** onto the canvas, connect them, and set `bash_command` to `echo hello`. 3. The Problems tab shows the policy at work: without a description the deploy is blocked (`require_description: true`), and a DAG id with capitals breaks `policy.dag_id_pattern`. 4. Add a description, then **Deploy**. The file appears as `dags/autoair/.py`. 5. Find the DAG in Airflow, unpause it, trigger it and read the task log. [Build your first DAG](https://ctolon.github.io/autoair/docs/getting-started/first-dag) walks through the editor in detail. ### Two replicas, one state 1. Sign in on `http://localhost:8765/ui/` and `http://localhost:8766/ui/` with **Sign in with Airflow**. 2. Open `demo_pipeline` on both. Click **Collaborate** in both browsers and edit different task parameters. Both converge through PostgreSQL `LISTEN/NOTIFY`, or through Redis with profile `redis`. 3. Save an operator template on A and open it from B. In **Admin**, the audit trail and the replica list are the same on both. ## Troubleshooting | Symptom | Cause and fix | | --- | --- | | `port is already allocated` | Another service uses the port. Change the matching port variable in `.env` (all twelve are listed under [Ports and credentials](#ports-and-credentials)) | | `all predefined address pools have been fully subnetted` | Docker has no free network ranges. Remove unused networks (`docker network prune` affects every project), or add an override file that pins the default network to a free subnet with `networks.default.ipam.config[0].subnet` | | `postgres` stays unhealthy, or `password authentication failed` | The volume was initialised with other passwords. Initialisation runs only once: `down -v` or `ALTER ROLE` | | `autoair-migrate` fails | `docker compose logs autoair-migrate`. Usually the database password does not match the volume. Every replica waits for this job | | Plugin page empty, `main.js` returns HTML | The AutoAir plugin failed to initialise. See `docker compose logs airflow-apiserver \| grep -A20 "AutoAir plugin failed"`. With `git`, check that `git-bootstrap` succeeded before Airflow started | | A replica is healthy but `/api/ready` answers 503 | Readiness checks the operator catalog, the deploy target (`server.ready_check_storage`; a target that could not be built at startup reports `storage: error: ...`) and, with `server.ready_check_airflow=true`, Airflow's health endpoint. The JSON body (`ready`, `checks`) names the failing check ([Observability](https://ctolon.github.io/autoair/docs/admin/observability)) | | Git deploy: `Authentication failed` | AutoAir re-reads the Airflow connection after an authentication failure and every `storage.git.credentials_ttl` seconds, so a rotated token is picked up on the next deploy. If it persists, the token in `gitea_dags` itself is wrong: rerun `git-bootstrap` | | A merged Git DAG does not appear | Wait for `refresh_interval` (30 s), then check `docker compose logs airflow-dag-processor`. The bundle's `git_conn_id` in `env/git.env` must match `DEMO_GIT_CONN_ID` (`gitea_dags`) | | `seed-dags` exited 1 | Its log lists the failing DAG and the reason. It is idempotent: `docker compose up seed-dags` | | Replicas show the sign-in page | Normal in a new browser profile. Use **Sign in with Airflow** after opening Airflow once, or paste the output of `curl -s localhost:8080/auth/token` | | Marketplace shows nothing | `docker compose logs marketplace-seed`. The replicas read `sources.json` from the `marketplace` volume at startup | | AI answers 503 | AI is off by default. Use profile `ai` with a preset. `/api/ai/*` errors with `DEMO_AI_SERVICE_URL` set mean `autoair-ai` is not running | | Ollama answers very slowly | CPU inference. Pick a smaller `DEMO_OLLAMA_MODEL` | | `x509: Missing Authority Key Identifier` | Certificates from an older `certs-init`. Remove the `certs` volume of this project and start again | | Everything is slow to start the first time | The image build (Node and Python) and Airflow's first migration. Later starts reuse both | Useful commands: ```bash docker compose -f docker-compose.demo.yml ps -a # one-shot jobs included docker compose -f docker-compose.demo.yml logs -f autoair-a docker compose -f docker-compose.demo.yml exec autoair-a autoair db current curl -s localhost:8765/api/ready ``` More in the [FAQ](https://ctolon.github.io/autoair/docs/troubleshooting/faq). ## Stop and tear down ```bash docker compose -f docker-compose.demo.yml stop # stop, keep containers and data docker compose -f docker-compose.demo.yml down # remove containers, keep volumes docker compose -f docker-compose.demo.yml down -v # also erase databases, DAGs, Gitea ``` Pass the same `--profile` and `--env-file` flags as for `up`. Without them, `down` leaves the profile services running. To run a second copy next to the first, for example to test a branch, use another project name and other ports: ```bash docker compose -p autoair-demo-2 -f docker-compose.demo.yml --env-file my-ports.env up -d --wait docker compose -p autoair-demo-2 -f docker-compose.demo.yml down -v # removes only that copy ``` ## The single-container setup `docker-compose.yaml` builds `deploy/docker/Dockerfile` (the official Airflow image plus a prebuilt AutoAir wheel and a precomputed operator catalog) and runs `airflow standalone` with the plugin: ```bash cd autoair # repository root make build # frontend bundles + dist/autoair-*.whl docker compose -f deploy/compose/docker-compose.yaml up --build ``` Open `http://localhost:8080` and go to **Browse → DAG Builder**. The container sets: ```yaml AIRFLOW__CORE__LOAD_EXAMPLES: "False" AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINS: "True" # development only AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: "5" AUTOAIR_STORAGE__MODE: local AUTOAIR_ENVIRONMENTS: '["dev","prod"]' ``` DAGs are written to the `dags` volume, and two environments, `dev` and `prod`, are configured ([Environments](https://ctolon.github.io/autoair/docs/user-guide/environments)). The port is published on all interfaces (`8080:8080`) and the all-admins mode is on: do not run it on a shared network. `make build` needs Node.js 22 and uv on the host; the image build fails with a `COPY dist/autoair-*.whl` error when the wheel is missing. # Helm Source: https://ctolon.github.io/autoair/docs/installation/helm AutoAir does not ship its own chart. On Kubernetes it runs as a plugin inside the official [Apache Airflow Helm chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) (version 1.22 or newer), with an Airflow image that contains AutoAir. `deploy/helm/values-autoair.yaml` is a ready-made values file for a cloud-native setup: - AutoAir commits DAG specs and generated files to Git, in pull-request mode; - Airflow's `GitDagBundle` reads the same repository; - no shared volume is needed and the API server replicas are stateless. For a standalone AutoAir Deployment next to Airflow, an architecture overview and git-sync variants, see [Kubernetes](https://ctolon.github.io/autoair/docs/deployment/kubernetes), [Helm plugin deployment](https://ctolon.github.io/autoair/docs/deployment/helm-plugin) and [Git bundles](https://ctolon.github.io/autoair/docs/deployment/git-bundles). ## 1. Build the image Build the wheel and an Airflow image that contains it, with the providers your DAGs use: ```bash make build # frontend + dist/autoair-*.whl docker build -f deploy/docker/Dockerfile \ --build-arg AIRFLOW_VERSION=3.3.2 \ -t registry.example.com/my-airflow-autoair:3.3.2 . docker push registry.example.com/my-airflow-autoair:3.3.2 ``` `deploy/docker/Dockerfile` installs `autoair[git,mcp,ai,ai-anthropic,ai-openai,format,...]` (`AUTOAIR_EXTRAS`) plus `apache-airflow-providers-git` (needed by `GitDagBundle`) and `apache-airflow-providers-common-messaging`, then precomputes the operator catalog with `autoair catalog build --output /opt/airflow/autoair-catalog.json` and points `AUTOAIR_CATALOG_FILE` at it. Every API server replica therefore starts with an identical palette and does not scan providers at startup. See [Airflow plugin](https://ctolon.github.io/autoair/docs/installation/plugin#into-a-derived-image). Add to the `pip install` line whatever the values below rely on: - `s3fs` (or `gcsfs`, `adlfs`) when `AUTOAIR_TEMPLATES_URL` / `AUTOAIR_AUDIT_URL` point at object storage; - `postgres` and `collab` extras when you move state to PostgreSQL or enable live collaboration across replicas. ## 2. Create the secrets ```bash kubectl create secret generic autoair-git --from-literal=token="$GITHUB_TOKEN" kubectl create secret generic autoair-ai --from-literal=api-key="$ANTHROPIC_API_KEY" ``` The Git token needs permission to push branches and open pull requests on the repository (a fine-grained token, or use a GitHub App: `AUTOAIR_STORAGE__GIT__GITHUB_APP_ID`, `AUTOAIR_STORAGE__GIT__GITHUB_INSTALLATION_ID`, `AUTOAIR_STORAGE__GIT__GITHUB_PRIVATE_KEY`). The Airflow connection `git_dags` used by the bundle must exist as well (for example through the chart's connections or a secrets backend). ## 3. Install ```bash helm repo add apache-airflow https://airflow.apache.org helm upgrade --install airflow apache-airflow/airflow \ -f deploy/helm/values-autoair.yaml \ -f my-site-values.yaml ``` Copy the file and change the repository, registry and names to yours; keep site-specific values (ingress, executor, database) in your own values file. ## The values explained ### Image and API server ```yaml images: airflow: repository: registry.example.com/my-airflow-autoair tag: "3.3.2" apiServer: replicas: 2 ``` The custom image is used for every Airflow component. AutoAir runs in each API server pod; two replicas means two independent AutoAir processes behind the same Service. ### DAG bundle ```yaml config: dag_processor: dag_bundle_config_list: >- [{"name": "dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"git_conn_id": "git_dags", "tracking_ref": "main", "subdir": "dags", "refresh_interval": 60}}] ``` Airflow's dag-processor reads DAGs from the `main` branch of the repository behind the `git_dags` connection, sub-folder `dags`, refreshing every 60 seconds. Runs are pinned to the commit they were created from (`bundle_version`). A merged pull request therefore reaches Airflow within about a minute. ### Deploy target | Variable | Value | Meaning | | --- | --- | --- | | `AUTOAIR_STORAGE__MODE` | `git` | Deploys commit to a Git repository | | `AUTOAIR_STORAGE__GIT__PROVIDER` | `github` | Use GitHub's API (also `gitlab`, `bitbucket`, `azure`, `gitea` for Gitea/Forgejo, or `local` for the git CLI) | | `AUTOAIR_STORAGE__GIT__REPO` | `acme/airflow-dags` | `owner/name` on GitHub | | `AUTOAIR_STORAGE__GIT__BASE_BRANCH` | `main` | Branch pull requests target, the one the bundle tracks | | `AUTOAIR_STORAGE__GIT__SUBDIR` | `dags/autoair` | Folder inside the repository; it is under the bundle's `subdir` (`dags`) so Airflow sees the files | | `AUTOAIR_STORAGE__GIT__MODE` | `pr` | Each deploy opens (or updates) a branch `autoair/` and a pull request; `push` commits directly | | `AUTOAIR_STORAGE__GIT__USE_AIRFLOW_BUNDLE` | `true` | Fill repository, branch, sub-folder and credentials from the `GitDagBundle` configuration where not set explicitly | `AUTOAIR_STORAGE__GIT__TOKEN` comes from the `autoair-git` secret (see `secret:` below). ### State on object storage | Variable | Value | Meaning | | --- | --- | --- | | `AUTOAIR_TEMPLATES_URL` | `s3://acme-airflow/autoair/templates` | Saved templates, shared by both replicas | | `AUTOAIR_AUDIT_URL` | `s3://acme-airflow/autoair/audit` | Audit trail, shared by both replicas | These are fsspec URLs: the image needs `s3fs` and the pods need S3 credentials (for example an IAM role for the service account). Other state (deploy proposals, AI memory, the deploy target chosen in the UI) stays in the per-pod state folder unless you configure it too. With `AUTOAIR_IAM__REQUIRE_APPROVAL=true` and several replicas, set `AUTOAIR_IAM__PROPOSALS_URL` to shared storage, or, simpler, move all state to PostgreSQL: ```yaml env: - { name: AUTOAIR_DATABASE__BACKEND, value: "postgresql" } - { name: AUTOAIR_DATABASE__URL, value: "postgresql+psycopg://autoair@autoair-db:5432/autoair" } - { name: AUTOAIR_DATABASE__MIGRATE_ON_START, value: "false" } secret: - { envName: AUTOAIR_DATABASE__PASSWORD, secretName: autoair-db, secretKey: password } ``` and run `autoair db upgrade` once per release (for example as a Helm pre-upgrade Job using the same image). See [Database](https://ctolon.github.io/autoair/docs/admin/database) and [PostgreSQL](https://ctolon.github.io/autoair/docs/deployment/postgres). ### Organisation policy | Variable | Value | Meaning | | --- | --- | --- | | `AUTOAIR_REQUIRE_OWNER` | `true` | Every task must have an owner | | `AUTOAIR_MIN_RETRIES` | `1` | Every task must retry at least once | | `AUTOAIR_ENVIRONMENTS` | `["dev","prod"]` | Known environments for overlays and promotion | | `AUTOAIR_DEFAULT_ENVIRONMENT` | `prod` | Environment preselected in the UI | Violations are validation errors and block deploys. See [Policy](https://ctolon.github.io/autoair/docs/admin/policy) and [Environments](https://ctolon.github.io/autoair/docs/user-guide/environments). ### IAM | Variable | Value | Meaning | | --- | --- | --- | | `AUTOAIR_IAM__STRATEGIES` | `["airflow_permissions","role_mapping","custom_view"]` | A user's AutoAir roles are the union of: Airflow DAG/config permissions, Airflow role names mapped to AutoAir roles, and AutoAir permissions defined in the auth manager | | `AUTOAIR_IAM__DIRECT_PUSH_ROLES` | `["Admin"]` | Only Admins may skip the pull request and push directly | | `AUTOAIR_IAM__REQUIRE_APPROVAL` | `true` | Four-eyes rule: going live needs another reviewer's approval of the exact version (pull requests are exempt) | With the FAB auth manager run `autoair iam sync-fab --create-roles` once (for example as a Job) so that the `custom_view` permissions and `AutoAir ` roles exist. See [IAM](https://ctolon.github.io/autoair/docs/admin/iam). ### Features, AI and configuration exposure | Variable | Value | Meaning | | --- | --- | --- | | `AUTOAIR_FEATURES__MARKETPLACE` | `false` | Hide template marketplaces (all features are on by default) | | `AUTOAIR_AI__PROVIDER` | `anthropic` | LLM provider | | `AUTOAIR_AI__MODEL` | `claude-sonnet-5-5` | Model id | | `AUTOAIR_AI__LLM_MODE` | `central` | Everyone uses the central model (`per_user` and `both` let users bring their own) | | `AUTOAIR_AI__SERVICE_URL` | commented out | Proxy `/api/ai/*` to a separate AI service (`autoair ai-serve`) | | `AUTOAIR_EXPOSE_CONFIG` | `non-sensitive-only` | Admins can read the effective configuration without secrets, like Airflow's `[api] expose_config` | ### Secrets ```yaml secret: - envName: AUTOAIR_STORAGE__GIT__TOKEN secretName: autoair-git secretKey: token - envName: AUTOAIR_AI__API_KEY secretName: autoair-ai secretKey: api-key # - envName: AUTOAIR_AI__SERVICE__SHARED_SECRET # secretName: autoair-ai # secretKey: service-secret ``` The chart turns each entry into an environment variable from a Kubernetes Secret. `AUTOAIR_AI__API_KEY` is optional: without it AI features stay off while rule-based review keeps working. `AUTOAIR_AI__SERVICE__SHARED_SECRET` is required only when the AI service runs separately; it signs the forwarded caller identity. ## Verify ```bash kubectl exec deploy/airflow-api-server -- airflow plugins | grep autoair kubectl exec deploy/airflow-api-server -- autoair config show --changed kubectl port-forward svc/airflow-api-server 8080:8080 curl -s localhost:8080/autoair/api/health ``` Then open **Browse → DAG Builder**, deploy a test DAG, and check that a pull request appears in the repository. After merging, the DAG shows up in Airflow within the bundle's `refresh_interval`. The chart names the API server Deployment and Service `-api-server`, so `deploy/airflow-api-server` and `svc/airflow-api-server` above assume the release name `airflow` used in the install command. With `useStandardNaming: true` the prefix becomes `fullnameOverride`, or `-airflow` when the release name does not contain `airflow`. ## Upgrading Build a new image with the new wheel, bump `images.airflow.tag`, and `helm upgrade`. Run `autoair db upgrade` first when state is in PostgreSQL. See [Upgrading](https://ctolon.github.io/autoair/docs/installation/upgrading). # Upgrading Source: https://ctolon.github.io/autoair/docs/installation/upgrading An AutoAir upgrade touches four things: the Python package (which includes the frontend), the state database schema (SQL backends only), the operator catalog cache, and the browsers that have the old UI loaded. Deployed DAG files are not touched. ## Before you upgrade 1. **Read the release notes** for changed defaults and removed settings. Check your configuration against the new version with `autoair config show --changed` after installing it in a test environment; invalid settings stop the CLI with exit code 2 and make the plugin log `AutoAir: invalid configuration`. 2. **Check Airflow compatibility.** AutoAir supports Airflow 3.0 and newer below 4. When you upgrade Airflow at the same time, rebuild the catalog (below) and review [validation](https://ctolon.github.io/autoair/docs/user-guide/validation) results: features are gated by the installed Airflow version, or by `AUTOAIR_TARGET_AIRFLOW_VERSION` when set. 3. **Back up state.** With `database.backend=postgresql` or `sqlite`, back up the database; with the file backend, back up the state folder (`//.autoair` by default) and any `*_url` locations you configured. Specs live in the deploy target (folder, bucket or Git). See [Backup and restore](https://ctolon.github.io/autoair/docs/admin/backup-restore). ## Upgrade the package ### Local mode ```bash uvx --refresh --from 'autoair[standalone,airflow]' autoair local # newest release uvx --from 'autoair[standalone,airflow]==' autoair local # a specific version pip install -U 'autoair[standalone,airflow]' # in a virtual environment ``` ### Standalone server ```bash pip install -U 'autoair[standalone,format,git,mcp,collab,postgres]==' autoair db upgrade # SQL backends, see below ``` With several replicas, run the migration once, then roll the replicas. ### Airflow plugin Rebuild the Airflow image with the new wheel (and, when you upgrade Airflow too, the new base image), then roll out the API server: ```bash make build docker build -f deploy/docker/Dockerfile --build-arg AIRFLOW_VERSION=3.3.2 \ -t registry.example.com/my-airflow-autoair:3.3.2-autoair . ``` With Helm, bump `images.airflow.tag` and `helm upgrade`. Installing into a running virtual environment works too (`pip install -U "autoair[]" "apache-airflow==$(airflow version)"`, with the extras you installed before), followed by a restart of the API server; pinning Airflow keeps pip from changing it. ## Migrate the state database The file backend has no schema; `autoair db upgrade` reports `database.backend is 'file': nothing to migrate` and exits with code 2. For SQLite and PostgreSQL: ```bash autoair db current # : version None (head 1) # pending: [1] - run `autoair db upgrade` autoair db upgrade # applied: [1]; head: 1 autoair db current --json # for scripts ``` - `autoair db upgrade` takes a database lock, so it is safe to run from several replicas at once; it only applies missing versions. - `--revision N` stops at schema version `N` (it cannot go below the applied version). - With `database.migrate_on_start=true` (the default) each server migrates on startup. For several replicas set it to `false` and run `autoair db upgrade` from an init container or a Helm pre-upgrade Job, as the Compose demo does with its `init-state` service. - Administrators can see the schema version and pending migrations in the admin panel, and apply them from there. The command reads the same configuration as the server (`AUTOAIR_DATABASE__BACKEND`, `AUTOAIR_DATABASE__URL`, `AUTOAIR_DATABASE__PASSWORD` and the other [`database`](https://ctolon.github.io/autoair/docs/reference/configuration#database) options, environment variables or configuration files), so run it with the server's environment. ## Rebuild the operator catalog The catalog cache is keyed by the installed `apache-airflow*` distributions, the AutoAir version, the catalog code and the discovery settings, so an upgrade never serves a stale on-disk cache: the first start after an upgrade rebuilds it (in `AUTOAIR_CACHE_DIR`, default `$XDG_CACHE_HOME/autoair`). A **precomputed** catalog file (`AUTOAIR_CATALOG_FILE`) built by an older version is detected as stale and ignored, with the log line `catalog file /opt/airflow/autoair-catalog.json is missing or stale; ignoring it`; AutoAir then builds the catalog at startup, which is slow with many providers. Rebuild it as part of the image build: ```bash autoair catalog build --output /opt/airflow/autoair-catalog.json # wrote /opt/airflow/autoair-catalog.json: 1234 operators, 87 providers, 0 errors ``` Do the same whenever you add or upgrade providers. Users with the `catalog:refresh` permission can also trigger a catalog refresh through the API. ## Frontend assets The UI is part of the wheel; there is nothing to deploy separately. Content-hashed files are served with `Cache-Control: public, max-age=, immutable`; entry files (`/ui/` pages and the plugin's `/autoair/static/main.js`) are served with `no-cache`, so browsers pick up the new UI on the next load. A browser tab opened before the upgrade may still reference chunks that the new build no longer contains; reload the page if a dialog fails to open after an upgrade. During a rolling update, old and new pods serve different builds; prefer short rollouts or session affinity if users report missing chunks. A wheel built from source without `npm run build` contains no UI and `/ui/` answers `503 Frontend bundle not built`. Released wheels always contain it. ## Specs and generated files - **Specs** are migrated in memory when read (`spec_version` 1 to 2 to 3). They are written in the new format the next time the DAG is saved or deployed. AutoAir 0.2.0 writes `spec_version` 3. A spec written by a newer AutoAir than the one reading it is rejected (`spec_version N is newer than this AutoAir supports (M); upgrade AutoAir`); AutoAir 0.1.0 refuses every spec that 0.2.0 has saved. - **Upgrade every AutoAir process that reads the same specs together**: the Airflow plugin, `autoair serve` replicas, `autoair mcp-serve`, `autoair ai-serve`, the validator service, and the `autoair` package installed in the Airflow dag-processor and workers. In the loader output mode the stub DAG file parses `.autoair.json` with the AutoAir installed there, so an older AutoAir in Airflow fails to import every loader DAG that 0.2.0 has deployed. - **Generated DAG files** are not rewritten by the upgrade. Changes in code generation (formatting, imports for a newer Airflow) appear the next time each DAG is deployed; the version diff shows them. Loader-mode DAGs build from the spec at parse time with the `autoair` installed on the dag-processor, so upgrade it there at the same time. ## Rollback 1. Reinstall the previous package version (or roll back the image tag / Helm release). 2. **Database**: migrations only go forward; there is no downgrade command. Restore the backup taken before the upgrade if the new version applied migrations. 3. **Specs** saved by 0.2.0 have `spec_version` 3, which AutoAir 0.1.0 refuses ("upgrade AutoAir"); loader-mode DAGs deployed by 0.2.0 fail to import under 0.1.0. To roll back to 0.1.0, restore the specs (and loader-mode spec files) from before the upgrade, for example from the git history or object-storage versions of the deploy target; otherwise stay on 0.2.0. 4. **Catalog**: the cache key includes the version, so the previous version rebuilds or reuses its own cache; rebuild a precomputed `AUTOAIR_CATALOG_FILE` with the previous version. 5. Ask users to reload the UI. ### Older AutoAir against a newer database schema Whether a rolled-back AutoAir starts against a database that a newer version already migrated depends on `database.migrate_on_start`: | `database.migrate_on_start` | Result | |---|---| | `true` (default) | The migration step finds nothing to apply (every known migration is already recorded) and the server starts. It then runs against tables that may contain columns or tables it does not know; nothing checks that the schema is not newer. | | `false` | Startup fails with ``database schema is at version , AutoAir needs : run `autoair db upgrade` ``, because the recorded version differs from the version this AutoAir expects. Running `autoair db upgrade` with the older version does not downgrade anything. | Restoring the pre-upgrade backup (step 2) is the only supported way back to an older schema. ## Related - [CLI reference](https://ctolon.github.io/autoair/docs/reference/cli) - [Database](https://ctolon.github.io/autoair/docs/admin/database) - [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling) - [Helm](https://ctolon.github.io/autoair/docs/installation/helm) # User guide overview Source: https://ctolon.github.io/autoair/docs/user-guide/overview The AutoAir builder consists of the canvas where a DAG is drawn, the inspector where tasks are configured, the code views, the validation pipeline, and the lifecycle features: versions, deploy, reviews and runs. Below are the workbench layout and the guide page for each topic. ## The workbench [Screenshot: The AutoAir workbench: toolbar at the top, operator palette on the left, a DAG on the canvas in the middle, the inspector on the right and the bottom panel with the Problems tab (The workbench with a DAG open: palette, canvas, inspector and bottom panel.)] The builder is one page made of five areas: | Area | Position | What it holds | |---|---|---| | Toolbar | Top | Product name, DAG id, status chips (unsaved, deployed, new, open pull request, deploy target, target Airflow version), the **File**, **Run**, **Versions & Git** and **AI** menus, undo / redo, auto layout, the **Visual / Code** editor-mode toggle, collaboration control, environment selector, the **Deploy** button, the user menu and the **More** (`⋯`) menu. | | Palette | Left | Searchable list of everything that can be inserted: built-in nodes (`@task` variants, task group, note, extension nodes), operator templates, template fragments and every discovered operator, grouped by provider. | | Canvas | Centre | The DAG graph: tasks, task groups, sticky notes and dependencies. See [Canvas](https://ctolon.github.io/autoair/docs/user-guide/canvas). | | Inspector | Right | Settings of the selected node or dependency. With nothing selected it shows **DAG settings** (General, Schedule, Params, Defaults, Alerts, Environments, Docs). | | Bottom panel | Below the canvas | Tabs **Problems**, **Code**, **Runs**, **Task**, **Versions**, **Airflow** and **Audit**. **Runs** and **Airflow** need a reachable Airflow server (`features.airflow`; **Runs** also needs `features.runs`); **Audit** needs the `audit:read` permission. | Every side panel and the bottom panel can be hidden, resized and restored; the layout is remembered per browser. See [Panels: collapse and resize](https://ctolon.github.io/autoair/docs/user-guide/canvas#panels-collapse-and-resize). When the builder is narrower than 1100 px the inspector opens as an overlay drawer instead of a docked panel; below 720 px the palette does too. The width that counts is the builder's own width, not the browser window's: inside Airflow the builder is narrower than the page. ## Map of the user guide ### Building a DAG | Page | Covers | |---|---| | [Canvas](https://ctolon.github.io/autoair/docs/user-guide/canvas) | Palette, canvas, inspector, bottom panel, task groups, notes, context menu, insert popover, alignment guides, auto layout, copy / paste / duplicate, undo / redo, minimap, search. | | [Keyboard shortcuts](https://ctolon.github.io/autoair/docs/user-guide/shortcuts) | Complete shortcut table for macOS and Windows/Linux, the command palette and the shortcuts sheet. | | [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode) | Editing the generated Python file and syncing it back to the canvas. | | [Field editors](https://ctolon.github.io/autoair/docs/user-guide/editors) | Code editors, built-in IntelliSense, the optional IDE language server, the **Validate** button and the Markdown editor. | | [Operators and parameters](https://ctolon.github.io/autoair/docs/user-guide/operators-and-parameters) | Operator catalog, parameter forms, type badges, required and optional parameters, connection and pool pickers. | | [Jinja, XCom and Python](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python) | Templated fields, upstream outputs, Python references, inline functions and inline scripts. | | [TaskFlow and loops](https://ctolon.github.io/autoair/docs/user-guide/taskflow-and-loops) | `@task` decorators, dynamic task mapping, branching, task groups, task loops, trigger rules, setup and teardown. | | [Scheduling and assets](https://ctolon.github.io/autoair/docs/user-guide/scheduling-and-assets) | Schedule editor, cron presets, timetables, asset conditions, time zones, catchup, start and end dates, outlets and inlets. | | [Params and trigger forms](https://ctolon.github.io/autoair/docs/user-guide/params-and-trigger-forms) | DAG params, the Airflow trigger form preview and the trigger dialog. | | [Human-in-the-loop](https://ctolon.github.io/autoair/docs/user-guide/human-in-the-loop) | HITL operators and their dedicated editor. | | [Callbacks, retries and deadlines](https://ctolon.github.io/autoair/docs/user-guide/callbacks-retries-deadlines) | Notifier callbacks, retries, backoff, exception retry policies and deadline alerts. | | [Connections and Variables](https://ctolon.github.io/autoair/docs/user-guide/connections) | Connection pickers, creating connections, Variables and per-environment aliases. | | [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation) | The Problems panel, live checks, the server validation pipeline and severities. | ### Reuse, lifecycle and teamwork | Page | Covers | |---|---| | [Operator templates](https://ctolon.github.io/autoair/docs/user-guide/operator-templates) | Reusable, pre-configured tasks with locked and required fields. | | [Templates and fragments](https://ctolon.github.io/autoair/docs/user-guide/templates-and-fragments) | DAG templates and reusable groups of tasks. | | [Marketplace](https://ctolon.github.io/autoair/docs/user-guide/marketplace) | Installing templates from configured marketplaces. | | [Versions and diff](https://ctolon.github.io/autoair/docs/user-guide/versions-and-diff) | Version history and comparing versions on the canvas. | | [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy) | Writing the DAG to its deploy target. | | [Reviews and approvals](https://ctolon.github.io/autoair/docs/user-guide/reviews-and-approvals) | Four-eyes approvals before deployment. | | [Runs](https://ctolon.github.io/autoair/docs/user-guide/runs) | Test runs, triggers, backfills and the run overlay. | | [Environments](https://ctolon.github.io/autoair/docs/user-guide/environments) | Per-environment overlays and promotion. | | [Import and export](https://ctolon.github.io/autoair/docs/user-guide/import-export) | Importing Python, YAML and Airflow DAGs; exporting Python, YAML, dag-factory and spec JSON. | | [Collaboration](https://ctolon.github.io/autoair/docs/user-guide/collaboration) | Real-time co-editing and presence. | | [Multi-team](https://ctolon.github.io/autoair/docs/user-guide/multi-team) | Team-scoped DAGs, pickers and deploy targets. | ## Plugin and standalone UI AutoAir ships one frontend that runs in two hosts. The builder itself (canvas, inspector, panels, dialogs, shortcuts) is identical; the differences are in how it is reached, themed and signed into. | Aspect | Airflow plugin | Standalone (`autoair serve`) and local mode (`autoair local`) | |---|---|---| | Where it opens | Airflow 3.1 and later: a navigation entry named **DAG Builder** (`plugin.nav_name`, in the menu set by `plugin.nav_category`, default `browse`) rendered inside Airflow's own UI. Airflow 3.0: a full page at `/autoair/ui/`. | Its own page at `/ui/` on the AutoAir server (the root URL redirects there). | | DAG tab | An **AutoAir** tab on every DAG page (`plugin.dag_tab`, `plugin.dag_tab_name`; on Airflow 3.4 and newer `plugin.dag_tab_tags` limits it to tagged DAGs). It opens that DAG in the builder; for a DAG that exists only as Python in Airflow it offers to import it. | Not applicable. | | Light and dark mode | Follows the Airflow UI theme. There is no light/dark toggle in the builder and the **Toggle light / dark mode** command is not offered. | The user chooses light or dark from the **More** (`⋯`) menu or the command palette; the choice is stored in the browser. The initial mode follows the operating system preference. | | Page title | Airflow's. | `ui.title` (default `AutoAir`). | | Sign-in | The Airflow UI session. AutoAir watches the session, warns before it expires and shows **Sign in to Airflow** when it ends. | A sign-in screen: Airflow username and password, an Airflow access token, or **Sign in with Airflow** when available. `autoair local` can require a token (`autoair local --token`). | | Airflow features | Always available (pickers, DAG status, runs, import from Airflow). | Available when `airflow_url` points at a reachable Airflow API. In `autoair local` the Airflow-backed features (`features.airflow`) are off: connection, pool and Variable pickers fall back to free text and the **Runs** and **Airflow** tabs are hidden. | The following settings change the shell in both hosts: `ui.banner_text` and `ui.banner_color` show an environment banner above the builder (for example `PRODUCTION`), `ui.accent_color` adds an accent strip, `ui.default_language` (`en` or `tr`) sets the language for users who never chose one, and `ui.default_timezone` / `ui.date_locale` control how times and dates are displayed. See the [`ui` settings](https://ctolon.github.io/autoair/docs/reference/configuration#ui) and the [`plugin` settings](https://ctolon.github.io/autoair/docs/reference/configuration#plugin). ```bash AUTOAIR_PLUGIN__NAV_NAME="Pipelines" AUTOAIR_PLUGIN__DAG_TAB=false # Show a red PRODUCTION banner above the builder AUTOAIR_UI__BANNER_TEXT="PRODUCTION" AUTOAIR_UI__BANNER_COLOR=red ``` ### Editor availability Administrators decide which editors exist with `features.visual_editor` and `features.code_editor` (both on by default). With both on, the toolbar shows the **Visual / Code** toggle; with only one on, the builder opens directly in that editor and the toggle is hidden. See [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode#editor-modes). ### Language The UI is available in English and Turkish. Change it from the **More** (`⋯`) menu under **Language**; the choice is stored in the browser. ## Related pages - [Quick start](https://ctolon.github.io/autoair/docs/getting-started/quick-start) - [Installation overview](https://ctolon.github.io/autoair/docs/installation/overview) - [Feature flags](https://ctolon.github.io/autoair/docs/admin/feature-flags) # Canvas and workbench Source: https://ctolon.github.io/autoair/docs/user-guide/canvas The canvas is the visual editor of a DAG. Each task is a node, each dependency is an edge, task groups are frames that contain tasks, and sticky notes document the graph without becoming tasks. Everything drawn here is stored in the AutoAir DAG spec and turned into a Python DAG file by the code generator. [Screenshot: A DAG on the canvas with tasks, a task group and a sticky note; the palette on the left and the inspector on the right] ## Palette The palette on the left lists everything that can be inserted. | Section | Contents | |---|---| | **Built-in** | **Python task** (`@task`) and the other `@task.` variants the server's Airflow provides, **Task group**, **Note**, and nodes contributed by installed extensions. | | **Operator templates** | Pre-configured tasks saved by your team. See [Operator templates](https://ctolon.github.io/autoair/docs/user-guide/operator-templates). | | **Templates** | Template fragments: reusable groups of tasks. See [Templates and fragments](https://ctolon.github.io/autoair/docs/user-guide/templates-and-fragments). | | One section per provider | Every operator, sensor and transfer the operator catalog discovered, grouped by provider package. See [Operators and parameters](https://ctolon.github.io/autoair/docs/user-guide/operators-and-parameters). | Tabs above the list filter it: **All**, **Operators**, **Sensors**, **Transfers**, **Built-in**, **Favorites** and **Recent**. The search box matches every word you type against the label, summary, provider, class path and keywords, and ranks label matches first. While you search, collapsed provider sections expand. To add an item, do one of the following: - Drag it onto the canvas. Dropping it inside a task group puts the task in that group. - Double-click it. - Move to it with the arrow keys in the search box and press **Enter**. Press **Alt+S** on the highlighted item, or click its star, to add it to **Favorites**. The ten most recently inserted items appear under **Recent**. Favorites and recents are stored in the browser. Items the builder cannot configure are listed but disabled, with the reason in their tooltip, for example `Not supported yet: needs Python callable parameter(s): …`. Operators that take Python objects such as `python_callable` are insertable; they are configured with a Python reference, and the hint names the equivalent `@task` node. See [Python references](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python#python-references). When the canvas is empty it shows **Start your DAG** with shortcuts to **Search operators…**, **Start from a template…**, **Generate with AI…** and **Write it as code**. ## Nodes and dependencies ### Task nodes A task node shows its task id (or its canvas label), the operator class or decorator, and badges: | Badge | Meaning | |---|---| | `disabled` | The task stays on the canvas but is left out of the generated DAG. Its upstream and downstream tasks are connected directly. | | `setup` / `teardown` | The task is a setup or teardown task. Edges touching it are drawn dashed. | | `mapped` | Dynamic task mapping is configured. | | `→ asset` | The task produces assets (outlets). | | `result` | The task is the DAG result (`@result`). | | `stub · ` | A `@task.stub` task routed to a language SDK queue. | | `lineage` / `no lineage` | OpenLineage is forced on or off for this task. | | `N errors`, `N warnings` | Validation problems on this task. | Markers also flag **Sensor**, **Deferrable**, **Human-in-the-loop**, **Branching** and **Short-circuit** tasks. Selecting a single task shows a toolbar above it: **Duplicate**, **Disable task** / **Enable task**, **Setup task**, **Teardown task**, **Test this task**, **Operator documentation** (operators only) and **Delete**. **Test this task** is disabled when test runs are not available (`features.test_runs`). The **Label (canvas only)** field in the inspector changes what the canvas shows without changing the task id Airflow sees. Icons can be changed per node or for all nodes of an operator class from **Icon (canvas only)**. ### Connecting tasks Drag from a task's output handle to another task's input handle. The canvas refuses a connection and keeps the graph unchanged when: - a node would depend on itself; - the two nodes are already connected; - a task group would be connected with one of its own members; - the connection would create a cycle. Edges to or from a task group count as edges to every task in the group, as in Airflow. Notes cannot be connected. If you drop a connection on empty canvas, the [insert popover](#insert-popover) opens at that point; the node you pick is created there and connected to the handle you dragged from. ### Dependency labels Double-click an edge, or right-click it and choose **Edit label**, to give it a label. Press **Enter** to save or **Escape** to cancel. Labels are generated as `airflow.sdk.Label`, for example to name a branch condition: ```python choose >> Label("weekday") >> load_weekday ``` Selecting an edge shows the **Dependency** inspector with the same **Label** field. ## Insert popover The insert popover is a compact searchable list for adding a node in context. It opens in three ways: | Trigger | Result | |---|---| | Hover or select an edge and click its **+** button (**Insert a node here**) | The new node is inserted between the two tasks: source → new → target. | | Drop a connection on empty canvas | The new node is created at the drop point and connected to the dragged handle. | | Double-click empty canvas | The command palette opens in insert mode at that position. See [Command palette](https://ctolon.github.io/autoair/docs/user-guide/shortcuts#command-palette). | Between two tasks the popover offers operators and `@task` variants only: notes and template fragments cannot be inserted on an edge. Press **Escape** or click outside the popover to close it. ## Task groups A task group is a frame on the canvas that becomes an Airflow `TaskGroup`. - Insert **Task group** from the palette, or select tasks and press **Ctrl+G** / **⌘G** (**Group selection**) to wrap them in a new group. - Drag a task into a group to add it; drag it out to remove it. Membership is decided by where the node's top centre lands when you release it. - Select a group and press **Ctrl+Shift+G** / **⌘⇧G** (**Ungroup**) to dissolve it. Its edges are rewired onto the member tasks that were its entry and exit points. - Click the chevron in the group header, or press **Alt+C** / **⌥C**, to collapse or expand the selected groups. A collapsed group is drawn as one box with its task count and, when a run is shown, an aggregated run state; edges into its members are drawn to the box. Collapsing is a view setting only: it is not saved in the DAG and does not mark it as changed. - Resize an expanded, selected group from its edges (minimum 240 × 140 px). - Deleting a group keeps its tasks: they move to the enclosing group, or to the top level. The group inspector holds **Group id**, **Label (canvas only)**, **Tooltip**, **Prefix task ids with group id** (on by default: tasks inside become `.`) and **Loop this group (Task Loop)**. The resulting Airflow task id of every task is shown under its **Task id** field as **Airflow task id: …**. Task loops are described in [TaskFlow and loops](https://ctolon.github.io/autoair/docs/user-guide/taskflow-and-loops#task-loops). ## Sticky notes Notes document the DAG on the canvas. They are not tasks and are never connected. - Add one with **Shift+N** (**Add note**), with **Add note here** in the canvas context menu, or by inserting **Note** from the palette. - Double-click a note to edit it. The text supports a small Markdown subset (headings, lists, bold, inline code). Press **Ctrl+Enter** / **⌘Enter** or click outside to save; press **Escape** to discard the edit. - Pick a colour and edit the text in the note inspector. Turn on **Include in DAG docs** to append the note's text to the DAG's `doc_md`. ## Context menu [Screenshot: The canvas context menu open on a task, with Copy, Cut, Duplicate, Group, Zoom to node and Delete] Right-click a node, an edge or empty canvas. The menu shows the keyboard shortcut next to each action that has one. | Target | Items | |---|---| | Node | **Copy**, **Cut**, **Duplicate**, **Save as operator template…** (tasks only, with the `template:write` permission and templates enabled), **Group into task group** / **Group N nodes**, **Ungroup** and **Collapse group** / **Expand group** (task groups only), **Zoom to node**, **Delete**. | | Edge | **Edit label**, **Delete dependency**. | | Empty canvas | **Paste here**, **Add note here**, **Select all**, **Group N nodes** (with more than one node selected), **Auto layout**, **Fit view**. | Right-clicking an unselected node selects it first. **Paste here** and **Add note here** use the position you right-clicked. ## Selection, copy, paste and duplicate - Click a node to select it. **Shift**+drag on empty canvas draws a selection box; **Ctrl**+click (**⌘**+click on macOS) adds a node to the selection. **Ctrl+A** / **⌘A** selects every visible node and edge; **Escape** clears the selection. - Arrow keys move the selected nodes by 1 px; **Shift** + arrow keys move them by 16 px. Holding the key repeats the move. - **Ctrl+C** / **⌘C** and **Ctrl+X** / **⌘X** copy or cut the selection to the system clipboard as JSON. Copying a task group copies all its members. When the system clipboard cannot be read (browser permissions, insecure context), the menu and palette commands fall back to an in-memory clipboard. - **Ctrl+V** / **⌘V** pastes under the mouse pointer when it is over the canvas, otherwise next to the original position. Pasting inside a task group puts the pasted nodes in that group. You can also paste a whole AutoAir DAG spec JSON copied from elsewhere. - **Ctrl+D** / **⌘D** duplicates the selection 40 px down and right; duplicates of grouped tasks stay in the same group. Pasted and duplicated nodes get new internal ids. Task ids that would collide get a numeric suffix (`extract` becomes `extract_1`), parameters and the edges between the copied nodes are kept, and XCom references between copied tasks are rewired to the copies. Keyboard copy only acts on the canvas: when text is selected on the page, or focus is in a field or code editor, **Ctrl+C** copies text as usual. ## Undo and redo **Ctrl+Z** / **⌘Z** undoes and **Ctrl+Shift+Z** / **⌘⇧Z** (or **Ctrl+Y** / **⌘Y**) redoes. The toolbar has the same two buttons. History keeps the last 100 changes of the document: node and edge edits, inserts, deletes, DAG settings and code-mode applies. Selection, panel layout, collapsed groups and validation results are not part of the history. A drag is recorded as one step however long it lasts. In a collaboration session undo and redo apply to your own edits only. See [Collaboration](https://ctolon.github.io/autoair/docs/user-guide/collaboration). ## Auto layout **Auto layout** (**Shift+L**, the toolbar button, or the canvas context menu) arranges the graph left to right with a layered algorithm (ELK). It understands task groups: members are laid out inside their group and the group is resized to fit. Collapsed groups are laid out as single boxes and their hidden members keep their positions. The view is fitted after the layout. Auto layout is one undoable step; if it fails, an **Auto layout failed** notification shows the reason and the graph is unchanged. ## Alignment guides and snapping While you drag a single node, alignment guides appear when its left, centre or right edge, or its top, middle or bottom edge, comes within a few pixels of another visible node, and the node snaps onto the guide. Guides are off while snap to grid is on. **Shift+S** (**Toggle snap to grid**) snaps node positions to a 16 px grid; the background changes from dots to crosses while it is on. ## Viewport, minimap and search - Drag empty canvas to pan; scroll or pinch to zoom, or use the zoom controls in the lower left corner (zoom in, zoom out, fit view, toggle interactivity). Double-clicking empty canvas does not zoom: it opens the insert palette. - **Shift+F** (**Fit view**) fits the whole graph in view. - **Shift+M** (**Toggle minimap**) shows or hides the minimap in the lower right corner. Nodes are coloured by kind, or by run state while a run is shown. The minimap can be dragged and zoomed. Snap and minimap settings are stored in the browser. - To find a task, open the command palette (**Ctrl+K** / **⌘K**) and type its task id, label or operator: the **Go to task** group selects the task, expands any collapsed group around it and zooms to it. **Zoom to node** in the context menu does the same for a node you can see. Graphs with more than 150 nodes only render the nodes inside the viewport, can zoom out further to an overview, and show low-detail cards when zoomed far out. See [Large DAGs](https://ctolon.github.io/autoair/docs/user-guide/large-dags). ## Inspector The inspector shows the settings of the selected element: | Selection | Inspector | |---|---| | Nothing | **DAG settings** with the tabs **General**, **Schedule**, **Params**, **Defaults**, **Alerts**, **Environments** and **Docs**. | | Task | Task id, label, icon and task group, then the tabs **Parameters** (**Code** for Python tasks), **Execution**, **Behaviour**, **Data** and **Docs**. | | Task group | Group settings and the task loop. | | Note | Text, colour and **Include in DAG docs**. | | Dependency | **Label**. | [Screenshot: The inspector of an operator task: task id and label, the Parameters tab with required parameters, type badges and the Jinja badge] A tab with validation errors shows their count. At the top of the task inspector, a problem banner lists the task's problems; click one to switch to the tab that owns the field and focus it. **Delete** in the inspector header removes the node. The task tabs are covered in detail in [Operators and parameters](https://ctolon.github.io/autoair/docs/user-guide/operators-and-parameters), [Callbacks, retries and deadlines](https://ctolon.github.io/autoair/docs/user-guide/callbacks-retries-deadlines), [TaskFlow and loops](https://ctolon.github.io/autoair/docs/user-guide/taskflow-and-loops) and [Scheduling and assets](https://ctolon.github.io/autoair/docs/user-guide/scheduling-and-assets#assets-produced-and-consumed-by-tasks). ## Bottom panel | Tab | Contents | |---|---| | **Problems** | Validation findings grouped by task. See [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation). The tab shows a spinner while checks run and a badge with the problem count. | | **Code** | The generated output: **Python**, **Airflow YAML**, **dag-factory** and **Spec JSON**. See [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode#the-code-tab). | | **Runs** | Runs of the deployed DAG and the run overlay. See [Runs](https://ctolon.github.io/autoair/docs/user-guide/runs). | | **Task** | Details of the selected task. | | **Versions** | Version history and comparison. See [Versions and diff](https://ctolon.github.io/autoair/docs/user-guide/versions-and-diff). | | **Airflow** | The DAG's state in Airflow. | | **Audit** | The DAG's audit trail (needs `audit:read`). | ## Panels: collapse and resize The palette, the inspector and the bottom panel can each be hidden and resized. - **Hide**: click the small round button on the panel's edge, use **Ctrl+B** / **⌘B** (palette), **Ctrl+I** / **⌘I** (inspector) or **Ctrl+J** / **⌘J** (bottom panel), or use the **More** (`⋯`) menu. A hidden panel leaves a slim rail (side panels) or bar (bottom panel) labelled **Operators**, **Inspector** or the active bottom tab; click it to show the panel again. The bottom bar keeps the problem count visible. - **Resize**: drag the line between the panel and the canvas. With keyboard focus on the line, the arrow keys resize by 16 px, **Home** and **End** go to the minimum and maximum size, and double-click restores the default size. The canvas always keeps at least 320 × 160 px. | Panel | Default size | Minimum size | |---|---|---| | Palette | 280 px wide | 200 px | | Inspector | 380 px wide | 280 px | | Bottom panel | 280 px tall | 120 px | Hidden panels and sizes are stored per browser. The canvas refits when a panel is shown or hidden. On a narrow builder the side panels open as overlay drawers instead (see [The workbench](https://ctolon.github.io/autoair/docs/user-guide/overview#the-workbench)). If a panel fails to render, it shows **Something went wrong in the …** with **Try again** and, where it applies, **Undo last change**, so a bad edit never takes the whole builder down. ## Related pages - [Keyboard shortcuts](https://ctolon.github.io/autoair/docs/user-guide/shortcuts) - [Operators and parameters](https://ctolon.github.io/autoair/docs/user-guide/operators-and-parameters) - [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation) - [Large DAGs](https://ctolon.github.io/autoair/docs/user-guide/large-dags) - [Collaboration](https://ctolon.github.io/autoair/docs/user-guide/collaboration) # Keyboard shortcuts and command palette Source: https://ctolon.github.io/autoair/docs/user-guide/shortcuts Every builder action that has a keyboard shortcut is a registered command. The same registry feeds the keyboard dispatcher, the command palette (**Ctrl+K** / **⌘K**) and the shortcuts sheet (**?**), so the three always agree. ## How shortcuts are dispatched - **Mod** is **Ctrl** on Windows and Linux and **⌘** (Command) on macOS. On macOS, **Ctrl+K** is not **⌘K**: the other platform's modifier never matches. - Shortcuts are ignored while you type in an input, a text area, a select box or a code editor, so **Delete**, the arrow keys and letters edit text there instead of the canvas. - Shortcuts only act when focus is inside the builder or on nothing in particular. Inside Airflow, keys pressed in Airflow's own UI and in open dialogs or menus keep their normal behaviour. - A shortcut only runs when its command is currently available. For example **Ctrl+G** does nothing without a selection, and **Ctrl+Shift+A** does nothing when AI is disabled. - When the builder handles a key it prevents the browser's default action (for example the browser's bookmark dialog for **Ctrl+D**). - Symbols such as **?** are typed with **Shift** on most keyboard layouts; the dispatcher accepts them with or without **Shift**. - Copy, cut and paste go through the browser's native clipboard events, so they work without clipboard permissions. ## Global shortcuts Each table heading below is the group the command has in the shortcuts sheet and the command palette. ### General | Command | macOS | Windows / Linux | Notes | |---|---|---|---| | Command palette | ⌘K | Ctrl+K | Opens the palette. Not listed inside the palette itself. | | Insert node… | ⇧A | Shift+A | Opens the command palette in insert mode. | | Keyboard shortcuts | ? | ? | Opens the shortcuts sheet. | | Toggle palette | ⌘B | Ctrl+B | Shows or hides the operator palette. | | Toggle inspector | ⌘I | Ctrl+I | Shows or hides the inspector. | | Toggle bottom panel | ⌘J | Ctrl+J | Shows or hides the bottom panel. | | Toggle light / dark mode | (none) | (none) | Command palette only; standalone UI only. | ### Edit | Command | macOS | Windows / Linux | Notes | |---|---|---|---| | Undo | ⌘Z | Ctrl+Z | Repeats while held. In a collaboration session, undoes your own edits. | | Redo | ⌘⇧Z or ⌘Y | Ctrl+Shift+Z or Ctrl+Y | Repeats while held. | | Copy | ⌘C | Ctrl+C | Copies the selected nodes. Ignored when text is selected on the page. | | Cut | ⌘X | Ctrl+X | Copies, then deletes the selected nodes. | | Paste | ⌘V | Ctrl+V | Pastes under the mouse pointer when it is over the canvas. | | Duplicate | ⌘D | Ctrl+D | Duplicates the selection, offset by 40 px. | | Delete selection | Del or ⌫ | Delete or Backspace | Deletes the selected nodes and dependencies. | | Select all | ⌘A | Ctrl+A | Selects every visible node and dependency. | | Clear selection | Esc | Esc | Not listed in the command palette. | ### Canvas | Command | macOS | Windows / Linux | Notes | |---|---|---|---| | Group selection | ⌘G | Ctrl+G | Wraps the selected nodes in a new task group. | | Ungroup | ⌘⇧G | Ctrl+Shift+G | Dissolves the selected task groups. | | Collapse / expand group | ⌥C | Alt+C | Toggles the selected task groups. | | Add note | ⇧N | Shift+N | At the mouse pointer when it is over the canvas. | | Auto layout | ⇧L | Shift+L | | | Fit view | ⇧F | Shift+F | | | Toggle snap to grid | ⇧S | Shift+S | 16 px grid. | | Toggle minimap | ⇧M | Shift+M | | | Move selection left / right / up / down | ← → ↑ ↓ | ← → ↑ ↓ | 1 px per press, repeats while held. Not listed in the command palette. | | Move selection (large step) | ⇧ + arrow | Shift + arrow | 16 px per press. Not listed in the command palette. | ### File | Command | macOS | Windows / Linux | Notes | |---|---|---|---| | Open… | ⌘O | Ctrl+O | Opens the **Open** dialog. | | Deploy | ⌘S | Ctrl+S | Deploys the DAG (or opens a pull request when that is the default). See [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy). | ### AI | Command | macOS | Windows / Linux | Notes | |---|---|---|---| | Ask AI (chat) | ⌘⇧A | Ctrl+Shift+A | Opens or closes the AI chat panel. Needs AI enabled and the `ai:use` permission. See [AI assist](https://ctolon.github.io/autoair/docs/ai/assist). | ## Shortcuts in specific places These keys are handled by a particular control, not by the global dispatcher, and are not listed in the shortcuts sheet. | Where | Key (macOS / Windows, Linux) | Action | |---|---|---| | Palette search box | ↑ / ↓, Home / End | Move through the list. | | Palette search box | Enter | Insert the highlighted item. | | Palette search box | ⌥S / Alt+S | Add or remove the highlighted item from **Favorites**. | | Command palette, insert popover | ↑ / ↓, Enter, Esc | Navigate, run or insert, close. | | Canvas node or dependency with keyboard focus | Enter or Space | Select it. Tab moves focus between nodes. | | Sticky note being edited | ⌘Enter / Ctrl+Enter | Save the note. | | Sticky note being edited | Esc | Discard the edit. | | Dependency label being edited | Enter / Esc | Save / cancel. | | Code mode editor | ⌘Enter / Ctrl+Enter | **Apply to canvas**. See [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode). | | Markdown editor | ⌘B / Ctrl+B, ⌘I / Ctrl+I, ⌘K / Ctrl+K | Bold, italic, link. These win over the global panel shortcuts while you type in the editor. | | Markdown editor | `/` | Block and inline snippets. See [Markdown editor](https://ctolon.github.io/autoair/docs/user-guide/editors#markdown-editor). | | Any code editor | Ctrl+Space | Open completion. | | Any multi-line code editor | ⌘F / Ctrl+F | Search in the editor. | | Any multi-line code editor | Tab | Indent (instead of moving focus). | | Code editor resize handle (focused) | ↑ / ↓ | Shrink or grow the editor by 40 px. Double-click resets the height. | | Panel resize line (focused) | Arrow keys, Home, End | Resize by 16 px; minimum; maximum. Double-click restores the default size. | | Definition peek (IDE language server) | Esc | Close the peek panel. | | Problems list entry (focused) | Enter or Space | Go to the problem. | | Deploy dialog | ⌘Enter / Ctrl+Enter | Submit. | | AI chat input | Enter / ⇧Enter, Shift+Enter | Send / new line. | | AI prompt dialogs | ⌘Enter / Ctrl+Enter | Submit the prompt. | ## Command palette [Screenshot: The command palette with a search typed, listing matching commands with their group and shortcut, then built-in nodes and operators] Open the command palette with **Ctrl+K** / **⌘K**, the **More** (`⋯`) menu, or **Command palette** in the empty-canvas actions. It searches three kinds of entries: | Group | Contents | Shown | |---|---|---| | **Commands** | Every available command that is not hidden, with its group and shortcut. | First while you type; last otherwise. | | **Favorites & recent**, **Built-in**, **Operator templates**, **Templates**, **Operators** | The same items as the palette panel. **Operators** only appears once you type (at most the best matches are listed); **Favorites & recent** only appears before you type. | Always. | | **Go to task** | Every task and task group on the canvas, searchable by task id, label, operator class or decorator. Selecting one selects it, expands collapsed groups around it and zooms to it. | Not in insert mode. | Ranking prefers an exact title, then a title prefix, then the start of a word in the title (camelCase counts: `dag run` finds `TriggerDagRunOperator`), then a substring, then matches in secondary text, then fuzzy matches. Commands get a small boost, so typing `dep` lists **Deploy** before operators that merely contain those letters. **Insert mode** opens with **Shift+A** or by double-clicking empty canvas. Its title is **Insert node** and it lists only insertable items; with a double-click, the node is placed where you clicked. ### Commands in the palette Commands appear only when they are available: most AI commands need a configured AI model and the right permissions, **Trigger…**, **Backfill…** and **Promote environment…** need a deployed DAG, **Test selected task…** needs a selected task, and so on. | Group | Commands | |---|---| | General | **Insert node…**, **Keyboard shortcuts**, **Toggle palette**, **Toggle inspector**, **Toggle bottom panel**, **Toggle light / dark mode** (standalone only). | | Edit | **Undo**, **Redo**, **Copy**, **Cut**, **Paste**, **Duplicate**, **Delete selection**, **Select all**. | | Canvas | **Group selection**, **Ungroup**, **Collapse / expand group**, **Add note**, **Auto layout**, **Fit view**, **Toggle snap to grid**, **Toggle minimap**. | | File | **New DAG**, **Open…**, **Deploy**, **Deploy with message…**, **New from template…**, **Save as template…**, **Save selection as fragment…**, **Save selected task as operator template…**, **Import DAG…**, **Python file (.py)**, **Airflow YAML DAG**, **dag-factory YAML**, **AutoAir spec (JSON)** (the last four export the DAG). | | Run | **Validate in Airflow**, **Trigger…**, **Backfill…**, **Test selected task…**, **Test whole DAG…**. | | Versions & Git | **History…**, **Deploy target…**, **Promote environment…**. | | AI | **Generate DAG…**, **Edit with AI…**, **Explain problems**, **Explain this DAG**, **Review DAG**, **Scan for leaks**, **Optimise parameters**, **Fix with AI…**, **Write task code…**, **Ask AutoAir…**, **AI writer…**, **My AI model…**, **AI knowledge base…** (AI administrators), **Write docs**, **Ask AI (chat)**, **Ask the admin assistant** (AI administrators), **Airflow upgrade assistant**. | The AI commands are described in [AI assist](https://ctolon.github.io/autoair/docs/ai/assist) and [AI writers](https://ctolon.github.io/autoair/docs/ai/writers). ## Shortcuts sheet [Screenshot: The Keyboard shortcuts dialog listing commands with their keycaps in General, Edit and Canvas groups] Press **?** or choose **Keyboard shortcuts** in the **More** (`⋯`) menu to open the sheet. It lists every registered command that has a shortcut, grouped as above, with keycaps for your platform (**⌘ ⇧ ⌥ ⌃** on macOS, **Ctrl Shift Alt** elsewhere). Hidden commands such as the nudge keys and **Clear selection** are included. The sheet lists a command even while it is unavailable (for example **Ungroup** with no group selected). **Ask AI (chat)** is only registered, and therefore only listed, when the AI feature (`features.ai`) is on. The footer repeats the two rules that matter most: shortcuts are ignored while typing in a field or code editor, and **Ctrl** is **⌘** on macOS. ## Related pages - [Canvas](https://ctolon.github.io/autoair/docs/user-guide/canvas) - [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode) - [Field editors](https://ctolon.github.io/autoair/docs/user-guide/editors) # Code mode Source: https://ctolon.github.io/autoair/docs/user-guide/code-mode AutoAir has two kinds of DAG documents, and both can be edited as Python: | Document | Source of truth | Edited as code | |---|---|---| | **Visual DAG** (spec-backed) | The DAG spec (`.autoair.json`). The DAG file is generated from it. | In code mode: you edit the generated file and apply it back into the spec. The canvas, the inspector and the next deploy reflect it. | | **Raw code DAG** | The DAG file you write. AutoAir stores it in the spec document (`raw_code`) and deploys it as written. | In the raw code editor. No generated code, no spec rules, no canvas. See [Raw code DAGs](#raw-code-dags). | Code you apply in code mode becomes the DAG's hand-written file: later deploys patch it instead of regenerating it (source policy **Auto**), and code mode then shows that file with your canvas changes patched in. See [Editing existing Python DAGs](https://ctolon.github.io/autoair/docs/user-guide/editing-python-dags). [Screenshot: Code mode: the generated DAG file in a full-height editor, with the draft status badge, Discard edits and Apply to canvas buttons, and the sync panel listing what applying will change] [Video: Editing the generated code and applying it to the canvas] ## Editor modes Administrators choose which editors a deployment offers: | Setting | Default | Effect | |---|---|---| | `features.visual_editor` (`AUTOAIR_FEATURES__VISUAL_EDITOR`) | `true` | The canvas editor. | | `features.code_editor` (`AUTOAIR_FEATURES__CODE_EDITOR`) | `true` | Direct code editing: the code mode of visual DAGs, the server endpoint that imports code (`POST /api/interchange/python/sync`), and raw code DAGs. | | `features.code_edit_inline` (`AUTOAIR_FEATURES__CODE_EDIT_INLINE`) | `false` | Makes the **Python** view of the bottom panel's **Code** tab editable, with the same apply behaviour as code mode. | | Visual editor | Code editor | Result | |---|---|---| | on | on | Both (the default). The toolbar shows the **Visual \| Code** toggle; visual DAGs open in **Visual**. Raw code DAGs open in the raw code editor. | | on | off | Visual only. No toggle and no code mode. **New raw code DAG** is hidden, raw code saves answer `404`, and an existing raw code DAG opens read-only with a notice (its **Deploy** button is disabled). | | off | on | Code only. No canvas, palette, insert palette, auto-layout or canvas commands. Visual DAGs open in code mode, raw code DAGs in the raw code editor. Templates and imports still create visual DAGs, which you then edit as code. | | off | off | Not allowed. Settings validation fails at start-up with `at least one of features.visual_editor and features.code_editor must be on`, and the admin panel refuses the change (`422`). | ```bash AUTOAIR_FEATURES__VISUAL_EDITOR=false AUTOAIR_FEATURES__CODE_EDITOR=true ``` Admins can also switch these flags at runtime in the admin panel. The UI re-reads `GET /api/meta` when the window regains focus and every minute. When the mode changes, the open DAG stays open in the editor that remains, a warning explains the change, and an unapplied code draft is kept. If a runtime change would switch both editors off, the configured pair applies instead. See [Feature flags](https://ctolon.github.io/autoair/docs/admin/feature-flags#switching-flags-at-runtime) and the [`features` settings](https://ctolon.github.io/autoair/docs/reference/configuration#features). ### The Visual | Code toggle When both editors are enabled, the toolbar shows a **Visual | Code** switch (**Editor mode**) for visual DAGs: - **Visual**: edit the DAG on the canvas. - **Code**: edit the generated DAG file as Python; applying imports it back into the canvas. An orange dot on **Code** means there is an unapplied code draft. On a narrow toolbar the switch shows icons only. Code mode takes the whole work area: the palette and the inspector are hidden, and the bottom panel stays available. The canvas stays loaded in the background, so your viewport and selection are unchanged when you switch back. Canvas shortcuts and the node insert palette do not act on the hidden canvas. ### Who can edit code Applying code runs it on the server to import it, so editing needs all of: - the `spec:write` permission; - the `code:execute` permission; - Python code allowed on the server (`allow_python_code` and `features.python_code`). Without them, code mode shows the generated file read-only with a **read-only** badge. **Copy** and **Download** are always available. See [IAM](https://ctolon.github.io/autoair/docs/admin/iam). ## How sync works 1. **The generated code.** While you have no draft, the editor shows the code generated from the current DAG. It updates as the DAG changes. If the DAG is temporarily invalid, the editor keeps the last generated code; if there is none yet it shows **Generating the code…** or the reason it cannot be generated. 2. **A draft starts.** Your first keystroke starts a draft. The status badge changes from **generated** to **edited**. Typing the text back to exactly the generated code ends the draft. 3. **Automatic check.** About 1.2 seconds after you stop typing, the draft is sent to the server and imported without changing anything (`POST /api/interchange/python/sync`). The badge shows **checking…**, then either **ready: +N −N ~N** (nodes added, removed and changed) or **Fix N problems to apply**. **Check now** re-runs the check. The panel on the right lists the problems and **Applying will change**, a summary of the differences against the current DAG. 4. **Apply.** **Apply to canvas** (or **Ctrl+Enter** / **⌘Enter** in the editor) replaces the DAG with the imported version as one undoable step. A notification reports **Code applied to the canvas** with the counts and an **Undo** action. The draft ends. 5. **Discard.** **Discard edits** asks for confirmation and returns to the generated code. The server import works in two stages: 1. The code is parsed. A syntax error is reported with its line and column and nothing is executed. 2. The file is imported with Airflow's `DagBag` in an isolated subprocess, with AutoAir's secrets removed from its environment. The resulting DAG objects are converted back into a spec. Import errors are reported at the line of the last traceback frame in your file. Limits that apply to the check: the source must be no larger than `interchange.max_source_chars` (default 2,000,000 characters) and the import must finish within `python_import_timeout` (default 90 seconds), otherwise the check fails with `import timed out after 90s`. If the file defines several DAGs, the one with the current DAG id is used and the others are ignored with a warning; if none matches, the first one is used, also with a warning. A file that defines no DAG fails with `no DAG is defined in the code`. Warnings do not block an apply; errors do. A failed request (network error, missing permission) shows **check failed** and the apply is refused with **Cannot apply the code**. ### Unapplied drafts A draft lives in the browser tab. Reloading or closing the page while a draft exists asks for confirmation; an unapplied draft is lost if you continue. ## What round-trips AutoAir matches the imported tasks and task groups with the current ones by task id and group id, and carries over everything the Python file cannot express. An apply therefore only changes what the code changed. **Kept from the current DAG** (never read from the code): canvas positions and sizes, canvas labels, node icons, operator template links, internal node ids, sticky notes, and the AutoAir-only DAG fields output mode, team and environment overlays. **Read from the code:** | Area | Imported | |---|---| | Operators | Class path and the constructor arguments the code passes. Arguments left at their default, and values that only come from `default_args`, are not added (a plain `BashOperator` gets no `skip_on_exit_code`). Values can be JSON, `timedelta`, `datetime`, XCom references (`XComArg`, `task.output`, `task.output["key"]`) and scripts written in the builder (see below). | | TaskFlow tasks | `@task` and the decorators `branch`, `short_circuit`, `virtualenv`, `external_python`, `branch_virtualenv`, `branch_external_python`, `sensor`, `bash`, `docker` and `kubernetes`: the function name, body (docstring included), bound arguments, unbound parameters and `multiple_outputs`. | | Common task settings | `retries`, `pool`, `pool_slots`, `queue`, `owner`, `doc_md`, `priority_weight`, `weight_rule`, `task_display_name`, `max_active_tis_per_dagrun`, `depends_on_past`, `trigger_rule`, setup and teardown, asset outlets. Values equal to the DAG's `default_args` or to Airflow's defaults are left unset. | | Structure | Task groups (with `tooltip` and `prefix_group_id`, nested groups) and dependencies. | | Dynamic task mapping | `.partial(...).expand(...)` and `.partial(...).expand_kwargs(...)` of operators and TaskFlow tasks: the task keeps its operator (not `MappedOperator`), the `partial()` arguments become its parameters and the `expand` arguments (lists or XCom references) or the `expand_kwargs` input become its mapping, with `map_index_template`. | | Schedule | `None`, presets (`@hourly`, `@daily`, `@weekly`, `@monthly`, `@quarterly`, `@yearly`; `@annually` becomes `@yearly`, also as `CronTriggerTimetable("@daily", timezone=…)` in the DAG time zone), `@once`, `@continuous`, cron (`CronTriggerTimetable` and `CronDataIntervalTimetable`, with time zone), intervals (`DeltaTriggerTimetable`, `DeltaDataIntervalTimetable`), `MultipleCronTriggerTimetable`, and asset conditions (assets, aliases, `&` / `\|` combinations). | | DAG settings | `dag_id`, display name, description, `doc_md`, `catchup`, tags, `max_active_runs`, `max_active_tasks`, time zone, start and end dates, params, and `default_args` (`owner`, `retries`, `depends_on_past`, `email`, `email_on_failure`, `pool`, `queue`, `retry_delay`, `execution_timeout`). | ## What does not round-trip The importer reports what it drops as warnings in the sync panel. Check them before you apply. | Construct in the code | Result after apply | |---|---| | Mapping over transformed XCom (`.map()`, `.zip()`, `.concat()`) | The task is imported without mapping, with a warning naming the argument (`mapped argument '…' cannot be imported; the task is imported without mapping (re-create the mapping in the builder)`). Its `partial()` arguments are kept. | | Callables passed to operators (`python_callable=my_fn`, `on_failure_callback=…`) | Dropped with a warning (`callable parameter '…' dropped`). Python references and inline functions written in the builder are therefore lost if you apply the generated code back. Re-enter them in the inspector. | | Values of other types | Dropped with a warning (`value of type … dropped`). | | Timetables not listed above (events, partitioned, asset-and-time, asset-or-time, custom) | Imported without a schedule (warning: `timetable … is not supported; imported without schedule`). | | DAG callbacks, deadline alerts | Not imported (warnings `DAG callbacks are not imported`, `deadline alerts are not imported`). | | Task-level `retry_delay`, `execution_timeout`, `retry_exponential_backoff`, callbacks, retry policies, `executor`, `executor_config`, inlets, OpenLineage switches | Not read from the code: the task's values are reset. | | Decorator arguments (`@task.virtualenv(requirements=…)`, `@task.docker(image=…)`) | Not read from the code. | | Dependency labels (`Label(...)`) | Not read from the code: labels are removed. | | Task loops, `@result`, `@task.stub`, the `loop` context | Not read from the code. | | DAG fields `dagrun_timeout`, `is_paused_upon_creation`, `fail_fast`, `max_consecutive_failed_dag_runs`, `owner_links`, `render_template_as_native_obj`, `rerun_with_latest_version` | Not read from the code. | | DAG params | Only `type`, `title`, `description`, `description_md`, `section`, `format`, `enum`, `values_display`, `examples`, `minimum` and `maximum` are kept; length limits, `pattern`, `items` and `const` are dropped. | | Extension (custom) nodes | Imported as the operators their handler generated. | | Code outside the DAG (helper functions, module constants, comments) | Not kept. Only what ends up in the DAG objects is imported. | Two builder-only constructs are recognised in the generated code and restored, so applying the generated code again changes nothing: - **Scripts written in the builder** (a script file used as a parameter value, for example a shell script for `bash_command`): the generated DAG writes the script to a content-addressed temp file and passes its path. Code mode recognises that path and restores the script (name, content and interpreter), not the path. A path that does not point to a script written by the generated DAG stays a plain string. - **Sticky notes with Include in DAG docs**: codegen appends the note text to `doc_md`. The notes stay on the canvas, so their text is removed from the imported `doc_md` and is not duplicated on the next apply. A successful apply is one undoable step, so **Ctrl+Z** / **⌘Z** restores the previous DAG completely if the result is not what you wanted. **Warning:** Do not use code mode as a lossless converter. A successful apply means the code imported cleanly, not that every builder setting survived. Review the **Applying will change** summary and the warnings, then the canvas and the Problems tab, before you deploy. ## Conflicts A draft starts from the generated code of a moment in time. If the DAG changes in the visual editor afterwards (in another view, by an AI proposal or by a collaborator), code mode shows a **canvas changed** badge and the message: > The DAG was changed in the visual editor after you started editing the code. Applying replaces those changes; reload to start from the current code. You then have two options: - **Reload generated code** discards your draft and starts again from the current code. - **Apply to canvas** imports your draft as it is, which overwrites the newer visual changes for everything the code expresses. Undo restores them. Deploy conflicts (another editor deployed a newer version) are handled by the deploy flow. See [Deploy](https://ctolon.github.io/autoair/docs/user-guide/deploy). ## The Code tab The bottom panel's **Code** tab shows the generated output in four views: | View | Contents | |---|---| | **Python** | The generated DAG file (read-only), or editable with **Apply to canvas** when `features.code_edit_inline` is on and you can edit code. Problems from the generator are listed above the editor. The **Preview** selector shows the file of an environment, read-only (see [Environment preview](#environment-preview)). For a raw code DAG this view shows the raw code. | | **Airflow YAML** | The Airflow YAML DAG (labelled with the target Airflow version). It is editable: **Apply YAML to canvas** imports the YAML as one undoable step, keeping positions of matching nodes and laying out new ones; **Regenerate from canvas** discards the edit. | | **dag-factory** | A one-way export for teams using astronomer/dag-factory. | | **Spec JSON** | The document AutoAir stores and deploys. | Each view has **Copy** and **Download**. Code mode itself (the full-screen editor) also has **Copy** and **Download** for `.py`. Raw code DAGs have no Airflow YAML or dag-factory view. ## Environment preview The code you edit is always the base DAG. To see the file an environment gets, choose the environment in the **Preview** selector of the code mode header or of the **Code** tab's **Python** view (**Preview: base (editable)** is the code you edit). AutoAir renders the file with that environment's [overlay and connection and variable mapping](https://ctolon.github.io/autoair/docs/user-guide/environments#overlays) applied (`POST /api/codegen?environment=`) and shows it read-only, labelled **Preview: `` (read-only)**. The preview cannot be edited or applied. Applying code rendered for one environment would copy that environment's values into the base definition, and so into every environment. Edit the base code, or the environment's overlay in the DAG settings. Raw code DAGs have no preview selector: environment overlays and mapping do not change their file. See [Environments](https://ctolon.github.io/autoair/docs/user-guide/environments#raw-code-dags). ## Raw code DAGs A raw code DAG is a DAG file you write yourself, in free-form Python, independent of AutoAir's code generator and spec rules. It is a document type next to visual DAGs: stored, versioned, reviewed and deployed the same way. ### Create one - **File → New raw code DAG** (or **Write a raw code DAG** on an empty canvas) opens the raw code editor with a starter file that already uses the new DAG id. If you change the id, change it in the inspector's raw DAG settings and in the code. - **File → Import → Python file** has **Open as raw code DAG**: the file is kept as it is and nothing is executed. The DAG id is taken from the first literal `dag_id="…"` or `DAG("…")` in the file, else from the file name. - For a DAG that exists in Airflow but is not managed by AutoAir, the import dialog offers **Open as raw code DAG** with the original file. - In code mode of a visual DAG, **Convert to raw code…** turns it into a raw code DAG (see [Convert a visual DAG](#convert-a-visual-dag-to-raw-code)). - Agents use the MCP tool `save_raw_dag_code` (see [MCP](https://ctolon.github.io/autoair/docs/ai/mcp#tools)). The toolbar shows a **Raw code** badge next to the DAG id. In **File → Open…** every DAG carries a type badge (**Visual** or **Raw code**) and the list can be filtered by type. The inspector shows the raw DAG settings: DAG id, display name, description, tags and team. Schedule, default arguments, params and the other spec fields are written in the code instead. ### Who can use them Raw code is arbitrary Python, so saving a raw code DAG needs: - the `spec:write` permission, plus `deploy` when the save goes live, as for any DAG; - the `code:raw` permission (granted to the built-in Author and Admin roles; see [IAM](https://ctolon.github.io/autoair/docs/admin/iam)); - `features.code_editor` and `features.python_code` (and `allow_python_code`). While either feature is off, saving a raw code DAG (`PUT /api/specs/`, the deploy preview) answers `404`. Deploying an approved proposal and promoting between environments do not check `code:raw` again; the author was checked when the version was proposed or saved. Their validation still refuses raw code while Python code is off. ### Validation The live checks and every deploy validate a raw code DAG with these checks instead of the spec rules. Problems carry the line (and column) in your file: they are underlined in the editor and listed in the **Problems** tab, where clicking one moves the cursor to its line. | Check | Severity | |---|---| | Syntax (`ast`). A syntax error stops the other checks. | error | | ruff with the editor's lint rules (`editor.python_lint_select`, `editor.python_lint_ignore` and their `_extra` lists). Undefined names and syntax-level rules (`F82x`, `E9`) are errors, other rules warnings. | error or warning | | Hard-coded secrets: lines matching `policy.secret_assignment_pattern`. | warning | | Python code allowed: `allow_python_code` and `features.python_code`. | error | | Privileged operators (`policy.privileged_operators`, for example `BashOperator`) while `allow_privileged_operators` is false, and `forbidden_operators`. Found by a static scan of imports and calls; Airflow re-exports with the same class name and `@task.bash` count too. | error | | DAG id rules: `dag_id_prefix`, `policy.dag_id_pattern`, the team prefix. | error | | The code does not define a DAG with the document's DAG id literally (`DAG("…")`, `DAG(dag_id="…")`, `@dag`). | warning | | OPA (`policy.opa_url`): the input carries the spec, including `raw_code`, and `context.document_type` is `raw`. | per OPA decision | | The DagBag import check (`import_check_on_deploy`, for users with `code:execute` while `iam.import_check_requires_code_execute` is on). It also confirms the file defines the DAG. Runs only when no other error was found. | error | Spec rules do not apply: graph and catalog checks, operator templates, Jinja checks and `policy.rules` (which address spec nodes and fields). `require_owner`, `require_description`, `min_retries`, `required_tags` and `max_tasks` are not checked either. The operator scan is static and best effort: code that builds class paths at run time escapes it, which is why raw code needs its own permission. ### Deploy, versions and reviews A raw code DAG goes through the same deploy pipeline as a visual DAG: the same deploy targets (local folder, object storage, git push or pull request), the per-deploy git token, the deploy context in commit messages and pull requests, four-eyes approvals and deploy proposals, history, versions and diff (the diff shows the change of `raw_code`), promotion, delete and the audit trail. The deployed `.py` is your code as written; AutoAir never reformats it. Deploy targets only overwrite files that carry AutoAir's marker, so when the first 400 characters of your code do not contain `Generated by AutoAir`, AutoAir prepends one line: ```python # Generated by AutoAir from the raw code DAG ''. Edit it in AutoAir. ``` Environment overlays and the connection and variable mapping do not apply: every environment gets the same file. Read environment-specific values at run time from Airflow Variables, Connections or environment variables. With collaboration on, the raw code is shared as text, so several people can type in the same file. ### Convert a visual DAG to raw code With both editors on, the code mode of a visual DAG has **Convert to raw code…** (needs `code:raw`). The conversion is one way: the canvas, the node settings and the sync between code and spec are dropped, and the current generated code, or your unapplied draft, becomes the raw file. The confirmation dialog says so before anything changes. The conversion is one undoable step until you deploy; after the deploy, the stored DAG is a raw code DAG. ### Open a raw code DAG in the visual builder When both editors are on, the raw code editor has **Open in visual builder**. It imports the file with the same Python importer as code mode (`POST /api/interchange/python/sync`, which needs `spec:write`, `code:execute` and `python_code`), then shows what the import produced and the changes for review. Accepting replaces the document with a visual DAG as one undoable step. The conversion is best effort: everything listed in [What does not round-trip](#what-does-not-round-trip) is lost, as is code outside the DAG objects (helper functions, module constants, comments). Review the warnings before you accept. ## Editor features in code mode The code mode editor is the same CodeMirror editor as the field editors: Python highlighting, completion, hover documentation and signature help for the whole module, search, and the language server when `features.ide_language_server` is on. Problems from the sync check are shown in the gutter and underlined at their line. See [Field editors](https://ctolon.github.io/autoair/docs/user-guide/editors). ## Related pages - [Environments](https://ctolon.github.io/autoair/docs/user-guide/environments) - [IAM](https://ctolon.github.io/autoair/docs/admin/iam) - [Import and export](https://ctolon.github.io/autoair/docs/user-guide/import-export) - [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation) - [Feature flags](https://ctolon.github.io/autoair/docs/admin/feature-flags) - [Spec format](https://ctolon.github.io/autoair/docs/extending/spec-format) # Field editors Source: https://ctolon.github.io/autoair/docs/user-guide/editors Every field that holds code or long text uses the same CodeMirror 6 editor: Python task bodies, inline functions and scripts, bash commands, SQL, YAML, JSON, Jinja templates, Markdown documentation, the code mode file and the file of a [raw code DAG](https://ctolon.github.io/autoair/docs/user-guide/code-mode#raw-code-dags). Which of the visual and code editors a deployment offers is described in [Editor modes](https://ctolon.github.io/autoair/docs/user-guide/code-mode#editor-modes). ## Editor basics | Feature | Details | |---|---| | Languages | Python, bash, SQL, YAML, JSON, Markdown, Jinja, Go, Java, JavaScript, TypeScript and plain text. Script fields pick the language from the file extension (`.py`, `.sh`, `.sql`, `.hql`, `.yaml`, `.json`, `.md`, `.go`, `.java`, `.js`, `.ts`). | | Jinja | `{{ … }}`, `{% … %}` and `{# … #}` are highlighted inside any language, and templated fields get Airflow-aware completion. See [Jinja, XCom and Python](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python). | | Resizing | Drag the handle under a multi-line editor, or focus it and press **↑** / **↓** (40 px per step). Double-click the handle to reset the height. | | Large editor | The **Open … in a large editor** button in the top-right corner opens the field in a full-window dialog. Templated fields show the **Insert** menu and the rendered preview there. | | Search, indentation | **Ctrl+F** / **⌘F** searches inside multi-line editors. **Tab** and **Enter** indent with the indentation the text already uses (spaces or tabs, detected from the first indented lines). Without indented lines yet, Python and Java use 4 spaces, Go a tab, and other languages 2 spaces. This applies to task bodies, inline functions, scripts and the code mode file. | | Paste | A task body pasted with its function's indentation is dedented automatically. | | Loading | The editor's code is loaded the first time a code field appears; until then the field is a plain text box. You can start typing straight away: the text, the cursor position and the focus carry over to the editor when it arrives. | | Theme | The editor follows light and dark mode. | ## IntelliSense (built in) Python and bash editors have language services that need no extra installation. This is the default; it is what you get while `features.ide_language_server` is off or the language server is not installed. ### Python Completion, hover documentation and signature help are computed on the AutoAir server with [jedi](https://jedi.readthedocs.io/). The analysis is static (nothing is executed) and runs against the server's Python environment, where Airflow and the provider packages are installed, so `airflow.sdk`, hooks and operators complete with their real documentation. - **Task bodies.** The body you write is analysed wrapped in the same function signature the code generator emits. Function arguments, bound arguments, mapped arguments, extra parameters and the `loop` context of a task loop resolve, and so do the names of Airflow's task context, with their types: `ti`, `task_instance`, `dag_run`, `params`, `ds`, `ds_nodash`, `ts`, `ts_nodash`, `ts_nodash_with_tz`, `logical_date`, `data_interval_start`, `data_interval_end`, `prev_data_interval_start_success`, `prev_data_interval_end_success`, `prev_start_date_success`, `prev_end_date_success`, `run_id`, `task_instance_key_str`, `try_number`, `test_mode`, `templates_dict`, `outlet_events`, `triggering_asset_events`, `inlets`, `outlets`, `conn` and `var`. - **Whole modules.** Inline functions, embedded `.py` scripts, the code mode file and raw code DAG files are analysed as complete modules. - **Completion** appears as you type an identifier or after a dot; **Ctrl+Space** opens it explicitly. Items keep the server's ranking: keywords and prefix matches first. - **Hover** over an identifier shows its signature and docstring. - **Signature help** appears while you type the arguments of a call, with the current parameter highlighted. Diagnostics come from server validation rather than from the editor: syntax errors and ruff findings in a task body appear in the Problems panel and are underlined in the editor at their line. See [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation#python-lint). Limits are set in the [`editor` settings](https://ctolon.github.io/autoair/docs/reference/configuration#editor): `editor.max_source_chars` (largest code accepted, default 200,000 characters), `editor.max_completions` (items per request, default 100) and `editor.max_doc_chars` (docstring characters shown, default 4,000). ### Bash Bash editors (for example `bash_command` and `@task.bash` bodies) offer completion for shell keywords, builtins, common command-line tools, Airflow's task environment variables, the variables the script itself declares, and control-flow snippets. Diagnostics come from [ShellCheck](https://www.shellcheck.net/) on the server when it is installed (`pip install 'autoair[shell]'`); without ShellCheck the editor shows none. Jinja expressions are masked before ShellCheck runs. ## IDE language server With `features.ide_language_server` on, Python and bash editors talk to a real language server running on the AutoAir host. It replaces the built-in services and adds: - completion with resolved documentation, hover and signature help; - live diagnostics (for example an undefined name), mapped back into the task body; - **F12**: go to definition. Definitions in library files (Airflow, providers, the standard library) open read-only in a peek panel; **Escape** closes it; - **Shift+F12**: find references; - **F2**: rename; - **Shift+Alt+F**: format. Task bodies are analysed wrapped in their generated function, like the built-in services, so arguments and Airflow's context resolve. ### Servers | Language | Server | Install | Setting | |---|---|---|---| | Python | basedpyright (default) | `pip install 'autoair[lsp-pyright]'` (or `pip install basedpyright`; it bundles Node.js) | `lsp.python_server=basedpyright`, command `lsp.basedpyright_command` (default `["basedpyright-langserver", "--stdio"]`) | | Python | python-lsp-server with ruff | `pip install 'autoair[lsp]'` | `lsp.python_server=pylsp`, command `lsp.pylsp_command` (default `["pylsp"]`) | | Bash | bash-language-server | `npm i -g bash-language-server` | `lsp.bash_command` (default `["bash-language-server", "start"]`) | The commands are looked up on `PATH` and next to the running Python interpreter. When a server is not installed, `GET /api/lsp/servers` reports it unavailable and the editor keeps the built-in services for that language; nothing fails. ### Configuration ```bash AUTOAIR_FEATURES__IDE_LANGUAGE_SERVER=true AUTOAIR_LSP__PYTHON_SERVER=basedpyright # or pylsp AUTOAIR_LSP__MAX_SESSIONS=16 # language-server processes alive at once AUTOAIR_LSP__MAX_SESSIONS_PER_USER=4 AUTOAIR_LSP__IDLE_TIMEOUT=600 # seconds without a message before a session closes AUTOAIR_LSP__START_TIMEOUT=30 AUTOAIR_LSP__MEMORY_LIMIT_MB=2048 # RLIMIT_AS per process; empty for no limit AUTOAIR_LSP__CPU_LIMIT_SECONDS=3600 # RLIMIT_CPU per process; empty for no limit AUTOAIR_LSP__MAX_MESSAGE_BYTES=8388608 ``` Each browser page opens one WebSocket per language at `/api/lsp/ws/{language}`, authorised by a short-lived ticket from `POST /api/lsp/session`. The server starts one language-server process per session in a fresh temporary workspace and kills it, with its process group, when the WebSocket closes, after `lsp.idle_timeout`, or when it exceeds its resource limits. The editor reconnects when you type again after an idle timeout. **Note:** - A reverse proxy in front of AutoAir must forward WebSocket upgrades for `/api/lsp/ws/`. - Tickets are signed with the collaboration secret. When standalone runs several replicas, set `AUTOAIR_COLLAB_SECRET` to the same value on all of them. See [Scaling](https://ctolon.github.io/autoair/docs/admin/scaling). - Each session is a real process. Size `lsp.max_sessions` and the memory limit to the host. See the [`lsp` settings](https://ctolon.github.io/autoair/docs/reference/configuration#lsp). ## Validate button [Screenshot: A bash_command field after Validate: the unterminated if is underlined in red and the line under the editor reads 1 error, Checked with bash -n, shellcheck] Multi-line code fields have a **Validate** button (shield icon) next to the large-editor button; the large editor and the Markdown toolbar show it as a labelled **Validate** button. It sends the field's code to `POST /api/lsp/validate`, which checks it with the best tool installed on the server. Nothing is executed. | Language | Checks | |---|---| | Python | `ast` syntax check, then ruff with the editor's lint rules (`editor.python_lint_select`, `editor.python_lint_ignore` and their `_extra` lists). Task bodies are checked wrapped in a function. | | JSON | Strict parse with line and column. | | YAML | `yaml.safe_load_all`. | | Bash, sh | `bash -n` / `sh -n` syntax check, then ShellCheck (Jinja masked). | | SQL | sqlglot, when installed (`pip install sqlglot`). | | Go | `gofmt -e`. | | Java | `javac`, stopped after parsing, in a temporary directory. | | JavaScript, TypeScript | `tsc` syntax diagnostics; `esbuild` when `tsc` is missing. | | Markdown | Built-in checks (unclosed code fences). | Jinja-only and plain-text fields have no validator and no button. The result appears on one line under the editor and as underlines and gutter markers in the editor: - **No problems found**, or **N errors · M warnings**; - **Checked with …**: the tools that ran; - **Not available: …**: tools that are not installed on the server or failed, with the reason in the tooltip (for example `shellcheck is not installed on the server`, or `pip install sqlglot to validate SQL`); - **No validator is installed on the server for this language** when no tool could check the code. This is not a pass. The result belongs to the text it was computed for. As soon as you edit the field it disappears until you validate again. A request failure shows **Validation failed: …**. [Screenshot: The large editor dialog with a labelled Validate button and the validation result under the editor] External tools run as subprocesses in a temporary directory, with AutoAir's secrets removed from the environment and a timeout of `editor.validate_timeout` seconds (default 15). To use a tool that is not on `PATH`, set its command explicitly: ```bash AUTOAIR_EDITOR__VALIDATE_COMMANDS='{"tsc": ["/opt/node/bin/tsc"], "javac": ["/usr/lib/jvm/java-21/bin/javac"]}' ``` The **Validate** button checks one field in isolation. Whole-DAG validation is automatic; see [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation). ## Markdown editor Documentation fields use the Markdown editor: the DAG **Docs** tab (`doc_md`), the task **Docs** tab (`doc_md`), sticky-note text in the note inspector, the markdown description of DAG params, and the body of human-in-the-loop requests. [Screenshot: The DAG Docs tab with the Markdown editor: formatting toolbar, Edit, Split and Preview buttons and the editor] - **Toolbar**: heading, **Bold** (**Ctrl+B** / **⌘B**), **Italic** (**Ctrl+I** / **⌘I**), bulleted list, numbered list, quote, code, **Link** (**Ctrl+K** / **⌘K**) and table. While you type in the editor these keys format text instead of toggling panels or opening the command palette. - **Views**: **Edit**, **Split** (editor and live preview side by side) and **Preview**. The preview renders GitHub-flavoured Markdown. - **Full screen** opens the editor in a large dialog, in **Split** view by default. - **`/` snippets**: type `/` to insert blocks at the start of a line (`/h1`, `/h2`, `/h3`, `/list`, `/numbered`, `/todo`, `/python`, `/sql`, `/bash`, `/code`, `/table`, `/quote`, `/note`, `/warning`, `/hr`, and `/dagdoc`, an outline with Owner, Inputs, Outputs and On failure sections) or inline anywhere (`/link`, `/image`, `/bold`, `/italic`, `/inline-code`). - **Enter** continues lists and quotes. - **Validate** in the toolbar reports unclosed code fences. [Screenshot: The Markdown editor in Split view with the source on the left and the rendered preview on the right] [Screenshot: The Markdown editor in full screen with the formatting toolbar and the side-by-side preview] The DAG `doc_md` is shown on the DAG page in Airflow; a task's `doc_md` is shown in the task instance details. Notes with **Include in DAG docs** are appended to the DAG's `doc_md` when the code is generated. ## Related pages - [Jinja, XCom and Python](https://ctolon.github.io/autoair/docs/user-guide/jinja-xcom-python) - [Code mode](https://ctolon.github.io/autoair/docs/user-guide/code-mode) - [Validation](https://ctolon.github.io/autoair/docs/user-guide/validation) - [Configuration reference: `editor`](https://ctolon.github.io/autoair/docs/reference/configuration#editor) # Large DAGs Source: https://ctolon.github.io/autoair/docs/user-guide/large-dags The editor is tested with DAGs of 500 tasks (about 700 dependencies) and 1,000 tasks (about 1,400 dependencies). Both stay usable: panning, zooming and dragging run at 60 frames per second on a current laptop, and the live checks answer in well under a second. This page describes what changes when a DAG gets large, the measured numbers, and what to tune when your DAGs are bigger still. ## What changes on a large DAG All of these switch on by themselves; there is nothing to enable. | Behaviour | Applies above | Effect | |---|---|---| | Viewport rendering | 150 nodes | Only nodes and edges inside the visible area are rendered. Panning mounts the ones that come into view. | | Zoom out to an overview | 150 nodes | The canvas can zoom out to 5 % (instead of 50 %), so **Fit view** (**Shift+F**, the zoom controls or the context menu) shows the whole graph. | | Low-detail cards | zoom below 40 % | Task cards turn into plain boxes with the task name, tinted with the colour of their kind. Errors, warnings, run states, the version diff and the selection stay visible as border and fill colours. Dependencies are drawn without their label and **+** button. Zooming back in restores the full cards. | | Readable opening | 150 nodes | Opening a DAG, auto layout and resizing panels fit the view no further out than 50 %, centred on the graph, as before. Only **Fit view** zooms out to the overview. | | Slower live checks | 200 nodes | The [live checks](https://ctolon.github.io/autoair/docs/user-guide/validation#live-checks) wait 600 ms plus 1 ms per node above 200 after your last change (at most 2 s): 900 ms at 500 nodes, 1.4 s at 1,000. Typing in the inspector then sends one check per pause, not one per keystroke. | Undo and redo, copy and paste, collaboration, run-state overlays and the version diff overlay work the same on large DAGs. A drag is still one undo step. Collapsing task groups (the chevron in the group header) hides their members and is a quick way to work on one part of a large graph. ## Measured limits Numbers from the benchmark below on one developer machine (production build, headless Chrome, 1600×1000 window). Your numbers will differ; the ratios between rows are the useful part. | Step | 500 tasks | 1,000 tasks | |---|---:|---:| | Open the DAG until the first card shows | 0.7 s | 0.9 s | | Open until the first live check is back | 1.5 s | 2.2 s | | Server: validation, first time / again | 145 ms / 46 ms | 278 ms / 101 ms | | Server: code generation | 32 ms | 65 ms | | Auto layout (ELK, in a Web Worker) | 0.4–0.9 s | 0.9 s | | Open a DAG without positions (laid out on open) | 1.5 s | 2.2 s | | Pan and drag at a readable zoom | 60 fps | 60 fps | | Zoom out to the overview (mounts every card) | short pauses, 0.2 s in total | short pauses, 0.4 s in total | | Browser memory (JS heap) after opening | 27 MB | 30 MB | "Again" is a later check of the same DAG: unchanged Python task bodies and shell scripts are not linted a second time (see below), so live checks while you edit cost the second number. Beyond 1,000 tasks the editor keeps working, with costs that grow linearly: the server validates 2,000 tasks in about half a second the first time. The overview is the slowest view because it mounts every card; on very large graphs stay zoomed in or collapse task groups. The Airflow side has its own limits: a DAG file with thousands of tasks takes longer to parse in the DAG processor and to render in the Airflow UI. ## How validation scales Validation lints the body of every Python task with ruff and every Bash command or shell script with ShellCheck. On a large DAG they run in batches over all bodies that changed since the last check (ruff: one process; ShellCheck: one process per 50 scripts), and the results of unchanged bodies are remembered by the server process. A run that fails or times out reports no findings for that validation and is retried on the next one. Before 0.2.0 each task started its own process, which took seconds on DAGs with hundreds of tasks. ## Tuning | Setting | Default | When to change it | |---|---|---| | `AUTOAIR_EDITOR__PYTHON_LINT_TIMEOUT` | `15` | Seconds one ruff run may take. A first validation of thousands of Python tasks on a slow machine may need more; when it runs out, the lint findings are missing from that validation (nothing fails). | | `AUTOAIR_EDITOR__SHELLCHECK_TIMEOUT` | `5` | The same for one ShellCheck run (up to 50 scripts), which is slower than ruff per script. | | `AUTOAIR_EDITOR__SHELLCHECK_IN_VALIDATION` | `true` | `false` skips ShellCheck in validation entirely (the code editor still lints the script you are editing). | | `AUTOAIR_SERVER__MAX_BODY_BYTES` | `16777216` | Largest request body. A 1,000-task spec is about 0.4 MB, so only very large or code-heavy DAGs get near it. | | `AUTOAIR_IMPORT_CHECK_TIMEOUT` | `60` | Seconds of the Airflow import check (**Validate in Airflow**), which parses the generated file with Airflow. | | `AUTOAIR_MAX_TASKS` | none | Organisation policy: refuse DAGs above a task count. | The full list is in the [configuration reference](https://ctolon.github.io/autoair/docs/reference/configuration). ## Benchmarks Two scripts measure large DAGs on your own machine. Both use the same synthetic DAGs: stages of twelve tasks in task groups, a mix of Bash, Python, empty, sensor and trigger tasks, mapped tasks over a literal list and over an upstream result, setup and teardown pairs, labelled dependencies and sticky notes. ```bash uv run python scripts/gen_large_dag.py --tasks 500 -o large.autoair.json # Server: parse, validate, code generation, YAML and dag-factory export uv run python scripts/bench_large_dag.py --tasks 500 --tasks 1000 uv run python scripts/bench_large_dag.py --catalog --import-check # as the API, plus Airflow # Browser: load, layout, frame rate while panning, zooming and dragging, memory eval "$(.claude/skills/run-e2e/scripts/private-stack.sh start)" cd frontend && PERF_SIZES=500,1000 npx playwright test e2e/large-graph.spec.ts --reporter=list ``` The browser benchmark writes its numbers to `.artifacts/perf/large-graph-