Architecture

Wit is not a monolith. It is four pure-ish functions in a line — six with the two optional passes — each with a named, inspectable output type. Source text goes in one end; a string comes out the other. Once you see the shape you know where to intervene: swap the data loader, re-theme the renderer, write a renderer of your own, or catch an error at the right boundary.

The pipeline

scanCloseless (multi-file only) lists the names whose uses need no close anywhere in the reference graph. parse lexes, demotes stray markers back to prose, and builds a Document AST. loadExternalData (optional) rewrites @load nodes into data. resolve binds every use to the definition or data it names and merges partials and cross-file references. expand evaluates — it binds each use's parameters against the definition's header, inlines definition bodies, substitutes ::name:: and ::body::, runs conditionals and iteration, and runs <% %> scripts. renderHtml and renderMarkdown walk the fully-substituted tree to a string.

source ──parse──▶ Document ──loadExternalData──▶ Document ──resolve──▶ ResolvedDocument
   str   lex → stray → parse   (optional, @witlang/runtime)    (@witlang/runtime)
         (@witlang/parser)                                            │
                                                                    expand
                                                               (@witlang/runtime,
                                                                runs <% %> scripts)
                                                                      ▼
   str ◀──renderHtml / renderMarkdown── ExpandedDocument
        (@witlang/render-html / -markdown)

Three things to read off the diagram. loadExternalData sits between parse and resolve. Scripts run at the tail of expand — not as their own stage. Rendering is a pure tree walk: no I/O, no configuration beyond mode and css.

Inside parse, the lexer recognises markers anywhere; the stray-token pass then turns every marker no open construct can use (an email's @, C#, a lone !!, (if any)) back into its source text, so prose never loses characters. Inspect both token streams with wit tokens and wit tokens --parser.

The stages

Order is scanCloseless → parse → loadExternalData → resolve → expand → render, exactly as the CLI does it. loadExternalData is optional: skip it and a document with no @load behaves identically. scanCloseless matters only when a file uses a definition from another file: without it, a chapter's closeless @sig defined in a shared file raises E_UNCLOSED_NODE.

Which package owns what

• parser — lexer, stray-token pass, parser, formatter, AST types, and errors.
• runtime — closeless scan, resolver, expander, data loader, and the script bridge.
• render-html and render-markdown — the two reference renderers.
• cli — the wit command that chains the stages, plus wit migrate for 0.2 files.
• vscode — the extension and language server, running the same stages in the editor.

Where tools stop early

Renderers consume any post-expand AST — they never re-parse. Tools that only need structure, such as the language server and a formatter, stop after parse or resolve; they don't need expansion. This is why the boundary between stages is a real interface, not an implementation detail.

The whole pipeline

The whole machine, wired by hand — this is compile() in packages/cli/src/pipeline.ts:

import { parse } from '@witlang/parser';                    // → Document
import { loadExternalData, resolve, expand, scanCloseless } from '@witlang/runtime';
import { renderHtml } from '@witlang/render-html';          // → string

const closeless = scanCloseless(rootPath, source);           // → Set<string>
const doc      = parse(source, rootPath, { closeless });     // parse(source, file?, options?)
const prepare  = (d) => loadExternalData(d, dataSrc);        // also run on referenced files
const resolved = resolve(prepare(doc), { rootPath, closeless, prepare });
const expanded = expand(resolved);                           // → ExpandedDocument (runs scripts too)
const html     = renderHtml(expanded, { mode: 'document' });

For a document that needs no external data and no cross-file references, the whole thing composes into one line:

renderHtml(expand(resolve(parse('Hello from a *def*.'))))
// → <article class="wit-doc"><p>Hello from a <strong>def</strong>.</p></article>

Watch the tree change shape

Each stage's output is a plain object you can inspect (wit tour prints it as a readable tree). A pipeline inspector over one source makes the passes concrete — log the block kinds, then the definitions and bindings the resolver found, then the block kinds that survive expansion:

const src = '#greeting\nHello from a def.\ngreeting#\n\n@greeting';

const doc      = parse(src, 'demo.wit');
const resolved = resolve(doc);
const expanded = expand(resolved);

doc.children.map(c => c.kind);          // ['nodeDef', 'nodeUse']
[...resolved.definitions.keys()];       // ['greeting']
resolved.bindings.size;                 // 1
expanded.children.map(c => c.kind);     // ['paragraph']   ← the def is gone

The definition disappears because expand inlined its body at the use site. That is the whole story of the middle of the pipeline, in four lines.

Resolution timing

Not everything resolves at the same moment — this three-tier model is worth knowing before you build on the runtime.

• Eager. Data definitions and conditional operands outside definition bodies are bound once by the resolver.
• Expansion-time. A use's parameters are bound when it is expanded: the expander re-splits the raw (…) text against the definition's declared parameters, then re-resolves the body, so its ::name:: values reflect that call's arguments.
• Snapshot. An iteration freezes its collection at loop entry; the body then reads from that frozen sequence.

Design notes worth knowing

• Immutability. The resolver never mutates the parser AST. Bindings live in a side table — ResolvedDocument.bindings: Map<NodeUse, Binding> — not on the nodes.
• Re-resolution. expand re-resolves uses by name rather than reading that side table, because inlined bodies produce fresh NodeUse clones the table never saw.
• Scripts are part of expand. The lh bridge is built and <% %> scripts run at the tail of expand, so renderers never encounter a script node.
• Inspectability. ResolvedDocument carries Map objects (definitions, dataDefs, bindings, partials), so log .keys() or .size rather than JSON.stringify to see inside them.

Structural guarantees

Source locations — file, line, column, and length — ride every node through every stage, so an error or a highlight always points at an exact place in the source. Output is deterministic, and the AST is a serializable, exhaustive union, which is what lets tools and renderers attach at any stage boundary.

Common mistakes

• Handing a renderer a ResolvedDocument. Definitions, (each), and (if) are still present; renderers expect an ExpandedDocument. TypeScript catches it — the types are distinct on purpose.
• Assuming a renderer re-parses. It consumes the expanded AST; it never touches source.
• Expecting loadExternalData to be required. It is a no-op for documents with no @load.
• Expecting expansion at parse time. Parsing produces a raw tree with references and logic intact; expansion is a separate, later stage.
• Expecting scripts to run at render time. They ran during expand; their effects are already baked into the tree.

See also

Parsing → the AST · AST reference · Resolve & expand · Rendering · Errors reference.