Skip to content

Architecture

Data flow

flowchart LR
    subgraph TUI["virtufin-tui (your laptop)"]
        DA["Dashboard: tabs from the navigation structure"]
        subgraph ST["Shared DataPump"]
            W["workers store"]
            C["connections store"]
            TR["triggers store"]
            SS["states store"]
            E["EventsStore"]
        end
        DA --> ST

        subgraph PAGES["Per-pane widgets (mounted one at a time)"]
            HP["HomePage"]
            IP["FeedsPage / OrderBookPage / IndicatorsPage / StrategyPage / ScenarioPage"]
            WP["WorkersPage / ConnectionsPage / ScriptsPage / TriggersPage / EventsPage"]
            CP["ConfigPage"]
            XP["Custom panes (~/.config/virtufin/tui/panes/)"]
        end
        WP --> W
        WP --> C
        WP --> TR
        WP --> E
    end

    subgraph BG["Background tasks (DataPump)"]
        P["Poller (2s): QueryState while States is shown; every list while the lifecycle stream is down"]
        S["Subscriber: Pubsub.Subscribe"]
        F["Lifecycle: Pubsub.Subscribe worker lifecycle, feed metrics, trigger changes"]
    end
    F --> C
    P --> W
    P --> C
    P --> TR
    P --> SS
    S --> E

    API["virtufin-api (cluster)<br/>Gateway.* / State.* / Pubsub.* RPCs<br/>(talks to Dapr pubsub + state)"]

    TUI -- "gRPC" --> BG
    BG -- "gRPC" --> API

Components

1. App (src/virtufin_tui/app.py)

The root App subclass. Holds a single TuiConfig (from contexts.toml), connects to the active context, and pushes the Dashboard screen. Also configures file-based logging (~/.local/state/virtufin/tui/tui.log) before anything else runs, since Textual owns the terminal in alt-screen mode -- anything written to stderr is invisible while the TUI is running.

2. Dashboard (src/virtufin_tui/screens/dashboard.py)

A Screen that owns the DataPump and renders a two-level tab bar: the top-level tabs and, for any tab with more than one entry, a sub-tab bar underneath. Only one pane widget is mounted at a time -- switching tabs unmounts the current one and mounts the next.

The tabs are the navigation structure in use (navigation.py; openspec navigation-specs, tui#52): the user's navigation.json, else the context's shared cfg.navigation (read at bind, followed through the state-change events), else domain-devkit's default, drawn by the TUI's profile (navigation.tabs: top tabs, sub-tabs, deeper levels as Group › Leaf sub-tabs). A view opens a built-in page, a custom pane or a placeholder; a page a SharedPageView (screens/page_view.py, validated by page_specs.py). See Tabs, pages & custom panes. right/left cycle top-level tabs; 1-9 and 0 pick a sub-tab by position within the active tab (0 the tenth).

Dashboard also forwards a small set of generic keys (a, d, m, x, h, enter, space, /, T, c, C, s, p, w, z, l, t, ctrl+l, P), one meaning per key on every page, to whichever action_* method the currently-mounted pane implements. Each is a page(...) binding naming its candidate methods (see Keybindings for the full mapping); check_action hides a key from the footer entirely if the current pane implements none of them. There is no refresh key (tui#37): every pane updates itself from events; r is Run on Scripts only.

Liveness is pushed, not probed (tui#33). Each configured context gets a liveness.LivenessWatch: its own small gRPC channel with HTTP/2 keepalive (a ping every liveness_keepalive_seconds, default 20, dead after 10 s without its answer; 0 turns liveness off) and one grpc.health.v1.Health/Watch stream on it. The api pushes its serving status when it changes, and a broken stream (api pod gone, ingress refusing, network down) is down with its reason. Health checks need no API key, so each time a context comes up one ListServices with its key checks the key (a bad key is down with Invalid or missing API key). A down context is reconnected by the channel itself; nothing is called on a timer. The Dashboard matches the watchers to the configured contexts every 2 s (local only, no RPC).

3. Data store (src/virtufin_tui/store.py)

DataPump runs three background tasks against the active TuiClient and populates five reactive stores (Store, plus EventsStore for the stream):

Store Source Refresh
workers client.list_workers() once when the lifecycle stream is live, then once per burst of listing changes (registered, deleted, updated, loaded, load-failed, online, stopped, failed; WorkManager 0.41.7+)
connections feed_status.FeedTracker: client.list_feed_workers() (WebSocketFeed workers), each running feed's status once, then lifecycle and metrics events with the workers; feed state and counters pushed
triggers client.list_triggers() once when the stream is live, then once per burst of sys.api.trigger.scheduled / deleted events (api 0.15.12+)
states client.list_state_entries() 2s polling, only while the States pane is shown
events (EventsStore) Pubsub.Subscribe server-streaming Push, bounded deque (max 500)

Nothing is polled while the lifecycle stream is live (tui#34, tui#35): one subscription (_feed_loop) carries the worker lifecycle topic, the feeds' metrics and the trigger change events. Changes within 0.2 s are read once, and a change during a read reads again. The worker snapshot (_fetch_workers_raw, shared by every reader) stays valid until such an event or one of the TUI's own writes drops it. While the stream is down, the poll loop reads every list every 2 s, so no pane freezes; a store notifies its pages only when its value changed. Every list is read again on each resubscription, which also covers an event lost on the way (delivery is best effort).

The connections' feed state is not polled. The feed reports its state (connecting, connected, reconnecting, disconnected, degraded) through the WorkManager, which publishes it on sys.workmanager.worker.lifecycle (state=status, status, detail), the same topic as the worker transitions (online, stopped, failed, load-failed). A dedicated subscription to that topic applies them to the rows. A feed's full status (one InvokeWorkerCommand: URL, publish topic, counters) is fetched once: when it is first seen running, when it comes online, after every (re)subscription of the lifecycle stream (it waits for the stream's response headers, which the api sends once the subscription is live; events missed while the stream was down are covered by that fetch), and on demand (the Connections page opening, Enter). While the stream is down no status is fetched and the rows keep their last state; it is resubscribed with the same backoff as the event subscriber. The counters (Published, received, failures), the gauges Served and Subscribed (WebSocketFeed 0.5.7+, so a serve by anyone shows, tui#37) and the message rate come from the feeds' metrics events on wrk.websocketfeed.metrics (worker-devkit WorkerMetrics, pushed when they change), on the same subscription (tui#31); before a feed's first one they are its last status answer's.

The subscriber retries with exponential backoff (1s up to a 30s cap) on disconnect; events are not replayed on reconnect. The DataPump survives pane switches within a context; switching context (via the Config page) opens a fresh gRPC channel, checks the new context answers, starts a new pump on it and only then stops the old one (Dashboard.switch_context); a context that does not answer changes nothing.

FeedsPage, ScenarioPage and StrategyPage follow their registries (tui#36): every cfg.feed.*, cfg.scenario.* and cfg.strategy.* save and delete is announced (publish_change, by the TUI and the web app alike), and the pages reload on those sys.api.state.saved / deleted events (StorePage._follow) and when the workers store changes (atomic strategies); Feeds redraws its definitions' state from the connections store. No timer. The three share RegistryPage (screens/_registry_page.py): a page reads (_read) and shows (_show); a failed read keeps the rows and says they are stale.

OrderBookPage (tui#18) owns its data: it lists books with TuiClient.list_books (Pubsub.ListTopics read through the naming library: any topic of virtufin.market/orderbook), seeds from the saved book (get_book_state, service workmanager) and follows its own DataPump.stream (an EventStream: a live subscription to the book's changed and gap topics, raw events, released with the page). orderbook.py holds the book model (from the FlatBuffers image via virtufin.marketdata_devkit.orderbook, or from the saved JSON) and the ladder. New books join the selector by themselves (tui#37): the page follows sys.api.topic.catalog.added (api 0.15.14+, the topics new to the gateway's catalog) through StorePage._follow_new_topics and adds the books among them, keeping the one shown. Order Book, Portfolio, Indicators and Instruments share this through CatalogPage (screens/_catalog_page.py); the two with one picked ref (Order Book, Portfolio) through PickerPage, which also seeds the pick from its saved state and says when nothing produced it in QUIET_SEC.

IndicatorsPage (tui#19) works the same way: TuiClient.list_indicators (the changed topics, named through virtufin-indicator-devkit), get_indicator_values (one GetBulkState, service workmanager), its own DataPump.stream, and indicators.table_for, which picks the columns from the value's shape. New indicators and pairs arrive the same way as new books.

InstrumentsPage (tui#23) lists every instrument with TuiClient.list_instruments (the snapshot topics of virtufin.market/instrument), seeds the rows from get_instrument_states (one GetBulkState, service workmanager: BinanceOrderBook 0.12.0+ saves each tracked pair's instrument there, market-data-workers#32) and follows its own DataPump.stream of the snapshot topics. The State read is what fills it: a snapshot is published once per run and a subscription starts at the tail. New instruments arrive like new books.

The Dashboard builds a page on its first visit and keeps it (Dashboard._pages, hidden while another is shown, still following its stores), so a page keeps its filter, picked book, pause, tree view, cursor and log across tab switches; a context switch drops them (_drop_pages, released before removal). worker_history.py keeps the lifecycle events the DataPump's lifecycle subscription receives, per worker, for the worker detail view.

StatesPage and EventsPage show their entries as Values (tui#40, value-presentation spec): virtufin_tui.values decodes a state entry (Value.from_state) and an event (Value.from_event) one way and decides their kinds from the instance (values.kinds: the presentation fragments the devkits ship, composed by virtufin-domain-devkit's Presentation, the user's file matched first, loaded once per process; tui#41). virtufin_tui.filtering parses the / expression and matches Values (and, over its own fields, workers: the Workers page); screens/_filter_bar.py is the filter line the three pages use, and screens/_row_filter.py (RowFilterMixin) puts it over the rows of the table panes (a word in any cell, column:value in that column); name_tree.py builds the T tree. The shared conformance vectors ship with the fragments (each devkit's kind vectors; domain-devkit's generic kinds, filter, worker filter and name tree), and the TUI and the web app run every one from the installed packages. The Events log writes each event once: a store notification writes only the events received since the last write.

Both pages also keep a ValueStream per key or topic (tui#39): States appends each value read that differs from the key's newest, Events every event received (filter and pause aside). values.registry reads the composed values and streams lists (a user list replaces the shipped one of its kind) and gives a Value's or a stream's representation entries; values.representations implements the names as Textual widgets (plot.line with textual-plotext) and ValueView, which shows a key or topic with v/h/Esc and skips a representation that raises. The kind vectors (<fragment>.kinds.yaml) carry the expected value and stream lists too.

EventStoresPage (tui#20, tui#31) polls nothing either. Its EventStore workers come from the shared worker snapshot (TuiClient.list_eventstores). It follows its own DataPump.stream of wrk.eventstore.metrics (the stores' counts, pushed by worker-devkit's WorkerMetrics when they change) and the worker lifecycle topic (the stores' ok / degraded reports). It asks a store's status (rules, fallback table, last error) through TuiClient.worker_command once per activation, when its metrics show a new failure (the last error), and on an updated lifecycle event (new code or config). The pushed counts win over a status answer that arrives after them.

PortfolioPage (tui#24) polls nothing. Its accounts are the account.snapshot topics the gateway has seen (TuiClient.list_accounts, through virtufin-portfolio-devkit's names, names.account_of_topic), new ones joining from the topic catalog's announcements. The picked account is seeded from the account its portfolio worker saves at <world>.portfolio.<venue>.account.<account> (get_account_state, service workmanager), then follows its snapshot, balance.changed and openorder.changed events through its own DataPump.stream; portfolio.Account folds them.

PnLPage (tui#25) shows the book the scenario's P&L worker saves at sc.<S>.pnl.book (get_pnl_book, service workmanager; names from virtufin-portfolio-devkit's virtufin.positions fragment: names.pnl_book_key, names.pnl_topics). The scenarios come from the registry, followed through the cfg.scenario.* state changes. The worker's pnl.realized, pnl.unrealized and risk.exposure events carry no average cost or fees, so each one reads the book again (one read in flight, one more queued); pnl.py renders it.

RiskPage (tui#26) shows the P&L worker's risk.exposure.update (risk.from_exposure), seeded from its saved book with the registry entry's limits (risk.from_book, the same rules as the worker). l sets the limits (TuiClient.set_scenario_limits, a registry write announced on sys.api.state.saved, which the worker and the page both follow); save_scenario keeps them when the triplet is edited.

Instruments, Portfolio, P&L and Risk were placeholders without data (tui#22, PANE_ORDER_VERSION 6 dropped them from saved layouts once): 7 puts Instruments back before Order Book, 8 Portfolio after Indicators, 9 P&L and 10 Risk after it.

4. Config (src/virtufin_tui/config/)

One module per file the TUI reads or writes, all re-exported by the package: paths (the XDG locations, seeding the shipped defaults), user (config.toml and state.json), workers (workers.toml and source credentials), contexts (contexts.toml, the active context).

Two independent files, both XDG-aware ($XDG_CONFIG_HOME, default ~/.config):

  • virtufin/api/contexts.toml -- the list of virtufin-api endpoints ("contexts": host/port/TLS/API key/etc.) and which one is active. See Contexts.
  • virtufin/tui/config.toml -- everything else: theme, event-topic presets, the tab/pane layout (pane_order), and display toggles. Loaded into a TuiUserConfig dataclass and round-tripped through tomlkit on save, so comments and formatting you haven't touched survive. The TUI rewrites it only when the layout is edited on the Config page.
  • virtufin/tui/state.json (under $XDG_STATE_HOME) -- what the TUI sets as you use it: the active Scenario and event topics, the theme, the Events page's Wall/Msg time toggles, column widths. For the theme and the toggles, config.toml holds the default and the state the current value.

5. Client wrapper (src/virtufin_tui/client/)

A thin async wrapper (TuiClient) around virtufin.api.client.ApiClient exposing the operations the TUI needs as plain async methods (no Textual dependency) -- workers, connections, triggers, state, scenarios, feed definitions, strategies and the pubsub subscription. Backend RPCs go through Gateway.InvokeRaw with message classes built from the gateway's reflection descriptors.

TuiClient is put together from one part per backend area, each a mixin in its own module; everything is imported from virtufin_tui.client:

Module Part
_core.py ClientCore: the channel (connect, disconnect), the typed invoke (_invoke_unary, the descriptor pools), list_services, how gRPC and gateway failures read (describe_connection_error), retry_on_gateway_not_found
workers.py WorkersMixin: list (one shared snapshot), create, tag, start, stop, delete, worker_command; WorkerRow
feeds.py FeedsMixin: connections (WebSocketFeed workers), open_feed, serve_topics, feed_command; ConnectionRow
state.py StateMixin: state-store JSON entries, bulk reads, indexed registries; StateRow
pubsub.py PubsubMixin: the event streams, the topic catalog, order books and indicators
triggers.py TriggersMixin: scheduled triggers; TriggerRow
feed_definitions.py FeedDefinitionsMixin: the shared feed definitions (cfg.feed.*), connect and disconnect through market-data-devkit's orchestrator; SharedFeed
registries.py RegistriesMixin: strategies and scenarios

Connections are WebSocketFeed workers (general-workers, workmanager#103); the TUI no longer talks to a WebSocket service:

Operation WorkManager call
List (list_feed_workers, FeedTracker) ListWorkers, filtered to feed workers (the feed package, or an orchestrator's feedowner tag); InvokeWorkerCommand status once per running feed (feed_status), then lifecycle events (see "Data store")
Connect a feed definition (Feeds C, connect_feed_definition) market-data-devkit feeds.current (status of the owner's running feeds) → feeds.resolve (owner = the definition's name, its preset) → feeds.apply through the gateway's InvokeJson: serve {topics} on a running feed of the plan, else CreateWorker (the newest Virtufin.Worker.WebSocketFeed of the worker registry, the preset rendered for the world + topics + the definition's feed keys and autoreconnect, tags feedowner/name/source/world/universe, secrets by reference) + StartWorker; the owner's other feeds are deleted
Disconnect a feed definition (Feeds X) feeds.delete(owner): DeleteWorker for each feed tagged feedowner=<name>
Open a raw feed (open_feed) CreateWorker + StartWorker with a preset's config, then waits for its lifecycle status connected (wait_feed_connected); used by scripts and the examples
Serve topics (serve_topics) InvokeWorkerCommand serve {topics} (all or nothing)
Start / stop publish (raw feeds) InvokeWorkerCommand publish {topic} / unpublish
Tags GetTags / SetTag; clearing a tag rewrites the map with SetTags
Disconnect StopWorker + DeleteWorker

The feed subscribes its served topics' items and sends its subscribe frames after the login on every open (connect, reconnect, a move between Runtime pods), so the TUI never replays a subscription itself. A feed knows no consumer: workers subscribe to the topics, and nothing in the platform matches topics to feeds (the TUI, a notebook or a script decides which feed serves what: the shared feed definitions on the Feeds pane, market-data-devkit's feeds.resolve/apply underneath). Its state changes are lifecycle events on sys.workmanager.worker.lifecycle (state=status, with status and detail).

6. Per-pane widgets (src/virtufin_tui/screens/*.py)

Every pane is a plain textual.widget.Widget (not a Screen) following the same duck-typed convention: __init__(self, stores, client, **kwargs), compose(), optionally on_mount() and whichever action_* methods it wants Dashboard's generic keys routed to.

Tab Sub-tab (pane id) File
Home (single pane, no sub-tabs) home_page.py
Trading feeds / instruments / orderbook / indicators / portfolio / pnl / risk / strategy / scenario / ai / config feeds_page.py, instruments_page.py, orderbook_page.py, indicators_page.py, portfolio_page.py, pnl_page.py, risk_page.py, strategy_page.py, scenario_page.py, ai_page.py, config_page.py
Infra workers / connections / scripts / triggers / states / eventstores / alerts / changes / events / ai / config workers_page.py, connections_page.py, scripts_page.py, triggers_page.py, states_page.py, eventstores_page.py, alerts_page.py, changes_page.py, events_page.py, ai_page.py, config_page.py

Alerts (alerts.py, alert_watch.py): the App holds one AlertLog for every context. The Dashboard runs an AlertWatch for the bound context (worker lifecycle payloads through the DataPump's lifecycle listeners, and one breach topic per scenario, the pub/sub having no wildcards) and adds halt changes and context liveness changes; report_error adds every failed action. from_lifecycle, from_breach and from_halt are pure and pinned by domain-devkit's alerts/classify.yaml (openspec alerts), shared with the web app.

Changes (changes.py): every write of a TuiClient runs in its _write block, which asks first on a protected context and tells on_write the outcome (done, failed, declined); bootstrap.log_writes points that at the App's ChangeLog, which appends to changes.jsonl in the state directory.

The navigation view (the contexts table and your navigation) is ConfigPage, under the Config tab by default. Detail popovers (WorkerDetail, ConnectionDetail) and the action modal are pushed as modals on top of the active pane.

Shared pieces of the pages (tui#30):

  • _table.py: rebuild_rows replaces a table's rows and keeps the selected row (by key) and the horizontal scroll, which DataTable.clear() resets; selected_key reads the selection. Every list page rebuilds through it.
  • _modal.py: BaseModal, the base of every modal: Esc, a #btn-cancel or a #btn-back button close it with cancel_result() (None unless the modal says otherwise, e.g. the event-topic modals return the unchanged topics); FormModal adds the centred, bordered form box (60 cells wide, a modal changes what differs). A Select left empty holds Select.NULL, so the modals check the value is a str before they accept it.

Beyond the built-in pages, users can add their own panes and even new top-level tabs without touching the source tree -- see Custom panes.

7. Errors (src/virtufin_tui/errors.py)

Two kinds of failure, each shown one way (tui#28):

  • An action fails (create, delete, tag, publish, compose, ...): the page calls report_error(page, action, exc), which shows one error toast, <Action> failed: <reason>, and writes the traceback to tui.log. The reason is readable: a gRPC status as describe_connection_error puts it (e.g. Server unavailable), a call the backend refused as STATUS: message (the api client's Gateway invoke failed: STATUS - message), anything else as its own text. Input that is not valid (a tab id taken, a form field) stays a plain message.
  • Reading the data shown fails: the page keeps what it shows and says it is stale in its status line, stale since 12:03:04: <reason>, until the next good read. The stores (workers, connections, triggers, states) carry it (Store.fail, Store.stale_note); Feeds, Strategy and Scenario, which read their own data, keep a Staleness of their own. The log gets one traceback per failure, through repeat_log, not one per attempt.

Lifecycle

  1. virtufin-tui is invoked (from uvx, pip install, or local).
  2. app.main() configures file-based logging, then parses --config and --context flags.
  3. config.load_config() reads contexts.toml and applies env-var overrides (VIRTUFIN_TUI_API_HOST/_PORT/VIRTUFIN_TUI_CONTEXT).
  4. VirtufinTuiApp.on_mount() resolves the active context. If one resolves, it builds a TuiClient, connects, and pushes the Dashboard screen, which starts the DataPump and mounts the first tab's first pane after a short delay.
  5. If no context resolves -- no contexts.toml, or the stored active-context name isn't among the configured contexts -- the TUI still launches. It pushes the same Dashboard in its unbound state (no TuiClient, no DataPump polling) and lands on the Config page, which lists every configured context with a red "not connected" dot. A pane that needs a client (all but Home and Config, whose page classes set WORKS_UNBOUND) shows a NoContextPage stand-in until then. Pages derive from Page (screens/_page.py): the Dashboard hands each one the connection state through show_connection(Connection), and a page that changes the context row (the active Scenario) posts TradeSelectionChanged; neither touches the other's fields. Selecting one there (Enter) binds a real client and resumes the normal connected lifecycle. See Contexts.
  6. On q (quit), the TUI cancels background tasks, closes the gRPC channel, and exits 0.

Names

Every topic and state key the TUI builds comes from the naming library (lib.docs.virtufin.com/naming), through src/virtufin_tui/names.py:

Name Value Used for
cfg.scenario.<SCENARIO>, cfg.scenario.index scenario registry Scenario tab
cfg.feed.<NAME>, cfg.feed.index shared feed definitions (market-data-devkit feed_registry; the key upper-cased, the index as named) Feeds tab
cfg.strategy.<NAME>, cfg.strategy.index composite strategies Strategy tab
cfg.tui.startdate start date system_startdate.py init script
sc.<SCENARIO>.* scenario scope keys purged when a scenario is deleted
sys.api.state.saved, sys.api.state.deleted state changes default event topics
sys.workmanager.worker.lifecycle worker lifecycle event topic preset; the workers and connections stores
sys.api.trigger.scheduled, sys.api.trigger.deleted trigger changes (api's fragment) the triggers store
wrk.<type>.command.{worker_id} worker command topic New Worker form default; a feed's command topic (wrk.websocketfeed.command.{worker_id})
wrk.<type>.metrics a worker type's pushed counters Connections (wrk.websocketfeed.metrics), EventStores (wrk.eventstore.metrics)

Composite strategy names are uppercase identifiers ([A-Z0-9_], up to 32), like scenario ids; the compose dialog rejects anything else. Atomic strategies keep their tag values. Feed definition names are lower-case identifiers, as in the web app.

src/virtufin_tui/naming/ holds the TUI's own fragment (virtufin.tui, which defines cfg.tui.startdate). The others come from the installed packages that own them, through the virtufin.naming entry point: virtufin.worlds (virtufin-domain-devkit), virtufin.market (virtufin-marketdata-devkit) and the api and workmanager service fragments (virtufin-api, virtufin-workmanager).