Architecture¶
Data flow¶
flowchart LR
subgraph TUI["virtufin-tui (your laptop)"]
DA["Dashboard: tabs from the navigation structure"]
subgraph ST["Shared DataPump"]
W["workers store"]
C["connections store"]
TR["triggers store"]
SS["states store"]
E["EventsStore"]
end
DA --> ST
subgraph PAGES["Per-pane widgets (mounted one at a time)"]
HP["HomePage"]
IP["FeedsPage / OrderBookPage / IndicatorsPage / StrategyPage / ScenarioPage"]
WP["WorkersPage / ConnectionsPage / ScriptsPage / TriggersPage / EventsPage"]
CP["ConfigPage"]
XP["Custom panes (~/.config/virtufin/tui/panes/)"]
end
WP --> W
WP --> C
WP --> TR
WP --> E
end
subgraph BG["Background tasks (DataPump)"]
P["Poller (2s): QueryState while States is shown; every list while the lifecycle stream is down"]
S["Subscriber: Pubsub.Subscribe"]
F["Lifecycle: Pubsub.Subscribe worker lifecycle, feed metrics, trigger changes"]
end
F --> C
P --> W
P --> C
P --> TR
P --> SS
S --> E
API["virtufin-api (cluster)<br/>Gateway.* / State.* / Pubsub.* RPCs<br/>(talks to Dapr pubsub + state)"]
TUI -- "gRPC" --> BG
BG -- "gRPC" --> API
Components¶
1. App (src/virtufin_tui/app.py)¶
The root App subclass. Holds a single TuiConfig (from contexts.toml),
connects to the active context, and pushes the Dashboard screen. Also
configures file-based logging (~/.local/state/virtufin/tui/tui.log) before
anything else runs, since Textual owns the terminal in alt-screen mode --
anything written to stderr is invisible while the TUI is running.
2. Dashboard (src/virtufin_tui/screens/dashboard.py)¶
A Screen that owns the DataPump and renders a two-level tab bar:
the top-level tabs and, for any tab with more than one entry, a sub-tab bar
underneath. Only one pane widget is
mounted at a time -- switching tabs unmounts the current one and mounts
the next.
The tabs are the navigation structure in use (navigation.py;
openspec navigation-specs, tui#52): the user's navigation.json, else the
context's shared cfg.navigation (read at bind, followed through the
state-change events), else domain-devkit's default, drawn by the TUI's
profile (navigation.tabs: top tabs, sub-tabs, deeper levels as Group › Leaf
sub-tabs). A view opens a built-in page, a custom pane or a placeholder; a
page a SharedPageView (screens/page_view.py, validated by page_specs.py).
See Tabs, pages & custom panes. right/left cycle top-level tabs;
1-9 and 0 pick a sub-tab by position within the active tab (0 the tenth).
Dashboard also forwards a small set of generic keys (a, d, m, x, h,
enter, space, /, T, c, C, s, p, w, z, l, t, ctrl+l, P),
one meaning per key on every page, to whichever
action_* method the currently-mounted pane implements. Each is a
page(...) binding naming its candidate methods (see
Keybindings for the full mapping); check_action hides a
key from the footer entirely if the current pane implements none of them.
There is no refresh key (tui#37): every pane updates itself from events; r is
Run on Scripts only.
Liveness is pushed, not probed (tui#33). Each configured context gets a
liveness.LivenessWatch: its own small gRPC channel with HTTP/2 keepalive
(a ping every liveness_keepalive_seconds, default 20, dead after 10 s without
its answer; 0 turns liveness off) and one
grpc.health.v1.Health/Watch stream on it. The api pushes its serving status
when it changes, and a broken stream (api pod gone, ingress refusing, network
down) is down with its reason. Health checks need no API key, so each time a
context comes up one ListServices with its key checks the key (a bad key is
down with Invalid or missing API key). A down context is reconnected by the
channel itself; nothing is called on a timer. The Dashboard matches the
watchers to the configured contexts every 2 s (local only, no RPC).
3. Data store (src/virtufin_tui/store.py)¶
DataPump runs three background tasks against the active TuiClient and
populates five reactive stores (Store, plus EventsStore for the stream):
| Store | Source | Refresh |
|---|---|---|
workers |
client.list_workers() |
once when the lifecycle stream is live, then once per burst of listing changes (registered, deleted, updated, loaded, load-failed, online, stopped, failed; WorkManager 0.41.7+) |
connections |
feed_status.FeedTracker: client.list_feed_workers() (WebSocketFeed workers), each running feed's status once, then lifecycle and metrics events |
with the workers; feed state and counters pushed |
triggers |
client.list_triggers() |
once when the stream is live, then once per burst of sys.api.trigger.scheduled / deleted events (api 0.15.12+) |
states |
client.list_state_entries() |
2s polling, only while the States pane is shown |
events (EventsStore) |
Pubsub.Subscribe server-streaming |
Push, bounded deque (max 500) |
Nothing is polled while the lifecycle stream is live (tui#34, tui#35): one
subscription (_feed_loop) carries the worker lifecycle topic, the feeds'
metrics and the trigger change events. Changes within 0.2 s are read once, and
a change during a read reads again. The worker snapshot (_fetch_workers_raw,
shared by every reader) stays valid until such an event or one of the TUI's
own writes drops it. While the stream is down, the poll loop reads every list
every 2 s, so no pane freezes; a store notifies its pages only when its value
changed. Every list is read again on each resubscription, which also covers an
event lost on the way (delivery is best effort).
The connections' feed state is not polled. The feed reports its state
(connecting, connected, reconnecting, disconnected, degraded)
through the WorkManager, which publishes it on
sys.workmanager.worker.lifecycle (state=status, status, detail), the
same topic as the worker transitions (online, stopped, failed,
load-failed). A dedicated subscription to that topic applies them to the
rows. A feed's full status (one InvokeWorkerCommand: URL, publish topic,
counters) is fetched once: when it is first seen running, when it comes
online, after every (re)subscription of the lifecycle stream (it waits for
the stream's response headers, which the api sends once the subscription is
live; events missed while the stream was down are covered by that fetch),
and on demand (the Connections page opening, Enter). While the stream
is down no status is fetched and the rows keep their last state; it is
resubscribed with the same backoff as the event subscriber. The counters
(Published, received, failures), the gauges Served and Subscribed
(WebSocketFeed 0.5.7+, so a serve by anyone shows, tui#37) and the message
rate come from the feeds' metrics events on wrk.websocketfeed.metrics
(worker-devkit WorkerMetrics, pushed when they change), on the same
subscription (tui#31); before a feed's first one they are its last status
answer's.
The subscriber retries with exponential backoff (1s up to a 30s cap) on
disconnect; events are not replayed on reconnect. The DataPump
survives pane switches within a context; switching context (via the
Config page) opens a fresh gRPC channel, checks the new context answers,
starts a new pump on it and only then stops the old one
(Dashboard.switch_context); a context that does not answer changes nothing.
FeedsPage, ScenarioPage and StrategyPage follow their registries
(tui#36): every cfg.feed.*, cfg.scenario.* and cfg.strategy.* save and
delete is announced (publish_change, by the TUI and the web app alike), and
the pages reload on those sys.api.state.saved / deleted events
(StorePage._follow) and when the workers store changes (atomic strategies);
Feeds redraws its definitions' state from the connections store. No timer. The three share RegistryPage
(screens/_registry_page.py): a page reads (_read) and shows (_show); a
failed read keeps the rows and says they are stale.
OrderBookPage (tui#18) owns its data: it lists books with TuiClient.list_books
(Pubsub.ListTopics read through the naming library: any topic of
virtufin.market/orderbook), seeds from the saved book (get_book_state, service
workmanager) and follows its own DataPump.stream (an EventStream: a live
subscription to the book's changed and gap topics, raw events, released with
the page). orderbook.py holds the book model (from the FlatBuffers image via
virtufin.marketdata_devkit.orderbook, or from the saved JSON) and the ladder.
New books join the selector by themselves (tui#37): the page follows
sys.api.topic.catalog.added (api 0.15.14+, the topics new to the gateway's
catalog) through StorePage._follow_new_topics and adds the books among them,
keeping the one shown. Order Book, Portfolio, Indicators and Instruments share
this through CatalogPage (screens/_catalog_page.py); the two with one
picked ref (Order Book, Portfolio) through PickerPage, which also seeds the
pick from its saved state and says when nothing produced it in QUIET_SEC.
IndicatorsPage (tui#19) works the same way: TuiClient.list_indicators (the
changed topics, named through virtufin-indicator-devkit), get_indicator_values
(one GetBulkState, service workmanager), its own DataPump.stream, and
indicators.table_for, which picks the columns from the value's shape. New
indicators and pairs arrive the same way as new books.
InstrumentsPage (tui#23) lists every instrument with TuiClient.list_instruments
(the snapshot topics of virtufin.market/instrument), seeds the rows from
get_instrument_states (one GetBulkState, service workmanager: BinanceOrderBook
0.12.0+ saves each tracked pair's instrument there, market-data-workers#32) and follows
its own DataPump.stream of the snapshot topics. The State read is what fills it: a
snapshot is published once per run and a subscription starts at the tail. New
instruments arrive like new books.
The Dashboard builds a page on its first visit and keeps it (Dashboard._pages,
hidden while another is shown, still following its stores), so a page keeps its
filter, picked book, pause, tree view, cursor and log across tab switches; a context
switch drops them (_drop_pages, released before removal). worker_history.py keeps
the lifecycle events the DataPump's lifecycle subscription receives, per worker, for
the worker detail view.
StatesPage and EventsPage show their entries as Values (tui#40,
value-presentation spec): virtufin_tui.values decodes a state entry
(Value.from_state) and an event (Value.from_event) one way and decides their
kinds from the instance (values.kinds: the presentation fragments the devkits ship,
composed by virtufin-domain-devkit's Presentation, the user's file matched first,
loaded once per process; tui#41).
virtufin_tui.filtering parses the / expression and matches Values (and, over
its own fields, workers: the Workers page);
screens/_filter_bar.py is the filter line the three pages use, and
screens/_row_filter.py (RowFilterMixin) puts it over the rows of the table panes
(a word in any cell, column:value in that column); name_tree.py builds
the T tree. The shared conformance vectors ship with the fragments (each devkit's
kind vectors; domain-devkit's generic kinds, filter, worker filter and name tree), and
the TUI and the web app run every one from the installed packages. The Events log writes each event once: a store notification writes
only the events received since the last write.
Both pages also keep a ValueStream per key or topic (tui#39): States appends each
value read that differs from the key's newest, Events every event received
(filter and pause aside). values.registry reads the composed values and streams
lists (a user list replaces the shipped one of its kind) and
gives a Value's or a stream's representation entries; values.representations
implements the names as Textual widgets (plot.line with textual-plotext) and
ValueView, which shows a key or topic with v/h/Esc and skips a
representation that raises. The kind vectors (<fragment>.kinds.yaml) carry the
expected value and stream lists too.
EventStoresPage (tui#20, tui#31) polls nothing either. Its EventStore workers come
from the shared worker snapshot (TuiClient.list_eventstores). It follows its own
DataPump.stream of wrk.eventstore.metrics (the stores' counts, pushed by
worker-devkit's WorkerMetrics when they change) and the worker lifecycle topic (the
stores' ok / degraded reports). It asks a store's status (rules, fallback table,
last error) through TuiClient.worker_command once per activation, when its metrics
show a new failure (the last error), and on an updated lifecycle event (new code or
config). The pushed counts win over a status answer that arrives after them.
PortfolioPage (tui#24) polls nothing. Its accounts are the account.snapshot
topics the gateway has seen (TuiClient.list_accounts, through
virtufin-portfolio-devkit's names, names.account_of_topic), new ones joining from
the topic catalog's announcements. The picked account is seeded from the account its
portfolio worker saves at <world>.portfolio.<venue>.account.<account>
(get_account_state, service workmanager), then follows its snapshot,
balance.changed and openorder.changed events through its own DataPump.stream;
portfolio.Account folds them.
PnLPage (tui#25) shows the book the scenario's P&L worker saves at sc.<S>.pnl.book
(get_pnl_book, service workmanager; names from virtufin-portfolio-devkit's
virtufin.positions fragment: names.pnl_book_key, names.pnl_topics). The scenarios
come from the registry, followed through the cfg.scenario.* state changes. The
worker's pnl.realized, pnl.unrealized and risk.exposure events carry no average
cost or fees, so each one reads the book again (one read in flight, one more queued);
pnl.py renders it.
RiskPage (tui#26) shows the P&L worker's risk.exposure.update (risk.from_exposure),
seeded from its saved book with the registry entry's limits (risk.from_book, the
same rules as the worker). l sets the limits (TuiClient.set_scenario_limits, a
registry write announced on sys.api.state.saved, which the worker and the page both
follow); save_scenario keeps them when the triplet is edited.
Instruments, Portfolio, P&L and Risk were placeholders without data (tui#22,
PANE_ORDER_VERSION 6 dropped them from saved layouts once): 7 puts Instruments back
before Order Book, 8 Portfolio after Indicators, 9 P&L and 10 Risk after it.
4. Config (src/virtufin_tui/config/)¶
One module per file the TUI reads or writes, all re-exported by the package:
paths (the XDG locations, seeding the shipped defaults), user (config.toml
and state.json), workers (workers.toml and
source credentials), contexts (contexts.toml, the active context).
Two independent files, both XDG-aware ($XDG_CONFIG_HOME, default
~/.config):
virtufin/api/contexts.toml-- the list ofvirtufin-apiendpoints ("contexts": host/port/TLS/API key/etc.) and which one is active. See Contexts.virtufin/tui/config.toml-- everything else: theme, event-topic presets, the tab/pane layout (pane_order), and display toggles. Loaded into aTuiUserConfigdataclass and round-tripped throughtomlkiton save, so comments and formatting you haven't touched survive. The TUI rewrites it only when the layout is edited on the Config page.virtufin/tui/state.json(under$XDG_STATE_HOME) -- what the TUI sets as you use it: the active Scenario and event topics, the theme, the Events page's Wall/Msg time toggles, column widths. For the theme and the toggles,config.tomlholds the default and the state the current value.
5. Client wrapper (src/virtufin_tui/client/)¶
A thin async wrapper (TuiClient) around virtufin.api.client.ApiClient
exposing the operations the TUI needs as plain async methods (no Textual
dependency) -- workers, connections, triggers, state, scenarios, feed definitions,
strategies and the pubsub subscription. Backend RPCs go through
Gateway.InvokeRaw with message classes built from the gateway's
reflection descriptors.
TuiClient is put together from one part per backend area, each a mixin in
its own module; everything is imported from virtufin_tui.client:
| Module | Part |
|---|---|
_core.py |
ClientCore: the channel (connect, disconnect), the typed invoke (_invoke_unary, the descriptor pools), list_services, how gRPC and gateway failures read (describe_connection_error), retry_on_gateway_not_found |
workers.py |
WorkersMixin: list (one shared snapshot), create, tag, start, stop, delete, worker_command; WorkerRow |
feeds.py |
FeedsMixin: connections (WebSocketFeed workers), open_feed, serve_topics, feed_command; ConnectionRow |
state.py |
StateMixin: state-store JSON entries, bulk reads, indexed registries; StateRow |
pubsub.py |
PubsubMixin: the event streams, the topic catalog, order books and indicators |
triggers.py |
TriggersMixin: scheduled triggers; TriggerRow |
feed_definitions.py |
FeedDefinitionsMixin: the shared feed definitions (cfg.feed.*), connect and disconnect through market-data-devkit's orchestrator; SharedFeed |
registries.py |
RegistriesMixin: strategies and scenarios |
Connections are WebSocketFeed workers (general-workers, workmanager#103); the TUI no longer talks to a WebSocket service:
| Operation | WorkManager call |
|---|---|
List (list_feed_workers, FeedTracker) |
ListWorkers, filtered to feed workers (the feed package, or an orchestrator's feedowner tag); InvokeWorkerCommand status once per running feed (feed_status), then lifecycle events (see "Data store") |
Connect a feed definition (Feeds C, connect_feed_definition) |
market-data-devkit feeds.current (status of the owner's running feeds) → feeds.resolve (owner = the definition's name, its preset) → feeds.apply through the gateway's InvokeJson: serve {topics} on a running feed of the plan, else CreateWorker (the newest Virtufin.Worker.WebSocketFeed of the worker registry, the preset rendered for the world + topics + the definition's feed keys and autoreconnect, tags feedowner/name/source/world/universe, secrets by reference) + StartWorker; the owner's other feeds are deleted |
Disconnect a feed definition (Feeds X) |
feeds.delete(owner): DeleteWorker for each feed tagged feedowner=<name> |
Open a raw feed (open_feed) |
CreateWorker + StartWorker with a preset's config, then waits for its lifecycle status connected (wait_feed_connected); used by scripts and the examples |
Serve topics (serve_topics) |
InvokeWorkerCommand serve {topics} (all or nothing) |
| Start / stop publish (raw feeds) | InvokeWorkerCommand publish {topic} / unpublish |
| Tags | GetTags / SetTag; clearing a tag rewrites the map with SetTags |
| Disconnect | StopWorker + DeleteWorker |
The feed subscribes its served topics' items and sends its subscribe frames
after the login on every open (connect, reconnect, a move between Runtime
pods), so the TUI never replays a subscription itself. A feed knows no
consumer: workers subscribe to the topics, and nothing in the platform matches
topics to feeds (the TUI, a notebook or a script decides which feed serves
what: the shared feed definitions on the Feeds pane, market-data-devkit's
feeds.resolve/apply underneath). Its state changes are lifecycle events on
sys.workmanager.worker.lifecycle (state=status, with status and
detail).
6. Per-pane widgets (src/virtufin_tui/screens/*.py)¶
Every pane is a plain textual.widget.Widget (not a Screen) following
the same duck-typed convention: __init__(self, stores, client, **kwargs),
compose(), optionally on_mount() and whichever action_* methods it
wants Dashboard's generic keys routed to.
| Tab | Sub-tab (pane id) | File |
|---|---|---|
| Home | (single pane, no sub-tabs) | home_page.py |
| Trading | feeds / instruments / orderbook / indicators / portfolio / pnl / risk / strategy / scenario / ai / config | feeds_page.py, instruments_page.py, orderbook_page.py, indicators_page.py, portfolio_page.py, pnl_page.py, risk_page.py, strategy_page.py, scenario_page.py, ai_page.py, config_page.py |
| Infra | workers / connections / scripts / triggers / states / eventstores / alerts / changes / events / ai / config | workers_page.py, connections_page.py, scripts_page.py, triggers_page.py, states_page.py, eventstores_page.py, alerts_page.py, changes_page.py, events_page.py, ai_page.py, config_page.py |
Alerts (alerts.py, alert_watch.py): the App holds one AlertLog for every
context. The Dashboard runs an AlertWatch for the bound context (worker lifecycle
payloads through the DataPump's lifecycle listeners, and one breach topic per scenario,
the pub/sub having no wildcards) and adds halt changes and context liveness changes;
report_error adds every failed action. from_lifecycle, from_breach and from_halt
are pure and pinned by domain-devkit's alerts/classify.yaml (openspec alerts), shared
with the web app.
Changes (changes.py): every write of a TuiClient runs in its _write block,
which asks first on a protected context and tells on_write the outcome (done,
failed, declined); bootstrap.log_writes points that at the App's ChangeLog,
which appends to changes.jsonl in the state directory.
The navigation view (the contexts table and your navigation) is ConfigPage,
under the Config tab by default. Detail popovers (WorkerDetail, ConnectionDetail)
and the action modal are pushed as modals on top of the active pane.
Shared pieces of the pages (tui#30):
_table.py:rebuild_rowsreplaces a table's rows and keeps the selected row (by key) and the horizontal scroll, whichDataTable.clear()resets;selected_keyreads the selection. Every list page rebuilds through it._modal.py:BaseModal, the base of every modal:Esc, a#btn-cancelor a#btn-backbutton close it withcancel_result()(None unless the modal says otherwise, e.g. the event-topic modals return the unchanged topics);FormModaladds the centred, bordered form box (60 cells wide, a modal changes what differs). A Select left empty holdsSelect.NULL, so the modals check the value is astrbefore they accept it.
Beyond the built-in pages, users can add their own panes and even new top-level tabs without touching the source tree -- see Custom panes.
7. Errors (src/virtufin_tui/errors.py)¶
Two kinds of failure, each shown one way (tui#28):
- An action fails (create, delete, tag, publish, compose, ...): the page calls
report_error(page, action, exc), which shows one error toast,<Action> failed: <reason>, and writes the traceback totui.log. The reason is readable: a gRPC status asdescribe_connection_errorputs it (e.g.Server unavailable), a call the backend refused asSTATUS: message(the api client'sGateway invoke failed: STATUS - message), anything else as its own text. Input that is not valid (a tab id taken, a form field) stays a plain message. - Reading the data shown fails: the page keeps what it shows and says it is
stale in its status line,
stale since 12:03:04: <reason>, until the next good read. The stores (workers, connections, triggers, states) carry it (Store.fail,Store.stale_note); Feeds, Strategy and Scenario, which read their own data, keep aStalenessof their own. The log gets one traceback per failure, throughrepeat_log, not one per attempt.
Lifecycle¶
virtufin-tuiis invoked (fromuvx,pip install, or local).app.main()configures file-based logging, then parses--configand--contextflags.config.load_config()readscontexts.tomland applies env-var overrides (VIRTUFIN_TUI_API_HOST/_PORT/VIRTUFIN_TUI_CONTEXT).VirtufinTuiApp.on_mount()resolves the active context. If one resolves, it builds aTuiClient, connects, and pushes theDashboardscreen, which starts theDataPumpand mounts the first tab's first pane after a short delay.- If no context resolves -- no
contexts.toml, or the stored active-context name isn't among the configured contexts -- the TUI still launches. It pushes the sameDashboardin its unbound state (noTuiClient, noDataPumppolling) and lands on the Config page, which lists every configured context with a red "not connected" dot. A pane that needs a client (all but Home and Config, whose page classes setWORKS_UNBOUND) shows aNoContextPagestand-in until then. Pages derive fromPage(screens/_page.py): the Dashboard hands each one the connection state throughshow_connection(Connection), and a page that changes the context row (the active Scenario) postsTradeSelectionChanged; neither touches the other's fields. Selecting one there (Enter) binds a real client and resumes the normal connected lifecycle. See Contexts. - On
q(quit), the TUI cancels background tasks, closes the gRPC channel, and exits 0.
Names¶
Every topic and state key the TUI builds comes from the naming library
(lib.docs.virtufin.com/naming), through
src/virtufin_tui/names.py:
| Name | Value | Used for |
|---|---|---|
cfg.scenario.<SCENARIO>, cfg.scenario.index |
scenario registry | Scenario tab |
cfg.feed.<NAME>, cfg.feed.index |
shared feed definitions (market-data-devkit feed_registry; the key upper-cased, the index as named) |
Feeds tab |
cfg.strategy.<NAME>, cfg.strategy.index |
composite strategies | Strategy tab |
cfg.tui.startdate |
start date | system_startdate.py init script |
sc.<SCENARIO>.* |
scenario scope | keys purged when a scenario is deleted |
sys.api.state.saved, sys.api.state.deleted |
state changes | default event topics |
sys.workmanager.worker.lifecycle |
worker lifecycle | event topic preset; the workers and connections stores |
sys.api.trigger.scheduled, sys.api.trigger.deleted |
trigger changes (api's fragment) | the triggers store |
wrk.<type>.command.{worker_id} |
worker command topic | New Worker form default; a feed's command topic (wrk.websocketfeed.command.{worker_id}) |
wrk.<type>.metrics |
a worker type's pushed counters | Connections (wrk.websocketfeed.metrics), EventStores (wrk.eventstore.metrics) |
Composite strategy names are uppercase identifiers ([A-Z0-9_], up to 32),
like scenario ids; the compose dialog rejects anything else. Atomic strategies
keep their tag values. Feed definition names are lower-case identifiers, as in
the web app.
src/virtufin_tui/naming/ holds the TUI's own fragment (virtufin.tui,
which defines cfg.tui.startdate). The others come from the installed
packages that own them, through the virtufin.naming entry point:
virtufin.worlds (virtufin-domain-devkit), virtufin.market
(virtufin-marketdata-devkit) and the api and workmanager service fragments
(virtufin-api, virtufin-workmanager).