[rust] T4-3 compile wiring: al serve + /rust-code page (#16) #35
Loading…
Reference in a new issue
No description provided.
Delete branch "task/16-compile-wiring"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is
The JS CODE path has always run in the browser:
/code.jsserves a slot and the clientevalsit. Rust has no equivalent, so this makes the other half of the architecture real:
al serve, alocalhost-only HTTP control endpoint in front of
al buildandal run, and/rust-code, thepage that drives it. Edit Rust → compile → run against the local server, and the auth token never
passes through the browser or the backend in either direction.
Branch
task/16-compile-wiring, issue #16. Three commits: the endpoint, the page + live test,the runbook.
1. The endpoint table
al serve [--port N] [--workspace-root PATH] [--crate NAME] [--creds FILE] [--character ID] [--server URL]POST /compile{"crate","source","restore"?}{"ok","log","binary","crate","on_disk","restored"?}POST /run{"crate"?,"once"?}{"ok","pid","error","lines"}POST /stop{"ok","exit"}POST /reload{"crate"?,"build"?}{"ok","pid","exit","rebuilt","error"?}GET /status{"running","pid","restarts","last_exit","log_tail","crate","binary","creds","dropped","log_ring"}GET /health{"ok":true}GET /options{"crates","default_crate","log_ring","bind","creds"}/healthis the brief's six;/optionsis the one addition, because otherwise the page has tohardcode the allow-list and the ring size and both drift from the server's copy the moment either
changes. It answers from static state, so polling it costs the endpoint nothing.
Behaviour, in the four places where it is a decision rather than an obvious reading:
shows, and silently reverting it would be a lie. The last good text is kept beside it as
src/main.rs.good, put back by the next successful build or by{"restore":true}./runanswersalready running (pid N)instead of starting a secondsession for the same character.
/reloadstops (grace), rebuilds if asked, and always restarts.al servewashanded at startup. No response body and no log line carries it;
/statusreports the credentialset only through the masked
describeview./compiledrivesal build's ownBuildCommand, so the incremental cache isthe same one
al builduses: an unchanged Compile → Run costs milliseconds.2. The dependency added, and why
tiny_http0.12.0 — the onlyCargo.tomlchange, plusCargo.lock.It is one small crate, no framework, no async runtime of its own, and it exposes exactly the two
things needed (
Server::httpon aListenAddr,Request::respond) without inventing a routinglayer to learn. The alternatives were all worse for a tool whose whole job is to not be a server
product. Four of its behaviours cost real time and are documented at the call sites so the next
reader does not rediscover them:
HeaderField::equivtakes a&'static str;Header::from_bytes'error type is
();Request::respondconsumes the request (which is what makesanswer-exactly-oncable a property of the type system rather than of a comment); and
ListenAddrhasa unix variant behind a cfg.
Loopback is not a setting.
Server::bindtakes a port and always binds127.0.0.1; anon-loopback
--port-shaped argument is a parse error, not a warning. This endpoint compiles andruns arbitrary Rust as the local user, so a public bind would be a remote-code-execution listener.
CORS is answered for loopback origins only (
localhost,127.0.0.1,[::1]). Crate names arematched whole-string against one fixed allow-list, so a traversal is simply not on the list, and the
only writes the endpoint ever makes are
<crate>/src/main.rsand its.goodsibling.3. Structure, and the one bug the first draft had
serve/control.rsowns every rule (routes, bodies, allow-list, ring, supervised child) and nevertouches a socket, so the suite runs offline.
serve/http.rsis transport only: accept, decode,encode, CORS.
serve.rsis the CLI front door.Two things looked cheaper and were wrong:
Mutex<Option<Arc<Runtime>>>for the child. A supervisor keeps its children to itself, so a/runracing a retire would have started a character outside the one-child rule with nobody leftto stop it. The child is now guarded by an actor thread that owns the one current supervisor
outright, and handler threads send lifecycle messages and block for replies. A
Currentstructholds the runtime and its log pump together, so a retired supervisor's late output cannot appear
under the next run.
repo_root/rust, soexamples/hellobecame<repo>/examples/hello— a directory with noCargo.toml, and theendpoint answered "no manifest" about a crate that plainly exists.
ControlandServeCommandtake a
workspace_rootnow,--repo-rootis--workspace-root, and the live probe'saccidentally-created
<repo>/examples/debris is gone.4. The page
htmls/rust_code.html+ one additive route inmain.js(next to/vscode; nothing else in thebackend changed). No build step, no framework, no jQuery — the JS is inline.
A textarea prefilled with the crate's real source, a crate dropdown from
/options, Compile / Run/ Stop / Reload, a status line, and a
<pre>log pane polling/statusevery 1.5 s (the brief's"1–2 s"; no websockets).
Three rules the endpoint's shape forces on the client:
a second copy is a copy that goes stale.
on_diskexists because of a restore: that request carries no source, so without the fieldthe editor would go on showing the attempt that failed while the disk held the good copy.
/reloadrebuilds from the source alreadyon disk — otherwise it quietly restarts yesterday's code.
Signed out, the page renders a sign-in notice and no script at all.
5. Tests
Offline (in the gate): 40
serve::tests, incrates/al-tool/src/serve/tests.rs, drivingControland the transport directly. The brief asked for the struct over a socket and this followsit, with one socket test as its exception:
source and the good one recoverable; every compile answer reports what the file holds now;
counted as
dropped);/runrefuses; reloadrestarts with and without a rebuild;
al serve's CLI: therust/default,--workspace-root,--cratechecked against the sameallow-list the endpoint uses, and every argument it cannot honour;
the accept loop;
/statusand/optionscarry no token.Build steps and binaries are injected (
Builder,/bin/shfixtures in the style of T4-2's tests),so nothing here waits for cargo or a game server.
LIVE, one
#[ignore]d test —crates/al-tool/tests/live_serve.rs. Starts the realal serveon a port the OS hands out, then walks the buttons' sequence:/health→/options→/compileof the committed hello source →/run→ poll/statusuntil the crate's own linereaches the ring →
/stop, asserting/statusagrees about the exit. It asserts the token appearsneither in the ring nor in
al serve's stderr. Teardown isDrop, so a failure cannot leave a boundport or a logged-in character, and it removes the
main.rs.goodthat/compilecreates in atracked crate. Skips without creds unless
AL_TEST_REQUIRE=1, matching the other live suites.6. Gate
bash rust/dev/check.sh— fmt +clippy --workspace --all-targets -D warnings+test --workspace:576 individual
test … oklines across the workspace, zero failures, clippy pedantic-clean under--all-targets -D warnings. The newal servetests are 40 ofal-tool's 101 unit tests.7. Manual proof
The stack was already running and was not touched (
:8090→ 200,:7192→ 200 before andafter).
al serveran on its own port 8193, backgrounded to a file rather than through a pipe.A character logged in, walked to its target and logged about it, through a source the browser sent
over HTTP and cargo built on this machine. Port 8193 was closed again after the run and the tree is
clean.
8. What a backend restart activates
http://localhost:8090/rust-codecurrently answers 404 — the backend on:8090predates thisbranch and was deliberately not restarted. The route is verified by code inspection plus a
nunjucks render check, not by an HTTP request.
The render check stands in for the live request as far as it can: it renders the template under
nunjucks 3.2.4 with
autoescapeon (asscripts/precompile_templates.jsconfigures it) and thesame
domain/userglobals the repo passes, and asserts the doctype, the title fromdomain.title,the cache-busted stylesheet, the textarea, all four buttons, the status line, the
<pre>pane, the127.0.0.1:8192default, the 1.5 s poll, that the prefilled source round-trips through the textareabyte-identically, that the one
<script>block parses, that signed-out renders the notice with noeditor and no script, and that a source containing a close-tag sequence inside a string literal
cannot escape either the textarea or the script. 26 checks, all passing.
What it cannot prove is the route itself:
get_user,get_domain, and Express's dispatch. That isthe part a restart activates — after merge, restart the backend and
curl -s localhost:8090/rust-code | head -5should return the page instead of a 404.9. Assumptions
rust/dev/LOCAL.mdis extended with a "Rust CODE loop" section. The brief's §3 names that fileas the one to touch and says explicitly "this file is yours to touch";
PROGRESS.mdandPLAN.mdare untouched.
examples/hellomeansrust/examples/hello, matchingal build rust/examples/hello.--workspace-rootoverrides it;--repo-rootno longer exists onal serve./optionsadded to the brief's six endpoints, so the page does not carry a second copy of theallow-list.
are the user-CODE slots today; anything else is added to the list in source
(
AL_SERVE_CRATESoverrides it for tests only)./stopalways answersok:true. Even an unresponsive supervisor is reported as an exit text("stop not confirmed (…)"), because "this endpoint wants nothing running" is what the request asked
for either way.
--creds dev/.localwith no--character, so it injectsthe
AL_DEV_*trio — the runbook's manual/browser character. Noted inLOCAL.mdwith how to moveit if Rusty is playing in the browser.
al-runtime'skill_on_dropbehaviour is used, not worked around:al servestops the childexplicitly on
SIGINT/SIGTERMbefore exiting, precisely so the default is not the path a sessionends by (it is
SIGKILLwith no grace, which leaks a session the server still believes it owns).Closes #16
The CODE page has always compiled in the browser: `eval` under a shim. Rust has no equivalent, so `al serve` puts a local control endpoint in front of `al build` and `al run` and the editor calls it over HTTP. The endpoint compiles and runs arbitrary Rust as the local user, so it binds 127.0.0.1 and nothing else: `--port` moves the port, it cannot move the address. Two halves, deliberately not sharing anything but types: serve::control owns the semantics — routes, bodies, the crate allow-list, the ring, the supervised child — with no socket in sight, so the suite runs offline; serve::http is the transport: accept, decode, encode, CORS. `Control` is a cheap handle onto shared state, and the child is guarded by an actor thread that owns the one current supervisor outright. A plain `Mutex<Option<Arc<Runtime>>>` looked cheaper and was wrong: a supervisor keeps its children to itself, so a `/run` racing a retire would have started a character outside the one-child rule with nobody left to stop it. 38 tests, in the crate's own `tests.rs` (the workspace's shared style). Three are the live probe's bugs, pinned: a build log of any length cannot outrun the 200-line ring; the ring counts what fell off it; and `examples/hello` resolves inside `rust/`, not the repo root — a path level up meant no manifest about a crate that plainly exists. Crate names are matched whole-string against one fixed allow-list, so a traversal is simply not on the list, and the only writes the endpoint makes are <crate>/src/main.rs and its `.good` sibling.`al serve` is only worth having if something drives it, so this adds the browser page and the route, plus the one #[ignore]d live test. The page (htmls/rust_code.html) is one file with no build step: a textarea prefilled with the crate's real source (read from disk by the route, so there is no stale copy in a template), Compile / Run / Stop / Reload, a status line, and a log pane polling /status every 1.5 s. It talks to 127.0.0.1:8192 and nothing else; the auth token never crosses this process in either direction. Three rules the endpoint's shape forces on it: A failed build leaves the broken text on disk, so the answer carries `on_disk` — what the file holds now. On a restore the request sent no source at all, and without that field the editor would keep showing the attempt that failed while the disk held the good copy. `/reload` rebuilds from the source already on disk, so Reload compiles first when the editor is dirty. Two different intentions, two requests; Reload never quietly restarts yesterday's code. The source reaches the script from the textarea rather than a second copy emitted into it. The template escapes the file already; a close-tag sequence inside a Rust string literal would otherwise end the script block where it stands. (The comment explaining that had to be reworded for the same reason.) main.js gains one route, additive, next to /vscode. The running backend on :8090 predates it and answers 404; a restart after merge activates it. Until then the template is checked by rendering it under nunjucks with the repo's own globals and autoescape, signed in and out, and with a hostile source. tests/live_serve.rs runs the real `al serve` on a port the OS hands out and walks the buttons' sequence against the live stack: health, options, compile the committed hello source, run, poll /status until the crate's own line lands in the ring, stop, and agree about the exit. Teardown is Drop, so a failure cannot leak a bound port or a logged-in character, and it removes the .good file /compile leaves in a tracked crate.Review: CHANGES_REQUESTED. Verified-good: token masking (0 leaks in responses/log ring), HTML escaping (autoescape + textarea-only insertion), main.js purely additive, workspace resolution matches al build (regression test present), actor design sound (no deadlock).
Blocking:
servearm is untested. Restore them + add serve dispatch coverage.creds::read_filereturns Ok-with-warning for a missing file; the failure later ismissing credential(s): ..., which matches nothing.al servewith empty config home exits 1, contradicting the docs. Make the branch real or drop it and fix docs.if let Some(port)block in serve.rs,SupervisorStatus::has_runnever read,Captured::errornever read, emptyimpl Drop for Server, unusedpub use control::Control, 5 new intra-doc warnings from cargo doc.Non-blocking nits: main.js always serves the hello source even when --crate differs; stale comment in main.rs; LOCAL.md hardcodes the al-t43 worktree path; PLAN.md edited an unrelated task's status line; the escaping guarantee exists only as manual work — commit it as a test.
serve.rs detected the degradation with is_absent(): text matching on "not found" / "no such file" / "does not exist". creds::read_file reports a missing file as Ok-with-a-warning, so the failure that actually arrives is "missing credential(s): AL_DEV_USER …" and the branch never fired: a fresh machine (no ~/.config/al/credentials) made `al serve` exit 1 at startup, while the help page, /options' creds:false and the page's creds_note all promised compile-only mode. Decided structurally instead: * creds::Incomplete { missing, file } is a typed error resolve_parts returns through anyhow's source chain. Incomplete::nothing_found() (all three keys unset) is "nobody set this up"; one key present is a BROKEN setup, and that distinction is what matching on text could never make — an empty file and a typo'd token key produced the same substring hunt and would have had to produce the same answer. * ServeCommand::credentials_against(env, default_file) is the offline seam. Two injections, not one, because creds::default_path() reads $XDG_CONFIG_HOME/$HOME which a test cannot move: Resolver:: resolve_against takes the stand-in default file, so the rule is testable without touching anybody's config or environment (check.sh --live exports the real trio). * is_absent is gone. Degradation requires creds_file.is_none() AND nothing_found: a --creds path that is not a file stays a startup failure (a typed flag is a promise), and so does a half-written file. Four tests: the degraded endpoint compiles, refuses /run naming --creds and reports creds:false; env-alone is a set not a missing one; a partial file fails loudly (default path and --creds); a named absent file is a path error, not a key error. `al serve --help` and execute's # Errors now describe the rule that runs.Four tests, plus the seam the 503 one needs. race() parks both workers on one barrier and this thread — the third party on it — raises it, so the two requests really are in flight together instead of "probably overlapping": the interleaving a test names is the one that runs. * two_simultaneous_runs_start_exactly_one_child — the lifecycle guard. Checked for teeth by deleting the `let _lifecycle = lock(..)` line in Control::run: the test then fails with BOTH runs ok:true and two distinct pids. With the guard, exactly one answers ok, the loser says "already running (pid N)", /status shows the winner's single pid, and one /stop leaves nothing. * a_run_racing_a_stop_leaves_answers_and_status_agreeing — either may legitimately win, so it asserts what must hold afterwards: /status shows a pid only when an answer claims one, a refusal says why, and one stop is enough to leave no orphan (the failure mode being a child the page believes is stopped). * a_full_handler_pool_answers_503_and_keeps_listening — a Builder that parks inside the build step holds the slot, the next request gets 503 naming the bound and "retry in a moment", and after the release the same socket answers 200: full, not broken. * a_busy_refusal_is_the_json_the_page_renders — 503 plus {"ok":false,"error":…}, and no secret in the text a browser shows. Server::serve_with(stop, max_inflight, queue_wait) is the transport seam: at the shipped 6 slots and 10 s the branch needs seven concurrent compiles and ten seconds, which would only prove that a timer runs. MAX_INFLIGHT and busy() became pub(crate) so the suite pins the shipped number and the refusal body; SlotCount carries the bound into the message, so the number a caller configured and the number the answer states cannot drift apart.Three findings, all invisible to the compiler because each had *some* use somewhere — which is why they are listed here rather than assumed to be lint-caught. * `SupervisorStatus::has_run` was set in `status_of` and read by nobody. It is gone. The field and `last_exit` looked like the same fact twice, and the reviewer's question was exactly that, so the struct now says which one answers "did this ever run?" and why the other cannot: `last_exit` is `None` before the first run ends and `Some` after, and a run that started and finished keeps its `Current` (only shutdown takes it), so the exit receipt survives and no flag could contradict it. Two tests already read that pair — `a_status_snapshot_reads_the_ring_ and_the_supervisor_together` asserts `last_exit` is null before anything ran, `run_then_stop_reports_a_pid_the_log_and_the_exit` asserts it is `Some` afterwards — which is what stops the duplicate coming back. * `Captured::error` was always `None` on the path that built the struct with real output, and the one path that set it (`cargo` could not be started) already put the same text in `log`. The field is gone, so `run_captured` now has one failure channel instead of two and a handler cannot show the pane one and the JSON the other. Its remaining contract is stated where the code decides it, and `cargo_builder`'s `Err` arm — the other half of "cargo is missing" — already reports it as a build result rather than a panic, which the production-builder test asserts. * serve.rs: `if let Some(port) = port { command.port = port }` appeared twice, back to back (copy/paste from the CLI flag work). One remains. Behaviour-identical, but the duplicate is the kind that makes a later reader look for a difference that isn't there. rustdoc is clean now: `cargo doc -p al-tool --no-deps` generates with zero warnings. Six were unresolved/redundant links, all introduced in this PR — `Self::new` and `Self::supervisor_for` (targets that never existed; the constructors are `with_allowed`/`with_builder`, now linked to the one that is real, with a note on why there is no `Control::new`), `handle` in http.rs (now `Control::handle`), `HeaderField::equiv` (a tiny_http item this crate does not import — backticks, not a link), `DEFAULT_CRATE` in serve.rs (now `control::DEFAULT_CRATE`), and one redundant `al_runtime::Error::CrashLoop` target in run.rs.`main.js` seeded the editor with `readFileSync("rust/examples/hello/src/main.rs")`, so `al serve --crate examples/capture` opened the page on the wrong crate's code: Compile would have built `examples/capture` from text that came out of `examples/hello`. That was a visible bug, not a stylistic one, and it is the nit's own words — "always serves the hello source even when --crate differs". Fixing it in the place that knows the answer means `GET /source` on the endpoint, because three facts decide which file it is and only `Control` holds them: the allow-list, `--workspace-root`, and which crate `--crate` put in play. The backend can reach none of them, so it cannot seed the editor correctly — and it used to route the file's bytes through nunjucks to try, which is the second half of why this is the right shape (see the next commit). The route: GET /source {"crate"} (optional; ?crate=examples/capture) -> {"ok","crate","path","source"} * An unnamed one means "the crate in play", the same fallback `/run` and `/reload` use, so a page that does nothing clever opens on the right file. * `path` is in the answer because "matches the last build" is a claim about a file and a reader is entitled to know which one. * It is guarded like the writes. `/status`, `/health` and `/options` answer anybody — they leak nothing but a ring of log lines already on this screen. `/source` answers with the contents of a file on this machine, so it gets the loopback Host/Origin rule too, and `classify` now says so where the guard is applied (`post_only || route == "/source"`) rather than leaving "guarded" a synonym for "POST". * A `/source` fetch does not change which crate is in play: `remember_crate` skips it. Reading a file is not an intention to build it, and a page that fetches three crates to pick one must not have its browsing decide what a later unnamed `/run` starts. `--crate` survives the same reads; `/compile` still moves it. The page: the textarea opens empty and `load_source` fills it from the endpoint, so Rust source reaches the DOM by exactly one route — assigning `textarea.value` from a parsed JSON string. `main.js` now renders with `domain` and `user` and reads no Rust file at all; `get_rust_code_source` is gone. Two consequences of the editor now following the select, both handled rather than discovered later: * Changing the crate with unsaved edits is refused (the select goes back, the pane says why). Compile writes the box into whichever crate is selected *now*, so a switch that kept the old crate's text would build the wrong thing out of what looks like the right one. The edits are the expensive thing here; the click is not. * A crate with no `src/main.rs` yet answers 400, and the page leaves the editor alone rather than showing an empty box somebody would then Compile into a real crate. The live test's `handshake` now asks `/source?crate=examples/hello` and compares it to the file it is about to compile, so the page's seed and the CLI's workspace resolution are checked against the same bytes on a real `al serve` (the offline suite pins the same three properties on the routing side: the fallback, the guard, and that a read does not move the crate). `al serve --help`, this module's endpoint table, and LOCAL.md all list the route; LOCAL.md's page section says where the seed comes from now. Verified by hand against a live endpoint on 8197 with `--crate examples/capture`: /options says `examples/capture`, `/source` answers that crate's file byte-identically (30970 bytes, `==` on the strings), `Host: evil.example` and `Origin: http://evil.example` both 403, `?crate=crates/al-sdk` is refused with the list, `POST /source` says GET-only, and `/nope` names the eight routes. The smoke-test `al serve` was the only process started and stopped; :8090 and :7192 were left running.The review's words were that this property "exists only as manual work — commit it as a test". It was verified by reading (`autoescape: true` in common/init.js, source into a textarea and nowhere else) and reading is not a thing the next change has to pass. node/test/rust_code_page.test.js runs the page's inline script against a hand-built DOM and a `fetch` stub — no browser, no new dependency, and no dependency on `common/` or Mongo either, which matters because `node --test test/` already has 12 pre-existing failures in this worktree from a missing sibling checkout and these 6 do not join them. They run in 60ms and they run in CI's Node. Six properties, and the interesting ones are negative: * the seed follows the endpoint. `/options` says the crate in play, and the source request has to name *that* crate: `?crate=examples%2Fcapture`, exactly one request per load. This is the previous commit's bug, as a test — mutate `load_source(els.crate.value)` to `load_source("examples/hello")` and it is the test that fails. * the source reaches the DOM as a text property. No `innerHTML`, `outerHTML`, `insertAdjacentHTML`, `document.write` or `createContextualFragment` anywhere in the script; `els.source.*` is assigned twice and both times it is `.value`; the crate dropdown is built with `createElement`/`textContent`. The fake element has *no* `innerHTML` to write to, so a page that started using one throws in the harness rather than passing quietly — the difference between a test that greps and one that runs. * the template has two holes and both are constants (`domain.title`, `domain.v`), no `| safe` filter anywhere, the textarea is empty in the markup and carries no `value` attribute, and main.js renders it with `domain`/`user` only. `main.js` passing a `source`, or `get_rust_code_source` coming back, fails the test. * the hazard itself, checked rather than described: a `<script>` block ends at the first close-tag token *including inside a comment*, which is why the page's own prose cannot spell it and this file (plain JS, built from split literals) can. So the page's script is scanned for both tokens, and it is empty of them. * switching crates with unsaved edits refuses the switch (select back, edits intact, no fetch of the other crate over the top of them) and the same switch succeeds once the box is clean — the difference between a guard and a block. * an endpoint that cannot answer for a crate leaves the editor alone and does not retry into a loop. Verified by mutation, all three real ones: re-seeding the template from `{{ source }}` fails the interpolation test; hardcoding the hello crate fails the seed test; assigning `els.source.innerHTML` fails both the seed test and the sink test. Reverted all three. Formatting is CI's, not mine: `npx prettier --check .` from `node/` is what `.github/workflows/code_quality.yml` runs, on that directory's own `.prettierrc` (printWidth 120, tabs), and the file passes it. `node --test test/rust_code_page.test.js` passes 6/6; `bash rust/dev/check.sh` is green end to end.