Vibe Architecture — ATAM Evaluation

The LLM emits a destructive shell command (e.g. a recursive delete) during vibe -p in a CI pipeline.

Programmatic mode, which forces the auto-approve agent (BR-5).

There is no human approval prompt. Only the bash allow/deny prefix lists and arity checks stand between the LLM and execution (arc42 §8.2); sudo always asks, but a non-sudo destructive command is not on any deny list by default.

Undefined — no arc42 quality scenario measures the fraction of destructive commands the allow/deny list blocks.

AP-5 + BR-5.

This is the sharpest autonomy-vs-safety tradeoff in the system. Programmatic mode trades the entire human safety gate for unattended throughput. The arc42 Security concept (§8.2) lists "Tiered tool permissions" as the mitigation for threat T-001, but in auto-approve the ASK tier collapses to ALWAYS — the mitigation is disabled by the very mode that most needs it.

Rests on Q3.8.1 (no STRIDE threat model — the residual risk of T-001 cannot be claimed bounded) and Q4.9 (whether Safety outranks the unattended-throughput use case is unranked). → [T-1], [R-1].

An interactive developer selects the accept-edits profile; the LLM proposes a file edit and, separately, a shell command.

Interactive TUI.

The edit auto-applies without a prompt; the mutating shell command still resolves to ASK and prompts (arc42 §8.2, BR-4).

Edits applied with zero prompts; mutating bash prompts 100 % of the time.

AP-5 (agent safety profiles).

This is the architecture’s designed middle of the autonomy-safety spectrum, and it is sound: the per-tool permission tier is orthogonal to the per-profile autonomy level, so the profile can relax edits without relaxing shell execution. → [NR-1], [S-1], [S-2].

A developer cd`s into a freshly cloned repository whose `.vibe/ ships hostile tool, hook and agent definitions.

Interactive TUI, first run in that folder.

The trust-folder gate blocks loading of project config until the developer explicitly accepts (arc42 §8.2, BR-1); the decision persists in ~/.vibe/trusted_folders.toml.

Untrusted project config loaded: never. One trust prompt per new folder.

AP-6.

The gate cleanly closes threat T-005. The cost is one usability prompt per new folder — a small, bounded friction.

Mildly rests on Q2.6.BR.intent — whether the trust gate is a hard security guarantee or a convenience is deferred, so its strength cannot be asserted. → [NR-2], [S-6].

The LLM issues a write_file / search_replace targeting a path outside the working directory.

Any session.

The working-directory boundary forces an extra approval (arc42 §8.2, T-003); --add-dir widens the permitted root set.

Out-of-boundary writes without approval: none — provided the boundary is a guarantee.

AP-7.

The boundary is an approval trigger, not a hard block — and --add-dir plus the "extra approval" escape hatch mean an autonomous profile can still write widely.

Rests on Q2.6.BR.intent ("is the workdir boundary a security guarantee or a convenience?", explicitly deferred) and Q3.8.1. Until answered, SC-4’s response cannot be classified as safe or unsafe. → [T-1], [R-2].

In a single turn the LLM emits two search_replace calls that touch the same file.

Any session.

The Agent Loop runs tool calls as concurrent asyncio tasks (arc42 §6, Conversation Turn). The arc42 runtime view documents the concurrency but documents no serialization of file writes; the outcome is interleaving-dependent.

None — arc42 §6 covers only the happy path; there is no scenario for concurrent-write conflict.

AP-9.

Concurrency buys turn speed (the §10 quality tree credits "concurrent tools" to Performance Efficiency) at a correctness cost the documentation does not bound. Two patches to one file can lose an edit or apply against stale content. → [T-3], [R-3].

An interactive session’s context_tokens crosses the model’s auto_compact_threshold.

Long-running interactive session.

AutoCompactMiddleware returns COMPACT; the loop summarises history into [system, summary] and forks a new session id (arc42 §6).

Continuity preserved (the session does not fail); fidelity of the summary is not measured anywhere in arc42 §10.

AP-3 + AP-8.

Compaction trades correctness (detail in the dropped history) for reliability/continuity (the session survives). The §10 scenario gives the trigger as a symbolic auto_compact_threshold with a 50 % warning, but no measure of how much task-relevant context survives.

The acceptability of the fidelity loss depends on the Reliability -vs- Functional-Suitability ranking, which is Q4.9 (deferred). → [T-2], [S-3].

A programmatic run is bounded with --max-price; the session accumulates cost.

Programmatic mode.

PriceLimitMiddleware stops the loop when session_cost exceeds the ceiling — but session_cost is a self-described rough estimate that ignores prompt caching (arc42 §11, TD-001).

The --max-price guarantee is approximate; arc42 §11 states this explicitly. The divergence size is unmeasured.

AP-3.

Speed of unattended operation (no human watching the bill) is bought against the correctness of the cost bound. The run can stop early (wasting budget) or late (overspending). → [R-4], [S-4].

The LLM reads a file larger than the read_file cap.

Any session.

Content is returned truncated at 64 000 bytes with was_truncated set (arc42 §10).

Read cap 64 000 bytes/call; grep 100 matches / 64 000 bytes; bash 16 000-byte output cap.

Bounded-I/O caps (arc42 §10).

The caps bound turn latency and context growth (speed) but hand the LLM partial input (correctness) — and was_truncated is the only signal that this happened. A reasoning error built on a truncated read is silent. → [T-5], [S-5].

The LLM provider returns 429s, then becomes unavailable for an extended period.

Any session.

The Mistral backend retries with exponential backoff — 500 ms initial, 1.5× exponent, 300 s cap (arc42 §8.5, §10). Behaviour after retry exhaustion / on a sustained outage is not specified (arc42 §11, R-009).

A single turn can block up to the 300 s cap; the sustained-outage end-state has no measure.

AP-8.

The retry cap is a reliability-vs-performance tradeoff — a hung turn costs up to 300 s of latency.

The sustained-outage end-state rests on Q5.3 (deferred to Operations). → [T-4], [R-5].

A contributor integrates a new provider.

Development / CI.

A new backend is registered in BACKEND_FACTORY or covered by a new API-style adapter behind the BackendLike port; the Agent Loop is unchanged (ADR-002).

Agent-loop files changed: zero.

AP-1 + AP-2.

Hexagonal ports plus the backend factory deliver Compatibility and Maintainability with no opposing attribute materially harmed — ADR-002’s only negative is config complexity (-1). A clean non-risk. → [NR-3].

An IDE integrator embeds the agent via vibe-acp.

ACP server over stdio.

The ACP bridge exposes the engine; ACP-specific tools delegate file/terminal operations to the ACP client (arc42 §5, §6).

One engine reused behind two front ends.

AP-1 (ports) + the ACP bridge.

Compatibility is delivered, but acp/tools/ re-implements several core/tools/ builtins (arc42 §11, TD-002) — a Compatibility-vs -Maintainability tradeoff: the protocol reach costs duplicated tool maintenance. → [T-6], [R-6].