Series: Building Backarch, engineering decisions from building backarch.com
An ML model gives you a health score of 67. The user asks why. The model doesn't know. A rule engine gives you a health score of 67 and a list of exactly which checks fired and why. That's not a technical difference. It's a trust difference. And trust is a product requirement.
Any scoring system that affects decisions will be questioned
Health score drops from 84 to 61 after adding a database node, and the user wants to know what changed. A compliance flag appears on a diagram, and the user's manager wants to know why. Architecture risk increases, and someone wants to argue with it.
If your answer is "the model assessed it that way," you've built a system that cannot be interrogated, appealed, or improved by anyone except whoever controls the training data. For a tool that influences engineering decisions, that's a dealbreaker before you ship.
The case for a rule engine isn't that ML is wrong. It's that for architecture validation, the input space is small (20-50 nodes), labelled training data doesn't exist at scale, and explainability is non-negotiable. The rule engine wins on all three.
Six weighted categories, not flat penalties
The scoring system at backarch.com uses six weighted categories rather than flat penalties from 100:
| Category | Weight | What it covers | |-----------------|--------|----------------| | Reliability | 0.25 | SPOFs, replicas, DLQs | | Security | 0.25 | Auth, encryption, direct DB access | | Scalability | 0.20 | Cache presence, load balancer, queue usage | | Observability | 0.15 | Prometheus, tracing, alerting | | Cost | 0.10 | Budget constraint violations | | Maintainability | 0.05 | Diagram complexity, undifferentiated patterns |
Each category runs its own rule set. Each rule has a condition (evaluated against the graph), a severity (high/medium/low), a weight (how much it costs within the category), and an action_payload (pre-filled fix action for the frontend).
Representative rules: r_spof (compute node with no replica), r_no_db_replica (database with no backup marker), r_no_dlq (async edge with no dead-letter queue), r_no_auth (unauthenticated external surface), r_no_encryption (unencrypted cross-boundary edge), r_no_observability (three or more service nodes with no monitoring), r_budget_violation (estimated cost exceeds constraint).
The same diagram run through the engine twice produces the same score. Same inputs, same rules, same output, always. This is the property that makes the score trustworthy enough to act on.
Tier gates on top of the score
The numeric score isn't the only gating mechanism. On top of it sit certification tier gates: dev, production, ha, enterprise.
A diagram cannot reach ha tier regardless of its numeric score if it carries r_spof or r_no_db_replica. A single point of failure is an automatic block on high-availability certification, independent of what the overall score calculates to. The score is a continuous signal. The tier is a certification level.
This separation matters for user communication. "Your score is 74" is information. "Your diagram cannot be certified ha because it has a single point of failure" is an actionable requirement. Both come from the same underlying engine; they're presented differently because they serve different purposes.
The tier gates also make escalation paths explicit. A team debating whether an architecture is "good enough for production" now has a binary answer they can point to: not a score to interpret, but a certification status that either passes or lists exactly what blocks it.
Why the LLM handles explanation, not scoring
The rule engine answers "what is wrong." The LLM answers "why it matters in this context and what to do about it first."
The issues list from the rule engine is what the AI panel receives as structured input:
{
"health_score": 62,
"issues": [
{
"rule": "r_no_db_replica",
"severity": "high",
"category": "reliability",
"affected_nodes": ["postgres_1"],
"message": "Database has no replica or backup configuration.",
"action_label": "Add read replica",
},
{
"rule": "r_no_dlq",
"severity": "medium",
"category": "reliability",
"affected_nodes": ["sqs_queue_1", "lambda_processor_1"],
"message": "Async edge has no dead-letter queue.",
"action_label": "Add SQS DLQ",
}
]
}The LLM takes this structured list and generates the explanation: why the SPOF is the higher priority given the user's traffic profile, what the trade-offs of adding a replica look like given the budget constraint, which finding to address first.
The context the LLM reasons over, matched patterns and related ADRs, comes from typed tools (pattern_suggest, adr_search) that query Postgres directly. No embeddings pipeline, no vector store. Semantic retrieval is a deliberate later-phase decision, not a missing piece of this one.
Using an LLM to compute the score would break this separation. The score would become probabilistic instead of deterministic. Unit tests would become LLM call mocks instead of pure function assertions. Reproducing a specific score to debug it would require re-running a model. None of that is acceptable for a feature users rely on.
Unit-testing rules is trivial; unit-testing a model is not
A rule engine can be tested with a plain Python dict:
def test_db_spof_fires_on_single_database():
diagram = {
"nodes": [
{"id": "api_1", "type": "fastapi", "data": {}},
{"id": "db_1", "type": "postgres", "data": {}}
],
"edges": [{"source": "api_1", "target": "db_1"}]
}
result = compute_health_score(diagram, constraints={})
assert result.score < 80
assert any(i["rule"] == "r_no_db_replica" for i in result.issues)
assert any(i["severity"] == "high" for i in result.issues)No mocking. No API calls. Fast. Deterministic. The test runs in milliseconds and fails or passes for exactly one reason: the rule logic.
Testing an ML scoring model requires a representative test set, a trained model, and an acceptance criterion on a performance metric. That's fundamentally different from asserting that a specific rule fires on a specific input. The rule engine test is about correctness; the model test is about performance. Both matter, but only one is the right primitive for "does this specific architectural pattern flag correctly."
When a rule misfires in production, the fix is a one-line condition change and a new unit test. When an ML model misfires, the fix is a retrain, a new evaluation run, and a deployment, with no guarantee the fix doesn't introduce regressions elsewhere. The iteration cycle is an order of magnitude shorter with rules.
At backarch.com, the temptation when designing the health score was to send the diagram to the LLM and ask for a score from 0-100. The problem: different runs of the same diagram produce different scores. Different model versions produce different scores. There's no unit test that confirms the SPOF rule fires on a database with no replica. That's not a health scoring system. It's a suggestion generator with a number attached.
The LLM is not the answer to "what score should I give this architecture?" It's the answer to "how should I explain this score to an engineer who needs to understand and act on it?"
Next: Don't show users your model picker: how to abstract model selection into effort levels that match what users actually want.