Matching combines structured constraints, incomplete external information, and judgments that cannot all be reduced to one deterministic rule.
Case study · AI engineering internship
Reasoning needs boundaries.
Probabilistic output is not authority. A production-oriented Python service for data processing, vector retrieval, ML-assisted matching, and agent execution—designed so model output never becomes unchecked system behavior.
RoleAI engineering intern
ArchitectureService separated from main platform
StackPython · FastAPI · Qdrant
FocusRetrieval · agents · validation
Design problem
An LLM response is not yet a trusted result.
The model may compare and recommend inside a candidate set. It does not own normalization, validation, final approval, or external effects.
The public case study describes the architecture in generalized form and excludes company data, internal prompts, and sensitive operational details.
Structure first
Normalize business data and expose explicit schemas before asking a model to reason over it.
Retrieve narrowly
Use vector infrastructure to bring relevant candidates into the workflow instead of expanding model context without control.
Validate separately
Keep deterministic checks, confidence rules, and approval boundaries outside probabilistic reasoning.
Execution model
The agent is treated as a stateful workflow rather than one large prompt. Each transition has a responsibility, expected output, and failure path.
01 · INPUTReceive and normalize structured business data.
02 · RETRIEVEFind relevant candidates from vector search.
03 · REASONApply model judgment only where ambiguity requires it.
Judgment loop
One orchestrator carries the whole judgment loop.
Search runs before judgment.
The orchestrator reasons over what search returns; it does not reason freely before a candidate set exists.
Design · ground judgment in retrieved evidenceThe orchestrator expands its own search.
When the candidate set looks insufficient, the same orchestrator issues one bounded tool call to widen recall, then re-judges—it does not hand off to a separate search agent.
Design · one loop, not a relay of agentsThe same orchestrator checks its own output.
Re-verifying a produced result is not a separate agent’s responsibility; the orchestrator that made the first judgment also validates it before anything is finalized.
Design · one accountable judge, not a hidden second opinionLow confidence routes to a person, not a guess.
When the orchestrator’s confidence falls below a threshold at any judgment point, the result goes to human review instead of being forced to a decision.
Design · uncertainty is a valid, visible outcomeAuthority model
AI proposes. Software decides what may proceed.
- 01RetrieveReturn an explicit candidate set
- 02ReasonUse model judgment inside that set
- 03ValidateReject schema and constraint violations
- 04ApproveRequire review before external impact
Candidate membership
A recommendation outside the retrieved candidate set is rejected instead of being repaired silently.
Schema and constraint checks
Structured validation remains outside model reasoning so failures are visible and reproducible.
Final approval
Work with external impact stops at a review boundary; autonomy does not imply authority.
Investigation
A result is not proof until the measurement survives scrutiny.
The benchmark had to be trustworthy first.
Original labels were collected in order of purchase frequency, so 53% clustered into just 3 of 166 categories—rare cases were effectively unmeasured. A frozen, stratified benchmark replaced the ad-hoc sample.
Record · a repeatable, unbiased measurement firstA promising result isolated the wrong cause.
A feature that looked like a real improvement traced back to the measurement instrument itself: feature extraction wasn’t deterministic, so re-running identical code shifted which cases were even in the holdout set.
Isolate · the instrument, not the model, was the variableOne fix at a time, not several at once.
Feature extraction was cached to make it fully deterministic, and the statistical resampling method was changed to treat same-query candidates as dependent, not independent, samples.
Change · fix the measurement, then re-testRe-run under the same, now-trusted conditions.
Every prior result was re-checked against the corrected, frozen benchmark. One apparent gain turned out to be measurement noise; a category-identity signal held up as a real improvement on both common and rare cases.
Verify · same conditions, corrected instrumentValidation
Model confidence was not treated as ground truth.
77.3%Accuracy vs. human label · n=282
18.2 / 23.5%False approve / false reject
12.7% → 46.2%Category coverage after remodel
LoRA fine-tuning
Seven fine-tuning attempts never reproduced a real improvement. The negative-example mining pipeline turned out to be contaminated with duplicates, and most positive training pairs were easy cases the base model already handled correctly—leaving little new signal to learn from. The approach was not adopted.
Failure paths
Recovery is designed before the happy path is trusted.
Fail schema or constraint validation, preserve the reason, and route to retry or review.
Reject results that escape the candidate set rather than treating fluent output as evidence.
Stop after a bounded retry policy and surface the unresolved state to a reviewer.
Require explicit approval before the workflow can produce an externally consequential final action.
Public boundary
Architecture is shown; confidential operations are not.
This page demonstrates service separation, retrieval, validation, approval, and recovery responsibilities. It intentionally omits private company data, internal evaluation values, prompts, customer information, and deployment details.
77.3% accuracy is a validation-set snapshot, not a live production dashboard, and a 23.5% false-reject rate was an accepted trade-off rather than a closed problem at the time this was written.