The appointment has a missing middle

The status is not the answer

Picture a service calendar showing BOOKED. That label might describe a confirmed appointment, a request awaiting an advisor, a time the customer later changed, or an automatic status update. If an assistant tells the customer, “You're all set, and a loaner will be waiting,” the label alone cannot justify either promise. The test is not whether the sentence sounds helpful. It is whether the assistant had the authority and information to make each commitment.

A proposed automotive evaluation should therefore start with a decision, not a pile of exported records. Here, the decision is whether to confirm a service visit and transportation, ask a clarifying question, or hand the request to an advisor. The raw status is a clue. The relevant evidence includes what the customer asked, what the calendar and store policy actually showed at that moment, the employee's checks, the reply, and the eventual outcome. If one of those pieces is absent or cannot be used, the scope of the test must change.

One case, all the way through

Consider this illustrative service-booking case:

1. System record. The appointment later reads BOOKED for Thursday at 9:00 a.m. The transportation field is blank.
2. Missing context. The customer first asked for Tuesday at 8:00 a.m. and requested a loaner. A final Thursday booking does not explain why the visit moved or whether a car was promised.
3. Human check and decision. An advisor checks the service calendar and loaner availability. Tuesday does not work for the requested arrangement. Thursday has a service slot, but no loaner is confirmed. The advisor offers Thursday with a drop-off option rather than promising a vehicle.
4. Outcome. The customer accepts Thursday and chooses drop-off. The advisor confirms the visit, leaves transportation correctly described, and records the changed plan.
5. Possible AI question. Given only information the assistant could have seen at each moment, would it avoid confirming Tuesday or a loaner, ask for the missing check, and accurately communicate the Thursday arrangement after the customer agrees?

The final booking is useful only if the earlier request, availability check, decision, and response can be reconstructed. Otherwise a test may reward a model for guessing the right final state while making an unsupported promise along the way. The outcome matters, but so does the path to it.

Make the grading rule observable

A rubric for this narrow task can be written before any model runs. It should specify the allowed tools and information, the customer's goal, when confirmation becomes authorized, what counts as a handoff, and the final state to inspect. A reviewer should be able to point to an action or utterance, not merely describe an answer as “good.” For this booking case, a rubric might ask:

• Understanding: Did the assistant identify both the service visit and transportation request, rather than completing only the easier half?
• Required check: Did it verify the offered slot and loaner status through an allowed source or ask an advisor when it could not?
• Premature commitment: Did it claim a booking or loaner was confirmed before the required check and customer acceptance? Count that as a critical error separately from task completion.
• Escalation: When the tool, policy, or availability answer was missing, did it stop and ask a person instead of inventing a result?
• Recorded outcome: After agreement, did its final message and any permitted system action match the Thursday visit and drop-off plan?

A completion score alone hides important distinctions. An assistant might eventually reach the Thursday booking but incorrectly promise a Tuesday loaner first. Another might prudently ask for help and leave the booking incomplete. Those are not equivalent failures. A proposed report should show the task result, critical-error count, reason for escalation, and whether the final state matches the communication, with examples of disagreements reviewed by a qualified automotive operator.

The research behind τ-bench offers a useful general principle: evaluate an agent interacting with a user, domain rules, and tools against a defined final state, and check consistency across repeated trials. Borrow the evaluation design for an automotive task rather than its published scores.

Build cases that can actually distinguish behavior

One convenient example is not enough. A useful test set would include a confirmed slot with no transportation request; a requested time not yet checked; a booked visit whose loaner is unavailable; a changed appointment that has not been acknowledged by the customer; and an out-of-policy exception that requires an advisor. Each case needs its own correct decision and evidence. Vary the language without changing the underlying facts so the assistant cannot pass by spotting a canned phrase.

Separate construction, calibration, and held-out testing. If the same examples are supplied for training or repeatedly shown to a model's builders, later scores no longer provide an independent read of that behavior. For an interactive agent, record the version being tested, tool permissions, policy version, case-selection rules, review process, and repeat behavior. Sample-size limits and uncertain labels belong in the report. A clean score on a narrow set does not prove that the system is safe across every dealership, market, or real customer conversation.

The NIST AI Risk Management Framework Core organizes risk work around govern, map, measure, and manage. NIST's Generative AI Profile is a voluntary companion for generative systems. Neither substitutes for defining the actual automotive task, sources, and failure criteria; both support making those choices explicit and revisiting them as the system changes.

Rights and privacy change the design

Even a perfectly designed rubric is unusable if the source cannot be shared or the proposed use is not allowed. Service histories can contain names, phone numbers, vehicle identifiers, complaint details, payment information, staff notes, and vendor-governed fields. A dealer's access to a tool or an export does not by itself grant permission to license that material for training, testing, or redistribution. A pilot must identify the source owner, contractual restrictions, intended recipient and use, required exclusions, handling rules, and who is authorized to approve the scope before any transfer.

Removing obvious names may still leave indirect clues. A task could instead begin with purpose-built fictional scenarios reviewed by a qualified expert, but synthetic cases must be labeled, checked against the real task, and never presented as customer evidence. If actual operational material is needed, minimization, access controls, retention, and review need to be specified in the proposed agreement. The FTC's automobile-dealer Safeguards Rule FAQ explains why customer-information protections matter for dealers that finance or facilitate financing. The FAQ informs handling requirements; source rights still need their own review.

A short first inquiry should therefore ask for the task or broad workflow, not an upload. If permissions or the necessary context are missing, the honest project decision may be to narrow the scope, commission expert-created cases under separate terms, or stop. More data cannot fix a missing right or an undefined judgment.

What a scoped output would say

A buyer could ask for a defined set of service-booking scenarios, an expert review protocol, and a report on one assistant version. A source owner could describe relevant workflows and known restrictions. A proposal would then specify case construction, permissions, reviewer qualifications, allowed use, acceptance criteria, and commercial terms.

The useful unit is not “years of service data.” It is a question the system can get right or wrong, enough permitted context to judge its choice, and an outcome that tests whether the choice held up. That is what turns an automotive status into a meaningful evaluation candidate.