How grading works

1. The deterministic gate runs first

Every shift is a scenario: a sequence of tickets, each with the actions you may take. Before anything is scored, a deterministic check reads the actions you actually took and applies the scenario’s rules:

• Forbidden actions. Some actions end the shift and zero every score: asking a customer for a one-time code, pushing unreviewed code to production, giving clinical advice, approving an AI agent’s action you could not trace to a source. These are the bright lines of the job, and a shift that crosses one is recorded as disqualified. Nothing you did afterwards can argue it back up.
• Required actions. A scenario can require a step (verify the customer before releasing funds). Missing it is recorded the same way.

2. Award scoring, not opinion

Daily shifts are built from a bank of tickets written and reviewed by people who have done the job. Each option on a ticket carries a grade the author set in advance: strong (4), ok (2) or weak (1). Your score on a competency is the mean of the awards you earned on its tickets. No model is involved in this path; the same choices always produce the same grade, which is why a proof can be recomputed and checked by anyone with the link.

Authored scenarios with free-text answers use a model grader with a deterministic fallback. The model grader reads the same rubric; its criteria, rationale and the model used are stored with the evaluation, and the gate above still rules.

3. From a score to confidence

A score says how you did on one ticket. A proof says how sure we are about a skill. Each graded ticket becomes an evidence item with a confidence:

confidence = (difficulty ÷ 5) × (score ÷ 4) × independence

• Difficulty is set by the author (1 to 5), so a hard ticket says more than an easy one.
• Independence is 1 when you worked alone and 0.5 when you took a colleague’s steer. Asking is allowed; the record says you asked.
• Combination. Independent evidence on the same skill combines as 1 − Π(1 − cᵢ): more corroborating shifts can raise confidence, never lower it, and it is capped at 1.

A shift can be revised once. The first attempt stays on the record, the revision is flagged as such, and the improvement can lift that shift’s simulated pay by up to 25%. Both attempts appear on the proof.

4. Judgment over AI

Most tickets carry an AI copilot’s suggested answer with a confidence. The author knows whether the suggestion is right. You do not. The strong option on such a ticket is the one that matches reality: accepting the suggestion when it is right, overruling it when it is wrong. Following a wrong suggestion into a forbidden action is the rubber-stamp, the automation-bias failure these roles exist to prevent.

Calibration is the share of copilot tickets where you made the right call, split into accepted when right and overruled when wrong, with rubber-stamps counted separately. It is computed from the record at grading time and shown on the result, the profile, the proof and the instructor dashboard. Across the platform so far, 100% of copilot calls were right.

Knowing when you are unsure. Before choosing on a copilot ticket you say how sure you are: sure, fairly sure or guessing. The answer is not graded. It is compared with what happened: sure claims about a 90% chance of being right, fairly sure 70%, guessing 50%. Over the newest thirty stated calls, and only once ten exist, the proof reports the mean claimed chance against the share actually right, with a Brier score (the mean squared gap between the two, lower is better). More than fifteen points of claimed over actual reads as overconfident, the reverse as underconfident, and anything within that band as well calibrated. “Caution when it counts” is the share of wrong-suggestion tickets on which the person overruled the copilot while at least fairly sure. Always answering sure shows as overconfident and always guessing as underconfident, so neither improves the figure; each stated level is shown on the replay beside the call it belongs to.

The copilot aims at your blind spots. From the third graded shift the daily queue is assembled from your own record: the copilot is wrong more often in the competency where your level is lowest and right more often where it is highest. The aim is fixed by the day and your profile, so the same shift can be rebuilt and replayed, and it changes as your levels change. A score earned against a copilot that targets your weak spots cannot be earned by memorising the bank. Because the mix is harder for some people than others, every judgment figure is shown with the share of wrong suggestions faced, and each replay says which competency the copilot was aimed at.

5. Takes feedback

A 1:1 with the manager turns the record of recent shifts into coaching points. Every correction carries a target that later shifts are measured against without anyone’s say-so: adopt an action, drop one, avoid a forbidden action, or raise a skill over a window of shifts. After each graded shift the point is re-measured and its status written to an append-only ledger: open, implementing, implemented, lapsed, or regressed when an implemented behaviour comes back.

“Takes feedback” is computed from that ledger over the newest six resolved correction points, and only once at least three have resolved. Uptake is the share that reached implemented. Speed is the median number of shifts from the 1:1 to implemented. Durability is the share of implemented points that did not later regress. The headline on the 0 to 4 scale is 4 × (0.5 × uptake + 0.25 × speed + 0.25 × durability), where speed scores 1 for a median of one shift and loses a fifth for every further shift. Below the minimum the proof says “not enough evidence yet” rather than showing a number, and every figure opens to the coaching points and shifts behind it.

6. Reasoning on key decisions

A good score can be a lucky guess. On the tickets that matter, every copilot ticket, every flagged ticket and any the author marks as a key decision, the shift asks for one line on why, up to 240 characters, optional on the first graded shift and required from the second. The line is graded 0 to 2 on a fixed rubric: it names the deciding fact (the mismatch, the policy, the risk), it is consistent with the option chosen, and it invents nothing. The rubric is written per ticket, or derived from the strong option and the facts the ticket carries.

The evaluator model grades each line with that rubric, the chosen option and whether the copilot was right. When no model is available the deterministic rules check grades it instead, by how many of the rubric’s deciding facts the line names, with a cap of 1 when the option chosen was not the right call; a line graded that way is labelled as such wherever it is shown. The reasoning level is the mean over the newest twenty graded lines, scaled to 0 to 4, shown only once eight exist. Every line appears verbatim on the replay with its grade, so a reader can judge the judgment.

7. Works with others

Team shifts, help requests and steers already record how someone behaves with colleagues. Four parts are read from that record, each shown only once it rests on at least five events, and each opening to the events behind it:

• Helps. The share of a colleague’s help requests the person took rather than declined, with the mean grade on those tickets beside the mean on their own shifts.
• Asks well. Of the shifts on which the person asked a colleague for a steer, the share where the shift’s weakest skill was one of their weaker skills overall: asking where it is needed, not to avoid work.
• Hands over cleanly. The share of tickets the person handed to a teammate that carried a note, with the resolution rate of tickets they gave and tickets they received.
• Reliability. Of the team-shift tickets the person claimed and then either finished or let go, the share resolved rather than released without work.

Claiming and releasing to look busy lowers reliability; taking help requests and failing them shows in the helps grade; asking for steers everywhere shows in asks well. Nothing here is self-reported.

8. Adapts to change

The world changes during a shift. On every daily queue a change lands part way through, at a position fixed by the day’s seed: a policy update or a manager’s note, posted to the shift and to the inbox, with the policy library reading the rule as it now stands. The two tickets that follow are only right if the change is applied, and the copilot keeps citing the old rule. Those tickets grade a competency of their own, Adapts to change, on the same strong, ok and weak scale as the rest, and the replay shows the change landing and what the person did next.

The change and its position are seeded per day, so they cannot be anticipated from a previous shift, and a sized shift (a colleague’s ticket, a team day, a live verification) carries none, so those stay comparable.

9. Works with AI

On one ticket of every daily queue the person may use a work assistant: free chat on the characters model, the same kind of general tool people reach for at work. It is told to hold one wrong belief and to state it confidently whenever it is relevant, which is how a general model behaves on a company’s own rules. The whole exchange is recorded with the shift and nothing about it changes the grade on the ticket itself. The person is told this before they type.

After the run the exchange is graded on four parts, each 0 to 2: asked a precise question naming the facts of the ticket; checked a claim before using it; rejected the wrong belief rather than acting on it; decided with their own reasons rather than the assistant’s. The evaluator model grades them from the transcript, the final call and the person’s one-line reason; when no model is available a documented rules check does, and the grade says so. Two deterministic checks sit beside the parts: whether the assistant’s error reached the final answer (the weak or forbidden option was taken), and whether the person leaned on the tool (more than five questions on one ticket).

Works with AI is the mean of the four parts over the newest ten graded exchanges, shown as a level out of 4 only once four exist. The wrong belief is seeded per shift, so copying the assistant’s answer whole is the thing the rubric catches. Sized shifts (a colleague’s ticket, a team day, a live verification) carry no assistant.

10. Benchmarks against working professionals

Practising customer-service agents, analysts, dispatchers and engineers play the same seeded shifts as learners, with the same copilot, inside benchmark cohorts run by an instructor who records each one’s job and years. Per role and competency their levels form a distribution, rebuilt at every deploy and on an admin’s request. A learner’s level is placed in it and the proof says “above p% of working roles”: the share of professionals strictly below that level.

Nothing is claimed until at least 30 professionals have a level on that competency in that role; below that the proof says so instead. Professionals are never shown their own standing while they play, and their proof is theirs to keep.

No professional sample yet, so no proof carries a standing.

11. Employer challenges, validation and the audit

An employer can set a challenge on a role of their own company: a brief, their policy documents and a deadline. A candidate who takes the invitation works one shift from that role’s bank, graded exactly as every shift here, which stays on their own proof. The employer reads the graded replay and the same signals the proof shows, ranked by score and then judgment over AI. A ranking is an ordering of evidence, not a verdict. Nothing on CareerLyfe decides who is hired, shortlisted or rejected; a person at the employer does, with the replay in front of them, and should be ready to explain why. We do not market any score as a prediction of job performance.

Validation pilot. Employers may later record their own hiring outcome and a 90-day performance rating for each candidate. Once 20 rated pairs exist we report the correlation between challenge score and rating here, whatever it is. So far: 0 rated pairs across 0 challenges, 0 decided; no correlation is claimed yet.

Bias audit. Candidates may give optional demographics (gender, ethnicity, age band, each with “prefer not to say”). Employers never see them and they never touch a grade. Admins see shortlisting rates by group and each group’s impact ratio against the best-selected group, in the New York City automated employment decision tool format, with groups under five withheld. This report is the gate on ever marketing a score as a hiring signal.

Retention. A challenge shift is the candidate’s own evidence and stays on their proof like any shift. The employer’s decision, note and outcome are kept for the validation pilot and the audit; demographics are kept only as long as the person keeps them on their account and can be cleared at any time.

12. Verified live, on request

Anyone with a profile link can ask the learner to work a fresh shift. The learner is emailed and has 48 hours. The shift is five tickets drawn from the role’s bank with a seed the learner has never played, worked alone: a colleague’s steer and the team channel are both refused on it. It is graded exactly as any other shift, written to the proof, and its replay appears on the same link marked as the live verification, with the date. A profile that has a live verification within the last 30 days shows it at the top.

One open request per link at a time, three a day per requesting address. The learner can decline; the requester is told that and nothing more. Nothing about the requester is shown to other viewers of the link. A live verification stays out of the manager’s coaching windows, so it cannot be used to clear a coaching point. Someone else sitting the shift remains possible, as with any remote test; the reasoning lines on the replay let a reader compare the live shift with the history.

13. Reliability

• Reproducibility. The award grader is deterministic by construction: the same recorded actions give the same scores. A shared proof link recomputes the proof from stored evidence and checks a signature, so a copied page cannot be forged.
• Grader agreement with hand grades. The Works with AI grader in use (rules/v1) matched the hand grade exactly on 76% of the 80 part scores in the calibration set (20 exchanges, four parts each), and was within one point on 95%. By part: asked 75%, verified 60%, rejected 95%, decided 75%. The set was written and graded by the CareerLyfe team, not by independent graders, and it covers only the Works with AI grader; the award grader is deterministic and needs no such figure, and the reasoning grader has no calibration set yet. When the figure names the rules check rather than a model, no model was configured on this deployment.
• Fairness checks. Tickets are reviewed for cultural and regional assumptions before publication, and the daily queue is sampled by seed, not by anything about the learner. Scores are never adjusted by who the learner is.

14. What we do not claim

• No job guarantee. A proof is evidence an employer can read and check, not a promise of an outcome.
• Salary is simulated. The pay on a payslip is a game mechanic; the evidence is the real product.
• A score is evidence, not a verdict. It shows what you did on specific tickets on specific days.
• Learners own their evidence. Links are private until shared, and a learner can revoke one at any time.