Score the evidence.
A shared rubric helps reviewers distinguish a polished submission from demonstrated engineering ability.
100 points, with critical checks kept visible.
Multiple choice contributes 10 automatic points. The remaining 90 require a reviewer, including 10 for the live walkthrough. An unfinished review is never presented as a completed score.
Tenant isolation and code ownership require separate reviewer confirmation. A failed check flags a critical concern regardless of total score. It does not automatically reject a candidate.
Use the same 0–4 anchors for every criterion
Weighted points = rating ÷ 4 × criterion weight. Record specific evidence in the review notes. Calibrate the rubric on known engineers before using it to make decisions.
What reviewers look for
Correct authorization and behavior
Active actor membership in the requested tenant; editor/admin only; project scoped to that tenant; indistinguishable missing/forbidden; correct result.
Meaningful edge-case tests
Tests cross-tenant access, inactive/viewer membership, missing project, unknown/non-string/blank/oversized name, success, and input immutability.
Types and reasoning
Narrows unknown without unsafe casts, trims and limits the name, returns a new object, explains why in-memory checks are not a transactional database authorization boundary.
Finds the exploitable tenant boundary
Explains A user accessing B project; membership query lacks tenant and active/role checks; resource read/write and audit tenant are not bound to authorized membership.
Consistency and concurrency
Identifies non-atomic update/audit, count-then-write race, process-local cache consistency and a credible concurrency fix. Does not incorrectly claim parameterized SQL is injectable.
Prioritization and regression plan
Prioritizes real cross-tenant exposure, then supplies tenant-scoped authorization, validation, atomic mutation/audit, appropriate errors, and adversarial regression cases.
Identity and tenant boundaries
Server-verified identity/membership, roles, tenant-scoped keys, cache and queue context; explicit application/RLS tradeoff; considers privileged-role bypass and pooled session state.
Growth and workload tradeoffs
Measures tenant skew, explains shared-schema versus stronger isolation, composite indexes and pagination, bounded pools, rate limits/backpressure and noisy-neighbor mitigation.
Operational maturity
Backward-compatible expand/backfill/contract migrations, observable tenant-scoped failures, idempotent jobs, safe rollback and tenant-aware restore/export thinking.
Working full-stack slice
Reviewer exercises real deployed read/create/rename; Postgres persistence survives refresh; React states work; reproducible schema and seed. Partial credit for runnable source with deployment failure evidence.
Isolation and authorization
Reviewer personally verifies A cannot read/write B, viewers cannot mutate, unauthenticated access is denied, and audit writes are atomic using the deployed runtime role.
Effective prompting and ownership
Prompt defines constraints, acceptance tests and boundaries; iterations respond to observed behavior; candidate evaluates output and can explain the resulting code. No score for prompt length or model brand.
Evidence and honest limitations
Actual commands/results, adversarial tests, inspected diff and reproducible commit. Separates passing local tests from verified deployed behavior and states remaining risks.
Explains the implementation
Traces UI to authorization to database and audit; explains a rejected alternative; predicts failure behavior without outsourcing the explanation.
Handles a small change
Implements or precisely sketches viewer-write denial or tenant-scoped archive behavior, writes the failing regression first, and explains impact. Use the same change prompt across the cohort.
The 15-minute walkthrough
- Minutes 0–5: Candidate traces a rename from React through session, membership, database write, and audit. Ask which AI suggestion they changed and why.
- Minutes 5–12: Use this standard change: “Add archive/unarchive for projects. Only admins may change archive state; viewers can read it. Show the failing authorization test first.” The candidate works without AI for this small change; an incomplete implementation can still demonstrate sound reasoning.
- Minutes 12–15: Run or inspect the test, discuss a failed case, and ask what they would change before production.
Keep the change prompt consistent across the cohort. Reproduction commands and a precise code sketch earn partial credit when time runs out.
Before a real candidate pilot
- Use the same clean starter, initial feature set, and AI-tool access budget for the cohort. Verify database, auth, and hosting setup before starting the timer.
- Offer this as a paid work sample; agree on accommodations and timing in advance. Extra time is recorded without a scoring penalty.
- Observe the independent section live if independent authorship matters. A browser cannot prove whether an external AI tool was used.
- Have two reviewers independently score a few known engineers. The initial score bands (85 and 70) are working calibration labels, not validated hiring cutoffs.
- This pilot uses one hiring-team reviewer password. Keep it private. Agree on access, retention, and deletion with candidates. Add individual accounts and invite management before expanding to multiple hiring teams.
- Do not run arbitrary candidate code on the assessment server. Review and execute submissions in an isolated environment.
Technical and assessment references
- Next.js authentication and authorization: checks belong at the data/mutation boundary.
- PostgreSQL row security policies: role privileges and ownership affect enforcement.
- OPM work samples and simulations: use tasks representative of the job with trained assessors.