Independent holdout designs can expose a core optimizer that accepts an infeasible candidate or selects the wrong one. Save the optimizer’s predictions before revealing the reference results, then test decision errors as well as numerical residuals. A small residual on ordinary designs does not establish reliable choices near a constraint.
Reserve cases that test a defined decision
State the optimizer’s application range, objective and every engineering constraint. Include ordinary points, limiting cases and competing candidates from the claimed grades and constructions. Separate correlated geometry or material families when the intended use includes a new family. Cases inspected while tuning the tool become development or regression data; reserve another untouched set for the next release decision.
These are an engineering validation protocol. NIST’s model-validation discussion supports examining residual behaviour beyond one fit statistic, and its estimation discussion requires uncertainty when comparing estimates [1]. Those sources do not prescribe a holdout count or certify an optimizer.
Check the material input before the optimizer output
Nippon Steel lists maximum strip loss of 0.85 W/kg for 23ZH85 and 0.90 W/kg for 23ZH90 at 1.7 T, 50 Hz [2]. Preserve grade, frequency, induction and specimen basis when importing the data. For an assumed 1,000 kg material account, these limits correspond to 850 W and 900 W of strip-level loss account. Neither number is a finished-core prediction. Wrong grade mapping can invalidate the downstream model even if the optimizer solves its equations correctly.
Expose false acceptance and a ranking reversal
All following design outputs and gates are assumptions for a deliberately constructed validation fixture. The objective is minimum finished-core no-load loss at 1.7 T, 50 Hz. Both candidates meet the other assumed constraints, and the maximum permitted no-load loss is 940 W. The optimizer and a separately executed reference evaluation return:
| Locked candidate | Optimizer prediction | Independent reference result | Signed residual: reference − prediction |
|---|---|---|---|
| A | 900 W | 950 W | +50 W |
| B | 910 W | 900 W | −10 W |
The optimizer marks both feasible and prefers A by 10 W. The reference marks A infeasible by 10 W and retains B with 40 W margin. It also reverses their order: B is 50 W lower than A. Under an assumed ±15 W prediction-error allocation, A fails by 35 W while B passes that error check. These are distinct failures: a residual violation, a false feasible classification and a reversed preference.
Assume the reference evaluations each have expanded uncertainty ±5 W on the declared basis. A’s reference interval is 945–955 W, entirely above 940 W; B’s is 895–905 W, entirely below it. The rejection is therefore resolved under this assumed uncertainty. Reference values and uncertainty intervals are assumed comparison inputs.
Repair the failed layer and retain an untouched check
Block release for the illustrated decision envelope. Trace A’s error to material mapping, geometry, manufacturing correction, surrogate coverage or solver implementation before changing the optimizer. Do not tune a correction merely to make this pair pass and then reuse it as untouched evidence.
Rebuild geometry and evaluate governing physical relationships independently where practical. A second calculation sharing the same unverified material curve tests implementation agreement, not that curve’s physical accuracy. Packet schedule, joint registration, coating and compression must remain identified when reference data represent an assembled core.
Report residuals, false acceptance, false rejection and candidate-order discrepancies separately. Release only the application range supported by independent cases and an identified physical-validation basis; this fixture demonstrates rejection logic rather than completion of product validation.
References
- NIST/SEMATECH, e-Handbook of Statistical Methods, Model Validation, section 4.4.4, and Estimation, section 4.1.3.1.
- Nippon Steel Corporation, Grain-Oriented Electrical Steel Sheets, catalogue D004je, April 2020, specification tables, size tolerances and Annealing by Customers.