summaryrefslogtreecommitdiff
path: root/plugins/code-modernization/scripts/tests/test_compare_tolerance.py
AgeCommit message (Collapse)Author
2026-09-24code-modernization: approve a difference once per inputMorgan Lunt
An independent check of a rewritten batch job found differences on two new inputs of one family (a blank numeric field the rewrite rejects on purpose and the old program reads as zero). Each input has several output files, so accepting the deviation meant three or four hand-written entries per input. A top-level "approvedInputs": {"F23": "reason"} in a cases file now approves every case that shares that input label. A reason is still required (blank approves nothing), a case's own reason wins, a missing file is never turned into a pass, and the result still lists the difference as approved.
2026-09-24code-modernization: harden the numeric tolerance and say what Rehost and ↵Morgan Lunt
Replace mean Follow-ups from review of the tolerance check: an absolute tolerance above 1e-6 is refused like a relative one above 1%, numbers are compared as exact decimals rather than doubles, dotted runs such as 1.2.3 or 10.0.0.1 are text that must match exactly, and the self-check now changes the leading digit of every sampled number to prove the tolerance cannot hide a real change. The business-rules header says how many rules there were at extraction, since later steps add rules. The assessment says what to do when it recommends Rehost or Replace, which change no code and so have no build command.
2026-09-24code-modernization: compare floating-point output within a declared toleranceMorgan Lunt
The equivalence check compared bytes only, which cannot pass for numeric code (Fortran, C, scientific and financial results) whose last digits differ between compilers and math libraries, so those cases pushed people to mask the numbers away. A case can now declare a tolerance, relative or absolute, with a required reason. Only numbers written with a decimal point or an exponent may differ within it; integers and every other byte must match, a relative tolerance above 1% is refused, and the tolerance and the largest difference are recorded in the result. The self-check also fails when a tolerance is loose enough to hide a change to the leading digit of a number.