Make transforms part of the public MPS

[rptb1]

Fixes #111 .

Progress towards resolving #110 .

Part of the plan to meet Configura's requirements by separating and reviewing each implementation.

This branch cherry-picks and edits changes from branch/2022-01-23/cet-merge-2.

EDIT: Original requirements are discussed and linked in design.mps.transform, introduced by this work.

[rptb1]

Executing proc.review.entry

1. Start time 11:29.
2. Entry criteria: entry.universal, entry.impl, entry.design.
3. Source documents for recent requirements are available and linked via issues, and original discussions with Configura are linked via the design document.
4. Most source documents are unreviewed.
5. proc.review.brief-check: @thejayps and @rptb1 observed that the test, ztfm.c would not pass review and needs reworking. We have raised Testing of transforms is unclear and overengineered #242 and ztfm.c fails to meet its requirement to test transforms on all platforms #243 to fix this. We propose excluding ztfm.c from review except as a source document.
6. Entry passed otherwise.
7. End time 11:45.
8. Entry took 16 minutes.

[rptb1]

Executing proc.review.plan

1. Start time 11:46.
2. plan.review.time: Git reports about 2400 lines, but we're not reviewing ztfm.c whic h is 1400 of those, leaving about 1000 lines. This is a bit over 1h @ 10 l/min but since this is mature tested and reviewed code, what we are mostly checking is consistency and correct merging of CET-specific code into master, so we can check faster over the tranforms code itself. I think we can do this within 1h of checking, and time during logging.
3. proc.review.plan.schedule: Review is scheduled for 2023-06-15 13:00 for about 3h with breaks.
4. proc.review.plan.source: Public MPS source code does not include transforms #111 , Ravenbrook MPS sources are out of sync with Configura's sources #110 , design references, and Testing of transforms is unclear and overengineered #242 , ztfm.c fails to meet its requirement to test transforms on all platforms #243 concerning why ztfm.c is excluded.
5. proc.review.plan.rule: All rules except rule.code.python.
6. proc.review.plan.roles: @thejayps is .role.leader, @UNAA008 could do .role.check.source and .role.check.clarity , @thejayps could do .role.check.consistency and .role.check.source , @rptb1 .role.check.consistency , .role.check.backwards and .role.check.correctness. Note that we do not consider correctness (of the transform code) to be a primary focus because it's mature and well tested code.
7. proc.review.plan.homework: none.
8. proc.review.plan.invite: done via Keybase.
9. proc.review.plan.doc: See above for links, and this pull request.
10. Planning finished 12:03.
11. Time taken: 17 minutes.

[rptb1]

@thejayps asks that we spend a bit of time at the end of the review dividing the issues into those that must be edited before 1.118.0 and those which can be raised for later.

[thejayps]

Executing proc.review.ko

1. Start time 1305
2. @thejayps specified that the product document does not include ztfm.c as it includes many defects. It is a source document.
3. @thejayps explained that FIXME checks started failing after entry, but this does not need hold up the review.
4. Checking roles as per suggested in plan: @thejayps to start from code and @rptb1 to start from documentation to achieve some role.check.backwards.
5. .improve @rptb1 notes that comments that aren't inline ones haven't specifically referenced exactly what they are talking about. References must be made unambiguous and clear by checkers.
6. Checking started at 1328, logging to commence at 1435 (5 mins break included in 1h check time).

[thejayps]

Checking
comment: this PR doesn't obviously link to the review comments made by @gareth-rees in #76 .

M1 - this pull request doesn't obviously refer to the original requirements from Configura

Fix - reference added to PR description

[thejayps]

performing proc.review.check

[thejayps]

1h checking, checked all of trans.c
Found: 1M 8m 2comments 2questions

[rptb1]

Executing proc.review.check

1. Start time 13:30.
2. Built and ran tests on Kiwi-Ubuntu (lii6ll) and build the manual, no errors.
3. m. design.mps.transform is missing from design/index.txt. rule.generic.complete
4. 10M, 16m.
5. Finished checking 14:22.
6. Checking took 52 minutes.

[UNAA008]

M: Issue #111 refers to speculation about the use of mps_pool_walk and
mps mps_arena_roots_walk similar objectives to transforms
but the narrative in this issue doesn't reach a conclusion about
whether these are resolved sufficiently for the issue to be closed.
for example
#111 (comment)

M: Issue #110 refers to PR #76 as something that would close this issue.
Unclear whether that branch has been completely mined.

m: Copyright dates require update in design/transform.txt

M: The "not yet written" section in design/transform.txt is extensive, requires explicit review
to determine if any of it is essential for general release.
M: The transform.rst section of the manual does not appear to me to be comprehensive enough for a general release. Perhaps a use case would make a big difference.

[thejayps]

Note to editor:

M: Issue #110 refers to PR #76 as something that would close this issue. Unclear whether that branch has been completely mined.

From #214 (review)
Check that #76 is completely mined as one of the steps pre release of 118

[UNAA008]

I: Estimates for review time based on lines of code don't tend to work for role.check.source because this requires reading linked documents which add significantly to the bulk of the work.
Perhaps this could be estimated after a bit of analysis of what's listed as source documents.

[rptb1]

m. design.mps.transform is missing from design/index.txt. rule.generic.complete

Fixed in 24417c2

[rptb1]

Added pending label so that we later deal with process improvements and meta issues raised in #214 (review)

[thejayps]

resolving conversations pre exit

[thejayps]

Executing proc.review.exit

1. Start time 0150

2. Transgression - Improvement not acted on: Make transforms part of the public MPS #214 (comment) - Raise Ways of estimating progress through checking are not clearly described for all checking roles #250

3. Transgression - no remaining defects estimated. Do now: ~10 defects discovered considering duplicates. -> Use "75%" rule: 10/0.75 = 13.3 -> 3.3 Major defects remaining. This is at face value above the acceptable level for exit.pass, however we note that the code itself is mature and well tested with Configura, and most of the defects are around the documentation. As we have no other clients actively using transforms now we choose to exceptionally be more accommodating than usual of this estimated number of remaining defects in considering the outcome of this exit. Raise Transforms is estimated to contain more than 2 undiscovered major defects #251

4. exit.pass

5. review.exit.calc:

   • hours used: 25-30 (very roughly estimated),
   • hours saved: 100 - 30 = 70
   • major defects remaining: 3.3

6. exit finished at 0222 - exit took ~30 mins

[thejayps]

proc.review.exit.pass

[thejayps]

executing proc.merge.pull-request (currently in #228)

1. Start time 0225
2. 3 'sort of': There is a test: ztfm.c which we believe tests transforms correctly. It has not been reviewed as it is likely to have many defects (not suspected to be around .correctness) and it needs redesigning. There are issues logged that will prompt this to happen.
3. Commit messages for this merge may be incorrect, I was never invited to prepare the correct commit message despite following the merge procedure (and adapting it reasonably to take account of the merge conflicts)
4. Merge finished at 0238

[rptb1]

Fixing up merge mistakes by catching up from master, re-merging, and force-pushing to master. See #253 for prevention.