Problems with host-expressions in a DSL with multi-pass compilation

[michaelballantyne]

Mike ran into this trying to support Racket subexpressions in the multi-pass ANF compilation example.

Racket references to DSL variables

Expansion of the Racket subexpressions is delayed, so they don't get renamed in each pass. Thus when we finally expand them as part of the final compilation, or as part of an analysis like get-racket-referenced-identifiers in a later pass, they only rename once via the reference compiler. The final name they reach is one that only had a binder in an intermediate syntax.

Possible solutions:

1. Require that compiler passes local-expand the Racket subexpressions to keep them in lockstep. (Would local-expand of the renamed reference fail because the binder is not yet present?)
2. Change compile-reference to follow a chain of compiled-ids entries.
3. Change binding-as-rkt (the implementation of reference compilers) to iterate the call to compile-reference (perhaps via a compile-reference* helper to follow a chain of compiled-ids entries.

I prefer option 3. I don't think that reference compilation outside of reference compilers should iterate the lookup, because that could hide mistakes in a DSL compiler implementation where renaming isn't run on all portions of the program the same number of times.

Return to DSL code inside the Racket subexpression

What if the host expression contains not only a reference to a DSL variable, but a whole DSL fragment that requires its own multi-pass compilation? We currently don't have a means to line up the passes of compilation for this inner fragment with the the corresponding passes for the outer fragment. A get-racket-referenced-identifiers would force the complete compilation of the inner syntax, which would ultimately fail because the binders for the compiled free references would not be bound yet.

For now, I think we should try and solve the first problem with option 3 and accept the second problem as a limitation.

[michaelballantyne]

Possible solutions to the second problem (full multi-pass DSL programs inside host expressions):

1. Compile the outer code independently of the inner code. For example, in a language whose compiler normally removes unreferenced variable bindings, don't try to do get-racket-referenced-identifiers on host expressions, and assume that any variable that is in scope around a host expression is potentially referenced, and conservatively avoid removing it. Then, it might work to do all the inner code compilation when the host-expression suspension resumes. If compilation takes N passes, there will be a chain of N bindings of the free variables from the surrounding compilation. (Though these bindings may not be in-context in the environment? I'm not sure... What does re-expansion with nonterminal-expander do to #%host-expressions anyway? Does it multiply-wrap them?)

2. Provide a local-expand-like API except that it creates Racket-variable-like bindings for all compiled names in a definition context that it uses to expand the expression. That way, the inner expression can completely compile first, to Racket code that refers to these compiled variables. And then the outer compilation can examine the inner code, and then create real final bindings of these names. I guess these latter bindings would need to shadow the temporary bindings for inner compilation, and I'm not sure how to get the scopes right for that.

3. Get fancier and make it possible to expand inner code without compiling it, and even invoke passes on inner code to make the expansion in lock-step.