do a search engine for stacks project

[kohlhase]

see https://stacks.math.columbia.edu/ this has a couple of thousands of math pages.

[kohlhase]

The stacks sources are at https://github.com/stacks/stacks-project They are simply LaTeX files with a coherent notation and well-designed cross-labeling (they have permalinks for all "results", i.e. theorems).

They are using a lightweight structure converter to html and then MathJax for formulae. This is not as nice for us, since they do not have permanent IDs for the Math elements.

So I guess we will have to run LaTeXML over it (but they are using xy.sty heavily, so we will see how that goes). And then we can do a search engine on our HTML5+MathML. Maybe we can then cross-ref into their's.

[kohlhase]

I have cloned the book and ran latex over it, it is about 7000 pages, so a formula search engine is certainly worthwile.

[kohlhase]

Unfortunately, LaTeXML does not run on my laptop any more, so i cannot try what goes wrong.

[typohnebild]

Regarding their Makefile, they have a target that creates a tex file containg all the contents and that is meant to be parsed by plastex to generate Html.
I tried to throw this big tex file into LaTeXML and got at least some output, but that looks somehow broken and not complete.
Up to now, I did know where exactly the problems are, but the fact that LaTeXML produces some output let me hope..

[tkw1536]

Have you tried just running latexml over each individual '.tex' file (with the exception of the preamble). It seems like that might just work.

[tkw1536]

Tried this myself, didn't work that well. Instead ran over book.tex as suggested above. For documentation purpoes, the procedure is:

make book.tex
latexmlc ./tmp/book.tex --dest ./tmp/book.xhtml --preload LaTeX.pool

(as suggested above by @typohnebild)

The main source of errors seems to be: Error:misdefined:&. I believe this comes from missing support for the xy package (see brucemiller/LaTeXML#450 (comment) for upstream discussion).

This leads to a Fatal:too_many_errors -- not sure if that can be ignored.

Maybe instead of waiting for latexml, we should to use their process for generating html. Then we can extract the formulae using a regex (it's just stuff between $s) and process those individually with LaTeXML?

[typohnebild]

I have used

make web
latexmlc  --dest ./book.xhtml --includestyles book.tex

This gave me at least some output, but it was not complete.

Maybe instead of waiting for latexml, we should to use their process for generating html. Then we >can extract the formulae using a regex (it's just stuff between $s) and process those individually
with LaTeXML?

That sounds reasonable, that is a similar approach as I used for the nLab.

[kohlhase]

the problem with this approach is that we do not get the IDs on the math formulae.
But if we can pull this off, then we can at least whet their appetite.
So we should do that.

[tkw1536]

I don't think it matters too much at this stage -- as far as I can tell their current website does not have any IDs to begin with. Furthermore if we figure out scheme to assign an id to each formula, we could always post-process the latexml output and add those.

Looking at e.g. https://stacks.math.columbia.edu/tag/08ZD I can see that some paragraphs have ids. We might want to assign ids to each formula by picking the closest ancestor element with an id, and then appending something like a running index. When rendering a link to a formula in the frontend, we could then remove this prefix and get the 'ancestor element' back.