Per-pathogen docs prototyping

docs/Makefile

@@ -0,0 +1,36 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first three.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+BUILDDIR      ?= build
+SOURCEDIR     = src
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help help-docker Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+livehtml:
+	sphinx-autobuild -b html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.ONESHELL:
+docker-html:
+	set -euox
+	docker build -t nextstrain-docs-builder --network=host .
+	docker run -it --rm \
+	--name=nextstrain-docs-builder-$(shell date +%s) \
+	--init \
+	--user=$(shell id -u):$(shell id -g) \
+	--volume=$(shell pwd):/home/user/src \
+	--workdir=/home/user/src \
+	--env 'TERM=xterm-256colors' \
+	nextstrain-docs-builder

docs/README.md

@@ -0,0 +1,56 @@
+# Docs about Zika Docs
+
+
+## Dependencies
+
+in `conda.yml` - install these in the same environment you run augur in
+
+## Testing (WIP!)
+
+To test with nextstrain run we either need to symlink the checked out repo:
+
+```bash
+mkdir ~/.nextstrain/pathogens/zika
+ln -sv ~/github/nextstrain/zika ~/.nextstrain/pathogens/zika/local=NRXWGYLM
+```
+
+and check it's installed:
+
+```console
+$ nextstrain version --pathogens
+Nextstrain CLI 10.2.1.post1+git 
+
+Pathogens
+  ...
+
+  zika
+    zika@local (default)
+```
+
+or you should be able to use `nextstrain setup` from this branch (UNTESTED!)
+
+```bash
+nextstrain setup 'zika@james/docs-prototyping'
+```
+
+> Assumptions:
+> 1. That we installed using the symlink method, if not replace `zika@local` with `zika@james/docs-prototyping`
+> 2. That you've installed the docs dependencies (see above) into the nextstrain runtime you want to use. The following commands use `--ambient`
+> 3. That you're running from an external analysis directory (shouldn't matter, but easier to clean up if things go wrong)
+
+We then need to build the docs - in the future this step will be done during `nextstrain setup`, or when we build the image for a repo (via buildpacks etc etc), or some other way - but for now we can achieve this in a ad-hoc way:
+
+```bash
+nextstrain run --ambient zika@local docs . build
+```
+
+And then open the docs (might only work on MacOS). Long term we wouldn't use `nextstrain run` for this, we'd have a nicer interface like `nextstrain docs zika@local`.
+
+```bash
+nextstrain run --ambient zika@local docs . open
+```
+
+## Helpful commands:
+* `make html && open build/html/index.html`
+* `make livehtml`
+* `make clean`

docs/Snakefile

@@ -0,0 +1,19 @@
+# -----------------------------------------------------------------------------
+# This snakefile is an ad-hoc way to test out concepts for docs using nextstrain run
+# -----------------------------------------------------------------------------
+
+rule build:
+    params: workflow.basedir
+    shell:
+        r"""
+        cd {params[0]}
+        make clean && make html
+        """
+
+rule open:
+    # open is MacOS specific I think?
+    params: workflow.basedir
+    shell:
+        r"""
+        open {params[0]}/build/html/index.html
+        """

docs/conda.yml

@@ -0,0 +1,18 @@
+name: ncov-docs
+channels:
+  - defaults
+dependencies:
+  - make
+  - pip
+  - pip:
+    - nextstrain-sphinx-theme>=2022.5
+    - recommonmark
+    - requests
+    - sphinx
+    - docutils
+    - sphinx-argparse
+    - sphinx-autobuild
+    - sphinx-copybutton
+    - sphinx-markdown-tables
+    - sphinx-tabs
+    - PyYAML

docs/make.bat

@@ -0,0 +1,37 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+if "%BUILDDIR%" == "" (
+	set BUILDDIR=build
+)
+set SOURCEDIR=src
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/src/_static/css/configuration-reference.css

@@ -0,0 +1,29 @@
+/* Custom CSS to be applied to the reference/workflow-config-file.rst
+   page. That page defines a custom class of .configuration-reference */
+
+
+/* We detail a lot of nested (snakemake) configuration entries in the
+page. The parent key (top-level config key) is <h3> and sub-keys are <h4>.
+The default nextstrain-theme renders <h3> and <h4> extremely similarly.
+The following style changes are intended to convey that certain config
+entries are children of a higher-level config, rather than being top-level
+config parameters themselves */
+
+.configuration-reference h4 {
+    font-size: 100%;
+}
+
+/* Pad lists generated by a (local) contents directive showing sub-keys */
+.configuration-reference section > section > div.contents.local.topic {
+    margin-left: 24px; /* same as a nested <li> */
+    margin-top: -20px; /* CSS can't select previous sibling, FYI */
+}
+.configuration-reference section > section > div.contents.local.topic > ul > li {
+    list-style: circle;
+}
+/* pad out their siblings (which come _after_ the list) so that they
+   are in line with the start of text in the preceding <li> element */
+.configuration-reference section > section > div.contents.local.topic ~ * {
+    margin-left: 48px; 
+}
+

docs/src/conf.py

@@ -0,0 +1,155 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('..'))
+sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+from datetime import date
+import subprocess
+
+def git_authors():
+    result = subprocess.run(
+        ["git", "shortlog", "--summary", "HEAD"],
+        stdout = subprocess.PIPE,
+        check  = True)
+
+    names = [
+        line.strip().split("\t")[1]
+            for line in result.stdout.decode("utf-8").splitlines()
+    ]
+
+    return names
+
+def prose_list(items):
+    if not items:
+        return ""
+    if len(items) == 1:
+        return items[0]
+    elif len(items) == 2:
+        return " and ".join(items)
+    else:
+        return ", ".join([*items[0:-1], "and " + items[-1]])
+
+project = 'Zika Workflow'
+#TODO copyright date change? copied from augur
+copyright = '2014–%d Trevor Bedford and Richard Neher' % (date.today().year)
+author = prose_list(git_authors())
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'recommonmark',
+    'sphinx.ext.autodoc',
+    'sphinxarg.ext',
+    'sphinx.ext.graphviz',
+    'sphinx.ext.napoleon',
+    'sphinx_markdown_tables',
+    'sphinx.ext.intersphinx',
+    'nextstrain.sphinx.theme',
+    'extensions.config_value_role',
+    'extensions.snakemake_dag',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# Add external directories to the source path
+# This allows including RST files from outside the docs/src directory
+html_extra_path = ['../../shared/vendored']
+
+# A string of reStructuredText that will be included at the end of every source
+# file that is read. This is a possible place to add substitutions that should
+# be available in every file.
+rst_epilog = f"""
+.. |authors| replace:: {author}
+"""
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'nextstrain-sphinx-theme'
+
+html_theme_options = {
+    'logo_only': False, # if True, don't display project name at top of the sidebar
+    'collapse_navigation': False, # if True, no [+] icons in sidebar
+    'titles_only': True, # if True, page subheadings not included in nav
+}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# These paths are either relative to html_static_path
+# or fully qualified paths (eg. https://...)
+html_css_files = [
+    'css/custom.css',
+    'css/configuration-reference.css'
+]
+
+# -- Cross-project references ------------------------------------------------
+
+intersphinx_mapping = {
+    'docs.nextstrain.org': ('https://docs.nextstrain.org/en/latest/', None),
+    'augur': ('https://docs.nextstrain.org/projects/augur/en/stable', None),
+    'auspice': ('https://docs.nextstrain.org/projects/auspice/en/stable', None),
+    'snakemake': ('https://snakemake.readthedocs.io/en/stable', None),
+}
+
+# -- Linkchecking ------------------------------------------------------------
+
+linkcheck_ignore = [
+    # we have links to localhost for explanatory purposes; obviously
+    # they will never work in the linkchecker
+    r'^http://127\.0\.0\.1:\d+',
+    r'^http://localhost:\d+',
+    # the top level bucket link 404s
+    r'^https://data\.nextstrain\.org$',
+    # they block the client, probably anti-scraping measure
+    r'^https://czgenepi\.org/resources',
+    r'^https://science\.sciencemag\.org/content/early/2020/06/05/science\.abb9263',
+    # this link is correct but the lack of a top-level dataset means
+    # it 404s initially — because the point of this link is showing
+    # the community page, allow it to fail here:
+    r'^https://nextstrain\.org/community/ESR-NZ/GenomicsNarrativeSARSCoV2$'
+]
+linkcheck_anchors_ignore_for_url = [
+    # Github uses anchor-looking links for highlighting lines but
+    # handles the actual resolution with Javascript, so skip anchor
+    # checks for Github URLs:
+    r'^https://github\.com',
+    # you need to be logged in to see the anchor (and it looks like
+    # Terra is using it for redirecting more than anchoring too…)
+    r'^https://app\.terra\.bio/',
+    # client is blocked but links work
+    r'^https://www\.science\.org/doi/10\.1126/science\.abb9263',
+    # linkchecker doesn't support text fragments, and we link to one
+    # anchored to this page
+    r'^https://en\.wikipedia\.org/wiki/Consensus_sequence',
+]

docs/src/customise.rst

@@ -0,0 +1,32 @@
+*****************************************************************
+Customisation of workflow: using your own data
+*****************************************************************
+
+.. note::
+   Text mostly lifted from Avian flu docs for expediency
+
+
+
+The aim is to allow easy customisation of the workflow via config overlays. This includes defining the data inputs, modification of parameters (e.g. clock rates, minimum lengths, subsampling criteria, DTA metadata) is possible through overlays without needing to modify the underlying base config or Snakemake pipeline itself.
+
+Config overlays allow you to essentially maintain one or more modifications of the workflow for your own purposes. Using an overlay keeps this change separate from the config used for Nextstrain automation
+
+
+We'll start by creating a config (overlay) YAML, ``config.yaml`` in our analysis directory
+
+.. note::
+    You can choose a different name for the file, but if you do you'll have to supply it to the command via --configfile <filename> as only config.yaml will be automatically detected.
+
+Currently the default inputs for the workflow are:
+
+:configvalue:`phylogenetic/defaults/config.yaml:inputs`
+
+
+Let's add a similar block in our custom ``config.yaml`` defining some additional inputs:
+
+.. code-block:: yaml
+
+   additional_inputs:
+     - name: unreleased-data
+       metadata: https://data.nextstrain.org/something.tsv.zst
+       sequences: https://data.nextstrain.org/something.fasta.zst

docs/src/extensions/__init__.py

@@ -0,0 +1 @@
+# Extensions package for Sphinx documentation