It's been a busy week for PlasmaPy. I recently found out about Binder support in sphinx-gallery. The latter is a package that we use to turn python scripts with comments into Sphinx pages and Jupyter Notebooks. I figured adding that could be a nice fit for our existing example gallery .

However, I quickly realized that the system in place is a bit unwieldy. Binder takes a link to an existing GitHub repository and executes .ipynb notebooks located there online. However, with sphinx-gallery, we don't have those notebooks in the repository - we have .py files with comments. The currently recommended way of setting this up with sphinx-gallery is keeping your built documentation in another repository and hosting it via something along the lines of GitHub Pages rather than ReadTheDocs, which we are currently using.

I added the results of this investigation to sphinx-gallery's docs, but I didn't want to switch away from RTD, so I figured I'd go ahead and find another way. I think I've got something that works well enough now!

Trigger warning: later on during this post, there may be monkeypatching of sphinx_gallery internals. Beware.

Using Jupytext

The Jupytext project is kind of like nbconvert, but two-way. It lets you turn notebooks into scripts and vice versa. The interesting thing is that, as per Jupytext's documentation, it is possible to let Binder's jupyter instance parse sphinx-gallery style .py files as jupyter notebooks. This was done in PlasmaPy#656 . First, let's instruct the in-binder Jupyter instance to parse .py files in .jupyter/jupyter_notebook_config.py via blatant copy-paste:

c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
c.ContentsManager.preferred_jupytext_formats_read = "py:sphinx"
c.ContentsManager.sphinx_convert_rst2md = True

And then let's also add a binder/requirements.txt file that lets Binder know what Python packages to download while building the repository's image. The version I had there was pretty shoddy, as CI/setup.py/packaging errors surfaced while I was tinkering with this. Long story short, something like this should do:

-r ../requirements/automated-documentation-tests.txt
jupytext
.

Where, in didactic order:

jupytext should be pretty self-explanatory,
. is the repository's package itself (here, PlasmaPy), as accessed by setup.py
-r ../requirements/automated-documentation-tests.txt reads a pip requirements file specifying our documentation requirements. I think with a proper extras_require specified in setup.py, these two lines could be collapsed simply into .[dev] or some such. Note that -r takes a path relative to the file, thus the ../

At this point all this really is is implementing what's mentioned in Jupytext's docs. The result is as follows:

image

But you might notice an inconsistency in the Sphinx-rendered gallery itself: even if we were to configure docs to display Binder links they will point to a path as imagined by the current implementation in Sphinx-Gallery, such as:

https://gke.mybinder.org/v2/gh/PlasmaPy/PlasmaPy/master?filepath=plasmapy/examples/auto_examples/plot_physics.ipynb

Note the spurious auto_examples directory supposedly including an .ipynb file. This obviously doesn't work for our use case, so we'd like to be able to change the generated links somehow...

Monkeypatching

This (or rather, PlasmaPy#658 ) is where it gets dirty. The solution developed in cooperation with Stuart Mumford (of SunPy fame, who contributed the idea which I implemented) is monkeypatching sphinx-gallery's link generation code. It's simple, yet effective.

Let's use this config for sphinx-gallery:

sphinx_gallery_conf = {
    # path to your examples scripts
    'examples_dirs': '../plasmapy/examples',
    # path where to save gallery generated examples
    'backreferences_dir': 'gen_modules/backreferences',
    'gallery_dirs': 'auto_examples',
    'binder': {
        'org': 'PlasmaPy',
        'repo': 'PlasmaPy',
        'branch': 'master',
        'binderhub_url': 'https://mybinder.org',
        'dependencies': ['../binder/requirements.txt'],
        'notebooks_dir': 'plasmapy/examples',
    }
}

and add this fragment of sphinx_gallery.binder code with a modification into Sphinx's conf.py file:

# Patch sphinx_gallery.binder.gen_binder_rst so as to point to .py file in repository
import sphinx_gallery.binder
def patched_gen_binder_rst(fpath, binder_conf, gallery_conf):
    """Generate the RST + link for the Binder badge.
    ...
    """
    binder_conf = sphinx_gallery.binder.check_binder_conf(binder_conf)
    binder_url = sphinx_gallery.binder.gen_binder_url(fpath, binder_conf, gallery_conf)

    # I added the line below:
    binder_url = binder_url.replace(gallery_conf['gallery_dirs'] + os.path.sep, "").replace("ipynb", "py")

    rst = (
        "\n"
        "  .. container:: binder-badge\n\n"
        "    .. image:: https://mybinder.org/badge_logo.svg\n"
        "      :target: {}\n"
        "      :width: 150 px\n").format(binder_url)
    return rst

# And then we finish our monkeypatching misdeed by redirecting sphinx-gallery to use our function:
sphinx_gallery.binder.gen_binder_rst = patched_gen_binder_rst

The current gallery is located here, and an example link is https://mybinder.org/v2/gh/PlasmaPy/PlasmaPy/master?filepath=plasmapy/examples/particle_stepper.py - and you should instantly see it points to the right spot!

Obviously, it would be better to implement such link customization in sphinx-gallery itself somehow, but it's up to their maintainers to decide whether this kind of combo usage with Jupytext is in scope for their project. For now, the monkeypatch solution works decently. I'll try to update this post if that comes about.

Update - requirements

@jdkent on GitHub suggests that if the above doesn't work for you, you should make sure the Sphinx version you're using is 2 or newer.