Testing

Papis uses pytest for its testing infrastructure and makes use of some of its more advanced features (such as fixtures) to set everything up. The command-line interface is based on click and is tested using their testing helpers.

We give here an overview of the pieces needed to test the various parts of the Papis codebase, but in general mimicking existing tests is your best choice.

Using the default configuration

The Papis configuration is automatically loaded (and cached) on the first call to a function such as papis.config.get() or papis.config.set(). By default, it loads the configuration of the current user, somewhere at ~/.config/papis/config. This is obviously not desired for testing purposes, as the configuration settings may differ, leading to incorrect results (that make the CI fail).

To handle this, the TemporaryConfiguration context manager can temporarily redirect all the configuration paths to a temporary location and load the default settings. It can be used as:

import papis.testing

def test_me() -> None:
    # Include test setup here

    with papis.testing.TemporaryConfiguration() as config:
        # Add here tests that use papis.config functionality
        # and that require default values

    # Any calls outside of the context manager will take values
    # from the user configuration file if it exists

With pytest, we can use a fixture to overwrite the configuration for the whole function scope. For example, this can look like:

import pytest
import papis.testing

@pytest.mark.config_setup(**kwargs)
def test_me(tmp_config: papis.testing.TemporaryConfiguration) -> None:
    # All calls to papis.config in this function will point
    # to the temporary configuration
    assert tmp_config.configfile == papis.config.get_config_file()

The tmp_config() fixture is bundled with Papis and can be used directly as above. It uses the config_setup marker to pass keyword arguments directly to the underlying context manager. Check out the documentation for papis.testing.TemporaryConfiguration to see additional attributes and functionality provided by this class.

If the test requires access to a Papis library, e.g. to add, remove, or load documents from the disk, the papis.testing.TemporaryLibrary context manager should be used instead. It also has a corresponding fixture called tmp_library that can be configured with library_setup as follows:

import pytest
import papis.testing

@pytest.mark.library_setup(populate=True)
def test_me(tmp_library: papis.testing.TemporaryLibrary) -> None:
    # This function inherits all functionality of TemporaryConfiguration
    # and also has a small library populated with a dozen-ish documents
    # with random files and metadata attached

    assert tmp_library.libname == papis.config.get_lib_name()

Warning

If the user also has a ~/.config/papis/config.py, this is always read and inserted into the Papis global scope using eval(). This cannot be handled in a clean fashion by TemporaryConfiguration, so a new pytest flag was introduced to point to an empty configuration before the tests are loaded. It can be used like (and is enabled by default in pyproject.toml):

python -m pytest --papis-tmp-xdg-home tests

Warning

The doctests also try to load the global user configuration and cannot easily use the TemporaryConfiguration context manager or the associated fixture. To deal with this an autouse=True fixture is introduced. It can be used like (and is enabled by default in pyproject.toml):

python -m pytest --papis-tmp-doctests papis

Testing commands

To test papis commands (such as papis add), we make use of the infrastructure from click.testing.CliRunner and, in particular, the customized papis.testing.PapisRunner. To run a papis command as it would be invoked from the command-line, use:

import papis.testing

def test_me(tmp_library: papis.testing.TemporaryLibrary) -> None:
    from papis.commands.add import cli

    cli_runner = papis.testing.PapisRunner()
    result = cli_runner.invoke(
        # The first argument needs to be a function that was wrapped by
        # @click.group or @click.command to have all the argument handling:
        cli,
        # The second argument is a list of command-line arguments that will
        # be passed to the cli similar to how subprocess works:
        ["--from", "doi", "10.1007/s11075-008-9193-8"]
    )
    assert result.exist_code == 0

The second argument to invoke() is a list of arguments that should match exactly what would be passed on the command-line. The invocation returns a click.testing.Result that has captured the STDOUT and STDERR streams and can be easily inspected for testing purposes.

Testing downloaders

Testing importers and downloaders generally requires handling some remote resources, which are then converted to the Papis format and saved as documents in the library. To help with downloading and caching these resources, we can use the papis.testing.ResourceCache class.

This class handles caching resources on disk so that they can be used and compared against in the test. In particular, testing a downloader involves the following steps:

Remote: download resource from a URL or retrieve from a local path (if it exists).
Convert: feed the remote resource to Papis for conversion.
Local: retrieve an expected result from a local path (if it exists), otherwise save the existing conversion.
Check: check current conversion against the cached local resource.

When first adding a test case for a downloader, the resources are downloaded and cached automatically, since they do not exist. To update the resources for a test, use the PAPIS_UPDATE_RESOURCES environment variable when running the tests locally. This is done in the following way:

PAPIS_UPDATE_RESOURCES=remote python -m pytest -v -s test/downloaders/test_acl.py
# ... or ...
PAPIS_UPDATE_RESOURCES=local python -m pytest -v -s test/downloaders/test_acl.py
# ... or ...
PAPIS_UPDATE_RESOURCES=both python -m pytest -v -s test/downloaders/test_acl.py

The resources can also be updated in the test itself by using the force argument to get_remote_resource() or get_local_resource(). The resource cache can also be accessed through a fixture called resource_cache() that can be configured through the resource_setup marker. For example, we can write something like

@pytest.mark.resource_setup(cachedir="downloaders/resources")
def test_me(tmp_config: papis.testing.TemporaryConfiguration,
            resource_cache: papis.testing.ResourceCache,
            monkeypatch: pytest.MonkeyPatch) -> None:
    # Pick a URL and some filenames to cache
    url = "https://aclanthology.org/2022.naacl-main.2/"
    infile = "ACL-2022-naacl-main-2.html"
    outfile = "ACL-2022-naacl-main-2-out.json"

    # Monkeypatch the downloader to use the resource_cache:
    downloader = papis.downloaders.get_downloader_by_name("acl")
    monkeypatch.setattr(downloader, "_get_body",
                        lambda: resource_cache.get_remote_resource(infile, url))

    # Fetch remote resource data and check it against the stored version:
    downloader.fetch()
    expected_data = resource_cache.get_local_resource(outfile, downloader.ctx.data)
    assert expected_data == downloader.ctx.data