Conda within a container#
This is a set of instructions for creating and using a conda environment within a container. This can be used on the HPC or elsewhere. We recommend Apptainer. This is tested as working on ARC4 only, although containers built this way on ARC4 should also work on ARC3.
Why you want to use containers for Conda environments#
Conda environments create thousands of small files, that needs to be written when you create it, and many are read everytime you run any software within your environment. This is not playing into the strengths of the Lustre filesystem that underpins /nobackup.
By making a container, you end up with something that’s faster, easier to transfer onto other systems, share with others, and backup into your OneDrive. Pretty much better in everyway.
Creating the container#
Create a YAML file for the environment#
Conda allows you to build containers from a YAML file, which describes the
content. We’re going to use this example, and write it to a file called
channels: - defaults - conda-forge dependencies: - matplotlib - python=3.9 - pip - pip: - vivarium
Write a recipe file#
A single recipe should work for most situations here. The content of
mamba-example.def is provided below:
Bootstrap: docker From: mambaorg/micromamba %files environment.yml %post micromamba create -q -y -f environment.yml -p /opt/conda-env micromamba clean -aqy %runscript micromamba run -p /opt/conda-env "$@"
This recipe starts with a minimal micromamba Docker container, which is a fast, self-contained binary that’s an alternative to Conda.
%post section, you can see it creates a conda environment using the
provided YAML file, and tries to clean up after itself, to keep the size of the
container as small as possible.
%runscript section, we have a line that runs any command provided to
the container within the created Conda environment.
Build the container#
This is a single command, to take the recipe and YAML file and create a SIF image, that we can use later:
$ module add apptainer $ apptainer build mamba-example.sif mamba-example.def
With a small example, this will take a minute or so, but will take longer with
more complex environments. Once complete, you should see you now have a
SIF files are great, in that they’re a single file that encapsulates everything you need for a container, unlike formats used by systems like Docker, which use more complex layered arrangements.
Using the container#
Let’s just prove that it has indeed installed the vivarium library we asked for, and run a python command within the container:
$ ./mamba-example.sif python -c "import vivarium;print(vivarium.__version__)" 1.2.7
You’re also free to use the apptainer command. The equivalent would be:
$ apptainer run mamba-example.sif python -c "import vivarium;print(vivarium.__version__)" 1.2.7
Using the apptainer command directly allows you alter bind mounts, configure it to use GPUs, and other more advanced options.