Reproducibility I: Software Environments¶

Learning Objectives

After this lesson, you should be able to:

Understand the value of reproducible computing
Know the challenges of reproducible computing
Define a computing environment
Share a software environment with a colleague
Set up a software project with an environment

Reproducible Scientific Computing¶

Defining Reproducibility

"Reproducing the result of a computation means running the same software on the same input data and obtaining the same results." Rougier et al. 2016

"Getting someone else's code to run on my computer" - Anonymous

As the graphic below suggests, Reproducibility is a spectrum of sharing behaviors.

Interactive (ie, point-and-click) Computing¶

Definition

Manually navigating a mouse across a graphical user interface (GUI) and running commands by selecting from menu options.

Advantages¶

Inuitive and easy to navigate a GUI and click buttons

Limitations¶

It can be slow to sequence through hundreds of clicks to accomplish an analysis.
Less reproducible - Cumbersome to write and follow a click-by-click tutorial

Scripted Computing¶

Definition

Removing the GUI and instead instructing the computer to run a series of custom commands using a scripting/coding language.

We are automating what used to take many manual clicks.

We can write scripts to install software, clean data, run analyses, and generate figures.

Advantages¶

Much faster to run through commands
The script runs identically every time, reducing the human element
Easy for someone else to quickly reproduce the exact analysis and result
Enables analysis tasks to scale up

Challenges¶

Requires deeper computer knowledge
More upfront effort to produce the script

Discussion Question

What are some tasks you have automated or want to automate?

Have you ever successfully automated a task?
Found a way to make something scale or take less time?
What was the task, and how did you do it?
Are there any things you wish you could automate?
What are some barriers to automating them?

Scripting Languages¶

The two most common open-source scripting languages (for science) are Python and R.

Both languages consist of base software (Python Standard Library or R Base Package) and MANY additional packages that can be downloaded and installed for increased capabilities.

Software Installation¶

When you download and install software onto your computer, it will typically install it in a set of specific directories that we call the System Path.

System Path¶

In the context of computing, the system path, often referred to simply as PATH, is the set of directories in which the operating system looks for executable files when a command is issued.

When you go to launch an application by clicking on a desktop icon or with a CLI command, the computer will search for the application within the PATH directories. If it finds the executable, it will launch.

Find the PATH on your computer

In Linux and Mac Terminal

echo $PATH

In Windows Terminal

echo %PATH%

Nice and Short Video Describing the PATH.

The PATH prefers one version of any given software.

Computing Environment¶

A computing environment is the combination of hardware, software, and network resources that provide the infrastructure for computing operations and user interactions.

Hardware: CPUs, GPUs, RAM
Operating system & version: many flavors of Linux, MacOS, Windows
Software versions: R, Python, etc.
Package versions: specific R or Python packages, which often depend on other packages

The scripts you create:¶

Were designed to work in your specific computing environment
May not work on your computer in the future, because your computing enviroment will probably change (eg., updated software versions)
May not work on someone else's computer because their computing environment is different

Software Dependency Hell¶

Sometimes, it can be nearly impossible to get your computing environment correct enough to run someone else's code.

This can caused by incorrect software versions of the packages you are using or their dependencies.

Updating software installed in the system path - to make new code work - can break old code!

Environment Managers¶

One solution to software dependency hell is to use an Environment Manager

An environment manager allows you to create software installation directories (similar to PATH) that are isolated your computer's PATH. You can create unique environments and install specific software version to run specific scripts.

Conda - Open Source Environment Manager¶

Conda is a popular and open source environment manager tool that can be installed on any operating system (Windows, MacOS, Linux).

Users can create environments that have their own set of packages, dependencies, and even their own version of Python.
Projects can have their own specific requirements without interfering with each other
It allows for consistent and reproducible results across different systems and setups

Renv ¶

R package that allows you to create unique environments for an R project

Package Managers¶

A software tool to find, download, and install software packages to PATH or virtual environment

Conda ¶

Software: Python, R, Django, Celery, PostgreSQL, nginx, Node.js, Java programs, C and C++, Perl, and command line tools

Repository: Conda-Forge.

Pip ¶

Software: python

Repository: PyPi

Note: Pip can be used together with Conda environment manager.

R¶

With the R language, a package manager is built directly into the R Base Package.

install.packages('ggplot2')

Repository: R Comprehensive R Archive Network (CRAN)

Whether you are using Conda, Pip, or Renv, you should be able to share the specifications of your software environment so colleagues can reproduce the environment.

The general sharing workflow:

Output an environment file that lists the software and versions of the environment
Share the file with colleagues through a platform like Github
Colleagues create an empty environment on their computer and populate it with the contents of the environment file

Conda ¶

Export your Conda Environment
```
conda env export > my_conda_env.yml
```
Share the .yml file through Github
Reproduce the Environment on a Different Computer
```
conda env create --file environment.yml
```

Conda exports your Pip environment as well

Exporting your environment using Conda (conda env export > my_conda_env.yml) will ALSO export your pip environment!

Python¶

Export python libraries present in your environment
```
pip3 freeze > requirements.txt 
```
Share the requirements.txt on Github
Reproduce the Environment on a Different Computer
```
pip install -r requirements.txt
```

Renv ¶

Create an isolated environment
```
renv::init()
```
Export R packages to the renv.lock file
```
renv:snapshot()
```
Share the renv.lock, .Rprofile, renv/settings.json and renv/activate.R files to Github
Reproduce the Environment on a Different Computer
```
renv::restore()
```

Reproducibility Tutorial¶

Installing Conda¶

When you download and install Conda it comes in two different flavors:

Miniconda - lightweight (500 mb) program that includes Conda, the environment and package manager, as well as a recent version of the Standard Python Library.

Anaconda - a larger (2.5GB) program that includes Conda and many more python libraries pre-installed (in Conda base environment), as well as graphical user interface, acccess to jupyter notebooks, and support for easily integrating the R language.

Installing Conda

For the appropriate installation package, visit https://docs.conda.io/en/latest/miniconda.html. Note: If you are using the WSL, install the Linux version!!

# Download conda and add right permissions
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh     # Modify this to match the OS you're using.
chmod +x Miniconda3-py39_4.12.0-Linux-x86_64.sh

# install conda silenty (-b) and update (-u) and initial conda run
./Miniconda3-py39_4.12.0-Linux-x86_64.sh -b -u
~/miniconda3/bin/conda init

# Restart bash so that conda is activated
source ~/.bashrc

You'll be able to tell when conda is active when next (base) is present next to the to the shell prompt such as

(base) user@machine

Conda should now be installed and can be used to install other necessary packages!

Tip: slow Conda? Try Mamba.

Conda is known to take time processing some software installation. A solution is to use Mamba, a reimplementation of Conda in C++ for quicker queries and installations. Mamba is then invoked by using mamba instead of conda (whilst keeping options and the rest of the command synthax the same).

The quickest way to install mamba is with conda install -c conda-forge mamba, or follow the official installation documentation here.

Conda on Cyverse¶

OS of choice

This tutorial will be performed using the CyVerse CLI (Command Line Interface) which is a Linux Command Line. This requires a Cyverse account.

However, if you'd like to use your own computer feel free to! If you're on Mac or Linux, open your terminal; If you're on Windows, please use the Windows Subsystem for Linux (WSL) so you can follow along.

How to Scroll in Cyverse (Tmux) Cloud Shell

If you're using the Cyverse Cloud Shell, you can scroll up and down by pressing Ctrl + b and then [ to enter scroll mode. You can then use the arrow keys to scroll up and down. Press q to exit scroll mode.

The CLI in CyVerse is controlled with Tmux, a software that allows to "window" the CLI; Here is a cheat sheet that will teach you more Tmux tricks!

Environment Management with Conda¶

When you start a Cyverse Cloud shell, the prompt will look something like this:

(base) jovyan@a12b272e0:/home/user/data-store$

Miniconda has already been pre-installed, and by default, you are started in a base Conda directory (base)

View the list of conda environments

conda env list

View the software installed in the base directory. Notice the version of Python.

conda list

Create our own custom environment (type y when prompted).

conda create --name myenv

Activate your new environment with

conda activate myenv

You will notice that the prompt changed to (myenv)

View the software that is installed in your new custom environment. It should be empty!

conda list

Package management with Conda¶

Within your new custom environment (ie, myenv) download and install a specific version of python. This may take a few minutes to complete.

conda install python=3.9

View the new software that has been install

conda list

Install Salmon and FastQC (genomics software) using Conda

conda install -c bioconda salmon fastqc

Conda channels

Conda operates through channels, specififc repositories where packages are stored. Specific packages sometimes may appear in multiple channels, however it is always helpful to specify a channel with the -c flag.

Export all of the software in your custom environment to a file

conda env export > myenv.yml

Let's view the contents of the .yml file

nano myenv.yml

Now we are going to pretend that we are reproducing a conda environment from a .yml file shared by a collegue.

Change the name of the environement within the .yml file from myenv to myenv2

Create a new environment and populate it with the .yml environment file

conda env create --file myenv2.yml

Package management with Pip¶

Pip works similarly to Conda, as Pip is the package management supported by the Python Software foundation. If you use Python for your work it is likely you have installed packages using Pip.

We only have to install a single package required for this tutorial, MultiQC. To install MultiQC using Pip, do:

pip install multiqc

Similar to Conda, you can export your pip environment by doing

pip3 freeze > requirements.txt

Why pip3?

pip3 freeze > requirements.txt is used to export the pip environment such that it is readable for Python 3. If you want to export an environment for Python 2, you can use pip freeze > requirements.txt.

Reproducibility I: Software Environments¶

Reproducible Scientific Computing¶

Interactive (ie, point-and-click) Computing¶

Advantages¶

Limitations¶

Scripted Computing¶

Advantages¶

Challenges¶

Scripting Languages¶

Software Installation¶

System Path¶

Computing Environment¶

The scripts you create:¶

Software Dependency Hell¶

Environment Managers¶

Conda - Open Source Environment Manager¶

Renv¶

Package Managers¶

Conda¶

Pip¶

R¶

Sharing your Environment with Colleagues¶

Conda¶

Python¶

Renv¶

Reproducibility Tutorial¶

Installing Conda¶

Conda on Cyverse¶

Environment Management with Conda¶

Package management with Conda¶

Share and Reproduce a Conda Environment¶

Package management with Pip¶

Renv ¶

Conda ¶

Pip ¶

Conda ¶

Renv ¶