storemagic - Don’t lose your variables in Jupyter Notebook

A post on IPython extension ‘storemagic’ to keep your important variables in persistent storage. Use this trick to keep them safe and stop pulling your hair (if there are any left)

Enviornment

!jupyter --version

Selected Jupyter core packages...
IPython          : 7.29.0
ipykernel        : 6.4.1
ipywidgets       : 7.6.5
jupyter_client   : 6.1.12
jupyter_core     : 4.8.1
jupyter_server   : 1.4.1
jupyterlab       : 3.2.1
nbclient         : 0.5.3
nbconvert        : 6.1.0
nbformat         : 5.1.3
notebook         : 6.4.5
qtconsole        : 5.1.1
traitlets        : 5.1.0

About

This post is about a very important (and highly underrated) Jupyter notebook extension “storemagic”. We will see how we can use it to store our important data, and recover safely even when the unforgivable happens.

Introduction

All Jupyter Notebook lovers remember their petrified moments when they are struck by a [dark terminal] force, their heart skips a beat, and they stare blanky at the screen. When they know that the unforgivable has happened but they still timidly move the cursor to the last cell and press Enter with the faintest hope that it might be still alive. But fate has chosen something else for us, and we poor people are left cursing ourselves as to why we did not save our precious work!

I have been to those despair valleys (many times) till I stumbled upon ‘storemagic’. It seems like archaic magic as so few people know or use it, but I can guarantee once you get used to it you will barely write more than five lines in your notebooks without using it. So let’s see how it works.

Store a variable

Use %store magic for lightweight persistence. It stores variables, aliases and macros in IPython’s database. Let’s create a variable and then store it using this magic.

##
# create a variable
var_hello = "hello world!"

# store this variable
%store var_hello

Stored 'var_hello' (str)

Now I am going to intentionally restart the kernel. We can check that our created variable is now gone from the memory.

var_hello

NameError: name 'var_hello' is not defined

But no worries. We have it stored safely using our magic. So let’s get it back.

## 
# get the variable back from store
%store -r var_hello

Okay, we have our variable back and (with a sigh of relief) we can use it again.

var_hello

'hello world!'

Let’s create a few more variables and do some more magic with them.

##
# create variables
var_foo = [1,2,3,4]
var_bar = {'a':var_hello}

# store multiple variables
%store var_foo var_bar

Stored 'var_foo' (list)
Stored 'var_bar' (dict)

Check all stored varaibles

We can check all the varaibles stored using the following magic command.

%store

Stored variables and their in-db values:
var_bar               -> {'a': 'hello world!'}
var_foo               -> [1, 2, 3, 4]
var_hello             -> 'hello world!'

Remove a variable from store

To remove a variable from our storage is also straight forward. Put its name after %store -d flag

##
# remove 'var_hello'
%store -d var_hello

‘var_hello’ is now gone. Forever …

##
# check the remaining variables in store
%store

Stored variables and their in-db values:
var_bar             -> {'a': 'hello world!'}
var_foo             -> [1, 2, 3, 4]

Remove all variables from store

If you want to remove all the varaibles from store and start clean then use -z flag

##
# remove all variables
%store -z

Kaboom! all variables are gone.

##
# check store
%store

Stored variables and their in-db values:

Reference

Check the official IPython storemagic documentation here: https://ipython.readthedocs.io/en/stable/config/extensions/storemagic.html