Note
Click here to download the full example code
IO Tutorial
This notebook is designed to demonstrate the pynapple IO. It is build around the specifications of the BIDS standard for sharing datasets. The key ideas are summarized as follow :
Navigating a structured dataset
First let's import the necessary packages.
mkdocs_gallery_thumbnail_path = '_static/treeview.png'
import numpy as np
import pynapple as nap
import os
import requests, math
import tqdm
import zipfile
Here we download a small example dataset.
project_path = "MyProject"
if project_path not in os.listdir("."):
r = requests.get(f"https://osf.io/a9n6r/download", stream=True)
block_size = 1024*1024
with open(project_path+".zip", 'wb') as f:
for data in tqdm.tqdm(r.iter_content(block_size), unit='MB', unit_scale=True,
total=math.ceil(int(r.headers.get('content-length', 0))//block_size)):
f.write(data)
with zipfile.ZipFile(project_path+".zip", 'r') as zip_ref:
zip_ref.extractall(".")
Out:
Let's load the project with nap.load_folder.
Out:
/mnt/home/gviejo/mambaforge/envs/pynapple/lib/python3.11/site-packages/hdmf/utils.py:668: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.7.0 is already loaded.
return func(args[0], **pargs)
📂 MyProject
└── 📂 sub-A2929
The pynapple IO Folders class offers a convenient way of visualizing and navigating a folder based dataset. To visualize the whole hierarchy of Folders, you can call the view property or the expand function.
Out:
📂 MyProject
└── 📂 sub-A2929
└── 📂 A2929-200711
├── 📂 derivatives
│ ├── spikes.npz | TsGroup
│ ├── sleep_ep.npz | IntervalSet
│ ├── position.npz | TsdFrame
│ └── wake_ep.npz | IntervalSet
└── 📂 pynapplenwb
└── A2929-200711 | NWB file
Here it shows all the subjects (in this case only A2929), all the sessions and all of the derivatives folders. It shows as well all the NPZ files that contains a pynapple object and the NWB files.
The object project behaves like a nested dictionary. It is then easy to loop and navigate through a hierarchy of folders when doing analyses. In this case, we are gonna take only the session A2929-200711.
Out:
I can expand to see what the folders contains.
Out:
📂 A2929-200711
├── 📂 derivatives
│ ├── spikes.npz | TsGroup
│ ├── sleep_ep.npz | IntervalSet
│ ├── position.npz | TsdFrame
│ └── wake_ep.npz | IntervalSet
└── 📂 pynapplenwb
└── A2929-200711 | NWB file
None
Loading files
By default, pynapple save objects as NPZ. It is a convenient way to save all the properties of an object such as the time support. The pynapple IO offers an easy way to load any NPZ files that matches the structures defined for a pynapple object.
spikes = session["derivatives"]["spikes"]
position = session["derivatives"]["position"]
wake_ep = session["derivatives"]["wake_ep"]
sleep_ep = session["derivatives"]["sleep_ep"]
Objects are only loaded when they are called.
Out:
Index rate location group
------- ------- ---------- -------
0 7.30333 adn 0
1 5.7325 adn 0
2 8.11917 adn 0
3 6.67833 adn 0
... ... ... ...
11 3.30833 ca1 1
12 1.09417 ca1 1
13 1.27917 ca1 1
14 1.32333 ca1 1
Metadata
A good practice for sharing datasets is to write as many metainformation as possible. Following BIDS specifications, any data files should be accompagned by a JSON sidecar file.
Out:
To read the metainformation associated with a file, you can use the functions doc, info or metadata :
Out:
â•â”€ MyProject/sub-A2929/A2929-200711/derivatives/spikes.npz ─╮
│ time : 2024-04-29 14:38:34.962905 │
│ info : │
╰───────────────────────────────────────────────────────────╯
â•â”€ MyProject/sub-A2929/A2929-200711/derivatives/position.npz ─╮
│ No metadata │
╰─────────────────────────────────────────────────────────────╯
Saving a pynapple object
In this case, we define a new Tsd and a new IntervalSet that we would like to save in the session folder.
tsd = position["x"] + 1
epoch = nap.IntervalSet(start=np.array([0, 3]), end=np.array([1, 6]))
session.save("x_plus_1", tsd, description="Random position")
session.save("stimulus-fish", epoch, description="Fish pictures to V1")
We can visualize the newly saved objects.
Out:
📂 A2929-200711
├── 📂 derivatives
│ ├── spikes.npz | TsGroup
│ ├── sleep_ep.npz | IntervalSet
│ ├── position.npz | TsdFrame
│ └── wake_ep.npz | IntervalSet
├── 📂 pynapplenwb
│ └── A2929-200711 | NWB file
├── x_plus_1.npz | Tsd
└── stimulus-fish.npz | IntervalSet
Out:
â•â”€ MyProject/sub-A2929/A2929-200711/stimulus-fish.npz ─╮
│ time : 2024-09-27 15:14:54.678729 │
│ info : Fish pictures to V1 │
╰──────────────────────────────────────────────────────╯
Out:
Time (s)
---------- --------
670.6407 0.957143
670.649 0.956137
670.65735 0.955147
670.66565 0.954213
...
1199.96995 1.01097
1199.97825 1.01079
1199.9866 1.01066
1199.99495 1.01062
dtype: float64, shape: (63527,)
Total running time of the script: ( 0 minutes 6.008 seconds)
Download Python source code: tutorial_pynapple_io.py