# Usage examples¶

This class is greatly suited for neuroimaging applications (or any other domain), where each sample needs to be uniquely identified with a subject ID (or something similar).

Key-level correspondence across data, labels (1 or 2), classnames (‘healthy’, ‘disease’) and the related helps maintain data integrity and improve the provenance, in addition to enabling traceback to original sources from where the features have been originally derived.

Just to given you a concrete examples, let’s look at how an ML dataset is handled traditionally.

You have a matrix X of size n x p, with n samples and p features, and a vector y containing the target values (or class labels or class identifiers). This X and y serves as training (and test set) for a classifier like SVM to fit the data X to match y as accurately as possible.

Let’s get a little more concrete:

import sys, os
import numpy as np
import matplotlib
%matplotlib
%matplotlib inline
import matplotlib.pyplot as plt

n = 10 # number of samples
p = 3  # number of features

X = np.random.random([n, p]) # random data for illustration
y = [1]*5 + [2]*5            # random labels ...

np.set_printoptions(precision=2) # save some screen space
print('X : \n{}'.format(X))
print('y : \n{}'.format(y))

Using matplotlib backend: TkAgg
X :
[[ 0.73  0.85  0.3 ]
[ 0.63  0.09  0.87]
[ 0.14  0.71  0.19]
[ 0.25  0.33  0.08]
[ 0.8   0.85  0.99]
[ 0.78  0.76  0.47]
[ 0.25  0.54  0.18]
[ 0.57  0.98  0.36]
[ 0.1   0.1   0.74]
[ 0.16  0.76  0.53]]
y :
[1, 1, 1, 1, 1, 2, 2, 2, 2, 2]


Almost all the machine learning toolboxes take their input in this form: X and y, regardless of the original source that produced these features in the first place.

This is all fine if all you ever wanted to do is to extract some features, do some machine learning and dispose these features away!

** But this is almost never the case!**

Because it doesn’t simply end there.

At a minimum, I often need to know * which samples are misclassified - meaning you need to know what the identifiers are and not simply their row indices in X? * what are the charecteristics of those samples? * what classes do they belong to?

And all this info needs to be obtained * without having to write lots of code connecting few non-obvious links to disparate sources of data (numerical features X, and sample identifiers in a CSV file) to find the relevant info * without having to track down who or which method originally produced these features * how the previous personnel or grad student organized the whole dataset, if you haven’t generated the features yourself from scratch

And if you are like me, you would be thinking about how would you organize your workflow such that the aforementioned tasks can be accomplished with ease.

This data structure attempts to accomplish that with ease. By always organizing the extracted features keyed-in into a dictionary with their sample id, and other important info such as target values and other identified info. This, by definition, preserves the integrity of the data (inability to incorrectly label samples etc).

No, this data structure doesn’t offer the full provenance tracking, which is quite a challenging problem. But it tries make your life a little easier in your ML workflows.

An example application is shown below, touching upon the following topics:

• Motivation
• Constructing a dataset
• Attributes
• Accessing samples
• Iteration over samples
• Subset selection
• Combining datasets and diving them into useful subsets
• Portability (e.g. with sklearn)

Improting the necessary modules and our fancy class definition:

from pyradigm import MLDataset


We can now instantiate it and give it a description:

dataset = MLDataset()
dataset.description = 'ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.'

dataset

ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
Empty dataset.


You can see the dataset some description attached to it, however we know it is empty. This can be verified in a boolean context as shown below:

bool(dataset)

False


Let’s add samples to this dataset which is when this dataset implementation becomes really handy. Before we do that, we will define some convenience routines defined to just illustrate a simple yet common use of this dataset.

def read_thickness(path):
"""Dummy function to minic a data reader."""

# in your actural routine, this might be:
return np.random.random(2)

def get_features(work_dir, subj_id):
"""Returns the whole brain cortical thickness for a given subject ID."""

# extension to identify the data file; this could be .curv, anything else you choose
ext_thickness = '.thickness'

thickness = dict()
for hemi in ['lh', 'rh']:
path_thickness = os.path.join(work_dir, subj_id, hemi + ext_thickness)

# concatenating them to build a whole brain feature set
thickness_wb = np.concatenate([thickness['lh'], thickness['rh']])

return thickness_wb


So now we have IO routines to read the data for us. Let’s define where the data will come from:

work_dir = '/project/ADNI/FreesurferThickness_v4p3'
class_set = ['Cntrl', 'Alzmr', 'MCI']
class_sizes = [15, 12, 18]


This would obviously change for your applications, but this has sufficient properties to illustrate the point.

Let’s look at what methods this dataset offers us:

dir(dataset)

['add_classes',
'class_set',
'class_sizes',
'classes',
'data',
'data_and_labels',
'del_sample',
'description',
'extend',
'feature_names',
'get_class',
'get_feature_subset',
'get_subset',
'glance',
'keys',
'num_classes',
'num_features',
'num_samples',
'random_subset',
'random_subset_ids',
'random_subset_ids_by_count',
'sample_ids',
'sample_ids_in_class',
'save',
'summarize_classes',
'train_test_split_ids',
'transform']


That’s a lot of methods of convenience to organize and retrieve dataset.

So let’s go through them by their usage sections.

# Constructor¶

You can see there few methods such as add_sample, get_subset etc: important method being add_sample, which is key to constructing this dataset. Let’s go ahead and some samples:

To contruct a dataset, one typically starts with a list of subject IDs to be added - we create few random lists, each to be considered as a separate class:

import random
from datetime import datetime
random.seed(datetime.now())

"Generates a random target list. In reality, you would do something like the commented code below."
target_list = list()
for idx in range(class_size):
target_list.append('{}{:04d}'.format(class_name[0],np.random.randint(1000)))

return target_list


Now we go through each of the above classes, and add each sample that class to the dataset.

for class_index, class_id in enumerate(class_set):
print('Working on class {:>5}'.format(class_id))

for subj_id in target_list:
print('\t reading subject {:>15}'.format(subj_id))
thickness_wb = get_features(work_dir, subj_id)

# adding the sample to the dataset
dataset.add_sample(subj_id, thickness_wb, class_index, class_id)

Working on class Cntrl
Working on class Alzmr
Working on class   MCI


Nice. Isn’t it?

So what’s nice about this, you say? The simple fact that you are constructing a dataset as you read the data in its most elemental form (in the units of the dataset such as the subject ID in our neuroimaging application). You’re done as soon as you’re done reading the features from disk.

What’s more - you can inspect the dataset in an intuitive manner, as shown below:

dataset

ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
45 samples, 3 classes, 4 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.
Class   MCI : 18 samples.


Even better, right? No more coding of several commands to get the complete and concise sense of the dataset.

# Convenient attributes¶

If you would like, you can always get more specific information, such as:

dataset.num_samples

45

dataset.num_features

4

dataset.class_set

['MCI', 'Cntrl', 'Alzmr']

dataset.class_sizes

Counter({'Alzmr': 12, 'Cntrl': 15, 'MCI': 18})

dataset.class_sizes['Cntrl']

15


If you’d like to take a look data inside for few subjects - shall we call it a glance?

dataset.glance()

{'C0170': array([ 0.37,  0.78,  0.5 ,  0.79]),
'C0241': array([ 0.11,  0.18,  0.58,  0.36]),
'C0408': array([ 0.49,  0.38,  0.05,  0.82]),
'C0562': array([ 0.64,  0.59,  0.01,  0.8 ]),
'C0760': array([ 0.12,  0.51,  0.95,  0.23])}


We can control the number of items to glance, by passing a number to dataset.glance() method:

dataset.glance(2)

{'C0408': array([ 0.49,  0.38,  0.05,  0.82]),
'C0562': array([ 0.64,  0.59,  0.01,  0.8 ])}


Or you may be wondering what are the subject IDs in the dataset.. here they are:

dataset.sample_ids

['C0562',
'C0408',
'C0760',
'C0170',
'C0241',
'C0980',
'C0822',
'C0565',
'C0949',
'C0041',
'C0372',
'C0141',
'C0492',
'C0064',
'C0557',
'A0034',
'A0768',
'A0240',
'A0042',
'A0141',
'A0888',
'A0032',
'A0596',
'A0969',
'A0215',
'A0074',
'A0229',
'M0760',
'M0434',
'M0033',
'M0942',
'M0034',
'M0868',
'M0595',
'M0476',
'M0770',
'M0577',
'M0638',
'M0421',
'M0006',
'M0552',
'M0040',
'M0165',
'M0256',
'M0127']


# Accessing samples¶

Thanks to elegant implementation, data for a given sample ‘M0299’ can simply be obtained by:

dataset['M0040']

array([ 0.27,  0.52,  0.61,  0.49])


Like a Python dict, it raises an error if the key is not in the dataset:

dataset['dlfjdjf']

---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

<ipython-input-22-4b19d52bac71> in <module>()
----> 1 dataset['dlfjdjf']

839             return self.__data[item]
840         else:
--> 841             raise KeyError('{} not found in dataset.'.format(item))
842
843     def __iter__(self):



A more graceful handling would be to use dataset.get to control what value to be returned in case the requested id is not found in the dataset.

dataset.get('dkfjd', np.nan)

nan


# Iteration¶

Thanks to builtin iteration, we can easily iterate over all the samples:

for sample, features in dataset:
print("{} : {:>10} : {}".format(sample, dataset.classes[sample], features))

C0562 :      Cntrl : [ 0.64  0.59  0.01  0.8 ]
C0408 :      Cntrl : [ 0.49  0.38  0.05  0.82]
C0760 :      Cntrl : [ 0.12  0.51  0.95  0.23]
C0170 :      Cntrl : [ 0.37  0.78  0.5   0.79]
C0241 :      Cntrl : [ 0.11  0.18  0.58  0.36]
C0980 :      Cntrl : [ 0.1   0.52  0.79  0.68]
C0822 :      Cntrl : [ 0.44  0.97  0.06  0.99]
C0565 :      Cntrl : [ 0.89  0.5   0.89  0.48]
C0949 :      Cntrl : [ 0.84  0.84  0.51  0.12]
C0041 :      Cntrl : [ 0.07  0.19  0.68  0.81]
C0372 :      Cntrl : [ 0.7   0.05  0.67  0.39]
C0141 :      Cntrl : [ 0.46  0.18  0.69  0.17]
C0492 :      Cntrl : [ 0.82  0.77  0.07  0.69]
C0064 :      Cntrl : [ 0.24  0.54  0.36  0.37]
C0557 :      Cntrl : [ 0.59  0.86  0.1   0.42]
A0034 :      Alzmr : [ 0.35  0.96  0.41  0.93]
A0768 :      Alzmr : [ 0.65  0.37  0.7   0.24]
A0240 :      Alzmr : [ 0.87  0.78  0.1   0.28]
A0042 :      Alzmr : [ 0.12  0.3   0.35  0.7 ]
A0141 :      Alzmr : [ 0.85  0.28  0.06  0.74]
A0888 :      Alzmr : [ 0.85  0.78  0.93  0.7 ]
A0032 :      Alzmr : [ 0.28  0.41  0.61  0.09]
A0596 :      Alzmr : [ 0.28  0.15  0.88  0.23]
A0969 :      Alzmr : [ 0.47  0.37  0.52  0.58]
A0215 :      Alzmr : [ 0.49  0.7   0.31  0.96]
A0074 :      Alzmr : [ 0.87  0.7   0.37  0.7 ]
A0229 :      Alzmr : [ 0.96  0.34  0.59  0.96]
M0760 :        MCI : [ 0.27  0.22  0.37  0.14]
M0434 :        MCI : [ 0.26  0.04  0.49  0.92]
M0033 :        MCI : [ 0.14  0.39  0.71  0.5 ]
M0942 :        MCI : [ 0.19  0.29  0.42  0.46]
M0034 :        MCI : [ 0.36  0.54  0.67  0.71]
M0868 :        MCI : [ 0.29  0.46  0.47  0.83]
M0595 :        MCI : [ 0.62  0.07  0.66  0.75]
M0476 :        MCI : [ 0.73  0.97  0.59  0.24]
M0770 :        MCI : [ 0.81  0.78  0.28  0.61]
M0577 :        MCI : [ 0.84  0.86  0.94  0.5 ]
M0638 :        MCI : [ 0.61  0.64  0.94  0.94]
M0421 :        MCI : [ 0.73  0.16  0.97  0.69]
M0006 :        MCI : [ 0.76  0.62  0.49  0.03]
M0552 :        MCI : [ 0.26  0.85  0.13  0.31]
M0040 :        MCI : [ 0.27  0.52  0.61  0.49]
M0165 :        MCI : [ 0.03  0.79  0.92  0.79]
M0256 :        MCI : [ 0.06  0.06  0.69  0.97]
M0127 :        MCI : [ 0.42  0.11  0.93  0.5 ]


Did you see that? It’s so intuitive and natural! Such a clean traversal of dataset.

Thanks to the choice of the OrderedDict() to represent the data, classes and labels underneath, the order of sample addition is retained. Hence the correspondence across samples in the dataset not only key-wise (by the sample id), but also index-wise.

# Subject-wise tranform¶

Quite often, we are interested in computing some statistics on data for a given subject (such as mean, or ROI-wise median). Typically this requires a loop, with some computation and organizing it in a new dataset! A simple routine pattern of usage, but can’t avoided if you are still fiddling with representing your dataset in medieval matrices! :).

If you organized your dataset in a pyradigm, such computation is trivial, thanks to builtin implementation of transform method. The mean value for each subject can be computed and organized in a new dataset, with an intuitive and single line:

mean_data = dataset.transform(np.mean)
mean_data.description = 'mean values per subject'
mean_data

mean values per subject
45 samples, 3 classes, 1 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.
Class   MCI : 18 samples.


As the transform accepts an arbitrary callable, we could do many more sophisticated things, such as access the subset of features e.g. cortical thickness for a particular region of interest (say posterior cingulate gyrus).

# let's make a toy function to return the indices for the ROI
def get_ROI_indices(x): return x[:3]


Using this “mask” function, we can easily obtain features for an ROI

pcg = dataset.transform(get_ROI_indices)


We can verify that the new dataset does indeed have only 3 features, for the same subjects/classes:

pcg

None
ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
45 samples, 3 classes, 3 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.
Class   MCI : 18 samples.

pcg.num_features

3


Let’s make a bar plot with the just computed numbers:

data, lbl, keys = pcg.data_and_labels()

n, bins, patches = plt.hist(data)


Remember as the original source of data was random, this has no units, property or meaning!

# Subset selection¶

In addition to the structured way of obtaining the various properties of this dataset, this implementation really will come in handy when you have to slice and dice the dataset (with large number of classes and features) into smaller subsets (e.g. for binary classification). Let’s see how we can retrieve the data for a single class:

ctrl = dataset.get_class('Cntrl')


That’s it, obtaining the data for a given class is a simple call away.

Now let’s see what it looks like:

ctrl

 Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
15 samples, 1 classes, 4 features.
Class Cntrl : 15 samples.


Even with updated description automatically, to indicate its history. Let’s see some data from controls:

ctrl.glance(2)

{'C0408': array([ 0.49,  0.38,  0.05,  0.82]),
'C0562': array([ 0.64,  0.59,  0.01,  0.8 ])}


We can also query a random subset of samples for manual inspection or cross-validation purposes. For example:

random_subset = dataset.random_subset(perc_in_class=0.3)
random_subset

 Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
12 samples, 3 classes, 4 features.
Class Cntrl : 4 samples.
Class Alzmr : 3 samples.
Class   MCI : 5 samples.


You can see which samples were selected:

random_subset.sample_ids

['C0562',
'C0565',
'C0372',
'C0492',
'A0240',
'A0032',
'A0229',
'M0034',
'M0770',
'M0552',
'M0165',
'M0127']


You can verify that it is indeed random by issuing another call:

# supplying a new seed everytime to ensure randomization
from datetime import datetime
dataset.random_subset(perc_in_class=0.3).sample_ids

['C0562',
'C0822',
'C0949',
'C0141',
'A0034',
'A0141',
'A0032',
'M0434',
'M0942',
'M0868',
'M0421',
'M0552']


Let’s see how we can retrieve specific samples by their IDs (for which there are many use cases):

data = dataset.get_subset(dataset.sample_ids[1:20])
data

 Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
19 samples, 2 classes, 4 features.
Class Cntrl : 14 samples.
Class Alzmr : 5 samples.


So as simple as that.

## Cross-validation¶

If you would like to develop a variant of cross-validation, and need to obtain a random split of the dataset to obtain training and test sets, it is as simple as:

train_set, test_set = dataset.train_test_split_ids( train_perc = 0.5)


This method returns two sets of sample ids corresponding to training set (which 50% of samples from all classes in the dataset) and the rest in test_set. Let’s see what they have:

train_set, test_set

(['C0760',
'C0822',
'C0565',
'C0170',
'C0562',
'C0141',
'C0041',
'A0768',
'A0888',
'A0032',
'A0969',
'A0141',
'A0034',
'M0434',
'M0421',
'M0577',
'M0256',
'M0127',
'M0033',
'M0760',
'M0476',
'M0165'],
['M0040',
'A0240',
'C0241',
'C0492',
'A0074',
'A0042',
'M0942',
'M0595',
'M0006',
'C0372',
'C0064',
'C0557',
'M0552',
'M0034',
'C0408',
'C0980',
'A0229',
'C0949',
'A0596',
'M0770',
'A0215',
'M0868',
'M0638'])


We can also get a train/test split by specifying an exact number of subjects we would like from each class (e.g. when you would like to avoid class imbalance in the training set):

train_set, test_set = dataset.train_test_split_ids( count_per_class = 3)


Let’s see what the training set contains - we expect 3*3 =9 subjects :

train_set

['C0557',
'C0041',
'C0949',
'A0768',
'A0888',
'A0229',
'M0165',
'M0476',
'M0040']


We can indeed verify that is the case, by creating a new smaller dataset from that list of ids and getting a summary:

training_dataset = dataset.get_subset(train_set)
training_dataset

 Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
9 samples, 3 classes, 4 features.
Class Cntrl : 3 samples.
Class Alzmr : 3 samples.
Class   MCI : 3 samples.


Another programmatic way to look into different classes is this:

class_set, label_set, class_sizes = training_dataset.summarize_classes()
class_set, label_set, class_sizes

(['MCI', 'Cntrl', 'Alzmr'], [2, 0, 1], array([ 3.,  3.,  3.]))


which returns all the classes that you could iterative over.

Using these two lists, we can easily obtain subset datasets, as illustrated below.

dataset

ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
45 samples, 3 classes, 4 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.
Class   MCI : 18 samples.

binary_dataset = dataset.get_class(['Cntrl','Alzmr'])
binary_dataset

 Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
27 samples, 2 classes, 4 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.


How about selecting a subset of features from all samples?

binary_dataset.get_feature_subset(range(2))

Subset features derived from:

Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
27 samples, 2 classes, 2 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.


Great. Isn’t it? You can also see the two-time-point history (initial subset in classes, followed by a subset in features).

# Serialization¶

Once you have this dataset, you can save and load these trivially using your favourite serialization module. Let’s do some pickling:

out_file = os.path.join(work_dir,'binary_dataset_Ctrl_Alzr_Freesurfer_thickness_v4p3.MLDataset.pkl')
binary_dataset.save(out_file)


That’s it - it is saved.

Let’s reload it from disk and make sure we can indeed retrieve it:

reloaded = MLDataset(filepath=out_file) # another form of the constructor!

reloaded

 Subset derived from: ADNI1 baseline: cortical thickness features from Freesurfer v4.3, QCed.
27 samples, 2 classes, 4 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.


We can check to see they are indeed one and the same:

binary_dataset == reloaded

True


# Dataset Arithmetic¶

You might wonder how can you combine two different types of features ( thickness and shape ) from the dataset. Piece of cake, see below ...

To concatenat two datasets, first we make a second dataset:

dataset_two = MLDataset(in_dataset=dataset) # yet another constructor: in its copy form!


How can you check if they are “functionally identical”? As in same keys, same data and classes for each key... Easy:

dataset_two == dataset

True


Now let’s try the arithmentic:

combined = dataset + dataset_two

Identical keys found. Trying to horizontally concatenate features for each sample.


Great. The add method recognized the identical set of keys and performed a horiz cat, as can be noticed by the twice the number of features in the combined dataset:

combined

45 samples, 3 classes, 8 features.
Class Cntrl : 15 samples.
Class Alzmr : 12 samples.
Class   MCI : 18 samples.


We can also do some removal in similar fashion:

smaller = combined - dataset

C0562 removed.
C0408 removed.
C0760 removed.
C0170 removed.
C0241 removed.
C0980 removed.
C0822 removed.
C0565 removed.
C0949 removed.
C0041 removed.
C0372 removed.
C0141 removed.
C0492 removed.
C0064 removed.
C0557 removed.
A0034 removed.
A0768 removed.
A0240 removed.
A0042 removed.
A0141 removed.
A0888 removed.
A0032 removed.
A0596 removed.
A0969 removed.
A0215 removed.
A0074 removed.
A0229 removed.
M0760 removed.
M0434 removed.
M0033 removed.
M0942 removed.
M0034 removed.
M0868 removed.
M0595 removed.
M0476 removed.
M0770 removed.
M0577 removed.
M0638 removed.
M0421 removed.
M0006 removed.
M0552 removed.
M0040 removed.
M0165 removed.
M0256 removed.
M0127 removed.

/Users/Reddy/dev/pyradigm/pyradigm/pyradigm.py:1169: UserWarning: Requested removal of all the samples - output dataset would be empty.
warnings.warn('Requested removal of all the samples - output dataset would be empty.')


Data structure is even producing a warning to let you know the resulting output would be empty! We can verify that:

bool(smaller)

False


# Portability¶

This is all well and good. How does it interact with other packages out there, you might ask? It is as simple as you can imagine:

from sklearn import svm
clf = svm.SVC(gamma=0.001, C=100.)

data_matrix, target, sample_ids = binary_dataset.data_and_labels()
clf.fit(data_matrix, target)

SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0,
decision_function_shape='ovr', degree=3, gamma=0.001, kernel='rbf',
max_iter=-1, probability=False, random_state=None, shrinking=True,
tol=0.001, verbose=False)


There you have it, a simple example to show you the utility and convenience of this dataset.

Thanks for checking it out.

I would appreciate if you could give me feedback on improving or sharpening it further.