Using External Datasets with HuggingFace Data Loader

mapama247
Thu, 21 Apr 2022 09:13:38 +0000

Hi!

I assume that you are using the official dataset from Stanford, which can be downloaded by doing:

wget http://cs231n.stanford.edu/tiny-imagenet-200.zip

After unzipping it I simply did:

from datasets import load_dataset
ds = load_dataset('imagefolder', data_dir='./tiny-imagenet-200')

This loads the 120k examples into a single set. To load a specific set (i.e. test) you could do something like:

ds_test = load_dataset('imagefolder', data_dir='./tiny-imagenet-200/test')

As you can see this uses an ImageFolder dataset builder, the alternative would be to build your own custom dataloader (e.g. food101.py). Here’s some information on how to write data loading scripts, I believe all the examples are for text data but you can just check the loading script of any image dataset available on the hub.

PS: Make sure that you have installed the “Image” feature from the datasets library!

pip install datasets[vision]

You can find more info in the official documentation.