Using External Datasets with HuggingFace Data Loader

PaulYoung
Wed, 20 Apr 2022 20:22:28 +0000

Hi, I am a beginner with HuggingFace and PyTorch and I am having trouble doing a simple task. I took the ViT tutorial Fine-Tune ViT for Image Classification with 🤗 Transformers and replaced the second block with this:

from datasets import load_dataset
ds = load_dataset(
‘./tiny-imagenet-200’)
#data_files= {“train”: “train”, “test”: “test”, “validate”: “val”})
ds

As the name implies, ./tiny-imagenet-200 is the directory containing the tiny-imagenet dataset in its default configuration. Here is the data structure of that dataset:
./tiny-imagenet-200
test/
images/
[all test JPEG images]
train/
[directories for each class number]/
images/
[current class JPEG images]
[class name]_boxes.txt
val/
images/
[all validation JPEG images]
val_annotations.txt
wnids.txt (appears to be a list of class numbers)
words.txt (table of class numbers to names)

The error returned is

Unable to resolve any data file that matches ‘[’*test’, ‘*eval’]’ at /home/omniverse03/Documents/BeansTransformerTutorial/tiny-imagenet-200 with any supported extension [‘csv’, ‘tsv’, ‘json’, ‘jsonl’, ‘parquet’, ‘txt’, ‘blp’, ‘bmp’, ‘dib’, ‘bufr’, ‘cur’, ‘pcx’, ‘dcx’, ‘dds’, ‘ps’, ‘eps’, ‘fit’, ‘fits’, ‘fli’, ‘flc’, ‘ftc’, ‘ftu’, ‘gbr’, ‘gif’, ‘grib’, ‘h5’, ‘hdf’, ‘png’, ‘apng’, ‘jp2’, ‘j2k’, ‘jpc’, ‘jpf’, ‘jpx’, ‘j2c’, ‘icns’, ‘ico’, ‘im’, ‘iim’, ‘tif’, ‘tiff’, ‘jfif’, ‘jpe’, ‘jpg’, ‘jpeg’, ‘mpg’, ‘mpeg’, ‘msp’, ‘pcd’, ‘pxr’, ‘pbm’, ‘pgm’, ‘ppm’, ‘pnm’, ‘psd’, ‘bw’, ‘rgb’, ‘rgba’, ‘sgi’, ‘ras’, ‘tga’, ‘icb’, ‘vda’, ‘vst’, ‘webp’, ‘wmf’, ‘emf’, ‘xbm’, ‘xpm’, ‘zip’]