Using External Datasets with HuggingFace Data Loader

PaulYoung
Fri, 22 Apr 2022 18:40:16 +0000

Continuing with my quest to run the tutorial with the tinyimagenet dataset, I got to this block

train_results = trainer.train()
trainer.save_model()
trainer.log_metrics(“train”, train_results.metrics)
trainer.save_metrics(“train”, train_results.metrics)
trainer.save_state()

and it threw “ValueError: operands could not be broadcast together with shapes (224,224) (3,)” on the first line. This appears to be indicating a numpy dimensions error based on my investigations, but it is deep within trainer.train(). Moreover, there is not any relevant 3’s or 224’s in all the earlier code.

colab.research.google.com

Google Colaboratory

All code is identical to the tutorial I mentioned with these exceptions:
the pip install mapama mentioned
block 2 replaced by mario’s code
Some printouts removed
save, eval, and logging steps multiplied by 10 to reduce output