PyTorch DataLoader equivalent for training large models with Flux

Specific Domains Machine Learning
14

I have been using extensively this one https://github.com/pevnak/DataIterators.jl, but it uses processes rather than threads (as at the time of writing threads were not available). I would like to consolidate this effort, as I would like to write this kind of thing for threads as well.

2 Likes