sup3r.preprocessing.batch_queues.abstract

sup3r.preprocessing.batch_queues.abstract#

Abstract batch queue class used for multi-threaded batching / training.

TODO:

  1. Figure out apparent “blocking” issue with threaded enqueue batches. max_workers=1 is the fastest?

(2) Setup distributed data handling so this can work with data distributed over multiple nodes.

Classes

AbstractBatchQueue(samplers[, batch_size, ...])

Abstract BatchQueue class.