sup3r.utilities.execution.DistributedProcess
- class DistributedProcess(max_nodes=1, n_chunks=None, max_chunks=None, incremental=False)[source]
Bases:
object
High-level class with commonly used functionality for processes distributed across multiple nodes
- Parameters:
max_nodes (int, optional) – Max number of nodes to distribute processes across
n_chunks (int, optional) – Number of chunks to split all processes into. These process chunks will be distributed across nodes.
max_chunks (int, optional) – Max number of chunks processes can be split into.
incremental (bool) – Whether to skip previously run process chunks or to overwrite.
Methods
chunk_finished
(chunk_index)Check if process for given chunk_index has already been run.
node_finished
(node_index)Check if all out files for a given node have been saved
Attributes
Check if all out files have been saved
Get the number of process chunks for this distributed routine.
Check whether any processes have failed.
Get uncapped max number of nodes to distribute processes across
Get the chunk indices for different nodes
Get the file lists for different nodes
Get the max number of nodes to distribute chunks across, limited by the number of process chunks
Get list of out files to write process output to
- node_finished(node_index)[source]
Check if all out files for a given node have been saved
- Parameters:
node_index (int) – Index of node to check for completed processes
- Returns:
bool – Whether all processes for the given node have finished
- chunk_finished(chunk_index)[source]
Check if process for given chunk_index has already been run.
- Parameters:
chunk_index (int) – Index of the process chunk to check for completion. Considered finished if there is already an output file and incremental is False.
- Returns:
bool – Whether the process for the given chunk has finished
- property all_finished
Check if all out files have been saved
- property out_files
Get list of out files to write process output to
- property max_nodes
Get uncapped max number of nodes to distribute processes across
- property chunks
Get the number of process chunks for this distributed routine.
- property nodes
Get the max number of nodes to distribute chunks across, limited by the number of process chunks
- property node_chunks
Get the chunk indices for different nodes
- property node_files
Get the file lists for different nodes
- property failed_chunks
Check whether any processes have failed.