sup3r.utilities.execution.DistributedProcess

class DistributedProcess(max_nodes=1, n_chunks=None, max_chunks=None, incremental=False)[source]

High-level class with commonly used functionality for processes distributed across multiple nodes

Parameters:

max_nodes (int, optional) – Max number of nodes to distribute processes across
n_chunks (int, optional) – Number of chunks to split all processes into. These process chunks will be distributed across nodes.
max_chunks (int, optional) – Max number of chunks processes can be split into.
incremental (bool) – Whether to skip previously run process chunks or to overwrite.

Methods

`chunk_finished`(chunk_index)	Check if process for given chunk_index has already been run.
`node_finished`(node_index)	Check if all out files for a given node have been saved

Attributes

`all_finished`	Check if all out files have been saved
`chunks`	Get the number of process chunks for this distributed routine.
`failed_chunks`	Check whether any processes have failed.
`max_nodes`	Get uncapped max number of nodes to distribute processes across
`node_chunks`	Get the chunk indices for different nodes
`node_files`	Get the file lists for different nodes
`nodes`	Get the max number of nodes to distribute chunks across, limited by the number of process chunks
`out_files`	Get list of out files to write process output to

node_finished(node_index)[source]

Check if all out files for a given node have been saved

Parameters:: node_index (int) – Index of node to check for completed processes
Returns:: bool – Whether all processes for the given node have finished

chunk_finished(chunk_index)[source]

Check if process for given chunk_index has already been run.

Parameters:: chunk_index (int) – Index of the process chunk to check for completion. Considered finished if there is already an output file and incremental is False.
Returns:: bool – Whether the process for the given chunk has finished

property max_nodes: Get uncapped max number of nodes to distribute processes across

property chunks: Get the number of process chunks for this distributed routine.

property nodes: Get the max number of nodes to distribute chunks across, limited by the number of process chunks