sup3r.preprocessing.feature_handling.FeatureHandler

class FeatureHandler

Bases: object

Feature Handler with cache for previously loaded features used in other calculations

Methods

extract_feature(file_paths, raster_index, ...)	Extract single feature from data source
get_input_arrays(data, chunk_number, f, ...)	Get only arrays needed for computations
get_inputs_recursive(feature, handle_features)	Lookup inputs needed to compute feature.
get_raw_feature_list(features, handle_features)	Lookup inputs needed to compute feature
has_exact_feature(feature, handle)	Check if exact feature is in handle
has_multilevel_feature(feature, handle)	Check if exact feature is in handle
has_surrounding_features(feature, handle)	Check if handle has feature values at surrounding heights.
lookup(feature, attr_name[, handle_features])	Lookup feature in feature registry
parallel_compute(data, file_paths, ...[, ...])	Compute features using parallel subprocesses
parallel_extract(file_paths, raster_index, ...)	Extract features using parallel subprocesses
pop_old_data(data, chunk_number, all_features)	Remove input feature data if no longer needed for requested features
recursive_compute(data, feature, ...)	Compute intermediate features recursively
serial_compute(data, file_paths, ...)	Compute features in series
serial_extract(file_paths, raster_index, ...)	Extract features in series
valid_handle_features(features, handle_features)	Check if features are in handle
valid_input_features(features, handle_features)	Check if features are in handle or have compute methods

Attributes

FEATURE_REGISTRY

classmethod valid_handle_features(features, handle_features)

Check if features are in handle

Parameters:

• features (str | list) – Raw feature names e.g. U_100m

• handle_features (list) – Features available in raw data

Returns:

bool – Whether feature basename is in handle

classmethod valid_input_features(features, handle_features)

Check if features are in handle or have compute methods

Parameters:

• features (str | list) – Raw feature names e.g. U_100m

• handle_features (list) – Features available in raw data

Returns:

bool – Whether feature basename is in handle

classmethod pop_old_data(data, chunk_number, all_features)

Remove input feature data if no longer needed for requested features

Parameters:

• data (dict) – dictionary of feature arrays with integer keys for chunks and str keys for features. e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

• chunk_number (int) – time chunk index to check

• all_features (list) – list of all requested features including those requiring derivation from input features

classmethod has_surrounding_features(feature, handle)

Check if handle has feature values at surrounding heights. e.g. if feature=U_40m check if the handler has u at heights below and above 40m

Parameters:

• feature (str) – Raw feature name e.g. U_100m

• handle (xarray.Dataset) – netcdf data object

Returns:

bool – Whether feature has surrounding heights

classmethod has_exact_feature(feature, handle)

Check if exact feature is in handle

Parameters:

• feature (str) – Raw feature name e.g. U_100m

• handle (xarray.Dataset) – netcdf data object

Returns:

bool – Whether handle contains exact feature or not

classmethod has_multilevel_feature(feature, handle)

Check if exact feature is in handle

Parameters:

• feature (str) – Raw feature name e.g. U_100m

• handle (xarray.Dataset) – netcdf data object

Returns:

bool – Whether handle contains multilevel data for given feature

classmethod serial_extract(file_paths, raster_index, time_chunks, input_features, **kwargs)

Extract features in series

Parameters:

• file_paths (list) – list of file paths

• raster_index (ndarray) – raster index for spatial domain

• time_chunks (list) – List of slices to chunk data feature extraction along time dimension

• input_features (list) – list of input feature strings

• kwargs (dict) – kwargs passed to source handler for data extraction. e.g. This could be {‘parallel’: True,

  ‘chunks’: {‘south_north’: 120, ‘west_east’: 120}}

  which then gets passed to xr.open_mfdataset(file, **kwargs)

Returns:

dict – dictionary of feature arrays with integer keys for chunks and str keys for features. e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

classmethod parallel_extract(file_paths, raster_index, time_chunks, input_features, max_workers=None, **kwargs)

Extract features using parallel subprocesses

Parameters:

• file_paths (list) – list of file paths

• raster_index (ndarray | list) – raster index for spatial domain

• time_chunks (list) – List of slices to chunk data feature extraction along time dimension

• input_features (list) – list of input feature strings

• max_workers (int | None) – Number of max workers to use for extraction. If equal to 1 then method is run in serial

• kwargs (dict) – kwargs passed to source handler for data extraction. e.g. This could be {‘parallel’: True,

  ‘chunks’: {‘south_north’: 120, ‘west_east’: 120}}

  which then gets passed to xr.open_mfdataset(file, **kwargs)

Returns:

dict – dictionary of feature arrays with integer keys for chunks and str keys for features. e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

classmethod recursive_compute(data, feature, handle_features, file_paths, raster_index)

Compute intermediate features recursively

Parameters:

• data (dict) – dictionary of feature arrays. e.g. data[feature] = array. (spatial_1, spatial_2, temporal)

• feature (str) – Name of feature to compute

• handle_features (list) – Features available in raw data

• file_paths (list) – Paths to data files. Used if compute method operates directly on source handler instead of input arrays. This is done with features without inputs methods like lat_lon and topography.

• raster_index (ndarray) – raster index for spatial domain

Returns:

ndarray – Array of computed feature data

classmethod serial_compute(data, file_paths, raster_index, time_chunks, derived_features, all_features, handle_features)

Compute features in series

Parameters:

• data (dict) – dictionary of feature arrays with integer keys for chunks and str keys for features. e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

• file_paths (list) – Paths to data files. Used if compute method operates directly on source handler instead of input arrays. This is done with features without inputs methods like lat_lon and topography.

• raster_index (ndarray) – raster index for spatial domain

• time_chunks (list) – List of slices to chunk data feature extraction along time dimension

• derived_features (list) – list of feature strings which need to be derived

• all_features (list) – list of all features including those requiring derivation from input features

• handle_features (list) – Features available in raw data

Returns:

data (dict) – dictionary of feature arrays, including computed features, with integer keys for chunks and str keys for features. e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

classmethod parallel_compute(data, file_paths, raster_index, time_chunks, derived_features, all_features, handle_features, max_workers=None)

Compute features using parallel subprocesses

Parameters:

• data (dict) – dictionary of feature arrays with integer keys for chunks and str keys for features. e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

• file_paths (list) – Paths to data files. Used if compute method operates directly on source handler instead of input arrays. This is done with features without inputs methods like lat_lon and topography.

• raster_index (ndarray) – raster index for spatial domain

• time_chunks (list) – List of slices to chunk data feature extraction along time dimension

• derived_features (list) – list of feature strings which need to be derived

• all_features (list) – list of all features including those requiring derivation from input features

• handle_features (list) – Features available in raw data

• max_workers (int | None) – Number of max workers to use for computation. If equal to 1 then method is run in serial

Returns:

data (dict) – dictionary of feature arrays, including computed features, with integer keys for chunks and str keys for features. Includes e.g. data[chunk_number][feature] = array. (spatial_1, spatial_2, temporal)

classmethod get_input_arrays(data, chunk_number, f, handle_features)

Get only arrays needed for computations

Parameters:

• data (dict) – Dictionary of feature arrays

• chunk_number – time chunk for which to get input arrays

• f (str) – feature to compute using input arrays

• handle_features (list) – Features available in raw data

Returns:

dict – Dictionary of arrays with only needed features

classmethod lookup(feature, attr_name, handle_features=None)

Lookup feature in feature registry

Parameters:

• feature (str) – Feature to lookup in registry

• attr_name (str) – Type of method to lookup. e.g. inputs or compute

• handle_features (list) – List of feature names (datasets) available in the source file. If feature is found explicitly in this list, height/pressure suffixes will not be appended to the output.

Returns:

method | None – Feature registry method corresponding to feature

classmethod get_inputs_recursive(feature, handle_features)

Lookup inputs needed to compute feature. Walk through inputs methods for each required feature to get all raw features.

Parameters:

• feature (str) – Feature for which to get needed inputs for derivation

• handle_features (list) – Features available in raw data

Returns:

list – List of input features

classmethod get_raw_feature_list(features, handle_features)

Lookup inputs needed to compute feature

Parameters:

• features (list) – Features for which to get needed inputs for derivation

• handle_features (list) – Features available in raw data

Returns:

list – List of input features

abstract classmethod extract_feature(file_paths, raster_index, feature, time_slice=slice(None, None, None), **kwargs)

Extract single feature from data source

Parameters:

• file_paths (list) – path to data file

• raster_index (ndarray) – Raster index array

• time_slice (slice) – slice of time to extract

• feature (str) – Feature to extract from data

• kwargs (dict) – Keyword arguments passed to source handler

Returns:

ndarray – Data array for extracted feature (spatial_1, spatial_2, temporal)