rex.resource_extraction.resource_extraction.MultiYearResourceX

class MultiYearResourceX(resource_path, years=None, tree=None, unscale=True, str_decode=True, res_cls=None, hsds=False, hsds_kwargs=None)[source]

Bases: ResourceX

Multi Year resource extraction class

Parameters:
  • resource_path (str) – Unix shell style pattern path with * wildcards to multi-file resource file sets. Files must have the same time index and coordinates but can have different datasets.

  • years (list, optional) – List of years to access, by default None

  • tree (str | cKDTree) – cKDTree or path to .pkl file containing pre-computed tree of lat, lon coordinates

  • unscale (bool) – Boolean flag to automatically unscale variables on extraction

  • str_decode (bool) – Boolean flag to decode the bytestring meta data into normal strings. Setting this to False will speed up the meta data read.

  • res_cls (obj) – Resource handler to use to open individual .h5 files

  • hsds (bool, optional) – Boolean flag to use h5pyd to handle .h5 ‘files’ hosted on AWS behind HSDS, by default False

  • hsds_kwargs (dict, optional) – Dictionary of optional kwargs for h5pyd, e.g., bucket, username, password, by default None

Methods

box_gids(lat_lon_1, lat_lon_2)

Get gids within bounding lat_lon coordinates

close()

Close res_cls instance

get_SAM_gid(gid[, out_path, write_time, ...])

Extract time-series of all variables needed to run SAM for nearest site to given resource gid

get_SAM_lat_lon(lat_lon[, check_lat_lon, ...])

Extract time-series of all variables needed to run SAM for nearest site to given lat_lon

get_box_df(ds_name, lat_lon_1, lat_lon_2)

Extract timeseries of of all sites in given bounding box and return as a DataFrame

get_box_ts(ds_name, lat_lon_1, lat_lon_2)

Extract timeseries of of all sites in given bounding box

get_gid_df(ds_name, gid)

Extract timeseries of site(s) nearest to given lat_lon(s) and return as a DataFrame

get_gid_ts(ds_name, gid)

Extract timeseries of site(s) neareset to given lat_lon(s)

get_grid_vectors(target[, meta])

Get vectors representing pure horizontal/vertical movements in the meta data coordinate system.

get_lat_lon_df(ds_name, lat_lon[, check_lat_lon])

Extract timeseries of site(s) nearest to given lat_lon(s) and return as a DataFrame

get_lat_lon_ts(ds_name, lat_lon[, check_lat_lon])

Extract timeseries of site(s) neareset to given lat_lon(s)

get_means_map(ds_name[, year, region, ...])

Extract given year(s) and compute means

get_raster_index(target, shape[, meta, ...])

Get meta data index values that correspond to a 2D rectangular grid of the requested shape starting with the target coordinate in the bottom left hand corner.

get_region_df(ds_name, region[, region_col])

Extract timeseries of of all sites in given region and return as a DataFrame

get_region_ts(ds_name, region[, region_col])

Extract timeseries of of all sites in given region

get_timestep_map(ds_name, timestep[, ...])

Extract a map of the given dataset at the given timestep for the given region if supplied

lat_lon_gid(lat_lon[, check_lat_lon])

Get nearest gid to given (lat, lon) pair or pairs

make_SAM_files(res_h5, gids, out_path[, ...])

A performant parallel entry point for making many SAM csv files for many gids

region_gids(region[, region_col])

Get the gids for given region

save_region(out_fpath, region[, datasets, ...])

Extract desired datasets from desired region and save to a new out_fpath .h5 file

save_subset(out_fpath, gids[, datasets])

Extract desired datasets for given gids and save to a new out_fpath .h5 file

timestep_idx(timestep)

Get the index of the desired timestep

Attributes

attrs

Global (file) attributes

coordinates

(lat, lon) pairs

counties

Available Counties

countries

Available Countires

data_version

Get the version attribute of the data.

datasets

Datasets available

distance_threshold

Distance threshold, calculated as half of the diagonal between closest resource points, with an extra 5% margin

dsets

Datasets available

global_attrs

Global (file) attributes

groups

Groups available

h5

Open h5py File instance.

lat_lon

Extract (latitude, longitude) pairs

meta

Resource meta data DataFrame

res_dsets

Available resource datasets

resource

Open res_cls instance to access res_h5 data

resource_datasets

Available resource datasets

shape

Resource shape (timesteps, sites) shape = (len(time_index), len(meta))

states

Available states

time_index

Resource DatetimeIndex

tree

Pre-initialized cKDTree on the resource lat, lon coordinates

DEFAULT_RES_CLS

alias of Resource

get_means_map(ds_name, year=None, region=None, region_col='state', max_workers=None, chunks_per_worker=5)[source]

Extract given year(s) and compute means

Parameters:
  • ds_name (str) – Dataset to extract

  • year (str | list, optional) – Year(s) to compute means for, by default None

  • region (str) – Region to extract all pixels for

  • region_col (str) – Region column to search

  • max_workers (None | int, optional) – Number of workers to use, if 1 run in serial, if None use all available cores, by default None

  • chunks_per_slice (int, optional) – Number of chunks to extract on each worker, by default 5

Returns:

ts_map (pandas.DataFrame) – DataFrame of map values

property attrs

Global (file) attributes

Returns:

attrs (dict)

box_gids(lat_lon_1, lat_lon_2)

Get gids within bounding lat_lon coordinates

Parameters:
  • lat_lon_1 (list | tuple) – One corner of the bounding box

  • lat_lon_2 (list | tuple) – The other corner of the bounding box

Returns:

gids (ndarray) – Gids in bounding box

close()

Close res_cls instance

property coordinates

(lat, lon) pairs

Returns:

lat_lon (ndarray)

Type:

Coordinates

property counties

Available Counties

Returns:

counties (ndarray)

property countries

Available Countires

Returns:

countries (ndarray)

property data_version

Get the version attribute of the data. None if not available.

Returns:

version (str | None)

property datasets

Datasets available

Returns:

list

property distance_threshold

Distance threshold, calculated as half of the diagonal between closest resource points, with an extra 5% margin

Returns:

float

property dsets

Datasets available

Returns:

list

get_SAM_gid(gid, out_path=None, write_time=True, extra_meta_data=None, **kwargs)

Extract time-series of all variables needed to run SAM for nearest site to given resource gid

Parameters:
  • gid (int | list) – Resource gid(s) of interset

  • out_path (str, optional) – Path to save SAM data to in SAM .csv format, by default None

  • write_time (bool) – Flag to write the time columns (Year, Month, Day, Hour, Minute)

  • extra_meta_data (dict, optional) – Dictionary that maps the names and values of extra meta info. For example, extra_meta_data={‘TMY Year’: ‘2020’} will add a column ‘TMY Year’ to the meta data with a value of ‘2020’.

  • kwargs (dict) – Internal kwargs for get_SAM_df

Returns:

SAM_df (pandas.DataFrame | list) – Time-series DataFrame for given site and dataset If multiple lat, lon pairs are given a list of DatFrames is returned

get_SAM_lat_lon(lat_lon, check_lat_lon=True, out_path=None, **kwargs)

Extract time-series of all variables needed to run SAM for nearest site to given lat_lon

Parameters:
  • lat_lon (tuple) – (lat, lon) coordinate of interest

  • check_lat_lon (bool, optional) – Flag to check to make sure the requested lat lons are inside the resource grid. This is done by comparing with the bounding box of the resource coordinates and by ensuring the nearest neighbor distance are below the distance threshold to ensure that requested lat, lon coordinates are within the resource grid, by default True

  • out_path (str, optional) – Path to save SAM data to in SAM .csv format, by default None

  • kwargs (dict) – Internal kwargs for get_SAM_df

Returns:

SAM_df (pandas.DataFrame | list) – Time-series DataFrame for given site and dataset If multiple lat, lon pairs are given a list of DatFrames is returned

get_box_df(ds_name, lat_lon_1, lat_lon_2)

Extract timeseries of of all sites in given bounding box and return as a DataFrame

Parameters:
  • ds_name (str) – Dataset to extract

  • lat_lon_1 (list | tuple) – One corner of the bounding box

  • lat_lon_2 (list | tuple) – The other corner of the bounding box

Returns:

box_df (pandas.DataFrame) – Time-series array of desired dataset for all sites in desired bounding box

get_box_ts(ds_name, lat_lon_1, lat_lon_2)

Extract timeseries of of all sites in given bounding box

Parameters:
  • ds_name (str) – Dataset to extract

  • lat_lon_1 (list | tuple) – One corner of the bounding box

  • lat_lon_2 (list | tuple) – The other corner of the bounding box

Returns:

box_ts (ndarray) – Time-series array of desired dataset for all sites in desired bounding box

get_gid_df(ds_name, gid)

Extract timeseries of site(s) nearest to given lat_lon(s) and return as a DataFrame

Parameters:
  • ds_name (str) – Dataset to extract

  • gid (int | list) – Resource gid(s) of interset

Returns:

df (pandas.DataFrame) – Time-series DataFrame for given site(s) and dataset

get_gid_ts(ds_name, gid)

Extract timeseries of site(s) neareset to given lat_lon(s)

Parameters:
  • ds_name (str) – Dataset to extract

  • gid (int | list) – Resource gid(s) of interset

Returns:

ts (ndarray) – Time-series for given site(s) and dataset

get_grid_vectors(target, meta=None)

Get vectors representing pure horizontal/vertical movements in the meta data coordinate system. Note that this can break down if a target is requested outside of the main grid area.

Parameters:
  • target (tuple) – Starting coordinate (latitude, longitude) in decimal degrees for the bottom left hand corner of the raster grid.

  • meta (pd.DataFrame | None) – Optional meta data input with latitude, longitude fields. Default is None which extracts self.meta from the resource data.

Returns:

  • gid_target (np.ndarray) – 1D array of shape (2,) with (latitude, longitude) corresponding to the meta data grid cell closest to the requested target.

  • vector_x (np.ndarray) – 1D array of shape (2,) with (delta_latitude, delta_longitude) corresponding to the vector for pure positive horizontal movement in the meta data

  • vector_y (np.ndarray) – 1D array of shape (2,) with (delta_latitude, delta_longitude) corresponding to the vector for pure positive vertical movement in the meta data

  • close (np.ndarray) – Meta data index values corresponding to the 3x3 box of pixels closest to gid_target.

get_lat_lon_df(ds_name, lat_lon, check_lat_lon=True)

Extract timeseries of site(s) nearest to given lat_lon(s) and return as a DataFrame

Parameters:
  • ds_name (str) – Dataset to extract

  • lat_lon (tuple) – (lat, lon) coordinate of interest

  • check_lat_lon (bool, optional) – Flag to check to make sure the requested lat lons are inside the resource grid. This is done by comparing with the bounding box of the resource coordinates and by ensuring the nearest neighbor distance are below the distance threshold to ensure that requested lat, lon coordinates are within the resource grid, by default True

Returns:

df (pandas.DataFrame) – Time-series DataFrame for given site(s) and dataset

get_lat_lon_ts(ds_name, lat_lon, check_lat_lon=True)

Extract timeseries of site(s) neareset to given lat_lon(s)

Parameters:
  • ds_name (str) – Dataset to extract

  • lat_lon (tuple | list) – (lat, lon) coordinate of interest or pairs of coordinates

  • check_lat_lon (bool, optional) – Flag to check to make sure the requested lat lons are inside the resource grid. This is done by comparing with the bounding box of the resource coordinates and by ensuring the nearest neighbor distance are below the distance threshold to ensure that requested lat, lon coordinates are within the resource grid, by default True

Returns:

ts (ndarray) – Time-series for given site(s) and dataset

get_raster_index(target, shape, meta=None, max_delta=50)

Get meta data index values that correspond to a 2D rectangular grid of the requested shape starting with the target coordinate in the bottom left hand corner. Note that this can break down if a target is requested outside of the main grid area.

Parameters:
  • target (tuple) – Starting coordinate (latitude, longitude) in decimal degrees for the bottom left hand corner of the raster grid.

  • shape (tuple) – Desired raster shape in format (number_rows, number_cols)

  • meta (pd.DataFrame | None) – Optional meta data input with latitude, longitude fields. Default is None which extracts self.meta from the resource data.

  • max_delta (int) – Optional maximum limit on the raster shape that is retrieved at once. If shape is (20, 20) and max_delta=10, the full raseter will be retrieved in four chunks of (10, 10). This helps adapt to non-regular grids that curve over large distances.

Returns:

raster_index (np.ndarray) – 2D array of meta data index values that form a 2D rectangular grid with latitudes descending from top to bottom and longitudes ascending from left to right.

get_region_df(ds_name, region, region_col='state')

Extract timeseries of of all sites in given region and return as a DataFrame

Parameters:
  • ds_name (str) – Dataset to extract

  • region (str) – Region to extract all pixels for

  • region_col (str) – Region column to search

Returns:

region_df (pandas.DataFrame) – Time-series array of desired dataset for all sites in desired region

get_region_ts(ds_name, region, region_col='state')

Extract timeseries of of all sites in given region

Parameters:
  • ds_name (str) – Dataset to extract

  • region (str) – Region to search for

  • region_col (str) – Region column to search

Returns:

region_ts (ndarray) – Time-series array of desired dataset for all sites in desired region

get_timestep_map(ds_name, timestep, region=None, region_col='state', box=None)

Extract a map of the given dataset at the given timestep for the given region if supplied

Parameters:
  • ds_name (str) – Dataset to extract

  • timestep (str) – Timestep of interest

  • region (str, optional) – Region to extract all pixels for, by default None

  • region_col (str, optional) – Region column to search, by default ‘state’

  • box (tuple, optional) – Bounding corners of box to extract pixels for

Returns:

ts_map (pandas.DataFrame) – DataFrame of map values

property global_attrs

Global (file) attributes

Returns:

global_attrs (dict)

property groups

Groups available

Returns:

groups (list)

property h5

Open h5py File instance. If _group is not None return open Group

Returns:

h5 (h5py.File | h5py.Group)

property lat_lon

Extract (latitude, longitude) pairs

Returns:

lat_lon (ndarray)

lat_lon_gid(lat_lon, check_lat_lon=True)

Get nearest gid to given (lat, lon) pair or pairs

Parameters:
  • lat_lon (ndarray) – Either a single (lat, lon) pair or series of (lat, lon) pairs

  • check_lat_lon (bool, optional) – Flag to check to make sure the requested lat lons are inside the resource grid. This is done by comparing with the bounding box of the resource coordinates and by ensuring the nearest neighbor distance are below the distance threshold to ensure that requested lat, lon coordinates are within the resource grid, by default True

Returns:

gids (int | ndarray) – Nearest gid(s) to given (lat, lon) pair(s)

classmethod make_SAM_files(res_h5, gids, out_path, write_time=True, extra_meta_data=None, max_workers=1, n_chunks=36, **kwargs)

A performant parallel entry point for making many SAM csv files for many gids

Parameters:
  • res_h5 (str) – Filepath to resource h5 file.

  • gids (list | tuple | np.ndarray) – Resource gid(s) of interset

  • out_path (str, optional) – Path to save SAM data to in SAM .csv format. A gid index “*_{gid}.csv” will be appended to the file path

  • write_time (bool) – Flag to write the time columns (Year, Month, Day, Hour, Minute)

  • extra_meta_data (dict, optional) – Dictionary that maps the names and values of extra meta info. For example, extra_meta_data={‘TMY Year’: ‘2020’} will add a column ‘TMY Year’ to the meta data with a value of ‘2020’.

  • max_workers (int | None) – Number of parallel workers. None for all workers.

  • n_chunks (int) – Number of chunks to split gids into for parallelization

  • kwargs (dict) – Internal kwargs for get_SAM_df

property meta

Resource meta data DataFrame

Returns:

meta (pandas.DataFrame)

region_gids(region, region_col='state')

Get the gids for given region

Parameters:
  • region (str) – Region to search for

  • region_col (str) – Region column to search

Returns:

gids (ndarray) – Vector of gids in given region

property res_dsets

Available resource datasets

Returns:

list

property resource

Open res_cls instance to access res_h5 data

Returns:

res_cls (rex.resource.Resource | rex.renewable_resource.*)

property resource_datasets

Available resource datasets

Returns:

list

save_region(out_fpath, region, datasets=None, region_col='state')

Extract desired datasets from desired region and save to a new out_fpath .h5 file

Parameters:
  • out_fpath (str) – Path to .h5 file to save region datasets to

  • region (str, optional) – Region to extract all pixels for, by default None

  • datasets (str | list, optional) – Dataset(s) to extract from given region and save to out_fpath, if None extract all datasets, by default None

  • region_col (str, optional) – Region column to search, by default ‘state’

save_subset(out_fpath, gids, datasets=None)

Extract desired datasets for given gids and save to a new out_fpath .h5 file

Parameters:
  • out_fpath (str) – Path to .h5 file to save region datasets to

  • gids (list) – List of gids to extract data from and save to .h5

  • datasets (str | list, optional) – Dataset(s) to extract from given region and save to out_fpath, if None extract all datasets, by default None

property shape

Resource shape (timesteps, sites) shape = (len(time_index), len(meta))

Returns:

shape (tuple)

property states

Available states

Returns:

states (ndarray)

property time_index

Resource DatetimeIndex

Returns:

time_index (pandas.DatetimeIndex)

timestep_idx(timestep)

Get the index of the desired timestep

Parameters:

timestep (str) – Timestep of interest

Returns:

ts_idx (int) – Time index value

property tree

Pre-initialized cKDTree on the resource lat, lon coordinates

Returns:

tree (cKDTree)