rex.resource.Resource

class Resource(h5_file, unscale=True, str_decode=True, group=None, hsds=False, hsds_kwargs=None)[source]

Bases: rex.resource.BaseResource

Base class to handle resource .h5 files

Examples

Extracting the resource’s Datetime Index

>>> file = '$TESTDATADIR/nsrdb/ri_100_nsrdb_2012.h5'
>>> with Resource(file) as res:
>>>     ti = res.time_index
>>>
>>> ti
DatetimeIndex(['2012-01-01 00:00:00', '2012-01-01 00:30:00',
               '2012-01-01 01:00:00', '2012-01-01 01:30:00',
               '2012-01-01 02:00:00', '2012-01-01 02:30:00',
               '2012-01-01 03:00:00', '2012-01-01 03:30:00',
               '2012-01-01 04:00:00', '2012-01-01 04:30:00',
               ...
               '2012-12-31 19:00:00', '2012-12-31 19:30:00',
               '2012-12-31 20:00:00', '2012-12-31 20:30:00',
               '2012-12-31 21:00:00', '2012-12-31 21:30:00',
               '2012-12-31 22:00:00', '2012-12-31 22:30:00',
               '2012-12-31 23:00:00', '2012-12-31 23:30:00'],
              dtype='datetime64[ns]', length=17568, freq=None)

Efficient slicing of the Datetime Index

>>> with Resource(file) as res:
>>>     ti = res['time_index', 1]
>>>
>>> ti
2012-01-01 00:30:00
>>> with Resource(file) as res:
>>>     ti = res['time_index', :10]
>>>
>>> ti
DatetimeIndex(['2012-01-01 00:00:00', '2012-01-01 00:30:00',
               '2012-01-01 01:00:00', '2012-01-01 01:30:00',
               '2012-01-01 02:00:00', '2012-01-01 02:30:00',
               '2012-01-01 03:00:00', '2012-01-01 03:30:00',
               '2012-01-01 04:00:00', '2012-01-01 04:30:00'],
              dtype='datetime64[ns]', freq=None)
>>> with Resource(file) as res:
>>>     ti = res['time_index', [1, 2, 4, 8, 9]
>>>
>>> ti
DatetimeIndex(['2012-01-01 00:30:00', '2012-01-01 01:00:00',
               '2012-01-01 02:00:00', '2012-01-01 04:00:00',
               '2012-01-01 04:30:00'],
              dtype='datetime64[ns]', freq=None)

Extracting resource’s site metadata

>>> with Resource(file) as res:
>>>     meta = res.meta
>>>
>>> meta
        latitude  longitude   elevation  timezone    country ...
0      41.29     -71.86    0.000000        -5           None ...
1      41.29     -71.82    0.000000        -5           None ...
2      41.25     -71.82    0.000000        -5           None ...
3      41.33     -71.82   15.263158        -5  United States ...
4      41.37     -71.82   25.360000        -5  United States ...
..       ...        ...         ...       ...            ... ...
95     41.25     -71.66    0.000000        -5           None ...
96     41.89     -71.66  153.720000        -5  United States ...
97     41.45     -71.66   35.440000        -5  United States ...
98     41.61     -71.66  140.200000        -5  United States ...
99     41.41     -71.66   35.160000        -5  United States ...
[100 rows x 10 columns]

Efficient slicing of the metadata

>>> with Resource(file) as res:
>>>     meta = res['meta', 1]
>>>
>>> meta
   latitude  longitude  elevation  timezone country state county urban ...
1     41.29     -71.82        0.0        -5    None  None   None  None ...
>>> with Resource(file) as res:
>>>     meta = res['meta', :5]
>>>
>>> meta
   latitude  longitude  elevation  timezone        country ...
0     41.29     -71.86   0.000000        -5           None ...
1     41.29     -71.82   0.000000        -5           None ...
2     41.25     -71.82   0.000000        -5           None ...
3     41.33     -71.82  15.263158        -5  United States ...
4     41.37     -71.82  25.360000        -5  United States ...
>>> with Resource(file) as res:
>>>     tz = res['meta', :, 'timezone']
>>>
>>> tz
0    -5
1    -5
2    -5
3    -5
4    -5
     ..
95   -5
96   -5
97   -5
98   -5
99   -5
Name: timezone, Length: 100, dtype: int64
>>> with Resource(file) as res:
>>>     lat_lon = res['meta', :, ['latitude', 'longitude']]
>>>
>>> lat_lon
    latitude  longitude
0      41.29     -71.86
1      41.29     -71.82
2      41.25     -71.82
3      41.33     -71.82
4      41.37     -71.82
..       ...        ...
95     41.25     -71.66
96     41.89     -71.66
97     41.45     -71.66
98     41.61     -71.66
99     41.41     -71.66
[100 rows x 2 columns]

Extracting resource variables (datasets)

>>> with Resource(file) as res:
>>>     wspd = res['wind_speed']
>>>
>>> wspd
[[12. 12. 12. ... 12. 12. 12.]
 [12. 12. 12. ... 12. 12. 12.]
 [12. 12. 12. ... 12. 12. 12.]
 ...
 [14. 14. 14. ... 14. 14. 14.]
 [15. 15. 15. ... 15. 15. 15.]
 [15. 15. 15. ... 15. 15. 15.]]

Efficient slicing of variables

>>> with Resource(file) as res:
>>>     wspd = res['wind_speed', :2]
>>>
>>> wspd
[[12. 12. 12. 12. 12. 12. 53. 53. 53. 53. 53. 12. 53.  1.  1. 12. 12. 12.
   1.  1. 12. 53. 53. 53. 12. 12. 12. 12. 12.  1. 12. 12.  1. 12. 12. 53.
  12. 53.  1. 12.  1. 53. 53. 12. 12. 12. 12.  1.  1.  1. 12. 12.  1.  1.
  12. 12. 53. 53. 53. 12. 12. 53. 53. 12. 12. 12. 12. 12. 12.  1. 53.  1.
  53. 12. 12. 53. 53.  1.  1.  1. 53. 12.  1.  1. 53. 53. 53. 12. 12. 12.
  12. 12. 12. 12.  1. 12.  1. 12. 12. 12.]
 [12. 12. 12. 12. 12. 12. 53. 53. 53. 53. 53. 12. 53.  1.  1. 12. 12. 12.
   1.  1. 12. 53. 53. 53. 12. 12. 12. 12. 12.  1. 12. 12.  1. 12. 12. 53.
  12. 53.  1. 12.  1. 53. 53. 12. 12. 12. 12.  1.  1.  1. 12. 12.  1.  1.
  12. 12. 53. 53. 53. 12. 12. 53. 53. 12. 12. 12. 12. 12. 12.  1. 53.  1.
  53. 12. 12. 53. 53.  1.  1.  1. 53. 12.  1.  1. 53. 53. 53. 12. 12. 12.
  12. 12. 12. 12.  1. 12.  1. 12. 12. 12.]]
>>> with Resource(file) as res:
>>>     wspd = res['wind_speed', :, [2, 3]]
>>>
>>> wspd
[[12. 12.]
 [12. 12.]
 [12. 12.]
 ...
 [14. 14.]
 [15. 15.]
 [15. 15.]]
Parameters
  • h5_file (str) – Path to .h5 resource file

  • unscale (bool, optional) – Boolean flag to automatically unscale variables on extraction, by default True

  • str_decode (bool, optional) – Boolean flag to decode the bytestring meta data into normal strings. Setting this to False will speed up the meta data read, by default True

  • group (str, optional) – Group within .h5 resource file to open, by default None

  • hsds (bool, optional) – Boolean flag to use h5pyd to handle .h5 ‘files’ hosted on AWS behind HSDS, by default False

  • hsds_kwargs (dict, optional) – Dictionary of optional kwargs for h5pyd, e.g., bucket, username, password, by default None

Methods

close()

Close h5 instance

df_str_decode(df)

Decode a dataframe with byte string columns into ordinary str cols.

get_SAM_df(site)

Placeholder for get_SAM_df method that it resource specific

get_attrs([dset])

Get h5 attributes either from file or dataset

get_dset_properties(dset)

Get dataset properties (shape, dtype, chunks)

get_meta_arr(rec_name[, rows])

Get a meta array by name (faster than DataFrame extraction).

get_scale_factor(dset)

Get dataset scale factor

get_units(dset)

Get dataset units

open_dataset(ds_name)

Open resource dataset

preload_SAM(h5_file, sites, tech[, unscale, ...])

Pre-load project_points for SAM

Attributes

ADD_ATTR

SCALE_ATTR

UNIT_ATTR

attrs

Dictionary of all dataset attributes

chunks

Dictionary of all dataset chunk sizes

coordinates

(lat, lon) pairs

data_version

Get the version attribute of the data.

datasets

Datasets available

dsets

Datasets available

dtypes

Dictionary of all dataset dtypes

global_attrs

Global (file) attributes

groups

Groups available

h5

Open h5py File instance.

lat_lon

Extract (latitude, longitude) pairs

meta

Resource meta data DataFrame

res_dsets

Available resource datasets

resource_datasets

Available resource datasets

scale_factors

Dictionary of all dataset scale factors

shape

Resource shape (timesteps, sites) shape = (len(time_index), len(meta))

shapes

Dictionary of all dataset shapes

time_index

Resource DatetimeIndex

units

Dictionary of all dataset units

property attrs

Dictionary of all dataset attributes

Returns

attrs (dict)

property chunks

Dictionary of all dataset chunk sizes

Returns

chunks (dict)

close()

Close h5 instance

property coordinates

(lat, lon) pairs

Returns

lat_lon (ndarray)

Type

Coordinates

property data_version

Get the version attribute of the data. None if not available.

Returns

version (str | None)

property datasets

Datasets available

Returns

list

static df_str_decode(df)

Decode a dataframe with byte string columns into ordinary str cols.

Parameters

df (pd.DataFrame) – Dataframe with some columns being byte strings.

Returns

df (pd.DataFrame) – DataFrame with str columns instead of byte str columns.

property dsets

Datasets available

Returns

list

property dtypes

Dictionary of all dataset dtypes

Returns

dtypes (dict)

get_SAM_df(site)

Placeholder for get_SAM_df method that it resource specific

Parameters

site (int) – Site to extract SAM DataFrame for

get_attrs(dset=None)

Get h5 attributes either from file or dataset

Parameters

dset (str) – Dataset to get attributes for, if None get file (global) attributes

Returns

attrs (dict) – Dataset or file attributes

get_dset_properties(dset)

Get dataset properties (shape, dtype, chunks)

Parameters

dset (str) – Dataset to get scale factor for

Returns

  • shape (tuple) – Dataset array shape

  • dtype (str) – Dataset array dtype

  • chunks (tuple) – Dataset chunk size

get_meta_arr(rec_name, rows=slice(None, None, None))

Get a meta array by name (faster than DataFrame extraction).

Parameters
  • rec_name (str) – Named record from the meta data to retrieve.

  • rows (slice) – Rows of the record to extract.

Returns

meta_arr (np.ndarray) – Extracted array from the meta data record name.

get_scale_factor(dset)

Get dataset scale factor

Parameters

dset (str) – Dataset to get scale factor for

Returns

float – Dataset scale factor, used to unscale int values to floats

get_units(dset)

Get dataset units

Parameters

dset (str) – Dataset to get units for

Returns

str – Dataset units, None if not defined

property global_attrs

Global (file) attributes

Returns

global_attrs (dict)

property groups

Groups available

Returns

groups (list) – List of groups

property h5

Open h5py File instance. If _group is not None return open Group

Returns

h5 (h5py.File | h5py.Group)

property lat_lon

Extract (latitude, longitude) pairs

Returns

lat_lon (ndarray)

property meta

Resource meta data DataFrame

Returns

meta (pandas.DataFrame)

open_dataset(ds_name)

Open resource dataset

Parameters

ds_name (str) – Dataset name to open

Returns

ds (ResourceDataset) – Resource for open resource dataset

classmethod preload_SAM(h5_file, sites, tech, unscale=True, str_decode=True, group=None, hsds=False, hsds_kwargs=None, time_index_step=None, means=False)

Pre-load project_points for SAM

Parameters
  • h5_file (str) – h5_file to extract resource from

  • sites (list) – List of sites to be provided to SAM

  • tech (str) – Technology to be run by SAM

  • unscale (bool) – Boolean flag to automatically unscale variables on extraction

  • str_decode (bool) – Boolean flag to decode the bytestring meta data into normal strings. Setting this to False will speed up the meta data read.

  • group (str) – Group within .h5 resource file to open

  • hsds (bool, optional) – Boolean flag to use h5pyd to handle .h5 ‘files’ hosted on AWS behind HSDS, by default False

  • hsds_kwargs (dict, optional) – Dictionary of optional kwargs for h5pyd, e.g., bucket, username, password, by default None

  • time_index_step (int, optional) – Step size for time_index, used to reduce temporal resolution, by default None

  • means (bool, optional) – Boolean flag to compute mean resource when res_array is set, by default False

Returns

SAM_res (SAMResource) – Instance of SAMResource pre-loaded with Solar resource for sites in project_points

property res_dsets

Available resource datasets

Returns

list

property resource_datasets

Available resource datasets

Returns

list

property scale_factors

Dictionary of all dataset scale factors

Returns

scale_factors (dict)

property shape

Resource shape (timesteps, sites) shape = (len(time_index), len(meta))

Returns

shape (tuple)

property shapes

Dictionary of all dataset shapes

Returns

shapes (dict)

property time_index

Resource DatetimeIndex

Returns

time_index (pandas.DatetimeIndex)

property units

Dictionary of all dataset units

Returns

units (dict)