nsrdb.data_model.gfs.GfsVar

class GfsVar(name, var_meta, date, **kwargs)[source]

Bases: AncillaryVarHandler

Framework for GFS source data extraction.

Parameters:

name (str) – NSRDB var name.
var_meta (str | pd.DataFrame | None) – CSV file or dataframe containing meta data for all NSRDB variables. Defaults to the NSRDB var meta csv in git repo.
date (datetime.date) – Single day to extract data for.

Methods

`pre_flight`()	Perform pre-flight checks - source file check.
`scale_data`(array)	Perform a safe data scaling operation on a source data array.
`unscale_data`(array)	Perform a safe data unscaling operation on a source data array.

Attributes

`DEFAULT_DIR`
`NN_METHOD`
`attrs`	Return a dictionary of dataset attributes for HDF5 dataset attrs.
`cache_file`	Get the nearest neighbor result cache csv file for this var.
`chunks`	Get the variable's intended storage chunk shape.
`data_source`	Get the data source.
`date`	Get the date for this handler
`date_stamp`	Get the GFS datestamp corresponding to the specified datetime date
`description`	Long variable description.
`doy`	Get the day of year for this handler
`dset_name`	Get the source dataset name for the NSRDB variable.
`dtype`	Get the data type attribute.
`elevation_correct`	Get the elevation correction preference.
`file`	Overrides the base handler file attribute
`file_set`	Get the source file set name for the NSRDB variable.
`files`	Get the GFS file paths for the target NSRDB day.
`final_dtype`	Get the variable's intended storage datatype.
`grid`	Return the GFS source coordinates with elevation.
`mask`	Get a boolean mask to locate the current variable in the meta data.
`name`	Get the NSRDB variable name.
`next_date`	Get the date after the date for this handler.
`next_file`	Get the file path for the date for the target NSRDB variable name based on the glob self.next_pattern.
`next_file_exists`	Check if file for next date exists
`next_pattern`	Get the next date source file pattern which is sent to glob().
`pattern`	Get the source file pattern which is sent to glob().
`physical_max`	Get the variable's physical maximum value.
`physical_min`	Get the variable's physical minimum value.
`scale_factor`	Get the variable's intended storage scale factor.
`shape`	Output dataset shape, (time, sites)
`source_data`	Get single day data from the GFS source file.
`source_dir`	Get the source directory containing the variable data files.
`spatial_method`	Get the spatial interpolation method.
`temporal_method`	Get the temporal interpolation method.
`time_index`	Get the albedo native time index.
`units`	Get the units attribute.
`var_meta`	Return the meta data for NSRDB variables.

property time_index

Get the albedo native time index.

Returns:: alb_ti (pd.DatetimeIndex) – Pandas datetime index for the current day at the albedo resolution (1-month).

property date_stamp

Get the GFS datestamp corresponding to the specified datetime date

Returns:: date (str) – Date stamp that should be in the GFS file, format is YYYY_MMDD

property file

Overrides the base handler file attribute

Returns:: list

property files

Get the GFS file paths for the target NSRDB day.

Returns:: files (list) – List of GFS file paths needed to fill given NSRDB day.

property shape

Output dataset shape, (time, sites)

Returns:: tuple

property grid

Return the GFS source coordinates with elevation.

Returns:: self._GFS_grid (pd.DataFrame) – GFS source coordinates with elevation

property source_data

Get single day data from the GFS source file.

Returns:: data (np.ndarray) – Flattened GFS time-series data. Note that the data originates as a 2D spatially gridded numpy array with shape (lat x lon).

pre_flight()[source]

Perform pre-flight checks - source file check.

Returns:: missing (str) – Look for the source file and return the string if not found. If nothing is missing, return an empty string.

property attrs

Return a dictionary of dataset attributes for HDF5 dataset attrs.

Returns:: attrs (dict) – Namespace of attributes to define the dataset.

property cache_file

Get the nearest neighbor result cache csv file for this var.

Returns:: _cache_file (False | str) – False for no caching, or a string filename (no path).

property chunks

Get the variable’s intended storage chunk shape.

Returns:: chunks (tuple) – Data storage chunk shape (row_chunk, col_chunk).

property data_source

Get the data source.

Returns:: data_source (str) – Data source.

property date

Get the date for this handler

Returns:: datetime.date

property description

Long variable description.

Returns:: description (str) – Description of the variable to provide more info than the sometimes opaque dset names.

property doy

Get the day of year for this handler

Returns:: int

property dset_name

Get the source dataset name for the NSRDB variable. This is typically the netcdf or h5 source dataset name for the variable such as T2M or TOTANGSTR (for MERRA temp and alpha)

Returns:: str

property dtype

Get the data type attribute.

Returns:: dtype (str) – Intended NSRDB disk data type.

property elevation_correct

Get the elevation correction preference.

Returns:: elevation_correct (bool) – Whether or not to use elevation correction for the current var.

property file_set

Get the source file set name for the NSRDB variable. This is typically used for MERRA source filesets such as tavg1_2d_aer_Nx or tavg1_2d_slv_Nx (for MERRA)

Returns:: str

property final_dtype

Get the variable’s intended storage datatype.

Returns:: dtype (str) – Data type for the current variable.

property mask: Get a boolean mask to locate the current variable in the meta data.

property name: Get the NSRDB variable name.

property next_date

Get the date after the date for this handler. This is used to get the data for the next date for temporal interpolation

Returns:: datetime.date

property next_file

Get the file path for the date for the target NSRDB variable name based on the glob self.next_pattern. The file is used to get the data for the next date for temporal interpolation

Returns:: str

property next_file_exists: Check if file for next date exists

property next_pattern

Get the next date source file pattern which is sent to glob().

Returns:: str | None

property pattern

Get the source file pattern which is sent to glob().

Returns:: str | None

property physical_max

Get the variable’s physical maximum value.

Returns:: physical_max (float) – Physical maximum value for the variable. Variable range can be truncated at this value. Must be consistent with the final dtype and scale factor.

property physical_min

Get the variable’s physical minimum value.

Returns:: physical_min (float) – Physical minimum value for the variable. Variable range can be truncated at this value. Must be consistent with the final dtype and scale factor.

scale_data(array)

Perform a safe data scaling operation on a source data array.

Steps:

Enforce physical range limits
Apply scale factor (mulitply)
Round if integer
Enforce dtype bit range limits
Perform dtype conversion
Return manipulated array

Parameters:: array (np.ndarray) – Source data array with full precision (likely float32).
Returns:: array (np.ndarray) – Source data array with final datatype.

property scale_factor

Get the variable’s intended storage scale factor.

Returns:: scale_factor (float) – Scale factor for the current variable. Data is multiplied by this scale factor before being stored.

property source_dir

Get the source directory containing the variable data files.

Returns:: source_dir (str) – Directory containing source data files (with possible sub folders).

property spatial_method

Get the spatial interpolation method.

Returns:: spatial_method (str) – NN or IDW

property temporal_method

Get the temporal interpolation method.

Returns:: temporal_method (str) – linear or nearest

property units

Get the units attribute.

Returns:: units (str) – NSRDB variable units.

unscale_data(array)

Perform a safe data unscaling operation on a source data array.

Parameters:: array (np.ndarray) – Scaled source data array with integer precision.
Returns:: array (np.ndarray) – Unscaled source data array with float32 precision.

property var_meta

Return the meta data for NSRDB variables.

Returns:: _var_meta (pd.DataFrame) – Meta data for NSRDB variables.