NSRDB Distributed (API Key Required)

from dask.distributed import LocalCluster, Client
from dotenv import load_dotenv
import pvdeg
import os

Setting Up

As in load_pvgis_distributed.ipynb we need to get ready to make our parallelized API calls. The notebook linked here goes through the process in more detail but we need to import our API key and email. This cell will not work for you unless you replace the api_key and email with your personal NSRDB api keys. REQUEST A KEY.

We also need to initalize a dask client. pvdeg.weather.weather_distributed will not work without it. It will fail silently and not populate and of the results in the resulting weather_ds called geo_weather in the example below. It is hard to recognize that this has occured so be careful. Make sure to initialize a dask client first. Visiting the link takes you to a daskboard that shows what dask is doing.

load_dotenv()

### REPLACE WITH YOUR API KEY AND EMAIL ###
api_key = "DEMO_KEY"
email = "user@mail.com"
###########################################

workers = 4

cluster = LocalCluster(
    n_workers=workers,
    processes=True,
)

client = Client(cluster)

print("Daskboard link")
print(client.dashboard_link)

Daskboard link
http://127.0.0.1:8787/status

Note on .env file: Create a file named .env in your project root directory (c:\Users\rdaxini\Documents\GitHub\PVDegradationTools_NREL\) with the following content:

api_key=YOUR_NREL_API_KEY
email=YOUR_EMAIL_ADDRESS

Replace YOUR_NREL_API_KEY and YOUR_EMAIL_ADDRESS with your actual NREL developer credentials.

Requesting Weather

As with the other script load_pvgis_distributed.ipynb. We will create a list of tuple (latitude, longitude) pairs and call the function on all of them at once. failed will represent a list of failed gids, unique location ID’s that correspond to points in space on the NSRDB. These are different than on PVGIS where they are arbitrary indexes that do NOT correspond to a spatial location on earth.

We will request “PSM4” data from the Physical Solar Model that represents a typical meteorological year (TMY) from the NSRDB. We will have to supply the api key and email from above here. Refer to the linked script to see this in further detail. The only difference between the scripts, lies in the NSRDB/PSM4 data requiring Api keys.

coords = [
    (25.783388, -80.189029),
    (24.783388, -80.189029),
]

geo_weather, geo_meta, failed = pvdeg.weather.weather_distributed(
    database="PSM4", coords=coords, api_key=api_key, email=email
)

Connected to a Dask scheduler | Dashboard: http://127.0.0.1:8787/status

Viewing Results

Same as in the other tutorial, our results are stored in an xarray dataset with a dask backend so you will have to use .compute() on the dataset to inspect the individual values of the dask arrays.

Click on the Data variables dropdown to expand the dataset viewer.

geo_weather

<xarray.Dataset> Size: 2MB
Dimensions:            (gid: 2, time: 8760)
Coordinates:
  * time               (time) datetime64[ns] 70kB 2022-01-01 ... 2022-12-31T2...
  * gid                (gid) int64 16B 0 1
Data variables: (12/15)
    Year               (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    Month              (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    Day                (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    Hour               (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    Minute             (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    temp_air           (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    ...                 ...
    ghi                (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    albedo             (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    pressure           (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    wind_direction     (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    wind_speed         (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>
    relative_humidity  (gid, time) float64 140kB dask.array<chunksize=(2, 8760), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • gid: 2
  • time: 8760
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2022-01-01 ... 2022-12-31T23:00:00

    array(['2022-01-01T00:00:00.000000000', '2022-01-01T01:00:00.000000000',
           '2022-01-01T02:00:00.000000000', ..., '2022-12-31T21:00:00.000000000',
           '2022-12-31T22:00:00.000000000', '2022-12-31T23:00:00.000000000'],
          shape=(8760,), dtype='datetime64[ns]')

  • gid
    (gid)
    int64
    0 1

    array([0, 1])

• Data variables: (15)
  • Year
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • Month
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • Day
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • Hour
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • Minute
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • temp_air
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • dew_point
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • dhi
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • dni
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • ghi
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • albedo
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • pressure
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • wind_direction
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • wind_speed
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

  • relative_humidity
    (gid, time)
    float64
    dask.array<chunksize=(2, 8760), meta=np.ndarray>

    Array Chunk Bytes 136.88 kiB 136.88 kiB Shape (2, 8760) (2, 8760) Dask graph 1 chunks in 3 graph layers Data type float64 numpy.ndarray

• Indexes: (2)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 00:00:00', '2022-01-01 01:00:00',
                   '2022-01-01 02:00:00', '2022-01-01 03:00:00',
                   '2022-01-01 04:00:00', '2022-01-01 05:00:00',
                   '2022-01-01 06:00:00', '2022-01-01 07:00:00',
                   '2022-01-01 08:00:00', '2022-01-01 09:00:00',
                   ...
                   '2022-12-31 14:00:00', '2022-12-31 15:00:00',
                   '2022-12-31 16:00:00', '2022-12-31 17:00:00',
                   '2022-12-31 18:00:00', '2022-12-31 19:00:00',
                   '2022-12-31 20:00:00', '2022-12-31 21:00:00',
                   '2022-12-31 22:00:00', '2022-12-31 23:00:00'],
                  dtype='datetime64[ns]', name='time', length=8760, freq='h'))

  • gid
    PandasIndex

    PandasIndex(Index([0, 1], dtype='int64', name='gid'))

• Attributes: (0)

geo_weather.compute()

<xarray.Dataset> Size: 2MB
Dimensions:            (gid: 2, time: 8760)
Coordinates:
  * time               (time) datetime64[ns] 70kB 2022-01-01 ... 2022-12-31T2...
  * gid                (gid) int64 16B 0 1
Data variables: (12/15)
    Year               (gid, time) float64 140kB 2.019e+03 ... 2.02e+03
    Month              (gid, time) float64 140kB 1.0 1.0 1.0 ... 12.0 12.0 12.0
    Day                (gid, time) float64 140kB 1.0 1.0 1.0 ... 31.0 31.0 31.0
    Hour               (gid, time) float64 140kB 0.0 1.0 2.0 ... 21.0 22.0 23.0
    Minute             (gid, time) float64 140kB 30.0 30.0 30.0 ... 30.0 30.0
    temp_air           (gid, time) float64 140kB 24.1 24.1 24.0 ... 24.7 24.7
    ...                 ...
    ghi                (gid, time) float64 140kB 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0
    albedo             (gid, time) float64 140kB 0.08 0.08 0.08 ... 0.0 0.0 0.0
    pressure           (gid, time) float64 140kB 1.019e+03 ... 1.02e+03
    wind_direction     (gid, time) float64 140kB 128.0 126.0 ... 121.0 123.0
    wind_speed         (gid, time) float64 140kB 3.6 3.5 3.4 3.4 ... 9.0 9.0 8.8
    relative_humidity  (gid, time) float64 140kB 87.0 86.47 85.93 ... 84.96 86.0

xarray.Dataset

• Dimensions:
  • gid: 2
  • time: 8760
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2022-01-01 ... 2022-12-31T23:00:00

    array(['2022-01-01T00:00:00.000000000', '2022-01-01T01:00:00.000000000',
           '2022-01-01T02:00:00.000000000', ..., '2022-12-31T21:00:00.000000000',
           '2022-12-31T22:00:00.000000000', '2022-12-31T23:00:00.000000000'],
          shape=(8760,), dtype='datetime64[ns]')

  • gid
    (gid)
    int64
    0 1

    array([0, 1])

• Data variables: (15)
  • Year
    (gid, time)
    float64
    2.019e+03 2.019e+03 ... 2.02e+03

    array([[2019., 2019., 2019., ..., 2024., 2024., 2024.],
           [2021., 2021., 2021., ..., 2020., 2020., 2020.]], shape=(2, 8760))

  • Month
    (gid, time)
    float64
    1.0 1.0 1.0 1.0 ... 12.0 12.0 12.0

    array([[ 1.,  1.,  1., ..., 12., 12., 12.],
           [ 1.,  1.,  1., ..., 12., 12., 12.]], shape=(2, 8760))

  • Day
    (gid, time)
    float64
    1.0 1.0 1.0 1.0 ... 31.0 31.0 31.0

    array([[ 1.,  1.,  1., ..., 31., 31., 31.],
           [ 1.,  1.,  1., ..., 31., 31., 31.]], shape=(2, 8760))

  • Hour
    (gid, time)
    float64
    0.0 1.0 2.0 3.0 ... 21.0 22.0 23.0

    array([[ 0.,  1.,  2., ..., 21., 22., 23.],
           [ 0.,  1.,  2., ..., 21., 22., 23.]], shape=(2, 8760))

  • Minute
    (gid, time)
    float64
    30.0 30.0 30.0 ... 30.0 30.0 30.0

    array([[30., 30., 30., ..., 30., 30., 30.],
           [30., 30., 30., ..., 30., 30., 30.]], shape=(2, 8760))

  • temp_air
    (gid, time)
    float64
    24.1 24.1 24.0 ... 24.8 24.7 24.7

    array([[24.1, 24.1, 24. , ..., 24.2, 24.2, 24.1],
           [24.6, 24.6, 24.6, ..., 24.8, 24.7, 24.7]], shape=(2, 8760))

  • dew_point
    (gid, time)
    float64
    21.8 21.7 21.5 ... 21.8 22.0 22.2

    array([[21.8, 21.7, 21.5, ..., 21.8, 21.9, 21.9],
           [22.3, 22.3, 22.3, ..., 21.8, 22. , 22.2]], shape=(2, 8760))

  • dhi
    (gid, time)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.]], shape=(2, 8760))

  • dni
    (gid, time)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.]], shape=(2, 8760))

  • ghi
    (gid, time)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.]], shape=(2, 8760))

  • albedo
    (gid, time)
    float64
    0.08 0.08 0.08 0.08 ... 0.0 0.0 0.0

    array([[0.08, 0.08, 0.08, ..., 0.08, 0.08, 0.08],
           [0.  , 0.  , 0.  , ..., 0.  , 0.  , 0.  ]], shape=(2, 8760))

  • pressure
    (gid, time)
    float64
    1.019e+03 1.019e+03 ... 1.02e+03

    array([[1019., 1019., 1019., ..., 1020., 1019., 1019.],
           [1020., 1020., 1020., ..., 1020., 1020., 1020.]], shape=(2, 8760))

  • wind_direction
    (gid, time)
    float64
    128.0 126.0 124.0 ... 121.0 123.0

    array([[128., 126., 124., ..., 128., 128., 128.],
           [124., 125., 125., ..., 119., 121., 123.]], shape=(2, 8760))

  • wind_speed
    (gid, time)
    float64
    3.6 3.5 3.4 3.4 ... 8.9 9.0 9.0 8.8

    array([[3.6, 3.5, 3.4, ..., 4. , 3.9, 3.8],
           [8.7, 8.5, 8.4, ..., 9. , 9. , 8.8]], shape=(2, 8760))

  • relative_humidity
    (gid, time)
    float64
    87.0 86.47 85.93 ... 84.96 86.0

    array([[87.00057525, 86.47066798, 85.93379377, ..., 86.48011647,
            87.00968036, 87.53332619],
           [87.04600782, 87.04600782, 87.04600782, ..., 83.42994899,
            84.96044656, 86.00216948]], shape=(2, 8760))

• Indexes: (2)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2022-01-01 00:00:00', '2022-01-01 01:00:00',
                   '2022-01-01 02:00:00', '2022-01-01 03:00:00',
                   '2022-01-01 04:00:00', '2022-01-01 05:00:00',
                   '2022-01-01 06:00:00', '2022-01-01 07:00:00',
                   '2022-01-01 08:00:00', '2022-01-01 09:00:00',
                   ...
                   '2022-12-31 14:00:00', '2022-12-31 15:00:00',
                   '2022-12-31 16:00:00', '2022-12-31 17:00:00',
                   '2022-12-31 18:00:00', '2022-12-31 19:00:00',
                   '2022-12-31 20:00:00', '2022-12-31 21:00:00',
                   '2022-12-31 22:00:00', '2022-12-31 23:00:00'],
                  dtype='datetime64[ns]', name='time', length=8760, freq='h'))

  • gid
    PandasIndex

    PandasIndex(Index([0, 1], dtype='int64', name='gid'))

• Attributes: (0)

Spot Check

We can plot the entire TMY air_temperature to check that our data has loaded correctly.

Explanation of steps

geo_weather is our weather xarray dataset. We can index into the first entry at the 0th index by using isel (index-select). This will grab the data from the first gid. Then we pick the air temperature attribute. This can be replaced with bracket notation so .temp_air becomes `[“temp_air”].

This selects a single array from the dataset that is labeled as “temp_air”. This array will be a dask array so the values will be stored out of memory, we would have to load it using .compute() to directly inspect it but when plotting with matplotlib it will load the array for us.

import matplotlib.pyplot as plt

plt.plot(geo_weather.isel(gid=0).temp_air)

[<matplotlib.lines.Line2D at 0x21023ef9ee0>]

Next Steps

Now we have data ready to use for geospatial calculations. This is shown in the other distributed script load_pvgis_distributed.ipynb. You can also see how to do this in Geospatial Templates.ipynb