1 Abstract

The Records API (application program interface) for Advanced Energy System Design (AESD) enables software that serves multidimensional record-oriented data to interoperate with software than uses such data. In the context of the Records API, multidimensional data records are simply tuples of real numbers, integers, and character strings, where each data value is tagged by a variable name, according to a pre-defined schema, and each record is assigned a unique integer identifier. Conceptually, these records are isomorphic to rows in a relational database, JSON objects, or key-value maps. Records servers might supply static datasets, sensor measurements that periodically update as new telemetry become available, or the results of simulations as the simulations generate new output. Records client software might display or analyze the data, but in the case of simulations the client request the creation of new ensembles for specified input parameters. It is also possible to chain records clients and servers together so that a client consuming data from a server might transform that data and serve it to additional clients.

This minimalist API avoids imposing burdensome metadata, structural, or implementation requirements on developers by relying on open-source technologies that are readily available for common programming languages. In particular, the API has been designed to place the least possible burden on services that provide data. This document defines the message format for the Records API, a transport mechanism for communicating the data, and the semantics for interpreting it. The message format is specified as Google Protocol Buffers (Google Developers 2017b) and the transport mechanism uses WebSockets (Internet Engineering Task Force 2017). We discuss three major use cases for serving and consuming records data: (i) static data, (ii) dynamically augmented data, (iii) on-demand simulations, (iv) with filters, and (v) with bookmarks. Separate implementations of the API exist in C++, Haskell, JavaScript, Python, and R.

2 Overview

Client-server communication in the Records API simply consists of clients sending Request messages to the server and servers asynchronously sending Response messages to the client. The request and response messages hold the specifics of the request or response and the responses are correlated with the requests; however, it is important to note that multiple responses may occur for a single request, as when record data are chunked into multiple response, or that an error response may be sent at any time. The nested messages within Request and Response may in turn contain nested fields and messages providing further details. The table below shows the correspondence between requests and responses, while the figure following that shows the containment relationships between message types.

Containment relationships between protocol buffer messages in the Records API.

Metadata messages describe “models”, which are just sources of data, and the variables they contain. Data record messages hold the data itself. Data records are simply tuples of real numbers, integers, and character strings, where each data value is tagged by a variable name, according to a pre-defined schema, and each record is assigned a unique integer identifier. Conceptually, these records are isomorphic to rows in a relational database, JSON objects, or key-value maps. For efficiency and compactness, RecordData may be provided in list format or tabular format, with the latter format obtained only when the contents of the table all have the same data type. The data records may be provided in toto or filtered using filter messages so that only certain fields or records are returned. The API contains a small embedded language for filtering via set and value operations. Sets of records may be bookmarked for sharing or later retrieval by (i) enumerating their unique record identifiers, (ii) defining a range of unique record identifiers, or (iii) specifying a filtering criterion.

Servers that perform computations or simulations can receive input parameters via a RequestWork message that contains those input parameters. After the server has completed its computations, it sends the results as RecordData messages.

In general the response to a request for data records comes in chunks numbered in sequence, where each chunk has an identifier, chunk_id, and the response specifies the identifier of the next chunk, next_chunk_id. Thus, the chunks form a linked list. The sending of additional chunks can be cancelled using a RequestCancel message. If the subscribe flag is set when making a request, the server will respond indefinitely with additional data as it becomes available, until the subscription is cancelled.

3 Use Cases

In this section we outline some standard use cases for the Records API. UML Sequence Diagrams (Fowler 2017) illustrate the flow of messages and the messages themselves are printed in the text format output by the Google protoc tool (Google Developers 2017a).

3.1 Static Data

The retrieval of static data records forms the simplest use case for the Records API. A user chooses a particular data source (a “model” in the parlance of the Records API) and then the data are retrieved and displayed. The visualization client software communicates with a Records server, which in turn accesses the static data. The figure below illustrates the process.

Visualizing data from a static source using the Records API.

A Request without model_id specified requests the server to list all models:

version: 4
id: 1
models_metadata {
}

The Response from the server provides metadata for all of the models:

version: 4
id: 1
models {
  models {
    model_id: "example-model-1"
    model_name: "Example Model #1"
    model_uri: "http://esda.nrel.gov/examples/model-1"
    variables {
      var_id: 0
      var_name: "Example Real Variable"
      type: REAL
    }
    variables {
      var_id: 1
      var_name: "Example Integer Variable"
      type: INTEGER
    }
    variables {
      var_id: 2
      var_name: "Example String Variable"
      type: STRING
    }
  models {
    model_id: "example-model-2"
    model_name: "Example Model #2"
    model_uri: "http://esda.nrel.gov/examples/model-2"
    variables {
      var_id: 0
      var_name: "POSIX Epoch"
      type: INTEGER
    }
    variables {
      var_id: 1
      var_name: "Measurement"
      type: REAL
    }
  }
  models {
    model_id: "example-simulation-3"
    model_name: "Example Simulation #3"
    model_uri: "http://esda.nrel.gov/examples/simulation-3"
    variables {
      var_id: 0
      var_name: "Input"
      type: REAL
    }
    variables {
      var_id: 1
      var_name: "Time"
      type: REAL
    }
    variables {
      var_id: 2
      var_name: "Value"
      type: REAL
    }
    inputs {
      var_id: 0
      interval {
        first_value: 0
        second_value: 100
      }
    }
  }
}

Note that the response above is tagged with the same id as the request: this allows the client to correlate responses with the particular requests it makes. Next the user might request three records from the first model:

version: 4
id: 2
records_data {
  model_id: "example-model-1"
  max_records: 3
}

The record data might be returned as two chunks, where the first chunk is

version: 4
id: 2
chunk_id: 1
next_chunk_id: 2
data {
  list {
    records {
      record_id: 10
      variables {
        var_id: 0
        value: 10.5
      }
      variables {
        var_id: 1
        value: -5
      }
      variables {
        var_id: 2
        value: "first"
      }
    }
    records {
      record_id: 20
      variables {
        var_id: 0
        value: 99.2
      }
      variables {
        var_id: 1
        value: 108
      }
      variables {
        var_id: 2
        value: "second"
      }
    }
  }
}

and the last chunk is:

version: 4
id: 2
chunk_id: 2
next_chunk_id: 0
data {
  list {
    records {
      record_id: 30
      variables {
        var_id: 0
        value: -15.7
      }
      variables {
        var_id: 1
        value: 30
      }
      variables {
        var_id: 2
        value: "third"
      }
    }
  }
}

3.2 Dynamic Data

As shown in the following figure retrieving data from a dynamic source proceeds quite similarly to retrieving data from a static source. The only essential difference is that the server repeatedly sends additional responses containing new data, until a request to cancel is sent.

Visualizing data from a dynamic source using the Records API.

When requesting dynamic data, it is advisable to set the subscribe flag in the request for data:

version: 4
id: 2
subscribe: true
records_data {
  model_id: "example-model-2"
}

The RequestCancel message is the cancel field Request and must include the id of the request to be cancelled:

version: 4
cancel {
  id: 2
}

3.3 Simulations

The model Example Simulation #3 in the Static Data use case is a simulation model, as evidenced by the presence of the inputs field in its metadata. The following figure shows a typical interaction with a simulation-based model via the Records API.

Steering and visualizing simulation results using the Records API.

The RequestWork message, which is contained in the work field of a Request, specifies the input for a simulation to be run:

version: 4
id: 3
work {
  model_id: "example-simulation-3"
  inputs {
    var_id: 0
    value: 50
  }
}

The response to this message will be data for the result of the simulation.

3.4 Bookmarks

Once data from a model is loaded, it may be bookmarked. One simply supplies a description of the data to be bookmarked. Bookmarks can be listed and loaded, as shown in the following figure.

Creating and retrieving a bookmark and its associated data.

To create a bookmark for a specific list of records, simply supply their record identifiers as part of a BookmarkMeta message in the save_bookmark field of Request:

version: 4
id: 4
save_bookmark {
  model_id: "example-model-1"
  new_bookmark {
    bookmark_name: "Sample Bookmark"
    set {
      record_ids: 10
      record_ids: 30
    }
  }
}

The response will be the same bookmark, but with the bookmark_id field added:

version: 4
id: 4
bookmarks {
  bookmark_metas {
    bookmark_id: "bookmark-1"
    bookmark_name: "Sample Bookmark"
    set {
      record_ids: 10
      record_ids: 30
    }
  }
}

The user or another user can retrieve the records corresponding to the bookmark:

version: 4
id: 5
records_data {
  model_id: "example-model-1"
  bookmark_id: "bookmark-1"
}

This will return precisely the bookmarked records:

version: 4
id: 5
data {
  list {
    records {
      record_id: 10
      variables {
        var_id: 0
        value: 10.5
      }
      variables {
        var_id: 1
        value: -5
      }
      variables {
        var_id: 2
        value: "first"
      }
    }
    records {
      record_id: 30
      variables {
        var_id: 0
        value: -15.7
      }
      variables {
        var_id: 1
        value: 30
      }
      variables {
        var_id: 2
        value: "third"
      }
    }
  }
}

3.5 Filtering

Filtering records can be used to select particular records for retrieval, via the RequestRecordsData message, or in defining bookmarks, via the BookmarkMeta message. Filtering of records is accomplished through expressions, FilterExpression, combining values for variables, DomainMeta, and the set operators not, union, and intersection, encoded in the messages FilterNot, FilterUnion, and FitlerIntersection, respectively. For example, the expression x ≤ 20 would be expressed as the following FilterExpression

filter_domain {
  interval {
    var_id: 0
    last_value: 20
  }
}

provided that x has var_id = 0. The expression (10 ≤ x ≤ 20)∪(y ∉ {4, 7}) would be expressed as

filter_union {
  filter_expressions {
    filter_domain {
      var_id: 0
      first_value: 10
      last_value: 20
    }
    filter_not {
      filter_expression {
        filter_domain {
          var_id: 1
          set {
            elements: 4
            elements: 7
          }
        
      }
    }
  }
}

provided that x has var_id = 0 and y has var_id = 1.

4 Records API, Version 4

The AESD Records API consists of Google Protobuf 3 (Google Developers 2017b) messages used to request and provid data and metadata for record-oriented information. This section contains the complete specification for version 4 of the Records API. Clients send Request messages and servers send Response messages, typically transported via WebSockets (Internet Engineering Task Force 2017).

4.1 Message Groups

The message types in the Records API are organized into thematic groups below.

4.1.1 Requests and Responses

Request messages are sent from client to server and Response messages are sent from server to client. Request messages contain a specific type of request and response messages contain a corresponding specific type of response.

• Request
• RequestModelsMeta
• RequestRecordsData
• RequestWork
• RequestBoomarkMeta
• RequestSaveBookmark
• RequestCancel
• Response

4.1.2 Metadata

Metadata messages describe data sources (“models”) and variables.

• ModelMeta
• ModelMetaList
• DomainMeta
• VarMeta
• VariableType
• VarSet
• VarInterval

4.1.3 Data Records

Data are represented as either lists of records or tables of them.

• Record
• VarValue
• Value
• RecordData
• RecordList
• RecordTable

4.1.4 Filtering

Records can be filtered by logical operations on conditions for values of variables in the records.

• FilterExpression
• FilterNot
• FilterIntersection
• FilterUnion
• DomainMeta

4.1.5 Bookmarks

Bookmarks record particular sets or records or conditions for record data.

• BookmarkMeta
• BookmarkMetaList
• BookmarkIntervalContent
• BookmarkSetContent

4.1.6 Miscellaneous

The following messages wrap data types for the content of records.

• DoubleList
• IntegerList
• StringList
• OptionalInt32
• OptionalUInt32
• OptionalString

4.2 General Conventions

All fields are technically optional in ProtoBuf 3, but some fields may be required in each message type in order for the message to be semantically valid. In the following specifications for the messages, fields are annotated as semantically required or semantically optional. Also, the specification notes when field in the protobuf oneof construct are required or mutually exclusive.

Furthermore, one cannot determine whether an optional value has been set or not if it is just a value, as opposed to a message. That is not true for fields that are messages, where the absence of the field truly indicates that the value is absent, not just a default or unset value. The message OptionalString, for example, is used in this API to indicate whether a character string value is truly present. Thus RequestModelsMeta has a model_id field that indicates whether the request is for all models, when the field has not been set, or for a specific one, when the field has been set.

Throughout this specification, the following types are used for identifiers: * var_id is int32 * model_id is string * record_id is int64

This specification conforms to Protocol Buffers version 3.

4.3 Messages

4.3.1 BookmarkIntervalContent

A range of record identifiers can specify the content of a bookmark. Bookmark interval content provides a convenient means to bookmark a contiguous selection of records in a model.

Both fields in this message are optional:

• If neither field is present, the bookmark interval designates all records in the model.
• If only first_recordis present, the bookmark interval designates all records starting from that record identifier.
• If only last_record is present, the bookmark interval designates all records ending at that record identifier. For a dynamic model, such a bookmark interval includes all “future” records.
• If both fields are present, the bookmark interval designates all records between the two identifiers, inclusively.

4.3.2 BookmarkMeta

A bookmark is metadata defining a subset of records in a model.

There are three alternatives to specifying a bookmark:

1. Interval content specifies a range of records in the bookmark.
2. Set content specifies a list of records in the bookmark.
3. A filter expression defines a set of logical conditions for determining whether a record is in the bookmark.

Exactly one of interval, set, or filter must be specified in this message.

4.3.3 BookmarkMetaList

Bookmarks may be grouped into lists (sets).

4.3.4 BookmarkSetContent

A list (set) of record identifiers can specify the contents of a bookmark. Bookmark-set content provides a convenient means to bookmark a specific selection of non-continuous records in a model.

4.3.5 DomainMeta

The domain (set of valid values) for a variable.

There are two alternatives to specifying a domain:

1. An interval specifies a range of values in the domain.
2. A set specifies a list of values in the domain.

Exactly one of interval or set must be specified in the message.

4.3.6 DoubleList

A list of real numbers.

4.3.7 FilterExpression

A filtering expression is a composition of logical conditions on a record. It can be used to filter records. There are four alternatives to specifying a filter expression:

1. The logical negation of another filtering expression.
2. The set union of multiple filtering expressions.
3. The set intersection of multiple filtering expressions.
4. Particular values of variables in a record.

Exactly one of filter_not, filter_union, filter_intersection, or filter_domain must be specified in this message.

4.3.8 FilterIntersection

Set intersection of filtering expressions. A record satisfies this expression if it satisfies all filter_expressions.

4.3.9 FilterNot

Logically negate a filtering expression. A record satisfies this expression if it does not satisfy filter_expression.

4.3.10 FilterUnion

Set union of filtering expressions. A record satisfies this expression if it satisfies any of filter_expressions.

4.3.11 IntegerList

A list of integers.

4.3.12 ModelMeta

Metadata for a model.

4.3.13 ModelMetaList

A list of metadata for models.

4.3.14 OptionalInt32

Wrapper for an optional signed integer.

4.3.15 OptionalString

Wrapper for an optional string.

4.3.16 OptionalUInt32

Wrapper for an optional unsigned integer.

4.3.17 Record

A record is a list of variables and their associated values.

4.3.18 RecordData

A collection of records.

There are two alternatives to specifying record data:

1. A list specifies a heterogeneously typed list.
2. A table specifies a homogeneously typed table.

Exactly one of list or table must be present in the message.

4.3.19 RecordList

A list of records. The list is heterogeneous in the sense that each variable may have a different type.

4.3.20 RecordTable

A homogeneously typed table of records, where each variable has each type, with a row for each record and a column for each variable.

This message represents the following table:

The underlying list is a single array, addressable using the following row-major index formula list[row][var] = array[var + NY * row] where NX = length of rec_ids and NY = length of var_ids.

Exacly one of reals, integers, or strings must be specified in the message.

4.3.21 Request

A request. There are six types of requests:

*Exactly one of models_metadata, records_data, bookmark_meta, save_bookmark, cancel, or work must be specified in the message.

4.3.22 RequestBookmarkMeta

A request for one or more bookmarks for a model.

The response to this request is BookmarkMetaList

4.3.23 RequestCancel

Cancel a previous request.

4.3.24 RequestModelsMeta

A request for metadata about model(s).

The response to this request is ModelMetaList.

4.3.25 RequestRecordsData

Request record data for a model.

There are three alternatives to requesting record data.

1. Request all records.
2. Request records in a bookmark.
3. Filter records according to a criterion.

The response to this request is RecordData.

No more than on of bookmark_id or expression may be present in the message.

4.3.26 RequestSaveBookmark

A request to create or update a bookmark.

The response to this request is BookmarkMetaList.

4.3.27 RequestWork

Request that the server compute new records based on input values.

The response to this request is RecordData.

4.3.28 Response

A response to a request.

Note that a server may send multiple responses to a single request, expressed as a linked list of chunks. It is strongly recommended that servers chunk by record_id so that each record is kept intact. A chunk may be empty.

4.3.29 StringList

A list of character strings.

4.3.30 Value

Value that may be a real number, an integer, or a character string

Exactly one of real_value, integer_value, or string_value must be specified in this message.

4.3.31 VarInterval

A range of values of a variable.

Both fields in this message are optional:

• If neither field is present, the interval designates all values in the domain.
• If only first_valueis present, the interval designates all values starting from that value.
• If only last_value is present, the bookmark interval designates all values ending at that value.
• If both fields are present, the interval designates all values between the two values, inclusive.

4.3.32 VarMeta

Metadata for a variable.

4.3.33 VarSet

A set of values for a variable.

4.3.34 VarValue

The value of a variable.

4.3.35 VariableType

The data type for a value.

4.4 Scalar Value Types

5 Implementations

This section provides an overview of the variety of libraries and applications implementing the Records API (see the table below). In particular, pre-built applications are available for serving text-based data sources, database queries, and sensor data feeds. Application Container Images (ACIs) (CoreOS 2017a) of each have been packed for use with the rkt container engine (CoreOS 2017b).

5.1 Haskell Client and Server Library and Applications

Both client and server applications in Haskell are available for the Records API. Full documentation resides at <https://github.com/NREL/AESD/lib/haskell>.

5.1.1 Client Library

The client library described below provides the basic functions for interacting with any Records API server.

5.1.1.1 Types

data State

State information for a client.

5.1.1.2 Entry Point

clientMain

Run a client.

close

Close a client.

5.1.1.3 Server Requests

fetchModels

Fetch model metadata.

fetchRecords

Fetch records from the server.

fetchBookmarks

Fetch bookmark(s).

storeBookmark

Save a bookmark.

5.1.2 Server Library

The server library provides two options for implementing a Records APIserver. The AESD.Records.Server module provides a main entry point serverMain, a type class ModelManager, and a monad ServiceM that implement a skeletal server which handles all of the WebSocket communication and Protocol Buffer serialization; an implementer need only create an instance of ModelManager. Furthermore, the AESD.Records.Server.Manager module provides such an instance InMemoryManager of the type class ModelManger to handle in-memory caching of data and on-disk persistence of bookmarks; here, an implementer just calls the function makeInMemoryManager and provides several functions that retrieve content:

makeInMemoryManager

Construct an in-memory model manager.

5.1.3 Server Backends

As previously mentioned, prebuilt servers have been implemented for standard types of data sources.

5.1.3.1 Tab-Separate-Value Files

Serving tab-separated-value (TSV) files is a simple as placing the TSV files in a directory and starting a server at the command line, with the arguments specified in the table below:

aesd-file-server <host> <port> <directory> <persistence> <chunkSize>

5.1.3.2 Database Queries

The Records API servers have been implemented for the most common database backends. Each server takes a single command-line argument specifying a YAML (Oren Ben-Kiki, Clark Evans, Ingy döt Net 2017) configuration file with the parametes in the table below.

5.1.3.3 Haystack Sensor Measurements and the “Internet of Things”

Furthermore, a server for Project Haystack (Project Haystack 2017) data feeds, typically sensor measurements from devices in the “internet of things”, has been implemented. The server takes a command-line arguments specified in the table below.

aesd-haystack-server <configuration> <host> <port> <startTime> <persistence> <chunkSize>

The parameters in the YAML configuration file like the one below and are described in the following table:

siteAccess     :
  server       : xv11skys01.nrel.gov
  root         : /api/nrel_wt_V7
  authorization: ["my username","my password"]
  secure       : false
  timeZone     : [-360, true, Denver]
siteIdentifier : NWTCv4
siteURI        : http://aesd.nrel.gov/records/v4/nwtc/
siteName       : NREL NWTC
siteDescription: Sensors from NREL National Wind Technology Center
siteTags       :
               ! 'DC.source'     : https://xv11skys01.nrel.gov/proj/nrel_wt_v7
               ! 'DC.creator'    : Brian W Bush <brian.bush@nrel.gov>
               ! 'DC.description': NREL NWTC sensors
siteMeters     :
               - 1dca834e-c6af46d6 NWTC Alstom Turbine Electricity Meter Turbine-Alstom kW Demand Forward
               - 1dca834e-69a3e57e NWTC Alstom Turbine Electricity Meter Turbine-Alstom kW Demand Reverse
               - 1dca834e-f56e11f0 NWTC Alstom Turbine Electricity Meter Turbine-Alstom kWh Delivered Forward

5.2 C++ Server and Client

Both client and server applications have been implemented in C++ for the Records API. See <https://github.nrel.gov/d-star/cpp-records> for details. There are GUI and command-line applications for serving comma-separated-value files and a GUI application for browsing Records API data sources.

5.3 JavaScript Client Library and Web-Based Browser

The client library for JavaScript relies on a few simple functions to interact with a Records API server. Full documentation for the JavaScript client library is available at <http://github.com/NREL/AESD/lib/javascript>. The figure below shows the user interface of the general purpose Records API browser using this JavaScript library.

User interface for the Records API browser.

5.3.1 Connect to a server

connect(wsURL)

Here wsURL is simply the URL of the server (e.g., ws://10.40.9.214:503761). This returns a connection object.

5.3.2 Disconnect from a server

disconnect(connection)

Here connection is the connection object returned by the connect function.

5.3.3 Retrieve list of data models

requestModelsMetadata(connection, modelId, notify, notifyError)

Here connection is the connection object returned by the connect function and modelId is either the string identifying the model or null if metadata for all models is requested. After all of the model metadata have been retrieved, the notify function is called with the list of model metadata objects as its argument; if an error occurs, notifyError is called with the error message as its argument. The function requestModelsMetadata returns a result object that contains a field done indicating whether all model metadata have been retrieved and a field models listing the model metadata retrieved so far.

5.3.4 Retrieve data records

requestRecordsData(connection, modelId, maxRecords, variableIds, bookmarkId, notify, notifyError)

Here connection is the connection object returned by the connect function and modelId is the string identifying the model. After all of the data records have been retrieved, the notify function is called with the list of data records as its argument; if an error occurs, notifyError is called with the error message as its argument. The maxRecords argument specifies the maximum number of records to retrieve, variableIds may list the variables of interest, and bookmarkId restricts the results to bookmarked records. The function requestRecordsData returns a result object that contains a field done indicating whether all data records have been retrieved and a field data listing the data records retrieved so far.

5.3.5 Retrieve list of bookmarks

requestBookmarkMeta(connection, modelId, bookmarkId, notify, notifyError)

Here connection is the connection object returned by the connect function, modelId is the string identifying the model, and bookmarkId is either the string identifying the bookmark or null if metadata for all bookmarks is requested. After all of the bookmark metadata have been retrieved, the notify function is called with the list of bookmark metadata as its argument; if an error occurs, notifyError is called with the error message as its argument. The function requestBookmarkMeta returns a result object that contains a field done indicating whether all bookmark metadata have been retrieved and a field bookmarks listing the bookmark metadata retrieved so far.

5.3.6 Create/update a bookmark

requestSaveBookmark(connection, modelId, name, filter, notify, notifyError)

Here connection is the connection object returned by the connect function, modelId is the string identifying the model, and bookmarkId is null for a new bookmark or the identifier for a bookmark being updated. The name field names the bookmark and the filter object describing the filtering operation for the bookmark. After the bookmark metadata has been created or updated, the notify function is called with the list of bookmark metadata as its argument; if an error occurs, then notifyError is called with the error message as its argument. The function requestSaveBookmark returns a result object that contains a field done indicating whether all bookmark metadata have been retrieved and a field bookmarks listing the bookmark metadata retrieved so far.

5.4 Python Client Library

Full documentation for the Python client library is available at <http://github.com/NREL/AESD/lib/python>.

5.4.1 Client API

new_server(self, server_url)

Change server url to which websocket will connnect
Parameters
----------
server_url : 'string'
    server url
 
Returns
---------
self.url : 'string'
    server url

send(self, request)

Closes event_loop
Parameters
----------
request : 'proto.request'
    proto request message
timeout : 'int'
    timeout in seconds for connection
 
Returns
---------
response : 'list'
    List of responses from the server, each response is a proto message

get_model_info(self, model_id)

Sends request of model metadata and extracts response
Parameters
----------
model_id : 'string'
    Id of model for which to requst models_metadata
    if None requests all models
 
Returns
-------
model_info : 'list'|'dict'
    List of model's metadata dictionaries for each model in models or
    dictionary for model_id

get_data(self, model_id, max_records=1000, variable_ids=None, bookmark_id=None)

Sends request of model metadata and extracts response
Parameters
----------
model_id : 'string'
    Id of model for which to requst records_data
max_records : 'int'
    Number or records being request (0 will return all records)
variable_ids : 'list'
    List of variable ids (ints) to be requested
    Will be returned in same order as request
    Default=None, all variables will be returned (order?)
bookmark_id : 'int'
    Request records_data based on bookmark id
 
Returns
-------
data : 'pd.DataFrame'
    Concatenated data from each response message
    Variable ids replaced with names from model_info

do_work(self, model_id, inputs)

Sends request of model metadata and extracts response
Parameters
----------
model_id : 'string'
    Id of model for which to requst records_data
inputs : 'dict'
    Dictionary of {var_id: value} pairs
 
Returns
-------
data : 'pd.DataFrame'
    Concatenated data from each response message
    Variable ids replaced with names from model_info

get_bookmark_info(self, model_id, bookmark_id)

Sends request of model metadata and extracts response
Parameters
----------
model_id : 'string'
    Id of model for which to requst bookmark_meta
bookmark_id : 'string'
    Id of bookmark for which to request models_metadata
    if None request all bookmarks
 
Returns
-------
model_info : 'list'|'dict'
    List of model's metadata dictionaries for each model in models or
    dictionary for model_id

save_bookmark(self, model_id, name, content)

Sends request to save new bookmark
Parameters
----------
model_id : 'string'
    Id of model for which to requst bookmark_meta
name : 'string'
    Name for new bookmark
content : 'list'|'tuple'
    Contents of bookmark
    list is a bookmark set
    tuple is a bookmark interval
 
Returns
-------
model_info : 'list'|'dict'
    List of model's metadata dictionaries for each model in models or
    dictionary for model_id

5.4.2 Example

The figure below shows example usage of the Python Records API client.

Example of a Python session using the Records API

6 Appendix

6.1 Protocol Buffers for Records API Version 4

syntax = "proto3";
package AesdRecords;

option optimize_for = LITE_RUNTIME;

message OptionalInt32 {
    int32 value = 1; /// [semantically required]
}

message OptionalUInt32 {
    uint32 value = 1; /// [semantically required]
}

message OptionalString {
    string value = 1; /// [semantically required]
}

message Value {
    oneof    value              /// [semantically required]
    {
      double real_value    = 1; 
      int64  integer_value = 2; 
      string string_value  = 3; 
    }
}

message DoubleList {
    repeated double values = 1; /// [semantically required]
}

message IntegerList {
    repeated sint64 values = 1; /// [semantically required]
}

message StringList {
    repeated string values = 1; /// [semantically required]
}

message BookmarkIntervalContent {
    int64 first_record = 1; /// [semantically optional]
    int64 last_record  = 2; /// [semantically optional]
}

message BookmarkSetContent {
    repeated int64 record_ids = 1; /// [semantically optional]
}

message BookmarkMeta {
    string                      bookmark_id   = 1; /// [semantically optional]
    string                      bookmark_name = 2; /// [semantically required]
    oneof                       content            /// [semantically required]
    {
        BookmarkIntervalContent interval      = 3; 
        BookmarkSetContent      set           = 4; 
        FilterExpression        filter        = 5; 
    }
}

message BookmarkMetaList {
    repeated BookmarkMeta bookmark_metas = 1; /// [semantically optional]
}

message RequestBookmarkMeta {
    string         model_id    = 1; /// [semantically required]
    OptionalString bookmark_id = 2; /// [semantically optional]
}

message RequestSaveBookmark {
    string       model_id     = 1; /// [semantically required]
    BookmarkMeta new_bookmark = 2; /// [semantically optional]
}

message FilterExpression {
  oneof                expression               /// [semantically required]
  {
    FilterNot          filter_not          = 1; 
    FilterUnion        filter_union        = 2; 
    FilterIntersection filter_intersection = 3; 
    DomainMeta         filter_domain       = 4; 
  }
}

message FilterNot {
  FilterExpression filter_expression = 1; /// [semantically required]
}

message FilterUnion {
  repeated FilterExpression filter_expressions = 1; /// [semantically required]
}

message FilterIntersection {
  repeated FilterExpression filter_expressions = 1; /// [semantically required]
}

enum VariableType
{
    REAL            = 0; 
    INTEGER         = 1; 
    STRING          = 2; 
}

message VarMeta {
    int32           var_id   = 1; /// [semantically required]
    string          var_name = 2; /// [semantically required]
    string          units    = 3; /// [semantically optional]
    repeated sint32 si       = 4; /// [semantically optional]
    double          scale    = 5; /// [semantically optional]
    VariableType    type     = 6; /// [semantically optional]
}

message ModelMeta {
    string              model_id   = 1; /// [semantically required]
    string              model_name = 2; /// [semantically required]
    string              model_uri  = 3; /// [semantically required]
    repeated VarMeta    variables  = 4; /// [semantically required]
    repeated DomainMeta inputs     = 5; /// [semantically optional]
}

message ModelMetaList {
    repeated ModelMeta models = 1; /// [semantically optional]
}

message RequestModelsMeta {
    OptionalString model_id = 1; /// [semantically optional]
}

message VarInterval {
    Value first_value = 1; /// [semantically optional]
    Value last_value  = 2; /// [semantically optional]
}

message VarSet {
    repeated Value elements = 1; /// [semantically optional]
}

message DomainMeta {
    int32           var_id   = 1; /// [semantically required]
    oneof           domain        /// [semantically required]
    {
        VarInterval interval = 2; 
        VarSet      set      = 3; 
    }
}

message RequestWork {
    string model_id          = 1; /// [semantically required]
    repeated VarValue inputs = 2; /// [semantically optional]
}

message VarValue {
    int32 var_id = 1; /// [semantically required]
    Value value  = 2; /// [semantically required]
}

message Record {
    int64    record_id          = 1; /// [semantically required]
    repeated VarValue variables = 2; /// [semantically optional]
}

message RecordList {
    repeated Record records = 1; /// [semantically optional]
}

message RecordTable {
    repeated int32  var_ids    = 1; /// [semantically required]
    repeated int64  rec_ids    = 2; /// [semantically required]
    oneof           list            /// [semantically required]
    {
        DoubleList  reals      = 3; 
        IntegerList integers   = 4; 
        StringList  strings    = 5; 
    }
}

message RecordData {
    oneof           style      /// [semantically required]
    {
        RecordList  list  = 1; 
        RecordTable table = 2; 
    }
}

message RequestRecordsData {
    string               model_id    = 1; /// [semantically required]
    uint64               max_records = 2; /// [semantically optional]
    repeated int32       var_ids     = 3; /// [semantically optional]
    oneof                filter           /// [semantically optional]
    {
        string           bookmark_id = 4; /// [semantically optional]
        FilterExpression expression  = 5; /// [semantically optional]
    }
}

message Response {
    uint32               version       = 1; /// [semantically required]
    OptionalUInt32       id            = 2; /// [semantically optional]
    int32                chunk_id      = 3; /// [semantically optional, but recommended]
    int32                next_chunk_id = 4; /// [semantically optional]
    oneof                type               /// [semantically optional]
    {
        string           error         = 5; 
        ModelMetaList    models        = 6; 
        RecordData       data          = 7; 
        BookmarkMetaList bookmarks     = 8; 
    }
}

message RequestCancel {
    OptionalUInt32 id = 1; /// [semantically required]
}

message Request {
    uint32                  version         = 1; /// [semantically required]
    OptionalUInt32          id              = 2; /// [semantically optional, but recommended]
    bool                    subscribe       = 3; /// [semantically optional]
    oneof                   type                 /// [semantically required]
    {
        RequestModelsMeta   models_metadata = 4; 
        RequestRecordsData  records_data    = 5; 
        RequestBookmarkMeta bookmark_meta   = 6; 
        RequestSaveBookmark save_bookmark   = 7; 
        RequestCancel       cancel          = 8; 
        RequestWork         work            = 9; 
    }
}

7 Glossary of Acronyms

8 References