Model Validation

Model Validation#

The purpose of this tutorial is to learn about model validation and how BuildingMOTIF can give useful feedback for fixing a model that has failed validation.

Note

This tutorial has the following learning objectives:

validating the model against an ontology to ensure that the model is a valid Brick model
validating the model against a manifest, which contains the metadata requirements for a specific model
validating the model against a use case for a specific application

Validating a model is the process of ensuring that the model is both correct (uses the ontologies correctly) and semantically sufficient (it contains sufficient metadata to execute the desired applications or enable the desired use cases). Validation is always done with respect to sets of Shapes using the Shapes Constraint Language (SHACL)[1].

Setup#

We create an in-memory BuildingMOTIF instance, load the model from the previous tutorial, and load some libraries to create the manifest with. The constraints.ttl library we load is a special library with some custom constraints defined that are helpful for writing manifests.

from rdflib import Namespace
from buildingmotif import BuildingMOTIF
from buildingmotif.dataclasses import Model, Library
from buildingmotif.namespaces import BRICK # import this to make writing URIs easier

# in-memory instance
bm = BuildingMOTIF("sqlite://")

# create the namespace for the building
BLDG = Namespace('urn:bldg/')

# create the building model
model = Model.create(BLDG, description="This is a test model for a simple building")

# load tutorial 1 model
model.graph.parse("tutorial1_model.ttl", format="ttl")

# load libraries included with the python package
constraints = Library.load(ontology_graph="../../buildingmotif/libraries/constraints/constraints.ttl")

# load libraries excluded from the python package (available from the repository)
brick = Library.load(ontology_graph="../../libraries/brick/Brick-subset.ttl")
g36 = Library.load(directory="../../libraries/ashrae/guideline36")

/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/pyshacl/extras/__init__.py:46: Warning: Extra "js" is not satisfied because requirement pyduktape2 is not installed.
  warn(Warning(f"Extra \"{extra_name}\" is not satisfied because requirement {req} is not installed."))

---------------------------------------------------------------------------
NoResultFound                             Traceback (most recent call last)
Cell In[1], line 23
     21 # load libraries excluded from the python package (available from the repository)
     22 brick = Library.load(ontology_graph="../../libraries/brick/Brick-subset.ttl")
---> 23 g36 = Library.load(directory="../../libraries/ashrae/guideline36")

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/dataclasses/library.py:208, in Library.load(cls, db_id, ontology_graph, directory, name, overwrite, infer_templates, run_shacl_inference)
    206     if not src.exists():
    207         raise Exception(f"Directory {src} does not exist")
--> 208     return cls._load_from_directory(
    209         src,
    210         overwrite=overwrite,
    211         infer_templates=infer_templates,
    212         run_shacl_inference=run_shacl_inference,
    213     )
    214 elif name is not None:
    215     bm = get_building_motif()

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/dataclasses/library.py:417, in Library._load_from_directory(cls, directory, overwrite, infer_templates, run_shacl_inference)
    415     lib._read_yml_file(file, template_id_lookup, dependency_cache)
    416 # now that we have all the templates, we can populate the dependencies
--> 417 lib._resolve_template_dependencies(template_id_lookup, dependency_cache)
    418 # load shape collections from all ontology files in the directory
    419 lib._load_shapes_from_directory(directory)

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/dataclasses/library.py:520, in Library._resolve_template_dependencies(self, template_id_lookup, dependency_cache)
    518         continue
    519     for dep in dependency_cache[template.id]:
--> 520         self._resolve_dependency(template, dep, template_id_lookup)
    521 # check that all dependencies are valid (use parameters that exist, etc)
    522 for template in self.get_templates():

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/dataclasses/library.py:470, in Library._resolve_dependency(self, template, dep, template_id_lookup)
    468 # if dep is a _template_dependency, turn it into a template
    469 if isinstance(dep, _template_dependency):
--> 470     dependee = dep.to_template(template_id_lookup)
    471     template.add_dependency(dependee, dep.bindings)
    472     return

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/dataclasses/library.py:87, in _template_dependency.to_template(self, id_lookup)
     84     return Template.load(id_lookup[self.template_name])
     85 # if not in the local cache, then search the database for the template
     86 # within the given library
---> 87 library = Library.load(name=self.library)
     88 return library.get_template_by_name(self.template_name)

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/dataclasses/library.py:216, in Library.load(cls, db_id, ontology_graph, directory, name, overwrite, infer_templates, run_shacl_inference)
    214 elif name is not None:
    215     bm = get_building_motif()
--> 216     db_library = bm.table_connection.get_db_library_by_name(name)
    217     return cls(_id=db_library.id, _name=db_library.name, _bm=bm)
    218 else:

File ~/work/BuildingMOTIF/BuildingMOTIF/buildingmotif/database/table_connection.py:230, in TableConnection.get_db_library_by_name(self, name)
    222 def get_db_library_by_name(self, name: str) -> DBLibrary:
    223     """Get database library by name.
    224 
    225     :param name: name of DBLibrary
   (...)
    228     :rtype: DBLibrary
    229     """
--> 230     return self.bm.session.query(DBLibrary).filter(DBLibrary.name == name).one()

File /opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/sqlalchemy/orm/query.py:2870, in Query.one(self)
   2852 def one(self):
   2853     """Return exactly one result or raise an exception.
   2854 
   2855     Raises ``sqlalchemy.orm.exc.NoResultFound`` if the query selects
   (...)
   2868 
   2869     """
-> 2870     return self._iter().one()

File /opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/sqlalchemy/engine/result.py:1522, in ScalarResult.one(self)
   1514 def one(self):
   1515     """Return exactly one object or raise an exception.
   1516 
   1517     Equivalent to :meth:`_engine.Result.one` except that
   (...)
   1520 
   1521     """
-> 1522     return self._only_one_row(
   1523         raise_for_second_row=True, raise_for_none=True, scalar=False
   1524     )

File /opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/sqlalchemy/engine/result.py:562, in ResultInternal._only_one_row(self, raise_for_second_row, raise_for_none, scalar)
    560 if row is None:
    561     if raise_for_none:
--> 562         raise exc.NoResultFound(
    563             "No row was found when one was required"
    564         )
    565     else:
    566         return None

NoResultFound: No row was found when one was required

Model Validation - Ontology#

BuildingMOTIF organizes Shapes into Shape Collections. The shape collection associated with a library (if there is one) can be retrieved with the get_shape_collection method. Below, we use Brick’s shape collection to ensure that the model is using Brick correctly:

# pass a list of shape collections to .validate()
validation_result = model.validate([brick.get_shape_collection()])
print(f"Model is valid? {validation_result.valid}")

Success! The model is valid according to the Brick ontology.

Model Validation - Manifest#

Writing a Manifest#

For now, we will write a manifest file directly; in the future, BuildingMOTIF will contain features that make manifests easier to write. Here is the header of a manifest file. This should also suffice for most of your own manifests.

@prefix brick: <https://brickschema.org/schema/Brick#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix constraint: <https://nrel.gov/BuildingMOTIF/constraints#> .
@prefix : <urn:my_site_constraints/> .

: a owl:Ontology ;
    owl:imports <https://brickschema.org/schema/1.3/Brick>,
                <https://nrel.gov/BuildingMOTIF/constraints>,
                <urn:ashrae/g36> .

We will now add a constraint stating that the model should contain exactly 1 Brick AHU.

:ahu-count a sh:NodeShape ;
    sh:message "need 1 AHU" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:AHU .

This basic structure can be changed to require different numbers of different Brick classes. Just don’t forget to change the name of the shape (:ahu-count, above) when you copy-paste!

Attention

As an exercise, try writing shapes that require the model to contain the following.

(1) Brick Supply_Fan
(1) Brick Damper
(1) Brick Cooling_Coil
(1) Brick Heating_Coil

Hint

:fan-count a sh:NodeShape ;
    sh:message "need 1 supply fan" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Supply_Fan .

:damper-count a sh:NodeShape ;
    sh:message "need 1 damper" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Damper .

:clg-coil-count a sh:NodeShape ;
    sh:message "need 1 cooling coil" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Cooling_Coil .

:htg-coil-count a sh:NodeShape ;
    sh:message "need 1 heating coil" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Heating_Coil .

Put all of the above in a new file called tutorial1_manifest.ttl. We’ll also add a shape called sz-vav-ahu-control-sequences, which is a use case shape to validate the model against in the next section.

The following block of code puts all of the above in the tutorial1_manifest.ttl file for you:

with open("tutorial1_manifest.ttl", "w") as f:
    f.write("""
@prefix brick: <https://brickschema.org/schema/Brick#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix constraint: <https://nrel.gov/BuildingMOTIF/constraints#> .
@prefix : <urn:my_site_constraints/> .

: a owl:Ontology ;
    owl:imports <https://brickschema.org/schema/1.3/Brick>,
                <https://nrel.gov/BuildingMOTIF/constraints>,
                <urn:ashrae/g36> .

:ahu-count a sh:NodeShape ;
    sh:message "need 1 AHU" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:AHU .

:fan-count a sh:NodeShape ;
    sh:message "need 1 supply fan" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Supply_Fan .

:damper-count a sh:NodeShape ;
    sh:message "need 1 damper" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Damper .

:clg-coil-count a sh:NodeShape ;
    sh:message "need 1 cooling coil" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Cooling_Coil .

:htg-coil-count a sh:NodeShape ;
    sh:message "need 1 heating coil" ;
    sh:targetNode : ;
    constraint:exactCount 1 ;
    constraint:class brick:Heating_Coil .

""")

Adding the Manifest to the Model#

We associate the manifest with our model so that BuildingMOTIF knows that we want validate the model against these specific shapes. We can always update this manifest, or validate our model against other shapes; however, validating a model against its manifest is the most common use case, so this is treated specially in BuildingMOTIF.

# load manifest into BuildingMOTIF as its own library!
manifest = Library.load(ontology_graph="tutorial1_manifest.ttl")
# set it as the manifest for the model
model.update_manifest(manifest.get_shape_collection())

Validating the Model#

We can now ask BuildingMOTIF to validate the model against the manifest and ask BuildingMOTIF for some details if it fails. By default, BuildingMOTIF will include all shape collections imported by the manifest (owl:imports). BuildingMOTIF will complain if the manifest requires ontologies that have not yet been loaded into BuildingMOTIF; this is why we are careful to load in the Brick and Guideline36 libraries at the top of this tutorial.

validation_result = model.validate()
print(f"Model is valid? {validation_result.valid}")

# print reasons
for entity, errors in validation_result.diffset.items():
    print(entity)
    for err in errors:
        print(f" - {err.reason()}")

Tip on supplying extra shape collections

We can also provide a list of shape collections directly to Model.validate; BuildingMOTIF will use these shape collections to validate the model instead of the manifest.

# gather shape collections into a list for ease of use
shape_collections = [
    brick.get_shape_collection(),
    constraints.get_shape_collection(),
    manifest.get_shape_collection(),
    g36.get_shape_collection(),
]

# pass a list of shape collections to .validate()
validation_result = model.validate(shape_collections)
print(f"Model is valid? {validation_result.valid}")

# print reasons
for entity, errors in validation_result.diffset.items():
    print(entity)
    for err in errors:
        print(f" - {err.reason()}")

Fixing the Model#

One of the reasons the model is failing is we don’t have a heating coil required by the manifest, which we forgot to add in the previous tutorial. It’s also failing the use case validation, which we’ll cover in the next section. To fix the manifest validation, use the equipment templates in the Brick library to create a heating coil, add it to the model, and connect it to the AHU using RDFLib’s graph.add() method.

# ahu name
ahu_name = "Core_ZN-PSC_AC"

# get template
htg_coil_template = brick.get_template_by_name(BRICK.Heating_Coil)

# add htg coil
htg_coil_name = f"{ahu_name}-Htg_Coil"
htg_coil_binding = {"name": BLDG[htg_coil_name]}
htg_coil_graph = htg_coil_template.evaluate(htg_coil_binding)
model.add_graph(htg_coil_graph)

# connect htg coil
model.graph.add((BLDG[ahu_name], BRICK.hasPart, BLDG[htg_coil_name]))

# print model to confirm component was added and connected
print(model.graph.serialize())

We can see that the heating coil was added to the model and connected to the AHU so let’s check if the manifest validation failure was fixed.

validation_result = model.validate()
print(f"Model is valid? {validation_result.valid}")

# print reasons
for entity, errors in validation_result.diffset.items():
    print(entity)
    for err in errors:
        print(f" - {err.reason()}")

Success! Our model is now valid.

Model Validation - Use Case#

Finding Use Case Shapes#

We can use a couple methods to search the libraries for shapes we might want to use. Let’s start by asking the g36 library for any system specifications it knows about. This library represents ASHRAE Guideline 36 High-Performance Sequences of Operation for HVAC Systems[2]. A system specification will specify all of the metadata required for an entity to run control sequences associated with that system type.

from buildingmotif.namespaces import BMOTIF
shapes = g36.get_shape_collection()
for shape in shapes.get_shapes_of_definition_type(BMOTIF["System_Specification"]):
    print(shape)

The model represents the Small Office Commercial Prototype Building model, which has single zone packaged AHUs, so we’re interested in validating it against Section 4.8 of Guideline 36 for single zone variable air volume (VAV) AHUs.

Let’s update our manifest to include the requirement that AHUs must match the “single zone AHU” shape from G36:

model.get_manifest().graph.parse(data="""
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix brick: <https://brickschema.org/schema/Brick#> .
@prefix : <urn:my_site_constraints/> .
:sz-vav-ahu-control-sequences a sh:NodeShape ;
    sh:message "AHUs must match the single-zone VAV AHU shape" ;
    sh:targetClass brick:AHU ;
    sh:node <urn:ashrae/g36/4.8/sz-vav-ahu/sz-vav-ahu> .
""")

Validating the Model#

Now we can run validation to see if our AHU is ready to run the “single zone AHU” control sequence:

validation_result = model.validate()
print(f"Model is valid? {validation_result.valid}")

The AHU fails validation because it doesn’t match the sz-vav-ahu-control-sequences requirements. Take a look at the first bit of output, which is the official SHACL validation report text format. This can be difficult to interpret without a background in SHACL, so BuildingMOTIF provides a more easily understood version of the same information.

# SHACL validation report
print(validation_result.report_string)

# separator
print("-"*79)

Here is BuildingMOTIF’s interpretation of that report.

# BuildingMOTIF output
print("Model is invalid for these reasons:")
for entity, errors in validation_result.diffset.items():
    print(entity)
    for err in errors:
        print(f" - {err.reason()}")

The model is failing because the AHU doesn’t have the required points. We could find those templates manually, evaluate them, and add the resulting graphs to the model. However, this can be a bit tedious. To address this issue, BuildingMOTIF can find those templates automatically for us. We’ll cover this feature in the next tutorial so let’s save the model.

#save model
model.graph.serialize(destination="tutorial2_model.ttl")

Model Validation

Contents

Model Validation#

Setup#

Model Validation - Ontology#

Model Validation - Manifest#

Writing a Manifest#

Adding the Manifest to the Model#

Validating the Model#

Fixing the Model#

Model Validation - Use Case#

Finding Use Case Shapes#

Validating the Model#