COMPASS Validation#

Here we give a brief overview of the results of known COMPASS validation efforts.

COMPASS validation is an ongoing effort, and we will update this page as new results become available.

Mini Wind Validation (June 2025)#

This validation was for the ordinance value extraction portion only (assume documents are correct and belong to the correct jurisdiction).

This validation effort is meant primarily as a sanity check and as a stepping stone to:

  • Perform a larger validation effort for solar ordinance values, and

  • Begin a CONUS-level COMPASS run for wind ordinance collection.

Warning

The number of documents examined is extremely small compared to the total number of jurisdictions. Interpret all results with extreme caution!

Info#

  • COMPASS Version: v0.5.0

  • Number of Documents: 19 (Assuming 6,000 jurisdictions, there is a 95% chance that the metrics are within ±22.45% of the reported value)

  • Features:

    • structures (participating)

    • structures (non-participating)

    • property line (participating)

    • property line (non-participating)

    • roads

    • railroads

    • transmission

    • water

    • noise

    • maximum height

    • maximum project size

    • minimum lot size

    • maximum lot size

    • shadow flicker

    • tower density

    • blade clearance

    • primary use districts

    • special use districts

    • accessory use districts

    • color

    • decommissioning

    • lighting

    • prohibitions

    • visual impact

    • ordinance effective year

  • Procedure Validated: Ordinance extraction from documents

  • LLM(s) used: OpenAI GPT-4o-mini

Results#

../_images/wind_mini_june_2025.png

Wind Validation (January 2024)#

This validation was for the ordinance value extraction portion only (assume documents are correct and belong to the correct jurisdiction).

This is the original model validation, focusing on the model ability to extract structured ordinance data from unstructured wind ordinance text documents.

Info#

  • COMPASS Version: alpha

  • Number of Documents: 83 (Assuming 3,000 jurisdictions, there is a 95% chance that the metrics are within ±10.61% of the reported value)

  • Features:

    • structures setbac

    • property lines

    • roads

    • railroads

    • transmission

    • water

    • noise

    • maximum height

    • minimum lot size

    • shadow flicker

    • tower density

  • Procedure Validated: Ordinance extraction from documents

  • LLM(s) used: OpenAI GPT-4

Results#

../_images/wind_jan_2024.png