Connector Property Transforms

Overview

Property transforms allow rules to be applied to add, or modify properties of a signal, condition, capsule, scalar, user group, or asset during indexing. These transforms are applied through transform directives that are configured on the connection.

A transform consists of two main elements, an input (or set of inputs) and an output (or set of outputs). Inputs and outputs are property/value pairs. In an input, the property field defines the property the transform should match, and the value defines the matching criteria. In an output the property value will define the property to modify, and the value will include the modification.

Examples Use Cases

• Change the Value Unit Of Measure to percent (%) for any signal whose Name ends with "_CONCENTRATION"

• Use a portion of the Description field to populate the Value Unit Of Measure

• Change the Maximum Interpolation for all signals whose Name contains the text "HIGH_FREQ"

• Change the interpolation method for signals whose Name ends with “STEP”

• Specify permissions to items in the datasource

Overview of Properties

Properties in Seeq are key, value pairs. Some properties affect how items can be searched for or how data queries are made, other properties can be informational only. In addition to properties listed here, custom properties can be added, or more specific properties (listed below) can be used.

Creating Transform Directives

The transform operation happens during the indexing operation, so you will need to wait for it to happen or force it by clicking Request Index for the connection in the Seeq Administration Datasources tab. \

If you change the Log field in the transform to true, then net-link.log or jvm-link.log (see General Troubleshooting of Connectors for which one is relevant to your connection) will contain entries for any transform activity that takes place:

DEBUG 2017-10-19T16:18:40.331-07:00 [Metadata sync: MYMACHINE1: OSIsoft PI: PISERVER] Seeq.Link.SDK.Services.PropertyTransformer Name: 'ASSET01_TEMP_HIGH_FREQ' Data ID: '7544' Property: 'Maximum Interpolation' Transform: '' --> '1s'

For more information on accessing logs, see Viewing Logs and Sending Log Files to Seeq.

You can temporarily disable a transform by setting the Enabled field to false. Note that if the Log field is true, you'll still get the output in the log file. This can be useful to test your transform without affecting the database.

Understanding Regular Expressions

The transform capability heavily utilizes a text matching and replacement scheme called Regular Expressions (aka "Regex"). This gives you the power to use "wildcards" and create precise matching directives for the transform.

Complex transforms will use a particular feature of Regular Expressions called Named Capture Groups. This allows you to grab a part of a property and use it in the transform output. Capture groups have the following syntax: (?<capture_group_name>logic)  and can be used in transform output like so: ${capture_group_name}

See Use Description Field For Units of Measure below for an example of this approach.

All regex specifications are case sensitive. The regex must make a full match on the input property.

To specify that a transform should be applied only for items that do not have a given property or for which that property is null in the input model posted to Seeq, one may specify the value of the Property as null (no quotes) in a transform input.

It is important to note that transforms apply only to the input model sent by the connector to Seeq and not to the item as seen in Seeq. That is, certain properties on an item in Seeq may have been defined by Seeq when the item was first posted, for example, the Interpolation Method property. If this property was not specified by the connector when the item was posted, it will have been assigned a value in the Seeq database based on whether it is a string or numeric item (step in the former case, linear in the latter). A property transform that is checking for “linear” as the Interpolation Method will, in this case, not have any effect, since the Interpolation Method is actually null on the input model being sent to Seeq from the connector. This also implies that properties created by transforms during previous indexes will not be available for use as input in subsequent indexes. If you would like a transform to depend on the results of other transforms, the transform producing the property on which the other transforms depend must be present in the list of transforms and must precede any dependent transforms.

There are online tools like Regex101 (for jvm-link connections) and Regex Storm (for net-link connections) that you can use to test your matching logic.

On Regex101, the EXPLANATION pane walks you through your expression syntax, and the MATCH INFORMATION shows you the named capture groups:

Examples

Use PILinear Interpolation in non-PI/AF Datasources

Seeq utilizes a special "PILinear" interpolation method when handling signals from OSIsoft PI or Asset Framework. This method "extrapolates" a flat line in two cases: (a) when the next sample is invalid; (b) when the next sample has not yet arrived.

While this interpolation mode is used by default for PI and AF datasources, you may wish to use it for other datasources like OPC-HDA. If so, you can use the following transforms list:

[
  {
    "Inputs": [
      {
        "Property": "Interpolation Method",
        "Value": "^linear$"
      }
    ],
    "Outputs": [
      {
        "Property": "Interpolation Method",
        "Value": "pilinear"
      }
    ]
  }
]

This simple transform matches on all "linear" interpolation methods (note the use of ^ and $ to guarantee that it matches exactly to the word "linear") and changes them to "pilinear".

Use Description Field For Units of Measure

Let's say you have a datasource where some tags have a Description field that contains units of measure embedded in parentheses at the end:

[
  {
    "Inputs": [
      {
        "Property": "Description",
        "Value": ".*\\((?<units>.+)\\)"
      }
    ],
    "Outputs": [
      {
        "Property": "Value Unit Of Measure",
        "Value": "${units}"
      }
    ]
  }
]

There are several things to note:

• To understand the Description's Value field specification, start with the named capture group called units. A group is always enclosed in parentheses, and the name is the first thing after the left parenthesis – in this case, ?<units>.

• We are trying to grab the units that are themselves enclosed within real parentheses. Therefore, there are escaped parentheses – which are encoded in your JSON as \\( and \\) -- on either side of the named capture group. Escaping a character with preceding backlashes causes the character to be treated literally instead of being interpreted as a Regex instruction.

  • When you use http://regex101.com, you use only a single preceding backlash. (You use two backslashes in JSON because backslashes themselves must be escaped in JavaScript.)

• In the output, we are using the named capture group units to overwrite the Value Unit Of Measure property on the signal.

• If you are adding multiple transforms, arguments must be separated by a comma. 

Filtering By Type

You can also filter on the "Type" property to apply Property Transforms on signals, scalars, assets, conditions, capsules or User Groups.

Sometimes a transform should only apply to certain types of items being indexed.

[
  {
    "Inputs": [
      {
        "Property": "Type",
        "Value": "Condition"
      }
    ],
    "Outputs": [
      {
        "Property": "Maximum Duration",
        "Value": "1hr"
      }
    ],
    "Enabled": true,
    "Log": false
  }
]

In the above example, "Maximum Duration" is only a meaningful property for Conditions, so the transform only matches that type. Another use would be for units of measure; Signals have both "Key Unit of Measure" and "Value Unit of Measure" properties, whereas Conditions and Scalars only have a single "Unit of Measure" property, so different type-filtered transforms are necessary.

Property transforms can only be applied to only signals, scalars, assets, and conditions..

Disabling Caching For a Child Signal of an Asset

Seeq is able to import asset hierarchies developed in OSIsoft Asset Framework through its respective connector. At times, an operation such as disabling the cache may be needed on a particular asset or particular child signal but not for the entire datasource.

[
  {
    "Inputs": [
      {
        "Property": "AF Path",
        "Value": ".*Pi World Barcelona\\\\Pump 1\\|Flow"
      }
    ],
    "Outputs": [
      {
        "Property": "Cache Enabled", 
        "Value": false
      }
    ]
  }
]

Note that in this regex, we were looking to disable the cache of the "Flow" signal within our asset hierarchy. JSON requires literal backslashes to be escaped with another backslash in order for the text to be recognized. Regex also requires an escape backslash and JSON requires another backslash to escape that regex escape, producing the 4 backslashes instead of the 1 used in an asset hierarchy path. There are 2 additional backslashes before "Flow" because Regex treats the vertical bar as a special character, so it needs to be escaped with a backslash. JSON then needs another backslash to recognize the Regex escape.

Applying Multiple Transforms

In this example, we'll look at how to apply a single transform with multiple inputs and how to apply multiple transforms. The former looks for data that satisfies both inputs prior to make any changes while the latter applies two separate transforms independent of each other. 

Transform with Multiple Inputs

[
  {
    "Inputs": [
      {
        "Property": "AF Path",
        "Value": ".*Optimizer"
      },
      {
        "Property": "Interpolation Method",
        "Value": "PILinear"
      }
    ],
    "Outputs": [
      {
        "Property": "Description",
        "Value": "PI Linear Optimizer"
      }
    ],
    "Enabled": true,
    "Log": true
  }
]

Here, we look for our Optimizer attribute within any hierarchy that also have a PI Linear interpolation method. We then change their descriptions to "PI Linear Optimizer". 

Multiple Transforms

[
  {
    "Inputs": [
      {
        "Property": "AF Path",
        "Value": ".*Optimizer"
      }
    ],
    "Outputs": [
      {
        "Property": "Description",
        "Value": "Testing Testing" 
      }
    ],
    "Enabled": true,
    "Log": true
  },
  {
    "Inputs": [
      {
        "Property": "Interpolation Method",
        "Value": "PILinear"
      }
    ],
    "Outputs": [
      {
        "Property": "Description",
        "Value": "Testing Testing"
      }
    ],
    "Enabled": true,
    "Log": true
  }
]

Here, we apply a description of "PI Linear Optimizer" to data that either is an Optimizer attribute or has a PI Linear Interpolation Method. 

Creating Unrecognized Units

By default, Seeq converts certain units into their Seeq equivalents, which are based on SI prefixes. For example, units with a prefix of "MM" in oil and gas often represent millions, so Seeq automatically replaces these units with their SI equivalent "M". A similar example of this can be found with gauge pressure (psig) and absolute pressure (psia), where Seeq makes these units psi by default. For cases where the renaming is not wanted, the transform below can be used as a template. This example makes all variations of psi (psia, psig, psid, etc.) unrecognized so that they'll remain in Seeq. Note that since these units are unrecognized, calculations involving signals with these units will not account them. A unit will have to be set in a Seeq formula.

Unrecognized PSIG/PSIA Units

[
  {
    "Inputs": [
      {
        "Property": "Value Unit Of Measure",
        "Value": "(?<CorrectUnit>(PSI|psi).+)"
      }
    ],
    "Outputs": [
      {
        "Property": "Source Value Unit Of Measure",
        "Value": "${CorrectUnit}"
      },
      {
        "Property": "Value Unit Of Measure",
        "Value": ""
      }
    ],
    "Enabled": true,
    "Log": true
  }
]

Indexing Exclusions

You can exclude items from indexing by applying a property transform that adds a property named "Exclude From Indexing" with a value of true.  Any items that have this property after the transform will be excluded from posts to the Seeq database.  The effect is to reduce the total time of indexing and to provide granular control over the indexing of databases that might include items that won't be of interest to end users.  If the property is set to any other value than true or if it it removed entirely, the next index operation will once again index items that were previously excluded.

[
  {
    "Inputs": [
      {
        "Property": "AF Path",
        "Value": ".*\\|Flow_Computed$"
      }
    ],
    "Outputs": [
      {
        "Property": "Exclude From Indexing", 
        "Value": true
      }
    ]
  }
]

Property Exclusions

While "Exclude From Indexing" causes an entire item to be excluded from indexing, the string "Exclude Property" may be used to exclude a single property from an item while still indexing the item. To do so, specify the name of the property to exclude using the Value field. For example, the following transform will prevent the Description property from being included as a property of any item for which the Name ends in Optimizer.

[
  {
      "Inputs": [
        {
          "Property": "Name",
          "Value": ".*Optimizer"
        }
      ],
      "Outputs": [
        {
          "Property": "Description",
          "Value": "Exclude Property" 
        }
      ],
      "Enabled": true,
      "Log": true
  }
]

This capability is most useful for excluding capsule properties, which can become excessive in number for certain datasources.

Removing Properties

There are cases where a property has been added to an item and is no longer appropriate or needed on that item. Indexing may update existing properties to new values, but it usually does not remove properties, since the connector does not always have a way of distinguishing between properties added via transforms, API requests, etc. after indexing versus those previously added by the connector itself but no longer added due to connector changes, datasource changes, etc. In such cases, there is a slightly stronger variant than “Exclude Property”: “Remove Property”. When an output property is specified to have value “Remove Property”, this will both prevent the property from being indexed and remove it from the Seeq item if it already exists. Since this transform causes a (usually intentional) loss of information by removing properties, it should be used with care. You may choose to set the transform’s “Log” property to true while leaving “Enabled” to false so that it’s possible to see details of the transform application without actually performing the removal, then change “Enabled” to true once you are confident in which items will be affected based on the inputs.

Property Name Matcher

Using the string "Property Name Matcher" as the Property of an input causes the input’s Value to be used as a regular expression to match the names of an item’s properties - this is in contradistinction to a regular input, for which the Property is the exact name of an item’s property and the Value field, if a string, is used as a regular expression to match and capture parts of the string value of the property whose name is specified by the Property field. The Property Name Matcher input must be the first in the list of inputs for each transform in which it is used. If Property Name Matcher is used as an input, any output that has a Property of "${Property Name Matcher} will be applied to each property matched by the Property Name Matcher input. This is mainly useful for excluding properties, since in most cases it won't make sense to apply the same output logic to multiple properties.

The following transform excludes all capsule properties ending in Recipe Data from the capsules of any conditions for the connection to which it is applied.

        {
          "Inputs": [
            {
              "Property": "Property Name Matcher",
              "Value": ".*Recipe Data$"
            },
            {
              "Property": "Type",
              "Value": "Capsule"
            }
          ],
          "Outputs": [
            {
              "Property": "${Property Name Matcher}",
              "Value": "Exclude Property",
              "UnitOfMeasure": null
            }
          ],
          "Enabled": true,
          "Log": true
        }

Transforming Capsules

Capsules of AF Event Frames conditions or other conditions indexed by a condition connector may also be transformed, with the following distinctions:

• For performance reasons, only transforms with input Type "Capsule" are applied to Capsules; transforms with no Type input are not applied to Capsules, even if they would otherwise match.

• Capsule transforms are not allowed to change Capsule properties, since the resulting divergence from the datasource would cause confusion for users comparing the source data and the data in Seeq. On the other hand, they are allowed to create new properties using any Capsule property as input.

• Besides using any capsule property as input, capsule transforms also have available an input property named "Condition Metadata: Data ID", whose value is the Data ID of the condition to which the capsule belongs. This allows transforming only capsules of specific conditions.

An example capsule transform is:

[
  {
    "Inputs": [
        {
            "Property": "Type",
            "Value": "Capsule"
        },
        {
            "Property": "Condition Metadata: Data ID",
            "Value": "Batch(?<BatchNumber>.+)"
        }
    ],
    "Outputs": [
        {
            "Property": "Batch Number",
            "Value": "${BatchNumber}",
            "UnitOfMeasure": "string"
        }
    ],
    "Enabled": true,
    "Log": false
  }
]

Given a condition with Data ID Batch123, the above will transform its capsules by creating a new capsule property named Batch Number with value 123.

Capsules from other connectors can be transformed in the same fashion.

Capsule properties may also be excluded using a transform. The example below shows how to exclude the Duration property. Note that as written, this transform will apply to all capsules of all conditions of the connection to which the transform is applied. For more granularity, use the "Condition Metadata: Data ID" input described above.

[
    {
        "Inputs": [
            {
                "Property": "Type",
                "Value": "Capsule"
            },
            {
                "Property": "Duration",
                "Value": ".*"
            }
        ],
        "Outputs": [
            {
                "Property": "Duration",
                "Value": "Exclude Property",
                "UnitOfMeasure": null
            }
        ],
        "Enabled": true,
        "Log": false
    }
]

Adding a Default Value

In this example, we will provide a default value for the Description field if it doesn’t exist. If it does exist, we’ll modify it slightly. Note that you must provide these transforms in the order specified:

[
  {
    "Inputs": [
      {
        "Property": "Description",
        "Value": "(?<existing>.+)"
      }
    ],
    "Outputs": [
      {
        "Property": "Description",
        "Value": "${existing} PLUS MY MODIFICATIONS"
      }
    ]
  },
  {
    "Inputs": [
      {
        "Property": "Description",
        "Value": null
      }
    ],
    "Outputs": [
      {
        "Property": "Description",
        "Value": "MY DEFAULT"
      }
    ]
  }
]

Applying an Uncertainty Override

As explained in Uncertainty Override, you can apply a property that dictates how much data is certain (and therefore cacheable) near the current time (aka “now”). In this example, we are telling the system that all signals that include VOLATILE in the name should not cache any data for the most recent 18 hours:

[
  {
    "Inputs": [
      {
        "Property": "Name",
        "Value": ".*VOLATILE.*"
      }
    ],
    "Outputs": [
      {
        "Property": "Uncertainty Override",
        "Value": 18,
        "UnitOfMeasure": "h"
      }
    ],
    "Enabled": true,
    "Log": false
  }
]

Applying a Request Interval Clamp

As explained in Request Interval Clamp, you can apply a property that dictates how far in the future a request to a datasource can extend. This example applies a 10h clamp to all signals that do not end with .forecast

[
  {
    "Inputs": [
      {
          "Property": "Name",
          "Value": ".+(?<!(.forecast))$"
      }
    ],
    "Outputs": [
      {
        "Property": "Request Interval Clamp",
        "Value": 10,
        "UnitOfMeasure": "h"
      }
    ],
    "Enabled": true,
    "Log": false
  }
]

Datasource Debounce

There are situations where calculations and updates to the UI are taking too long due to uncertain data near-now being constantly requested by calculations and user behavior, such as a frequently auto-updating signal or conditions.

Multiple requests for the same data can decrease Seeq and the datasource performance. To prevent this issue, you can apply a debounce duration property to each affected signal or condition in the datasource. This property ensures that Seeq does not retrieve the same items from the datasource within the specified duration.

The example below will apply a 1 minute debounce duration for all signals and conditions from a datasource.

[
  {
      "Inputs": [
          {
              "Property": "Name",
              "Value": ".*"
          }
      ],
      "Outputs": [
          {
              "Property": "Debounce Duration",
              "Value": 1,
              "UnitOfMeasure": "minutes"
          }
      ],
      "Enabled": true,
      "Log": false
  }
]