Connector Property Transforms

Overview

Property transforms allow rules to be applied to add, or modify properties of a signal, condition, capsule, scalar, user group, or asset during indexing. These transforms are applied through transform directives that are configured on the connection.

A transform consists of two main elements, an input (or set of inputs) and an output (or set of outputs). Inputs and outputs are property/value pairs. In an input, the property field defines the property the transform should match, and the value defines the matching criteria. In an output the property value will define the property to modify, and the value will include the modification.

Examples Use Cases

• Change the Value Unit Of Measure to percent (%) for any signal whose Name ends with "_CONCENTRATION"

• Use a portion of the Description field to populate the Value Unit Of Measure

• Change the Maximum Interpolation for all signals whose Name contains the text "HIGH_FREQ"

• Change the interpolation method for signals whose Name ends with “STEP”

• Specify permissions to items in the datasource

Overview of Properties

Properties in Seeq are key, value pairs. Some properties affect how items can be searched for or how data queries are made, other properties can be informational only. In addition to properties listed here, custom properties can be added, or more specific properties (listed below) can be used.

Creating Transform Directives

The transform operation happens during the indexing operation, so you will need to wait for it to happen or force it by clicking Request Index for the connection in the Seeq Administration Datasources tab. \

If you change the Log field in the transform to true, then net-link.log or jvm-link.log (see General Troubleshooting of Connectors for which one is relevant to your connection) will contain entries for any transform activity that takes place:

DEBUG 2017-10-19T16:18:40.331-07:00 [Metadata sync: MYMACHINE1: OSIsoft PI: PISERVER] Seeq.Link.SDK.Services.PropertyTransformer Name: 'ASSET01_TEMP_HIGH_FREQ' Data ID: '7544' Property: 'Maximum Interpolation' Transform: '' --> '1s'

For more information on accessing logs, see Viewing Logs and Sending Log Files to Seeq.

You can temporarily disable a transform by setting the Enabled field to false. Note that if the Log field is true, you'll still get the output in the log file. This can be useful to test your transform without affecting the database.

Understanding Regular Expressions

The transform capability heavily utilizes a text matching and replacement scheme called Regular Expressions (aka "Regex"). This gives you the power to use "wildcards" and create precise matching directives for the transform.

Complex transforms will use a particular feature of Regular Expressions called Named Capture Groups. This allows you to grab a part of a property and use it in the transform output. Capture groups have the following syntax: (?<capture_group_name>logic)  and can be used in transform output like so: ${capture_group_name}

See Use Description Field For Units of Measure below for an example of this approach.

All regex specifications are case sensitive. The regex must make a full match on the input property.

To specify that a transform should be applied only for items that do not have a given property or for which that property is null in the input model posted to Seeq, one may specify the value of the Property as null (no quotes) in a transform input.

It is important to note that transforms apply only to the input model sent by the connector to Seeq and not to the item as seen in Seeq. That is, certain properties on an item in Seeq may have been defined by Seeq when the item was first posted, for example, the Interpolation Method property. If this property was not specified by the connector when the item was posted, it will have been assigned as value in the Seeq database based on whether it is a string or numeric item (step in the former case, linear in the latter). A property transform that is checking for “linear” as the Interpolation Method will, in this case, not have any effect, since the Interpolation Method is actually null on the input model being sent to Seeq from the connector. This also implies that properties created by transforms during previous indexes will not be available for use as input in subsequent indexes. If you would like a transform to depend on the results of other transforms, the transform producing the property on which the other transforms depend must be present in the list of transforms and must precede any dependent transforms.

There are online tools like Regex101 (for jvm-link connections) and Regex Storm (for net-link connections) that you can use to test your matching logic.

On Regex101, the EXPLANATION pane walks you through your expression syntax, and the MATCH INFORMATION shows you the named capture groups:

Examples

Use PILinear Interpolation in non-PI/AF Datasources

Seeq utilizes a special "PILinear" interpolation method when handling signals from OSIsoft PI or Asset Framework. This method "extrapolates" a flat line in two cases: (a) when the next sample is invalid; (b) when the next sample has not yet arrived.

While this interpolation mode is used by default for PI and AF datasources, you may wish to use it for other datasources like OPC-HDA. If so, you can use the following transform:

      "Transforms": [
        {
          "Inputs": [
            {
              "Property": "Interpolation Method",
              "Value": "^linear$"
            }
          ],
          "Outputs": [
            {
              "Property": "Interpolation Method",
              "Value": "pilinear"
            }
          ]
        }
      ]

This simple transform matches on all "linear" interpolation methods (note the use of ^ and $ to guarantee that it matches exactly to the word "linear") and changes them to "pilinear".

Use Description Field For Units of Measure

Let's say you have a datasource where some tags have a Description field that contains units of measure embedded in parentheses at the end:

      "Transforms": [
        {
          "Inputs": [
            {
              "Property": "Description",
              "Value": ".*\\((?<units>.+)\\)"
            }
          ],
          "Outputs": [
            {
              "Property": "Value Unit Of Measure",
              "Value": "${units}"
            }
          ]
        }
      ]

There are several things to note:

• To understand the Description's Value field specification, start with the named capture group called units. A group is always enclosed in parentheses, and the name is the first thing after the left parenthesis – in this case, ?<units>.

• We are trying to grab the units that are themselves enclosed within real parentheses. Therefore, there are escaped parentheses – which are encoded in your JSON as \\( and \\) -- on either side of the named capture group. Escaping a character with preceding backlashes causes the character to be treated literally instead of being interpreted as a Regex instruction.

  • When you use http://regex101.com, you use only a single preceding backlash. (You use two backslashes in JSON because backslashes themselves must be escaped in JavaScript.)

• In the output, we are using the named capture group units to overwrite the Value Unit Of Measure property on the signal.

• If you are adding multiple transforms, arguments must be separated by a comma. 

Filtering By Type

Sometimes a transform should only apply to certain types of items being indexed.

"Transforms": [
  {
    "Inputs": [
      {
        "Property": "Type",
        "Value": "Condition"
      }
    ],
    "Outputs": [
      {
        "Property": "Maximum Duration",
        "Value": "1hr"
      }
    ],
    "Enabled": true,
    "Log": false
  }
]

In the above example, "Maximum Duration" is only a meaningful property for Conditions, so the transform only matches that type. Another use would be for units of measure; Signals have both "Key Unit of Measure" and "Value Unit of Measure" properties, whereas Conditions and Scalars only have a single "Unit of Measure" property, so different type-filtered transforms are necessary.

Property transforms can only be applied to only signals, scalars, assets, and conditions..

Disabling Caching For a Child Signal of an Asset

Seeq is able to import asset hierarchies developed in OSIsoft Asset Framework through its respective connector. At times, an operation such as disabling the cache may be needed on a particular asset or particular child signal but not for the entire datasource.

	"Transforms": [
        {
          "Inputs": [
            {
              "Property": "AF Path",
              "Value": ".*Pi World Barcelona\\\\Pump 1\\|Flow"
            }
          ],
          "Outputs": [
            {
              "Property": "Cache Enabled", 
              "Value": false
            }
          ]
        }
      ]

Note that in this regex, we were looking to disable the cache of the "Flow" signal within our asset hierarchy. JSON requires literal backslashes to be escaped with another backslash in order for the text to be recognized. Regex also requires an escape backslash and JSON requires another backslash to escape that regex escape, producing the 4 backslashes instead of the 1 used in an asset hierarchy path. There are 2 additional backslashes before "Flow" because Regex treats the vertical bar as a special character, so it needs to be escaped with a backslash. JSON then needs another backslash to recognize the Regex escape.

Applying Multiple Transforms

In this example, we'll look at how to apply a single transform with multiple inputs and how to apply multiple transforms. The former looks for data that satisfies both inputs prior to make any changes while the latter applies two separate transforms independent of each other. 

Transform with Multiple Inputs

 	"Transforms": [
        {
          "Inputs": [
            {
              "Property": "AF Path",
              "Value": ".*Optimizer"
            },
            {
              "Property": "Interpolation Method",
              "Value": "PILinear"
            }
          ],
          "Outputs": [
            {
              "Property": "Description",
              "Value": "PI Linear Optimizer"
            }
          ],
          "Enabled": true,
          "Log": true
        }
      ]

Here, we look for our Optimizer attribute within any hierarchy that also have a PI Linear interpolation method. We then change their descriptions to "PI Linear Optimizer". 

Multiple Transforms

	 "Transforms": [
        {
          "Inputs": [
            {
              "Property": "AF Path",
              "Value": ".*Optimizer"
            }
          ],
          "Outputs": [
            {
              "Property": "Description",
              "Value": "Testing Testing" 
            }
          ],
          "Enabled": true,
          "Log": true
        },
        {
          "Inputs": [
            {
              "Property": "Interpolation Method",
              "Value": "PILinear"
            }
          ],
          "Outputs": [
            {
              "Property": "Description",
              "Value": "Testing Testing"
            }
          ],
          "Enabled": true,
          "Log": true
        }
      ]

Here, we apply a description of "PI Linear Optimizer" to data that either is an Optimizer attribute or has a PI Linear Interpolation Method. 

Creating Unrecognized Units

By default, Seeq converts certain units into their Seeq equivalents, which are based on SI prefixes. For example, units with a prefix of "MM" in oil and gas often represent millions, so Seeq automatically replaces these units with their SI equivalent "M". A similar example of this can be found with gauge pressure (psig) and absolute pressure (psia), where Seeq makes these units psi by default. For cases where the renaming is not wanted, the transform below can be used as a template. This example makes all variations of psi (psia, psig, psid, etc.) unrecognized so that they'll remain in Seeq. Note that since these units are unrecognized, calculations involving signals with these units will not account them. A unit will have to be set in a Seeq formula.

Unrecognized PSIG/PSIA Units

	 "Transforms": [
        {
          "Inputs": [
            {
              "Property": "Value Unit Of Measure",
              "Value": "(?<CorrectUnit>(PSI|psi).+)"
            }
          ],
          "Outputs": [
            {
              "Property": "Source Value Unit Of Measure",
              "Value": "${CorrectUnit}"
            },
            {
              "Property": "Value Unit Of Measure",
              "Value": ""
            }
          ],
          "Enabled": true,
          "Log": true
        }
      ]

Indexing Exclusions

You can exclude items from indexing by applying a property transform that adds a property named "Exclude From Indexing" with a value of true.  Any items that have this property after the transform will be excluded from posts to the Seeq database.  The effect is to reduce the total time of indexing and to provide granular control over the indexing of databases that might include items that won't be of interest to end users.  If the property is set to any other value than true or if it it removed entirely, the next index operation will once again index items that were previously excluded.

	"Transforms": [
        {
          "Inputs": [
            {
              "Property": "AF Path",
              "Value": ".*\\|Flow_Computed$"
            }
          ],
          "Outputs": [
            {
              "Property": "Exclude From Indexing", 
              "Value": true
            }
          ]
        }
      ]

Adding a Default Value

In this example, we will provide a default value for the Description field if it doesn’t exist. If it does exist, we’ll modify it slightly. Note that you must provide these transforms in the order specified:

      "Transforms": [
        {
          "Inputs": [
            {
              "Property": "Description",
              "Value": "(?<existing>.+)"
            }
          ],
          "Outputs": [
            {
              "Property": "Description",
              "Value": "${existing} PLUS MY MODIFICATIONS"
            }
          ]
        },
        {
          "Inputs": [
            {
              "Property": "Description",
              "Value": null
            }
          ],
          "Outputs": [
            {
              "Property": "Description",
              "Value": "MY DEFAULT"
            }
          ]
        }
      ]

Applying an Uncertainty Override

As explained in https://seeq.atlassian.net/wiki/spaces/KB/pages/115185486/General+Troubleshooting+of+Connectors#Uncertainty-Override, you can apply a property that dictates how much data is certain (and therefore cacheable) near the current time (aka “now”). In this example, we are telling the system that all signals that include VOLATILE in the name should not cache any data for the most recent 18 hours:

      "Transforms": [
        {
          "Inputs": [
            {
              "Property": "Name",
              "Value": ".*VOLATILE.*"
            }
          ],
          "Outputs": [
            {
              "Property": "Uncertainty Override",
              "Value": 18,
              "UnitOfMeasure": "h"
            }
          ],
          "Enabled": true,
          "Log": false
        }
      ]

Applying a Request Interval Clamp

As explained in Request Interval Clamp, you can apply a property that dictates how far in the future a request to a datasource can extend. This example applies a 10h clamp to all signals that do not end with .forecast

      "Transforms": [
        {
          "Inputs": [
           {
                "Property": "Name",
                "Value": ".+(?<!(.forecast))$"
           }
          ],
          "Outputs": [
            {
              "Property": "Request Interval Clamp",
              "Value": 10,
              "UnitOfMeasure": "h"
            }
          ],
          "Enabled": true,
          "Log": false
        }
      ]

Datasource Debounce

There are situations where calculations and updates to the UI are taking too long due to uncertain data near-now being constantly requested by calculations and user behavior, such as a frequently auto-updating signal or conditions.

Multiple requests for the same data can decrease Seeq and the datasource performance. To prevent this issue, you can apply a debounce duration property to each affected signal or condition in the datasource. This property ensures that Seeq does not retrieve the same items from the datasource within the specified duration.

The example below will apply a 1 minute debounce duration for all signals and conditions from a datasource.

"Transforms": [
    {
        "Inputs": [
            {
                "Property": "Name",
                "Value": ".*"
            }
        ],
        "Outputs": [
            {
                "Property": "Debounce Duration",
                "Value": 1,
                "UnitOfMeasure": "minutes"
            }
        ],
        "Enabled": true,
        "Log": false
    }
]