Introduction to Datasources

Overview

Within the context of Seeq, a datasource refers to a repository of data. This includes both external time series data stores (such as historians, databases, and data lakes) and internal data stores (such as user-generated calculations and Data Lab generated data).

A datasource connection refers to a connection between Seeq and an external datasource. Seeq works directly with authoritative sources in the existing data architecture and does not migrate your data into another silo. This means that Seeq avoids unnecessary data duplication and maintains your datasources as the single version of truth.

Connections to datasources are established via connectors. Connectors are responsible for converting the request for data from Seeq into a request from the datasource and the resulting data into a data model that Seeq understands. The performance of data retrieval is highly dependent on the performance and configuration of the underlying data repository.

Connectors are hosted by agents. More information about agents can be found in Installing a Remote Agent.

Data Types

Connectors can access different datatypes from datasources:

Signals	A series of timestamp/value pairs
Scalars	A constant over time
Conditions	A collection of events between two times
Assets	Items that contain other items
Asset Trees	A set of relationships that define a hierarchical structure of Assets, Signals, Conditions, and Scalars

Indexing

Data available from a datasource is identified through the indexing process. This process uses the configuration of the connection to identify all the data the connector can access. Metadata is collected in the form of properties. These properties are stored in Seeq and used to discover/contextualize data during analysis. There are general properties listed here (insert link). In addition to the general properties, each connector may add properties automatically or through configuration. The properties that are added automatically will be listed on each datasource page.

Request Handling

Once indexed datasources provide data on request. The following diagram shows how a request flows from Seeq to a datasource.

The request is initiated by a user or an application accessing the API
Seeq determines what data is required to fulfill the request
Seeq checks the cache to see if this data over this range has been requested previously
Seeq requests the data from the connector for any uncached portion of the data
The connector converts the request into a datasource call
The connector retrieves the results and converts them into a Seeq data type
Seeq caches the response for future queries and responds to the request

Configuration

Connectors to datasources can be configured either in the Datasource Administration UI or a JSON file on the agent. When configuring via the Datasource Administration UI, each connection to a datasource is shown as a separate card. When configuring via the file on the agent, all the connections are in the same file.

The screenshot below shows two datasources from the Example Connector.

An equivalent Connector JSON file would look like the following:

CODE

{
  "Version" : "com.seeq.link.connectors.exampledata.config.ExampleConnectorConfigV1",
  "Connections" : [ {
    "Name" : "Connection 1",
    "Id" : "72a32436-561f-41e8-b5d2-aa11cba361f3",
    "Enabled" : true,
    "Indexing" : {
      "Frequency" : "1w",
      "OnStartupAndConfigChange" : false,
      "Next" : "2023-06-06T16:44:27.851287300Z"
    },
    "Transforms" : null,
    "MaxConcurrentRequests" : null,
    "MaxResultsPerRequest" : null,
    "ConfigParam1" : true,
    "ConfigParam2" : "Value"
  }, {
    "Name" : "Connection 2",
    "Id" : "db443a51-fe73-4a9c-af22-96050dcbb2a0",
    "Enabled" : true,
    "Indexing" : {
      "Frequency" : "1w",
      "OnStartupAndConfigChange" : false,
      "Next" : "2023-06-06T16:44:27.851287300Z"
    },
    "Transforms" : null,
    "MaxConcurrentRequests" : null,
    "MaxResultsPerRequest" : null,
    "ConfigParam1" : true,
    "ConfigParam2" : "Value"
  } ]
}

Every connector has common parameters that can be found in the Base Datasource Configuration Reference.

Datasource Authentication and Authorization

An account will have to be created on the datasource for Seeq access. This account should be authorized to all the data that you would like Seeq to be able to access. Access to this data can be managed within Seeq through the application of datasource permissions