Performance Considerations

Overview

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. However, there are some important factors that could determine the end user experience within Seeq.

Hot Cache vs Cold

Azure Data Explorer cache provides a granular cache policy that customers can use to differentiate between: hot data cache and cold data cache. Azure Data Explorer cache uses 95% of the local SSD disk to keep all data that falls into the hot data cache category. If the cache policy requires more disk space than the available local SSD disk, the most recent data will preferentially be kept in the cache. The remaining 5% of the local SSD space is used to hold data that isn't categorized as hot.

Understanding the implication of the hot cache policy is extremely important to determine any performance impacts of querying the data with Seeq.

For commonly accessed windows of data a Hot window policy could be employed https://learn.microsoft.com/en-us/azure/data-explorer/hot-windows

More information here: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/cachepolicy

Seeq caching can be leveraged to reduce the need to query the same range of data for the same signal repeatedly.

Materialized Views

When building asset hierarchies from denormalized tables it is often advisable to pre-compute the distinct asset paths. To see how to leverage a materialized view to access metadata information see here: Using Materialized Views to Increase Indexing Performance.