{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "3309313d-a902-4120-b145-0001f4e55467", "metadata": {}, "outputs": [], "source": [ "import math\n", "import re\n", "import textwrap\n", "from datetime import datetime, timedelta\n", "from seeq.sdk import *\n", "import pandas as pd\n", "from IPython.display import display, HTML" ] }, { "cell_type": "markdown", "id": "893dbd02", "metadata": {}, "source": [ "# Monitoring your Data Usage\n", "\n", "As a Seeq admin or champion of its use within your organization, you may wish to be alerted when significant usage occurs over a particular time period. This can highlight the value that users are realizing by using Seeq but can also highlight areas where calculations could be optimized to be more efficient (and therefore faster).\n", "\n", "This notebook uses the Seeq SDK to look at recent usage (the same as what is shown in the **Usage** tab of the Administration page) and notify via email if a (customizable) threshold is exceeded." ] }, { "cell_type": "markdown", "id": "ae063011-86df-44af-83b3-1c487984ca35", "metadata": {}, "source": [ "## Your preferences\n", "\n", "Change the values you see in the code block below to customize this script to your preferences." ] }, { "cell_type": "code", "execution_count": 2, "id": "b6cee990-b3e9-4acb-b845-1cbd983e3040", "metadata": {}, "outputs": [], "source": [ "# How far back do you want to look each time this script runs?\n", "look_back = timedelta(days=7)\n", "\n", "# For what threshold do you want to notify the stakeholders?\n", "threshold = '30 GB'\n", "\n", "# Who do you want to notify?\n", "emails_and_names = [\n", " ('jane.doe@mycompany.com', 'Jane Doe')\n", "]\n", "\n", "# What's the email subject line?\n", "email_subject = 'Seeq Data Usage Report'" ] }, { "cell_type": "markdown", "id": "b0e3e99f-c5da-45a3-aa44-4ef3ea11057c", "metadata": {}, "source": [ "## Utility functions\n", "\n", "We define here some utility functions to help us make things look good in the email." ] }, { "cell_type": "code", "execution_count": 3, "id": "f64ac26c-9755-4538-8be7-68dab6e42c3e", "metadata": {}, "outputs": [], "source": [ "def humanized(bytes_processed) -> str:\n", " \"\"\"\n", " Turns a raw bytes number into a human-readable representation with units like MB or GB.\n", " \"\"\"\n", " if bytes_processed == 0:\n", " return '0 B'\n", "\n", " suffixes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB']\n", " i = int(math.floor(math.log(bytes_processed, 1000)))\n", " p = math.pow(1000, i)\n", " s = int(bytes_processed / p)\n", " return '%s %s' % (s, suffixes[i])\n", "\n", "def to_bytes(s: str) -> int:\n", " \"\"\"\n", " Opposite of humanized()\n", " \"\"\"\n", " matcher = re.match(r'([\\d\\.]+)\\s*(\\w?)B', s)\n", " num = float(matcher.group(1))\n", " order = matcher.group(2).upper()\n", " unit_powers = {'': 0, 'K': 1, 'M': 2, 'G': 3, 'T': 4, 'P': 5, 'X': 6}\n", " power = unit_powers[order]\n", " return int(num * math.pow(1000, power))\n", "\n", "def link(row):\n", " \"\"\"\n", " Creates an HTML link if the Source URL is available in the DataFrame row.\n", " \"\"\"\n", " if row[\"Source URL\"]:\n", " return f'{row[\"Source\"]}'\n", " else:\n", " return row[\"Source\"]" ] }, { "cell_type": "markdown", "id": "8fc77fbe-6939-4d23-91d1-0bce0041e506", "metadata": {}, "source": [ "## Query the usage\n", "\n", "Seeq exposes a Usage API, which is the same API that powers the _Usage_ tab in the Seeq administration page.\n", "\n", "We'll use it here to query for the usage over the time period that was specified at the top." ] }, { "cell_type": "code", "execution_count": 4, "id": "337575f8-775c-41e9-afdf-e01a05fe1929", "metadata": {}, "outputs": [], "source": [ "usage_api = UsageApi(spy.client)\n", "\n", "start = (datetime.utcnow() - look_back).isoformat() + 'Z'\n", "end = datetime.utcnow().isoformat() + 'Z'\n", "\n", "usage_output_list = usage_api.get_usage(\n", " start_time=start,\n", " end_time=end,\n", " aggregate_by=['User', 'Source']\n", ")" ] }, { "cell_type": "markdown", "id": "63f00056-6714-4f24-814e-9317d75e9417", "metadata": {}, "source": [ "## Create a DataFrame\n", "\n", "Let's create a DataFrame of the output so it's easy to manipulate the information." ] }, { "cell_type": "code", "execution_count": 5, "id": "c3e39813-7e08-46a0-a95e-c480415a2512", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
UserSourceSource URLBytesData Processed
0Che TseAutoscaling - every 2min at various hourshttps://develop.seeq.dev/view/worksheet/0EE5D6...918086509504918 GB
1Che TseAutoscaling - every 3min at various hourshttps://develop.seeq.dev/view/worksheet/0EE5D6...546826414512546 GB
2Che TseAutoscaling - every 5min at various hourshttps://develop.seeq.dev/view/worksheet/0EE5D6...419919067232419 GB
3Che TseAutoscaling - every 10 min starting at :00https://develop.seeq.dev/view/worksheet/0EE5D6...319633503664319 GB
4AF Data ReferenceN/A13589569281 GB
5John CoxCLPM Demo June 2022 - Copy - Site CLPMhttps://develop.seeq.dev/view/worksheet/0EDB6F...258146432258 MB
6Sean TropsaEnterprise Demo - Daily Monitoring Viewhttps://develop.seeq.dev/view/worksheet/E71F77...176458688176 MB
7Sepide ZakeriProcess Health Solution Dashboard - Seeq ML en...https://develop.seeq.dev/view/worksheet/5DB566...125598672125 MB
8Mark SuchomelHydrogen Production via SMR - GREET Model Emis...https://develop.seeq.dev/view/worksheet/0EE476...116739136116 MB
9Sepide ZakeriProcess Health Solution Dashboard - Seeq ML en...https://develop.seeq.dev/view/worksheet/5DB566...6023520060 MB
\n", "
" ], "text/plain": [ " User Source \\\n", "0 Che Tse Autoscaling - every 2min at various hours \n", "1 Che Tse Autoscaling - every 3min at various hours \n", "2 Che Tse Autoscaling - every 5min at various hours \n", "3 Che Tse Autoscaling - every 10 min starting at :00 \n", "4 AF Data Reference N/A \n", "5 John Cox CLPM Demo June 2022 - Copy - Site CLPM \n", "6 Sean Tropsa Enterprise Demo - Daily Monitoring View \n", "7 Sepide Zakeri Process Health Solution Dashboard - Seeq ML en... \n", "8 Mark Suchomel Hydrogen Production via SMR - GREET Model Emis... \n", "9 Sepide Zakeri Process Health Solution Dashboard - Seeq ML en... \n", "\n", " Source URL Bytes \\\n", "0 https://develop.seeq.dev/view/worksheet/0EE5D6... 918086509504 \n", "1 https://develop.seeq.dev/view/worksheet/0EE5D6... 546826414512 \n", "2 https://develop.seeq.dev/view/worksheet/0EE5D6... 419919067232 \n", "3 https://develop.seeq.dev/view/worksheet/0EE5D6... 319633503664 \n", "4 1358956928 \n", "5 https://develop.seeq.dev/view/worksheet/0EDB6F... 258146432 \n", "6 https://develop.seeq.dev/view/worksheet/E71F77... 176458688 \n", "7 https://develop.seeq.dev/view/worksheet/5DB566... 125598672 \n", "8 https://develop.seeq.dev/view/worksheet/0EE476... 116739136 \n", "9 https://develop.seeq.dev/view/worksheet/5DB566... 60235200 \n", "\n", " Data Processed \n", "0 918 GB \n", "1 546 GB \n", "2 419 GB \n", "3 319 GB \n", "4 1 GB \n", "5 258 MB \n", "6 176 MB \n", "7 125 MB \n", "8 116 MB \n", "9 60 MB " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "usage_df = pd.DataFrame([{\n", " 'User': c.identity,\n", " 'Source': c.source_label,\n", " 'Source URL': f'{spy.session.public_url}{c.source_url}' if c.source_url is not None else '',\n", " 'Bytes': c.bytes\n", "} for c in usage_output_list.content])\n", "\n", "usage_df = usage_df.sort_values(by=['Bytes'], ascending=False)\n", "usage_df['Data Processed'] = usage_df['Bytes'].apply(humanized)\n", "usage_df.head(10)" ] }, { "cell_type": "markdown", "id": "b62f4bc4-8ce2-444e-8f60-6797f7e3f9da", "metadata": {}, "source": [ "## Filter down\n", "\n", "We only care about the rows that exceed our threshold, so filter out any other rows." ] }, { "cell_type": "code", "execution_count": 6, "id": "2553ebed-2b46-403c-9a08-688c4ec3391e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
UserSourceSource URLBytesData Processed
0Che TseAutoscaling - every 2min at various hourshttps://develop.seeq.dev/view/worksheet/0EE5D6...918086509504918 GB
1Che TseAutoscaling - every 3min at various hourshttps://develop.seeq.dev/view/worksheet/0EE5D6...546826414512546 GB
2Che TseAutoscaling - every 5min at various hourshttps://develop.seeq.dev/view/worksheet/0EE5D6...419919067232419 GB
3Che TseAutoscaling - every 10 min starting at :00https://develop.seeq.dev/view/worksheet/0EE5D6...319633503664319 GB
\n", "
" ], "text/plain": [ " User Source \\\n", "0 Che Tse Autoscaling - every 2min at various hours \n", "1 Che Tse Autoscaling - every 3min at various hours \n", "2 Che Tse Autoscaling - every 5min at various hours \n", "3 Che Tse Autoscaling - every 10 min starting at :00 \n", "\n", " Source URL Bytes \\\n", "0 https://develop.seeq.dev/view/worksheet/0EE5D6... 918086509504 \n", "1 https://develop.seeq.dev/view/worksheet/0EE5D6... 546826414512 \n", "2 https://develop.seeq.dev/view/worksheet/0EE5D6... 419919067232 \n", "3 https://develop.seeq.dev/view/worksheet/0EE5D6... 319633503664 \n", "\n", " Data Processed \n", "0 918 GB \n", "1 546 GB \n", "2 419 GB \n", "3 319 GB " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exceeds_threshold_df = usage_df[usage_df['Bytes'] > to_bytes(threshold)]\n", "exceeds_threshold_df" ] }, { "cell_type": "markdown", "id": "32765594-6292-4b7b-93fa-28e1ff21d356", "metadata": {}, "source": [ "## Prepare an email\n", "\n", "We want to populate the email with the relevant information." ] }, { "cell_type": "code", "execution_count": 7, "id": "43146498-a0e9-48b0-a88f-6d804b044eb1", "metadata": {}, "outputs": [], "source": [ "look_back_days = float(look_back.total_seconds()) / 86400.0\n", "table_rows = '\\n'.join([\n", " f'{row[\"User\"]}{link(row)}{row[\"Data Processed\"]}'\n", " for _, row in exceeds_threshold_df.iterrows()])\n", "email_body = textwrap.dedent(f\"\"\"\n", "

Data usage for a particular User and Source exceeded threshold of {threshold}:

\n", " \n", " \n", " {table_rows}\n", "
UserSourceData Processed
\n", "

Log into Seeq and go to the Usage tab of the Administration page if desired.

\n", "

Time period: Last {look_back_days} day(s)

\n", "\"\"\").strip()" ] }, { "cell_type": "markdown", "id": "adc887b7-d68d-4779-ae62-779f4e1f8e34", "metadata": {}, "source": [ "## Send the email (if necessary)\n", "\n", "If there was one or more rows in the filtered DataFrame, send an email." ] }, { "cell_type": "code", "execution_count": 8, "id": "2bb38244-eedd-4ca4-8187-a0536b652a1c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'status_message': 'The email was accepted'}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "notifier_api = NotifierApi(spy.client)\n", "\n", "email_result = ''\n", "if len(exceeds_threshold_df) > 0:\n", " email_result = notifier_api.send_email(body=SendEmailInputV1(\n", " to_emails=[\n", " SendEmailContactV1(email=pair[0], name=pair[1])\n", " for pair in emails_and_names\n", " ],\n", " subject=email_subject,\n", " content=email_body\n", " ))\n", " \n", "email_result" ] }, { "cell_type": "markdown", "id": "522e617e", "metadata": {}, "source": [ "### Sending the email via Sendgrid\n", "\n", "If you are not using Seeq's SaaS service, the `NotifierApi` is likely not set up. The cell below contains code that sends the email via Sendgrid, a common and easy-to-use emailer service. You will need a Sendgrid API key which, depending on current pricing plans, may require a purchase.\n", "\n", "You will need to `pip install sendgrid` to bring the Sendgrid API into this Data Lab project.\n", "\n", "To use this Sendgrid option, copy the code below into a cell and execute it. You will likely also want to delete the cell above that attempts to send via Seeq's `NotifierApi`.\n", "\n", "**Note: Your Seeq Data Lab Server will need to have internet access to the Sendgrid service.**" ] }, { "cell_type": "markdown", "id": "e684f9a2", "metadata": {}, "source": [ "```\n", "pip install sendgrid\n", "```" ] }, { "cell_type": "markdown", "id": "5b27f676", "metadata": {}, "source": [ "```\n", "from sendgrid import SendGridAPIClient, Attachment, FileContent, FileType, FileName, Disposition\n", "from sendgrid.helpers.mail import Mail\n", "```" ] }, { "cell_type": "markdown", "id": "986faf85", "metadata": {}, "source": [ "```\n", "def send_mail_via_sendgrid(sendgrid_api_key: str, sender: str, recipients: list, subject: str,\n", " html: str, attachment: str = None):\n", " message = Mail(\n", " from_email=sender,\n", " to_emails=recipients,\n", " subject=subject,\n", " html_content=html)\n", "\n", " if attachment is not None:\n", " with open(attachment, 'rb') as f:\n", " data = f.read()\n", " f.close()\n", " encoded_file = base64.b64encode(data).decode()\n", "\n", " mime_type, _ = mimetypes.guess_type(attachment)\n", " attached_file = Attachment(\n", " FileContent(encoded_file),\n", " FileName(os.path.basename(attachment)),\n", " FileType(mime_type),\n", " Disposition('attachment')\n", " )\n", "\n", " message.attachment = attached_file\n", "\n", " SendGridAPIClient(sendgrid_api_key).send(message)\n", "```" ] }, { "cell_type": "markdown", "id": "1e269f7a", "metadata": {}, "source": [ "```\n", "# This is a bogus key, just for demostration purposes. It will need to be replaced by a real key.\n", "# Log into your Sendgrid account and navigate to Settings > API Keys to create a new key.\n", "sendgrid_api_key = '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'\n", "\n", "send_mail_via_sendgrid(\n", " sendgrid_api_key,\n", " 'jane.doe@mycompany.com',\n", " ['john.doe@mycompany.com'],\n", " email_subject,\n", " email_body\n", ")\n", "```" ] }, { "cell_type": "markdown", "id": "16dda690", "metadata": {}, "source": [ "## Run this notebook on a schedule\n", "\n", "You likely want to have this notebook run in the background, on a schedule. Modify the schedule text below as desired." ] }, { "cell_type": "code", "execution_count": 9, "id": "fb4b3311-056b-4c69-8022-d50afd1a54f5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Scheduled the notebook Data Usage Monitoring.ipynb successfully.
Current context is INTERACTIVE. The jobs DataFrame was stored to _Job DataFrames/Data Usage Monitoring.pkl
ScheduleScheduledNext Run
0every day at 6:00amAt 06:00 AM2023-10-03 06:00:00 PDT
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ScheduleScheduledNext Run
0every day at 6:00amAt 06:00 AM2023-10-03 06:00:00 PDT
\n", "
" ], "text/plain": [ " Schedule Scheduled Next Run\n", "0 every day at 6:00am At 06:00 AM 2023-10-03 06:00:00 PDT" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "spy.jobs.schedule('every day at 6:00am')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.18" } }, "nbformat": 4, "nbformat_minor": 5 }