How to extract Historical Data from a Drift device¶
PANDA|Drift keeps a history of input data that might be images from CV cameras, sensor data or metrics. In this tutorial you will learn:
- How to connect a Drift device and explore topics
- How to download and parse a Drift Package
- How to extract an image from a CV camera
- How to extract time series data from an acceleration sensor
- Hot to get metrics
Connect and explore topics¶
To connect to a device, you need to know its hostname or IP address and password:
import os
import asyncio
from drift_client import DriftClient
loop = (
asyncio.new_event_loop()
) # normally we don't need a loop (only for Jupiter and async code)
drift_client = DriftClient("drift-dev2.local", os.getenv("DRIFT_PASSWORD"), loop=loop)
print(drift_client.get_topics())
/Users/atimin/Projects/panda-official/DriftPythonClient/venv/lib/python3.9/site-packages/drift_client/drift_client.py:83: RuntimeWarning: coroutine 'Client.info' was never awaited self._blob_storage = MinIOClient( RuntimeWarning: Enable tracemalloc to get the object allocation traceback
['acc-10', 'acc-11', 'acc-12', 'acc-5', 'acc-6', 'acc-7', 'acc-8', 'acc-9', 'energy-acc-10', 'energy-acc-11', 'energy-acc-12', 'energy-acc-5', 'energy-acc-6', 'energy-acc-7', 'energy-acc-8', 'energy-acc-9', 'opcua']
You can see that the Drift device has few topics. Unfortunately, the device doesn't provide information about the type, and you should know it in advance. Here we have 8 acceleration signals (time series), 8 energy distributions (metrics of accelerations) and one camera (images).
Download and parse a Drift package¶
When you know a topic name you can load some batch of packages for a time interval. Let's take packages of camera data for the last 5 seconds:
from datetime import datetime, timedelta
package_names = drift_client.get_package_names(
"acc-5", datetime.utcnow() - timedelta(seconds=5), datetime.utcnow()
)
print(f"Found {len(package_names)} packages")
print(package_names)
Found 5 packages ['acc-5/1676296193326.dp', 'acc-5/1676296194326.dp', 'acc-5/1676296195327.dp', 'acc-5/1676296196327.dp', 'acc-5/1676296197327.dp']
We should take ~5 packages because we capture an image each second. Let's download one from our object storage and parse it:
pkg = drift_client.get_item(package_names[0])
print(f"Download package id={pkg.package_id} status={pkg.status_code}")
KeyboardInterrupt
All packages have at least an ID
and status code
. The package ID is unique for a topic, it is a timestamp when an event in the system happened. If you know that topic 'A' has metrics for data from topic 'B' they will have the same IDs, so that you can align data from different topics that belong to the same event. A package has the status code 0 if it has valid data.
Extract an image from a CV camera¶
Because we download a Drift package it always contains WaveletBuffer. Let's take it and play a bit with:
buffer = pkg.as_buffer()
print(buffer)
From the previous step, we see that the buffer has some meta information about itself, and we can see that it contains a fullHD 3-channeled picture. We can extract it as a numpy array and encode into a JPEG image:
from wavelet_buffer.img import RgbJpeg, GrayJpeg
from IPython.display import Image
pic = buffer.compose(
scale_factor=1
) # we may use scale factor and restore 2^scaler_factor times smaller version of image
print(f"Scaled image {pic.shape}")
jpeg = RgbJpeg().encode(pic)
Image(data=jpeg)
Extract time series from an acceleration sensor¶
We can also use the wavelet decomposition for time series data when we have a very high sample rate (vibration or sound e.g). In our demo system we have 8 acceleration sensors acc-{0..7}
. Let's take one of them and see what is inside:
package_names = drift_client.get_package_names(
"acc-1", datetime.utcnow() - timedelta(seconds=5), datetime.utcnow()
)
pkg = drift_client.get_item(package_names[0])
buffer = pkg.as_buffer()
print(buffer)
The buffer contains a timeseries with ~48k points, we can show its scaled version:
signal = buffer.compose(scale_factor=7)
print(f"Scaled size {signal.shape}")
import matplotlib.pyplot as plt
plt.plot(signal)
plt.show()
Get metrics¶
The PANDA|Drift platform keep a history of metrics separated from input data. So, a user should use a different API to get them.
In our example, we have energy distributions (like frequency spectrums) of the acceleration senors as metrics. We can use the get_metrics
method to download them:
metrics = drift_client.get_metrics(
"energy-distr-1",
start=datetime.utcnow() - timedelta(seconds=3),
stop=datetime.utcnow(),
)
print(metrics)
Here is a short example of how to plot a bar diagram of an energy distribution.
energies = drift_client.get_metrics(
"energy-distr-1",
start=datetime.utcnow() - timedelta(seconds=5),
stop=datetime.utcnow(),
names=["d1", "d2", "d3", "d4", "d5", "d6"],
)
as_dict = dict(energies[0])
as_dict.pop("time")
plt.bar(as_dict.keys(), as_dict.values())
plt.show()