I'm using Qt 5.11.1 with Qt OPC Ua and the Open62541 backend to create an OPC Client application.
Is it at all possible to make a request for historical data with the Qt OPC UA module? E.g. Get the values for this variable (node) between these two times.
My server application has this functionality (FreeOpcUa) as I can set variables to be 'historized' and view previously stored values. But I cannot see an obvious solution to access this data easily on the client side.
At the moment I'm considering exposing a function on my server for each variable which would take in a start and end timestamp and manually gather the values and format them into a string or some object for the client to use.
Would anyone have any ideas or thoughts on a better way to do this?
I'm not overly familiar with OPC-UA or Qt so may just be missing something obvious.
To use the OPC UA History Feature, both your OPC UA Client and Server shall supports the HistoryRead/HistoryWrite Services.
I don't know the status of the feature for your Server but for your Client (Open62541) those Services are not completely functionnal yet. Check the FEATURES document from their GitHub here
Apparently those should be fully functionnal within the next 0.4 Release.
The Freeopcua-Server supports historization (https://python-opcua.readthedocs.io/en/latest/server.html).
You have to enable historization per node (i.e. per variable you want to historize):
historize_node_data_change(node, period=datetime.timedelta(7), count=0)
Start historizing supplied nodes
Args:
node: node or list of nodes that can be historized
(variables/properties)
period: time delta to store the history; older
data will be deleted from the storage
count: number of changes to
store in the history
e.g. if you want to serve the history of the temperature you have to use "server.historize_node_data_change(Temp, period=datetime.timedelta(7), count=0)" (after the server was started):
[Python]:
from opcua import Server
from random import randint
import datetime
import time
server = Server()
server.set_endpoint("opc.tcp://192.168.178.20:443")
addspace = server.register_namespace("OPCUA_BurkhardsTemperatureSensor")
node = server.get_objects_node()
Param = node.add_object(addspace, "Thermometer_1")
Temp = Param.add_variable(addspace, "Temperature", 0)
Temp.set_writable()
Time = Param.add_variable(addspace, "Time", 0)
Time.set_writable()
SerialNr = Param.add_variable(addspace, "SerialNr.", "2323784628346")
server.start()
server.historize_node_data_change(Temp, period=datetime.timedelta(7), count=0)
while True:
Temperature = randint (10,50)
TIME = datetime.datetime.now()
print (Temperature,TIME)
Temp.set_value(Temperature)
Time.set_value(TIME)
time.sleep (2)
Related
Consider this example - you need to load table1 from source database, do some generic transformations (like convert time zones for timestamped columns) and write resulting data into Snowflake. This is an easy one and can be implemented using 3 dagster ops.
Now, imagine you need to do the same thing but with 100s of tables. How would you do it with dagster? Do you literally need to create 100 jobs/graphs? Or can you create one job, that will be executed 100 times? Can you throttle how many of these jobs will run at the same time?
You have a two main options for doing this:
Use a single job with Dynamic Outputs:
With this setup, all of your ETLs would happen in a single job. You would have an initial op that would yield a DynamicOutput for each table name that you wanted to do this process for, and feed that into a set of ops (probably organized into a graph) that would be run on each individual DynamicOutput.
Depending on what executor you're using, it's possible to limit the overall step concurrency (for example, the default multiprocess_executor supports this option).
Create a configurable job (I think this is more likely what you want)
from dagster import job, op, graph
import pandas as pd
#op(config_schema={"table_name": str})
def extract_table(context) -> pd.DataFrame:
table_name = context.op_config["table_name"]
# do some load...
return pd.DataFrame()
#op
def transform_table(table: pd.DataFrame) -> pd.DataFrame:
# do some transform...
return table
#op(config_schema={"table_name": str})
def load_table(context, table: pd.DataFrame):
table_name = context.op_config["table_name"]
# load to snowflake...
#job
def configurable_etl():
load_table(transform_table(extract_table()))
# this is what the configuration would look like to extract from table
# src_foo and load into table dest_foo
configurable_etl.execute_in_process(
run_config={
"ops": {
"extract_table": {"config": {"table_name": "src_foo"}},
"load_table": {"config": {"table_name": "dest_foo"}},
}
}
)
Here, you create a job that can be pointed at a source table and a destination table by giving the relevant ops a config schema. Depending on those config options, (which are provided when you create a run through the run config), your job will operate on different source / destination tables.
The example shows explicitly running this job using python APIs, but if you're running it from Dagit, you'll also be able to input the yaml version of this config there. If you want to simplify the config schema (as it's pretty nested as shown), you can always create a Config Mapping to make the interface nicer :)
From here, you can limit run concurrency by supplying a unique tag to your job, and using a QueuedRunCoordinator to limit the maximum number of concurrent runs for that tag.
i am currently working on school project the goal is whenever i say 'pi'(machine name) 'sing for me'(google = assistant)
since it would be easy if i use hotword but it is currently unavailable and doesn't support our language
at env/lib/googleassistant/--/pushtotalk.py 457 line
wait_for_user_trigger = not once
while True:
if wait_for_user_trigger:
->#click.pause(info='Press Enter to send a new request...')
->time.sleep(1)
continue_conversation = assistant.assist()
# wait for user trigger if there is no follow-up turn in
# the conversation.
wait_for_user_trigger = not continue_conversation
# If we only want one conversation, break.
if once and (not continue_conversation):
break
i changed here(->) so that every time it is on state but i cannot find how to call only if it's name is called
could you help me?
and could you tell me if i work this on (env) how can i import GPIO i need to blink led
I was successful with snowboy.
example here:
https://busy.org/#neavvy/google-assistant-on-raspberry-or-part-3-custom-wake-word
While running pySpark SQL pipelines via Airflow I am interested in getting out some business stats like:
source read count
target write count
sizes of DFs during processing
error records count
One idea is to push it directly to the metrics, so it will gets automatically consumed by monitoring tools like Prometheus. Another idea is to obtain these values via some DAG result object, but I wasn't able to find anything about it in docs.
Please post some at least pseudo code if you have solution.
I would look to reuse Airflow's statistics and monitoring support in the airflow.stats.Stats class. Maybe something like this:
import logging
from airflow.stats import Stats
PYSPARK_LOG_PREFIX = "airflow_pyspark"
def your_python_operator(**context):
[...]
try:
Stats.incr(f"{PYSPARK_LOG_PREFIX}_read_count", src_read_count)
Stats.incr(f"{PYSPARK_LOG_PREFIX}_write_count", tgt_write_count)
# So on and so forth
except:
logging.exception("Caught exception during statistics logging")
[...]
Using graphite/Grafana to record the sizes of all collections in a mongodb instance. I wrote a simple (WIP) python script to do so:
#!/usr/bin/python
from pymongo import MongoClient
import socket
import time
statsd_ip = '127.0.0.1'
statsd_port = 8125
# create a udp socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client = MongoClient(host='12.34.56.78', port=12345)
db = client.my_DB
# get collection list each runtime
collections = db.collection_names()
sizes = {}
# main
while (1):
# get collection size per name
for collection in collections:
sizes[collection] = db.command('collstats', collection)['size']
# write to statsd
for size in sizes:
MESSAGE = "collection_%s:%d|c" % (size, sizes[size])
sock.sendto(MESSAGE, (statsd_ip, statsd_port))
time.sleep(60)
This properly shows all of my collection sizes in grafana. However, I want to get a rate of change on these sizes, so I build the following graphite query in grafana:
derivative(statsd.myHost.collection_myCollection)
And the graph shows up totally blank. Any ideas?
FOLLOW-UP: When selecting a time range greater than 24h, all data similarly disappears from the graph. Can't for the life of me figure out that one.
Update: This was due to the fact that my collectd was configured to send samples every second. The statsd plugin for collectd, however, was receiving data every 60 seconds, so I ended up with None for most data points.
I discovered this by checking the raw data in Graphite by appending &format=raw to the end of a graphite-api query in a browser, which gives you the value of each data point as a comma-separated list.
The temporary fix for this was to surround the graphite query with keepLastValue(60). This however creates a stair-step graph, as the value for each None (60 values) becomes the last valid value within 60 steps. Graphing a derivative of this then becomes a widely spaced sawtooth graph.
In order to fix this, I will probably go on to fix the flush interval on collectd or switch to a standalone statsd instance and configure as necessary from there.
I currently have an application using flask-sqlalchemy. My model is connected to a postgresql database, and now I would like to write unit tests (using nose). I was told to use SQLite to create a new database for testing, and after a lot of searching (and looking at the texting section on the flask-sqlalchemy website) I'm still confused as how to do it. Each class in my model.py looks something like the following:
db = SQLAlchemy(app)
class Prod(db.Model):
__tablename__ = 'prod'
id = db.Column(db.Integer, primary_key=True)
desc = db.Column(db.String)
def __init__(self, id, desc):
self.id = id
self.desc = desc
My config.py:
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres://name:pass#server/db
and I would like to test my insert functions in a new file by setting up and tearing down a new database for each test. If anyone can give me some example code that would be great. Thanks!
I can't answer your specific question, but will provide some general advise:
You will find that setting up and tearing down the complete database for each test will be too slow. Imagine in the future when you might have hundreds of tests or even thousands.
The approach we take is:
For testing purposes we have a database populated with test data. We have a script which creates a fresh database and populates it with this test data.
We run this script prior to running our test suite. All tests can assume this data exists.
Each test may create additional records if necessary, but it is their responsibility to undo any changes they make (delete new records, undo changes) - in order words to leave the database in the same state as it was before the test began. This prevents tests from interfering with each other.
In a project I manage we have a test suite of 1070 tests which runs in about 5 minutes using this approach.
What if we had taken your approach? Let's assume that 50% of these tests actually exercise the database (and need a fresh reload). That's 1070 * .50 * 20 seconds for the reload / 3600 = 2.97 hours. Oops - that's far too slow to be useful.
Even at a much smaller scale though, you'll be much happier if your test suite runs in 1 minute instead of 20 minutes.