Retrieve current metric value from graphite - graphite

Suppose I have a metric named a.b.c.count. I am trying to write a python script which reads the latest value of the metric a.b.c.count in graphite.
I went through the docs and figured out that we can use curl to retrieve metrics from graphite using functions http://graphite.readthedocs.org/en/0.9.13-pre1/functions.html.
But still unable to figure out how to achieve the same.

I haven't seen a way to ask Graphite for a single value, but you can ask for a summary of values over a configurable period, and take the last one. (This is just for minimizing the data returned, you could as easily pull out the last value from any series in a given timeframe.) Example render parameters:
target=summarize(a.b.c.count,'1hour','last')&from=-1h&format=json
The JSON returned will look like this:
[{"target": "summarize(a.b.c.count, \"1hour\", \"last\")",
"datapoints": [[5.1333330000000004, 1442160000],
[5.5499989999999997, 1442163600]]}]
Here is a Python snippet to retrieve and parse this, using the 'requests' HTTP library
import requests
r = requests.get("http://graphite.yourdomain.com/render/?" +
"target=summarize(a.b.c.count,'1hour','last')&from=-1h&format=json")
print r.json()[0][u'datapoints'][-1][0]

Related

Logic App using HTTP Action to access and GET Jira ticket

So I followed this answer and it works fine:
https://stackoverflow.com/a/74981947/20829088
Provided URL:
https://<YOUR_DOMAIN>.atlassian.net/rest/api/2/search?jql=project=<PROJECTID>&fields=issue,status,name&startAt=0&maxResults=8000
However, it take a lot of time. So, if I want the url to check for specific ticket depending on created time and type of ticket. For example
I want ticket that is created within 15 days and that are NOT a sub-task.
so I tried something like this.
.....&fields=issue,summary,issuetype&created>=-15d&hierarchylevel=0
I'm not sure of how it should be written I just tried this and it doesn't work.
Here is the request result in JSON:
It should be earthier [subtask=false] OR [hierarchylevel=0] OR [name=Task]
After reproducing from my end, I could able to achieve this using Condition connector of logic apps. I have initialized an array variable first and then tried to append each item that satisfies the condition. Below is the flow of my logic app.
and then I used Parse Json to retrieve the required values for condition comparision
You can use the below Code view to reproduce the same in your environment.
{"definition":{"$schema":"https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#","actions":{"For_each":{"actions":{"Compose":{"inputs":"#items('For_each')?['fields']?['created']","runAfter":{},"type":"Compose"},"Condition":{"actions":{"Append_to_array_variable":{"inputs":{"name":"Array","value":"#items('For_each')"},"runAfter":{},"type":"AppendToArrayVariable"}},"expression":{"or":[{"greaterOrEquals":["#formatDateTime(outputs('Compose'),'yyyy-MM-dd')","#formatDateTime(addDays(utcNow(),-15),'yyyy-MM-dd')"]},{"equals":["#items('For_each')?['fields']?['issuetype']?['subtask']",false]},{"equals":["#items('For_each')?['fields']?['issuetype']?['hierarchyLevel']",0]},{"equals":["#items('For_each')?['fields']?['status']?['statusCategory']?['name']","Task"]}]},"runAfter":{"Compose":["Succeeded"]},"type":"If"}},"foreach":"#body('Parse_JSON')?['issues']","runAfter":{"Parse_JSON":["Succeeded"]},"type":"Foreach"},"HTTP":{"inputs":{"authentication":{"password":"<API_KEY>","type":"Basic","username":"<USERNAME>"},"method":"GET","uri":"https://jira#<ProjectName>.atlassian.net/rest/api/2/search?jql=project=<ProjectID>"},"runAfter":{},"type":"Http"},"Initialize_variable":{"inputs":{"variables":[{"name":"Array","type":"array"}]},"runAfter":{"HTTP":["Succeeded"]},"type":"InitializeVariable"},"Parse_JSON":{"inputs":{"content":"#body('HTTP')","schema":{"properties":{"expand":{"type":"string"},"issues":{"items":{"properties":{"expand":{"type":"string"},"fields":{"properties":{"aggregateprogress":{"properties":{"progress":{"type":"integer"},"total":{"type":"integer"}},"type":"object"},"aggregatetimeestimate":{},"aggregatetimeoriginalestimate":{},"aggregatetimespent":{},"assignee":{},"components":{"type":"array"},"created":{"type":"string"},"creator":{"properties":{"accountId":{"type":"string"},"accountType":{"type":"string"},"active":{"type":"boolean"},"avatarUrls":{"properties":{"16x16":{"type":"string"},"24x24":{"type":"string"},"32x32":{"type":"string"},"48x48":{"type":"string"}},"type":"object"},"displayName":{"type":"string"},"emailAddress":{"type":"string"},"self":{"type":"string"},"timeZone":{"type":"string"}},"type":"object"},"customfield_10001":{},"customfield_10002":{},"customfield_10003":{},"customfield_10004":{},"customfield_10005":{},"customfield_10006":{},"customfield_10007":{},"customfield_10008":{},"customfield_10009":{},"customfield_10010":{},"customfield_10014":{},"customfield_10015":{},"customfield_10016":{},"customfield_10017":{"type":"string"},"customfield_10018":{"properties":{"hasEpicLinkFieldDependency":{"type":"boolean"},"nonEditableReason":{"properties":{"message":{"type":"string"},"reason":{"type":"string"}},"type":"object"},"showField":{"type":"boolean"}},"type":"object"},"customfield_10019":{"type":"string"},"customfield_10020":{},"customfield_10021":{},"customfield_10022":{},"customfield_10023":{},"customfield_10024":{},"customfield_10025":{},"customfield_10026":{},"customfield_10027":{},"customfield_10028":{},"customfield_10029":{},"customfield_10030":{},"customfield_10033":{},"description":{},"duedate":{},"environment":{},"fixVersions":{"type":"array"},"issuelinks":{"type":"array"},"issuetype":{"properties":{"avatarId":{"type":"integer"},"description":{"type":"string"},"entityId":{"type":"string"},"hierarchyLevel":{"type":"integer"},"iconUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"self":{"type":"string"},"subtask":{"type":"boolean"}},"type":"object"},"labels":{"type":"array"},"lastViewed":{},"priority":{"properties":{"iconUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"self":{"type":"string"}},"type":"object"},"progress":{"properties":{"progress":{"type":"integer"},"total":{"type":"integer"}},"type":"object"},"project":{"properties":{"avatarUrls":{"properties":{"16x16":{"type":"string"},"24x24":{"type":"string"},"32x32":{"type":"string"},"48x48":{"type":"string"}},"type":"object"},"id":{"type":"string"},"key":{"type":"string"},"name":{"type":"string"},"projectTypeKey":{"type":"string"},"self":{"type":"string"},"simplified":{"type":"boolean"}},"type":"object"},"reporter":{"properties":{"accountId":{"type":"string"},"accountType":{"type":"string"},"active":{"type":"boolean"},"avatarUrls":{"properties":{"16x16":{"type":"string"},"24x24":{"type":"string"},"32x32":{"type":"string"},"48x48":{"type":"string"}},"type":"object"},"displayName":{"type":"string"},"emailAddress":{"type":"string"},"self":{"type":"string"},"timeZone":{"type":"string"}},"type":"object"},"resolution":{},"resolutiondate":{},"security":{},"status":{"properties":{"description":{"type":"string"},"iconUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"self":{"type":"string"},"statusCategory":{"properties":{"colorName":{"type":"string"},"id":{"type":"integer"},"key":{"type":"string"},"name":{"type":"string"},"self":{"type":"string"}},"type":"object"}},"type":"object"},"statuscategorychangedate":{"type":"string"},"subtasks":{"type":"array"},"summary":{"type":"string"},"timeestimate":{},"timeoriginalestimate":{},"timespent":{},"updated":{"type":"string"},"versions":{"type":"array"},"votes":{"properties":{"hasVoted":{"type":"boolean"},"self":{"type":"string"},"votes":{"type":"integer"}},"type":"object"},"watches":{"properties":{"isWatching":{"type":"boolean"},"self":{"type":"string"},"watchCount":{"type":"integer"}},"type":"object"},"workratio":{"type":"integer"}},"type":"object"},"id":{"type":"string"},"key":{"type":"string"},"self":{"type":"string"}},"required":["expand","id","self","key","fields"],"type":"object"},"type":"array"},"maxResults":{"type":"integer"},"startAt":{"type":"integer"},"total":{"type":"integer"}},"type":"object"}},"runAfter":{"Initialize_variable":["Succeeded"]},"type":"ParseJson"}},"contentVersion":"1.0.0.0","outputs":{},"parameters":{},"triggers":{"manual":{"inputs":{"schema":{}},"kind":"Http","type":"Request"}}},"parameters":{}}

Cannot create Temperature Data Source?

I'm trying to create the temperature data source via REST and it keeps on failing with:
Data type does not match well-known data type with the same name
The data I'm sending in is:
{"application":{...Same as other working samples....},"dataType":{"field":[{"name":"body_temperature","format":"floatPoint"},{"name":"measurement_location","format":"integer","optional":true}],"name":"com.google.body.temperature"},"name":"Temperature Feed","type":"raw","dataStreamId":"raw:com.google.body.temperature:XXX"}
Per https://developers.google.com/fit/datatypes/health#body_temperature , I am not seeing any reason this shouldn't work -- what am I doing wrong? Does temperature require special permission?
Thanks.
Eric
The issue is that measurement_location must be body_temperature_measurement_location in the REST request. After changing that, it now works.

Kronos API v2 - including multiple query parameters in a request

I am trying to access the Time Entries object via the Kronos API v2.
The documentation says that there are two required Query Parameters: start_date and end_date.
I am able to query the endpoint including one of the parameters at a time but am not able to enter both. And, I find the documentation quite lacking.
The root of the endpoint is:
https://secure.saashr.com/ta/rest/v2/companies/{cid}/time-entries
Here are things I have tried to suffix to the above endpoint:
?start_date=2019-11-01&end_date=2019-12-01
?start_date=2019-11-01|end_date=2019-12-01
?start_date=2019-11-01 end_date=2019-12-01
?start_date=2019-11-01?end_date=2019-12-01
?start_date=2019-11-01:end_date=2019-12-01
?filter=start_date:=:2019-11-01|end_date:=:2019-12-01
I also tried including quotes around the dates.
Everything results in some 400 level error when querying the API. With most of the above suffixes, it recognizes start_date but not the end_date. In this case, the error is:
{'code': 400, 'message': 'Missing required: end_date'}]
Note, above {cid} is replaced with the company's id.
In summary, how should I include two query parameters in the Kronos API?
The first option is correct.
https://secure.saashr.com/ta/rest/v2/companies/{cid}/time-entries?start_date=2019-11-01&end_date=2019-12-01
should work just fine.
Could you provide full URL you set in request?

Airflow Custom Metrics and/or Result Object with custom fields

While running pySpark SQL pipelines via Airflow I am interested in getting out some business stats like:
source read count
target write count
sizes of DFs during processing
error records count
One idea is to push it directly to the metrics, so it will gets automatically consumed by monitoring tools like Prometheus. Another idea is to obtain these values via some DAG result object, but I wasn't able to find anything about it in docs.
Please post some at least pseudo code if you have solution.
I would look to reuse Airflow's statistics and monitoring support in the airflow.stats.Stats class. Maybe something like this:
import logging
from airflow.stats import Stats
PYSPARK_LOG_PREFIX = "airflow_pyspark"
def your_python_operator(**context):
[...]
try:
Stats.incr(f"{PYSPARK_LOG_PREFIX}_read_count", src_read_count)
Stats.incr(f"{PYSPARK_LOG_PREFIX}_write_count", tgt_write_count)
# So on and so forth
except:
logging.exception("Caught exception during statistics logging")
[...]

Graphite Derivative shows no data

Using graphite/Grafana to record the sizes of all collections in a mongodb instance. I wrote a simple (WIP) python script to do so:
#!/usr/bin/python
from pymongo import MongoClient
import socket
import time
statsd_ip = '127.0.0.1'
statsd_port = 8125
# create a udp socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client = MongoClient(host='12.34.56.78', port=12345)
db = client.my_DB
# get collection list each runtime
collections = db.collection_names()
sizes = {}
# main
while (1):
# get collection size per name
for collection in collections:
sizes[collection] = db.command('collstats', collection)['size']
# write to statsd
for size in sizes:
MESSAGE = "collection_%s:%d|c" % (size, sizes[size])
sock.sendto(MESSAGE, (statsd_ip, statsd_port))
time.sleep(60)
This properly shows all of my collection sizes in grafana. However, I want to get a rate of change on these sizes, so I build the following graphite query in grafana:
derivative(statsd.myHost.collection_myCollection)
And the graph shows up totally blank. Any ideas?
FOLLOW-UP: When selecting a time range greater than 24h, all data similarly disappears from the graph. Can't for the life of me figure out that one.
Update: This was due to the fact that my collectd was configured to send samples every second. The statsd plugin for collectd, however, was receiving data every 60 seconds, so I ended up with None for most data points.
I discovered this by checking the raw data in Graphite by appending &format=raw to the end of a graphite-api query in a browser, which gives you the value of each data point as a comma-separated list.
The temporary fix for this was to surround the graphite query with keepLastValue(60). This however creates a stair-step graph, as the value for each None (60 values) becomes the last valid value within 60 steps. Graphing a derivative of this then becomes a widely spaced sawtooth graph.
In order to fix this, I will probably go on to fix the flush interval on collectd or switch to a standalone statsd instance and configure as necessary from there.

Resources