OpenTSDB - Clean install rejects all metrics - opentsdb

I have installed an instance of openTSDB 2.0 for testing, but the server is rejecting all attempts to insert metrics.
These two are from the documentation:
$ telnet localhost 8020
Trying ::1...
Connected to localhost.
Escape character is '^]'.
put sys.cpu.nice 1346846400 50 hostname=test1
put: unknown metric: No such name for 'metrics': 'sys.cpu.nice'
put http.hits 1234567890 34877 host=A
put: unknown metric: No such name for 'metrics': 'http.hits'
My hbase server appears to be configured correctly:
hbase(main):012:0> list
TABLE
tsdb
tsdb-meta
tsdb-tree
tsdb-uid
4 row(s) in 0.0130 seconds
=> ["tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]
I have tried with the server set auto create metrics both ways:
tsd.core.auto_create_metrics = true
tsd.core.meta.enable_realtime_ts = true
tsd.core.meta.enable_realtime_uid = true
Any suggestions about what I have done wrong would be greatly appreciated.

Try rest api, it takes json as input on port 4242:
curl -i -H "Content-Type: application/json" -X POST -d '{"metric": "sys.cpu.nice", "timestamp": 1346846400,"value": 18, "tags": { "host": "web01"}}' http://localhost:4242/api/put/?details
If you want to do it with code:
import requests
import json
def SendTSDBMetrics(metrics):
response = requests.post(url=tsdburi, data=metrics,headers=headers)
print response.text # print what was inserted
metric = 'sys.cpu.nice'
metrics = []
metrics.append({'metric':metric, 'timestamp':time.now(), 'value':18, 'tags':{"device_id":"1"}})
metrics.append({'metric':metric, 'timestamp':time.now(), 'value':100, 'tags':{"device_id":"1"}})
SendTSDBMetrics(json.dumps(metrics))

Related

HTTP2 post request in pycurl

I am trying to hit a nginx server that is behind an alb. I am triggering a simple HTTP2 POST request using pycurl. My request looks like this'
import io
import pycurl
curl = pycurl.Curl()
curl.setopt(curl.URL, "https://alb-xxxx.us-east-1.elb.amazonaws.com/endpoint")
curl.setopt(pycurl.HTTP_VERSION, pycurl.CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE)
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.VERBOSE, 1)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.SSL_VERIFYHOST, 0)
with io.BytesIO() as buf:
curl.setopt(curl.WRITEFUNCTION, buf.write)
curl.perform()
print(buf.getvalue())
curl.close()
The error i am getting is
error Traceback (most recent call last)
<ipython-input-91-0756005b6c1b> in <module>()
16 with io.BytesIO() as buf:
17 curl.setopt(curl.WRITEFUNCTION, buf.write)
---> 18 curl.perform()
19 print(buf.getvalue())
20 curl.close()
error: (92, 'HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)')
the error occurrence is random. some times I get correct response and sometimes I get error like this. But when I try to do the same in curl I am getting correct response always. This is my curl command.
curl -v -k --http2-prior-knowledge -X POST https://alb-xxx.us-east-1.elb.amazonaws.com/endpoint
My post request response is a string({"field":"value"}). My NGINX configuration file looks like this
location =/endpoint {
if ($request_method = POST) {
return 200 '{"field":"value"}';
}
}
I don't know why I am getting correct response in curl but getting a different response in pycurl. Am I doing anything wrong in pycurl ?
Pycurl version -> 7.43.0.6
Python version -> 3.8.5
Thanks. Any help is appreciated.

Autoscaling using Ceilometer/Aodh failed to trigger an alarm in the openstack rocky

Here is the document I refer to
1.sample_server.yaml
type: os.nova.server
version: 1.0
properties:
name: cirros_server
flavor: m1.small
image: b86fb462-c5c2-4a08-9fe4-c9f86d05763d
networks:
- network: external-net
2.Execute the following command line
# openstack cluster create --profile pserver --desired-capacity 2 mycluster
# openstack cluster receiver create --type webhook --cluster mycluster --action CLUSTER_SCALE_OUT --params count=2 r_01
# export ALRM_URL01='http://vip:8777/v1/webhooks/aac3433a-40de-4d7d-830c-e0035f2a4d13/trigger?V=1&count=2'
# aodh alarm create --type gnocchi_resources_threshold --aggregation-method mean --name cpu-high --metric cpu_util --threshold 70 --comparison-operator gt --granularity 300 --evaluation-periods 1 --alarm-action $ALRM_URL01 --repeat-actions False --query metadata.user_metadata.cluster_id=$MYCLUSTER_ID --resource-type instance --resource-id f7e0e8a6-51a3-422d-b631-7ddaf65b3dfb
3.log into each cluster nodes and run some CPU burning workloads there to drive the CPU utilization high
I added log output to /usr/lib/python2.7/site-packages/aodh/notifier/rest.py when trigger the alert request
class RestAlarmNotifier(notifier.AlarmNotifier):
def notify(self, action, alarm_id, alarm_name, severity, previous,
current, reason, reason_data, headers=None):
body = {'alarm_name': alarm_name, 'alarm_id': alarm_id,
'severity': severity, 'previous': previous,
'current': current, 'reason': reason,
'reason_data': reason_data}
headers['content-type'] = 'application/json'
kwargs = {'data': json.dumps(body),
'headers': headers}
max_retries = self.conf.rest_notifier_max_retries
session = requests.Session()
LOG.info('#########################')
LOG.info(session)
LOG.info(kwargs)
LOG.info(action.geturl())
LOG.info('#########################')
session.mount(action.geturl(),
requests.adapters.HTTPAdapter(max_retries=max_retries))
resp = session.post(action.geturl(), **kwargs)
LOG.info('$$$$$$$$$$$$$$$$$$$$$$$')
LOG.info(resp.content)
LOG.info('$$$$$$$$$$$$$$$$$$$$$$$')
Some error messages are output in the /var/log/aodh/notifier.log log, as follows:
enter image description here
The reason is the error caused by adding the body request parameter, the direct post request can be successful, for example, using curl request without the body parameter
curl -g -i -X POST 'http://vip:8777/v1/webhooks/34e91386-7176-4b30-bc17-5c3503712696/trigger?V=1'
Aodh related version packages are as follows:
python2-aodhclient-1.1.1-1.el7.noarch
openstack-aodh-api-7.0.0-1.el7.noarch
openstack-aodh-common-7.0.0-1.el7.noarch
openstack-aodh-listener-7.0.0-1.el7.noarch
python-aodh-7.0.0-1.el7.noarch
openstack-aodh-notifier-7.0.0-1.el7.noarch
openstack-aodh-evaluator-7.0.0-1.el7.noarch
openstack-aodh-expirer-7.0.0-1.el7.noarch
Can anyone point me in the right direction? Thanks.
The problem has been solved. Here is the document I refer to
Modify aodh rest.py(aodh/notifier/rest.py)
https://github.com/openstack/aodh/blob/master/aodh/notifier/rest.py#L79
Under the headers['content-type'] , add this line: headers['openstack-api-version'] = 'clustering 1.10'
Restart aodh service

How to resolve octave machine learning submission error?

How to resolve submission error:
curl: (6) Couldn't resolve host 'www-origin.coursera.org'
m = 15
error] submission with curl() was not successful
!! Submission failed: Grader sent no response
Function: submitWithConfiguration>validateResponse
FileName: C:\Users\admin\Desktop\ex7\lib\submitWithConfiguration.m
LineNumber: 158
This error is probably caused because your computer is not able to connect to the internet at the moment.
This is the function working behind collecting the data from the online grader.
function response = validateResponse(resp)
% test if the response is json or an HTML page
isJson = length(resp) > 0 && resp(1) == '{';
isHtml = findstr(lower(resp), '<html');
if (isJson)
response = resp;
elseif (isHtml)
% the response is html, so its probably an error message
printHTMLContents(resp);
error('Grader response is an HTML message');
else
error('Grader sent no response');
end
end
Now the statement: "Grader sent no response" is printed when the response is null.
And the response can be null when the computer is not connected.
Hope this is the reason behind your error, if not then let me know.
Matlab?
This steps may helpful:
open file: submitWithConfiguration.m ; and: goto line 131 and 134;
then change:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -d #- %s', body, url);
to:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -s -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -s -d #- %s', body, url);
(both add -s)
it looks like the code is fine and the problem is that your computer can't connect to the internet.
You can simply solve this by using a VPN.
Good luck!

Access the OpenTimestamp api through R

I am working on a simple R package to submit hashes for trusted timestamping and to get timestamp info back through Origin Timestamps. I manage to get the information, but I do not manage to POST it OpenTimestamp post hash.
I am using the http package in R. My package ROriginStamp is on github, and the function which I do not get to work is store_hash_info().
Whenever I execute it, I get:
> store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e")
Error in store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e") :
Bad Request (HTTP 400).
3.
stop(http_condition(x, "error", task = task, call = call))
2.
httr::stop_for_status(result$response) at store_hash.R#29
1.
store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e")
>
The function is defined as follow:
store_hash <- function(
hash,
error_on_fail = TRUE,
information = NULL
) {
result <- new_OriginStampResponse()
##
url <- paste0("https://api.originstamp.org/api/", hash)
request_body_json <- jsonlite::toJSON( information, auto_unbox = TRUE )
result$response <- httr::POST(
url,
httr::add_headers(
Authorization = get_option("api_key"),
body = request_body_json
),
httr::content_type_json()
)
if (error_on_fail) {
httr::stop_for_status(result$response)
}
##
try(
{
result$content <- httr::content(
x = result$response,
as = "text"
)
result$content <- jsonlite::fromJSON( result$content )
},
silent = TRUE
)
##
return(result)
}
The function get_option("api_key") just returns my api key.
Any suggestions what I am doing wrong?
Edits
Thanks to Thomas Hepp, Here is a curl command which does work:
curl 'https://api.originstamp.org/api/ff55d7bc3fe6cb2958e4bdda3d4a4a8e528fb67d9194991e9539d97a55cda2a3' \
-H 'authorization: YOUR API KEY' \
-H 'content-type: application/json' \
-H 'accept: application/json' \
-H 'user-agent: OriginStamp cURL Test' \
--data-binary '{"url":null,"email":null,"comment":"this is a test","submit_ops":["multi_seed"]}'
I’m not familiar with R. But, it’s possible to timestamp a file using opentimestamps.org, by posting a hash of the file to one of their calendar servers, using the following methodology. There is no need for an API key, and this procedure can be used to prove the existence of the file at a point in time, via a reference to a value stored in the OP_RETURN field of a bitcoin transaction, in a block in the bitcoin blockchain.
As an example, first, create a test file:
$ echo -n 'this is a test... this is only a test...' > file.txt
Now, take a sha256 hash of the file:
$ sha256sum file.txt
This produces:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca
Now, to create a timestamp of the file, post the raw bytes of the hash to one of the calendar servers (e.g. https://a.pool.opentimestamps.org/). This can be done using curl, like so:
$ echo -n 'c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca' | xxd -r -p | curl -X POST --data-binary - https://a.pool.opentimestamps.org/digest > out.ots
[Optional: If you don’t want to disclose the hash of the file that you are timestamping to Opentimestamps, you can add a random salt to the file hash, then do sha256(original file hash + salt) and post the result of this.]
The response from the above request is redirected to a file out.ots. To get the status of the timestamp, we need to parse the raw bytes of out.ots.
First, view the raw bytes of the file using a hex editor, or xxd:
$ xxd out.ots
00000000: f010 95ee b35a b002 5b8b 5e76 3522 6970 .....Z..[.^v5"ip
00000010: 886c 08f1 0462 be3e 32f0 081a ff1c ae94 .l...b.>2.......
00000020: 4f00 4600 83df e30d 2ef9 0c8e 2e2d 6874 O.F..........-ht
00000030: 7470 733a 2f2f 616c 6963 652e 6274 632e tps://alice.btc.
00000040: 6361 6c65 6e64 6172 2e6f 7065 6e74 696d calendar.opentim
00000050: 6573 7461 6d70 732e 6f72 67 estamps.org
Some of the bytes represent instructions, as follows:
f0 xx: append xx bytes
f1 xx: prepend xx bytes
80: sha256 hash
00: stop
Start with the hash that we timestamped:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca
then, proceed by parsing the response of the POST request. The first two bytes are f0 10. This means append the next 16 bytes (10 in hex is 16 in decimal). This produces:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca95eeb35ab0025b8b5e7635226970886c
Continuing parsing the POST response, the next byte is 80. This means take the sha256 hash of the above.
echo -n ‘c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca95eeb35ab0025b8b5e7635226970886c’ | xxd -p -r | sha256sum
produces:
f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef35090
Continuing, we have f1 04. This means prepend the next 4 bytes to the above. This produces:
62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef35090
Next, we have f0 08. Append the next 8 bytes. This produces:
62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
Finally, we have 00. This means stop. At this point, skip the next 10 bytes, then extract starting from this point to get the URL that we’ll need to get the status of the timestamp:
https://alice.btc.calendar.opentimestamps.org
then, concatenate ‘/timestamp/’ followed by the result above:
https://alice.btc.calendar.opentimestamps.org/timestamp/62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
The status of the timestamp can be accessed by making a GET request to the above URL:
curl https://alice.btc.calendar.opentimestamps.org/timestamp/62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
returns:
Pending confirmation in Bitcoin blockchain
If you wait a few hours, the confirmation will be written to the bitcoin blockchain, and the above GET request will return a much longer proof like the one above, eventually chaining-up to a value that is written to the OP_RETURN field of a bitcoin transaction, in a block in the bitcoin blockchain. By saving this proof, you can verify the existence of the file at the point in time that the block was written to the blockchain, without the need to query the Opentimestamps servers.
The following python script automates the above procedure:
import hashlib
import requests
filehash=bytes.fromhex('c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca')
server='https://a.pool.opentimestamps.org/digest'
print('posting ' + filehash.hex() + ' to ' + server)
response=requests.post(url=server, data=filehash)
bytearray=response.content
print('saving response to ./out.ots')
f=open('./out.ots', 'wb')
f.write(bytearray)
f.close()
print('analysing response')
i=0
ptr=0x00
result=filehash
while(True):
print(i)
print('ptr:', hex(ptr))
nextinstruction=bytearray[ptr]
if(nextinstruction==0xf0):
#append
ptr+=1
numberofbytes=bytearray[ptr]
ptr+=1
bytesegment=bytearray[ptr:ptr+numberofbytes]
print('append ', bytesegment.hex())
result=result+bytesegment
ptr+=numberofbytes
elif(nextinstruction==0xf1):
#prepend
ptr+=1
numberofbytes=bytearray[ptr]
ptr+=1
bytesegment=bytearray[ptr:ptr+numberofbytes]
print('prepend ', bytesegment.hex())
result=bytesegment+result
ptr+=numberofbytes
elif(nextinstruction==0x08):
#sha256
print('sha256')
result=hashlib.sha256(result).digest()
ptr+=1
elif(nextinstruction==0xff):
#fork
print('fork')
ptr+=1
elif(nextinstruction==0x00):
#stop
print('stop')
ptr+=11
url=bytearray[ptr:].decode()
url=url + '/timestamp/' + result.hex()
print('url: ', url)
else:
print('invalid ots file format')
quit()
print('result:', result.hex())
print('-----')
i+=1
if(nextinstruction==0x00): break
print('to get status of timestamp, make a GET request to ' + url)
print('GET ' + url)
response=requests.get(url)
print(response.text)

Expanding Bash shell variable in Curl in Cmder

On a Window 7 machine in Cmder emulator in Git bash window, I'm trying to capture the ISO-formatted current date:
$ date +%s
1513354497
to be sent inside the body of curl POST request:
$curl.sh -X POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{"restricted":true,"marquardtRole":"ContentStreamReservedTagMARQUARDTBIWEEKLYUPDATE","description":"Bi-weekly update covering Marquee development and go-to-market functions.","assetClasses":["Commodities","Credit","Currencies","Economics","Emerging Markets","Equities Macro","Equities Micro","Interest Rates","Prime Services"],"authoringDivision":"SECDIV","titlePattern":"(?i)Marquardt Weekly Update.*","name":"Marquardt BIWEEKLY UPDATE", "updatedBy": "d37286ac275911d788f1b1f11ac60222","updated":"'"date +%s"'"}' "http://localhost.abc.com:8000/maq-app-cnts/services/pubs"
Here's the exception I'm getting:
<!--
The request did not match any of the requests defined for this endpoint.
Request definition 1: The JSON object in the body does not conform to the schema
error: instance type (string) does not match any allowed primitive type (allowed: ["integer"])
level: "error"
schema: {"loadingURI":"definition:/DateTimeInteger#","pointer":""}
instance: {"pointer":"/updated"}
domain: "validation"
keyword: "type"
found: "string"
expected: ["integer"]
-->
How do I pass the date +%s to curl for proper expansion in bash?
Thank you.
Instead of '... ,"updated":"'"date +%s"'"}', use '... ,"updated":"'"$(date +%s)"'"}'
See https://stackoverflow.com/a/30327963/4486184

Resources