Access the OpenTimestamp api through R - r

I am working on a simple R package to submit hashes for trusted timestamping and to get timestamp info back through Origin Timestamps. I manage to get the information, but I do not manage to POST it OpenTimestamp post hash.
I am using the http package in R. My package ROriginStamp is on github, and the function which I do not get to work is store_hash_info().
Whenever I execute it, I get:
> store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e")
Error in store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e") :
Bad Request (HTTP 400).
3.
stop(http_condition(x, "error", task = task, call = call))
2.
httr::stop_for_status(result$response) at store_hash.R#29
1.
store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e")
>
The function is defined as follow:
store_hash <- function(
hash,
error_on_fail = TRUE,
information = NULL
) {
result <- new_OriginStampResponse()
##
url <- paste0("https://api.originstamp.org/api/", hash)
request_body_json <- jsonlite::toJSON( information, auto_unbox = TRUE )
result$response <- httr::POST(
url,
httr::add_headers(
Authorization = get_option("api_key"),
body = request_body_json
),
httr::content_type_json()
)
if (error_on_fail) {
httr::stop_for_status(result$response)
}
##
try(
{
result$content <- httr::content(
x = result$response,
as = "text"
)
result$content <- jsonlite::fromJSON( result$content )
},
silent = TRUE
)
##
return(result)
}
The function get_option("api_key") just returns my api key.
Any suggestions what I am doing wrong?
Edits
Thanks to Thomas Hepp, Here is a curl command which does work:
curl 'https://api.originstamp.org/api/ff55d7bc3fe6cb2958e4bdda3d4a4a8e528fb67d9194991e9539d97a55cda2a3' \
-H 'authorization: YOUR API KEY' \
-H 'content-type: application/json' \
-H 'accept: application/json' \
-H 'user-agent: OriginStamp cURL Test' \
--data-binary '{"url":null,"email":null,"comment":"this is a test","submit_ops":["multi_seed"]}'

I’m not familiar with R. But, it’s possible to timestamp a file using opentimestamps.org, by posting a hash of the file to one of their calendar servers, using the following methodology. There is no need for an API key, and this procedure can be used to prove the existence of the file at a point in time, via a reference to a value stored in the OP_RETURN field of a bitcoin transaction, in a block in the bitcoin blockchain.
As an example, first, create a test file:
$ echo -n 'this is a test... this is only a test...' > file.txt
Now, take a sha256 hash of the file:
$ sha256sum file.txt
This produces:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca
Now, to create a timestamp of the file, post the raw bytes of the hash to one of the calendar servers (e.g. https://a.pool.opentimestamps.org/). This can be done using curl, like so:
$ echo -n 'c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca' | xxd -r -p | curl -X POST --data-binary - https://a.pool.opentimestamps.org/digest > out.ots
[Optional: If you don’t want to disclose the hash of the file that you are timestamping to Opentimestamps, you can add a random salt to the file hash, then do sha256(original file hash + salt) and post the result of this.]
The response from the above request is redirected to a file out.ots. To get the status of the timestamp, we need to parse the raw bytes of out.ots.
First, view the raw bytes of the file using a hex editor, or xxd:
$ xxd out.ots
00000000: f010 95ee b35a b002 5b8b 5e76 3522 6970 .....Z..[.^v5"ip
00000010: 886c 08f1 0462 be3e 32f0 081a ff1c ae94 .l...b.>2.......
00000020: 4f00 4600 83df e30d 2ef9 0c8e 2e2d 6874 O.F..........-ht
00000030: 7470 733a 2f2f 616c 6963 652e 6274 632e tps://alice.btc.
00000040: 6361 6c65 6e64 6172 2e6f 7065 6e74 696d calendar.opentim
00000050: 6573 7461 6d70 732e 6f72 67 estamps.org
Some of the bytes represent instructions, as follows:
f0 xx: append xx bytes
f1 xx: prepend xx bytes
80: sha256 hash
00: stop
Start with the hash that we timestamped:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca
then, proceed by parsing the response of the POST request. The first two bytes are f0 10. This means append the next 16 bytes (10 in hex is 16 in decimal). This produces:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca95eeb35ab0025b8b5e7635226970886c
Continuing parsing the POST response, the next byte is 80. This means take the sha256 hash of the above.
echo -n ‘c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca95eeb35ab0025b8b5e7635226970886c’ | xxd -p -r | sha256sum
produces:
f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef35090
Continuing, we have f1 04. This means prepend the next 4 bytes to the above. This produces:
62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef35090
Next, we have f0 08. Append the next 8 bytes. This produces:
62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
Finally, we have 00. This means stop. At this point, skip the next 10 bytes, then extract starting from this point to get the URL that we’ll need to get the status of the timestamp:
https://alice.btc.calendar.opentimestamps.org
then, concatenate ‘/timestamp/’ followed by the result above:
https://alice.btc.calendar.opentimestamps.org/timestamp/62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
The status of the timestamp can be accessed by making a GET request to the above URL:
curl https://alice.btc.calendar.opentimestamps.org/timestamp/62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
returns:
Pending confirmation in Bitcoin blockchain
If you wait a few hours, the confirmation will be written to the bitcoin blockchain, and the above GET request will return a much longer proof like the one above, eventually chaining-up to a value that is written to the OP_RETURN field of a bitcoin transaction, in a block in the bitcoin blockchain. By saving this proof, you can verify the existence of the file at the point in time that the block was written to the blockchain, without the need to query the Opentimestamps servers.
The following python script automates the above procedure:
import hashlib
import requests
filehash=bytes.fromhex('c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca')
server='https://a.pool.opentimestamps.org/digest'
print('posting ' + filehash.hex() + ' to ' + server)
response=requests.post(url=server, data=filehash)
bytearray=response.content
print('saving response to ./out.ots')
f=open('./out.ots', 'wb')
f.write(bytearray)
f.close()
print('analysing response')
i=0
ptr=0x00
result=filehash
while(True):
print(i)
print('ptr:', hex(ptr))
nextinstruction=bytearray[ptr]
if(nextinstruction==0xf0):
#append
ptr+=1
numberofbytes=bytearray[ptr]
ptr+=1
bytesegment=bytearray[ptr:ptr+numberofbytes]
print('append ', bytesegment.hex())
result=result+bytesegment
ptr+=numberofbytes
elif(nextinstruction==0xf1):
#prepend
ptr+=1
numberofbytes=bytearray[ptr]
ptr+=1
bytesegment=bytearray[ptr:ptr+numberofbytes]
print('prepend ', bytesegment.hex())
result=bytesegment+result
ptr+=numberofbytes
elif(nextinstruction==0x08):
#sha256
print('sha256')
result=hashlib.sha256(result).digest()
ptr+=1
elif(nextinstruction==0xff):
#fork
print('fork')
ptr+=1
elif(nextinstruction==0x00):
#stop
print('stop')
ptr+=11
url=bytearray[ptr:].decode()
url=url + '/timestamp/' + result.hex()
print('url: ', url)
else:
print('invalid ots file format')
quit()
print('result:', result.hex())
print('-----')
i+=1
if(nextinstruction==0x00): break
print('to get status of timestamp, make a GET request to ' + url)
print('GET ' + url)
response=requests.get(url)
print(response.text)

Related

Lua - encrypt / decrypt commands with aes cbc to pair with a Panasonic TV

I’m trying to rework a script I found online to control a Panasonic TV, which requires a secure/encrypted pairing to occur so I can control it remotely. (The full code here -> https://forum.logicmachine.net/showthread.php?tid=232&pid=16580#pid16580)
Because it seems to be built on LuaJIT and has some other proprietary Lua elements; I’m trying to find alternatives that will allow it to work with the 5.1 Lua install on a Vera Home Automation controller (a relatively closed system).
Also, and perhaps most important for me is that I’d love to make as much of the converted code have minimal requirements to call external modules. I should add I’ve only recently started learning Lua, but one way I like to learn is to convert/repurpose code I find online..
So far i’ve managed to find alternatives for a number of the modules being used, e.g
encdec.base64dec -> Lua Base64 Encode
lmcore.hextostr -> https://github.com/tst2005/binascii/blob/master/binascii.lua
storage.set -> Alternative found in Vera Home Controllers
storage.get -> Alternative found in Vera Home Controllers
bit.ban -> Bitware module in Vera Home Controllers
bit.bxor -> Bitware module in Vera Home Controllers
Where I’m stuck is with the following..
aes:new
aes.cipher
user.aes
encdec.hmacsha256
Here’s an extract of the code where the above are used.
function encrypt_soap_payload(data, key, hmac_key, iv)
payload = '000000000000'
n = #data
payload = payload .. string.char(bit.band(bit.rshift(n, 24), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 16), 0xFF))
payload = payload .. string.char(bit.band(bit.rshift(n, 8), 0xFF))
payload = payload .. string.char(bit.band(n, 0xFF))
payload = payload .. data
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 1)
ciphertext = aes_cbc:encrypt(payload)
sig = encdec.hmacsha256(ciphertext, hmac_key, true)
encrypted_payload = encdec.base64enc(ciphertext .. sig)
return encrypted_payload
end
function decrypt_soap_payload(data, key, hmac_key, iv)
aes_cbc, err = aes:new(key, nil, aes.cipher(128, 'cbc'), { iv = iv }, nil, 0)
decrypted = aes_cbc:decrypt(encdec.base64dec(data))
decrypted = string.gsub(string.sub(lmcore.strtohex(decrypted), 33), '%x%x', function(value) return string.char(tonumber(value, 16)) end)
return decrypted
end
I can get the the point where I can create the parameters for the payload encrypt request (example below), it’s the encryption/decryption I can do..
data="1234"
key="\\S„ßÍ}/Ìa5!"
hmac_key="¹jz¹2¸F\r}òcžÎ„ 臧.ª˜¹=¤µæŸ"
iv=" {¬£áæ‚2žâ3ÐÞË€ú "
I’ve found an aes.lua module online, but that requires loads of others modules most notably ffi.lua. Ideally I’d like to avoid using that. I also came across this aes128.lua but i’m not sure how that handles all the other parameters e.g cbc etc. Finally there’s this aes256ecb.lua script, could that be converted to aes 128 cbc and then used in the above?
Is anyone aware (or maybe has) a Lua script that can handle the aes cbc requirements above ?
Many thanks !
In the end I found out that I could do aes.cbc by calling openssl from the command line, e.g.
local payload = "ENTER HERE"
Local key = "ENTER HERE"
local iv = "ENTER HERE"
local buildsslcommand = "openssl enc -aes-128-cbc -nosalt -e -a -A "..payload.." -K "..key.." -iv "..iv
-- print("Command to send = " ..buildsslcommand)
local file = assert(io.popen(buildsslcommand, 'r'))
local output = file:read('*all')
file:close()
-- print(string.len(output)) --> just count what's returned.
-- print(output) -- > Prints the output of the command.
FYI - It looks like I could do encdec.hmacsha256 via openSSL as well, but I’ve not been able to do that :-( ..

Translate a working curl request into HTTP

I'm using curl to perform a POST request, but I can't assume my target platform to have curl available, so I'm trying to rewrite my curl request in HTTP (which is guaranteed to be available). My knowledge of both curl and HTTP is very limited, so I'm hoping someone can point out what I'm doing wrong.
My curl request (command line):
curl.exe POST https://xxxxxx.ingest.sentry.io/api/xxxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxx -F upload_file_minidump=#"C:\path\Minidump.dmp" -F upload_file_log=#"C:\path\program.log"
A relevant part of curl's output is shown below. This is after connecting to the server and sending it the POST request. The server now lets the client know the first file can be sent, and curl responds first by sending the file's own header, and then the data (clamped here)
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 175 bytes (0xaf)
0000: --------------------------f2a4a742c08bf427
002c: Content-Disposition: form-data; name="upload_file_minidump"; fil
006c: ename="UE4Minidump.dmp"
0085: Content-Type: application/octet-stream
00ad:
=> Send data, 16384 bytes (0x4000)
0000: MDMP..a..... .......m/S`.........................;..............
0040: 8Z......T...=...........`.......8...........T................[..
0080: .........\...........]..........= ..............................
00c0: ....................................aJ.......`......Lw..........
0100: ............T........?..i/S`........ ... ... ............ ......
0140: ............G.M.T. .S.t.a.n.d.a.r.d. .T.i.m.e...................
0180: ................................G.M.T. .D.a.y.l.i.g.h.t. .T.i.m.
01c0: e...................................................1.9.0.4.1...
..etc..
By reading the verbose output of curl, I've created a HTTP request looking like this (c++ code using unreal engine 4 libraries):
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> httpRequest = FHttpModule::Get().CreateRequest();
httpRequest->SetURL(TEXT("https://xxxxxx.ingest.sentry.io/api/xxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxx"));
httpRequest->SetVerb(TEXT("POST"));
const FString boundary(TEXT("------------------------f2a4a742c08bf427"));
httpRequest->SetHeader(TEXT("Content-Type"), TEXT("multipart/form-data; boundary=") + boundary);
const FString fileName(FPaths::Combine(path, crashToReport.folderName, TEXT("UE4Minidump.dmp")));
ensure(FPaths::FileExists(fileName));
const FString prefixBoundary(TEXT("\r\n--") + boundary + TEXT("\r\n"));
const FString fileHeader(TEXT("Content-Disposition: form-data; name=\"upload_file_minidump\"; filename=\"UE4Minidump.dmp\"\r\nContent-Type: application/octet-stream\r\n\r\n"));
FString fileContents;
FFileHelper::LoadFileToString(fileContents, *fileName);
const FString suffixBoundary(TEXT("\r\n--") + boundary + TEXT("--\r\n"));
const FString content(prefixBoundary + fileHeader + fileContents + suffixBoundary);
httpRequest->SetContentAsString(content);
This works to a degree, the server now accepts this, and will receive the file - however the file ends up being unreadable server-side, leading me to think I'm not sending it in the right format.
What kind of data is expected in a multipart/form-data request?
A thing I notice is that the curl request sends the file's header separately (the first chunk of 175 bytes). I would love some information on how to achieve that!
I finally figured it out. I'm not sure how to report what I was doing wrong, but I think it had to do with what happens under the hood in:
httpRequest->SetContentAsString(..)
, which reliably caused the backend to fail to interpret the binary file I was trying to send. I ended up reading the binary file .. as a binary file:
TArray<uint8> dumpFileData;
FFileHelper::LoadFileToArray(dumpFileData, *FPaths::Combine(path,crashToReport.folderName, TEXT("UE4Minidump.dmp")));
Then send it via the POST request, similarly as before, but adding the complete form data part as binary data:
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> httpRequest = FHttpModule::Get().CreateRequest();
httpRequest->SetURL(TEXT("https://xxxxx.ingest.sentry.io/api/xxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxxxxx"));
httpRequest->SetVerb(TEXT("POST"));
const FString boundary(TEXT("------------------------bb33b671b1212234"));
httpRequest->SetHeader(TEXT("Content-Type"), TEXT("multipart/form-data; boundary=") + boundary);
httpRequest->SetHeader(TEXT("Accept"), TEXT("*/*"));
httpRequest->SetHeader(TEXT("Expect"), TEXT("100-continue"));
{
const FString prefixBoundary(TEXT("--") + boundary + TEXT("\r\n"));
const FString fileHeader(TEXT("Content-Disposition: form-data; name=\"upload_file_minidump\"; filename=\"UE4Minidump.dmp\"\nContent-Type: application/octet-stream\r\n\r\n"));
const FString suffixBoundary(TEXT("\r\n--") + boundary + TEXT("--\r\n"));
TArray<uint8> CombinedContent;
CombinedContent.Append(FStringToUint8(prefixBoundary + fileHeader));
CombinedContent.Append(dumpFileData);
CombinedContent.Append(FStringToUint8(suffixBoundary));
httpRequest->SetContent(CombinedContent);
}
httpRequest->ProcessRequest();
For completeness, FStringToUint8 is defined as follows:
// Convert FString to UTF8 and put it in a TArray
TArray<uint8> FStringToUint8(const FString& InString)
{
TArray<uint8> OutBytes;
// Handle empty strings
if (InString.Len() > 0)
{
FTCHARToUTF8 Converted(*InString); // Convert to UTF8
OutBytes.Append(reinterpret_cast<const uint8*>(Converted.Get()), Converted.Length());
}
return OutBytes;
}

How to resolve octave machine learning submission error?

How to resolve submission error:
curl: (6) Couldn't resolve host 'www-origin.coursera.org'
m = 15
error] submission with curl() was not successful
!! Submission failed: Grader sent no response
Function: submitWithConfiguration>validateResponse
FileName: C:\Users\admin\Desktop\ex7\lib\submitWithConfiguration.m
LineNumber: 158
This error is probably caused because your computer is not able to connect to the internet at the moment.
This is the function working behind collecting the data from the online grader.
function response = validateResponse(resp)
% test if the response is json or an HTML page
isJson = length(resp) > 0 && resp(1) == '{';
isHtml = findstr(lower(resp), '<html');
if (isJson)
response = resp;
elseif (isHtml)
% the response is html, so its probably an error message
printHTMLContents(resp);
error('Grader response is an HTML message');
else
error('Grader sent no response');
end
end
Now the statement: "Grader sent no response" is printed when the response is null.
And the response can be null when the computer is not connected.
Hope this is the reason behind your error, if not then let me know.
Matlab?
This steps may helpful:
open file: submitWithConfiguration.m ; and: goto line 131 and 134;
then change:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -d #- %s', body, url);
to:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -s -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -s -d #- %s', body, url);
(both add -s)
it looks like the code is fine and the problem is that your computer can't connect to the internet.
You can simply solve this by using a VPN.
Good luck!

Writing Curl POST in R

What is the correct way of writing this Curl POST in R?
I would like to have R read the contents of a file as "values" in the post form.
curl -X POST https://api.priceapi.com/jobs \
-d "token=token" \
-d "country=country" \
-d "source=source" \
-d "currentness=currentness" \
-d "completeness=completeness" \
-d "key=key" \
-d 'values=<values>'
So far I have this-
library(RCurl)
library(RJSONIO)
url = "https://api.priceapi.com/jobs"
file.name = ".../output 1 .txt"
results = postForm(url, token="token",
country="country,
source="source",
currentness="currentness",
completeness="completeness,
key="key",
values=fileUpload(filename = file.name))
It returns "Error: Bad Request"
I also tried it using httr post request-
r = POST(url, body = list(token="token",
country="country,
source="source",
currentness="currentness",
completeness="completeness,
key="key",
values=upload_file(file.name)) )
Here upload_file is not uploading the contents of the file but, I am guessing it is passing the path to the file (as a string) into the "values" parmeter.
Naturally that does not return the correct results.
The result of the httr POST request is;
Response [https://api.priceapi.com/jobs]
Date: 2016-12-13 10:11
Status: 400
Content-Type: application/json; charset=utf-8
Size: 228 B
{
"success": false,
"reason": "parameter value invalid",
"parameter": "value",
"valid values": "An array or a string containing values separated by newline",
"comment": "Make sure the parameter 'value' has a valid value!"
I could solve this by using
file=readLines(".../output 1.txt")
inputValues <- paste(file,collapse="\n")
and then passing inputValues in the values parameter.

OpenTSDB - Clean install rejects all metrics

I have installed an instance of openTSDB 2.0 for testing, but the server is rejecting all attempts to insert metrics.
These two are from the documentation:
$ telnet localhost 8020
Trying ::1...
Connected to localhost.
Escape character is '^]'.
put sys.cpu.nice 1346846400 50 hostname=test1
put: unknown metric: No such name for 'metrics': 'sys.cpu.nice'
put http.hits 1234567890 34877 host=A
put: unknown metric: No such name for 'metrics': 'http.hits'
My hbase server appears to be configured correctly:
hbase(main):012:0> list
TABLE
tsdb
tsdb-meta
tsdb-tree
tsdb-uid
4 row(s) in 0.0130 seconds
=> ["tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]
I have tried with the server set auto create metrics both ways:
tsd.core.auto_create_metrics = true
tsd.core.meta.enable_realtime_ts = true
tsd.core.meta.enable_realtime_uid = true
Any suggestions about what I have done wrong would be greatly appreciated.
Try rest api, it takes json as input on port 4242:
curl -i -H "Content-Type: application/json" -X POST -d '{"metric": "sys.cpu.nice", "timestamp": 1346846400,"value": 18, "tags": { "host": "web01"}}' http://localhost:4242/api/put/?details
If you want to do it with code:
import requests
import json
def SendTSDBMetrics(metrics):
response = requests.post(url=tsdburi, data=metrics,headers=headers)
print response.text # print what was inserted
metric = 'sys.cpu.nice'
metrics = []
metrics.append({'metric':metric, 'timestamp':time.now(), 'value':18, 'tags':{"device_id":"1"}})
metrics.append({'metric':metric, 'timestamp':time.now(), 'value':100, 'tags':{"device_id":"1"}})
SendTSDBMetrics(json.dumps(metrics))

Resources