Sending compressed file (gzip) though HTTP messes with the encoding - http

I have a compressed file.
How do I send it to the browser and make it decode it?
ASP.NET Core suports compression like this:
services.AddResponseCompression();
...
app.UseResponseCompresssion
app.Run(async (context) =>
{
var file = System.IO.File.ReadAllText("file.txt");
await context.Response.WriteAsync(file);
});
But my file is already compressed. I tried just setting the headers, reading the file and sending the content, but it messes with the encoding.
app.Run(async (context) =>
{
context.Response.Headers.Add("Content-Type", "text/plain");
context.Response.Headers.Add("Content-Encoding", "gzip");
context.Response.Headers.Add("Vary", "Accept-Encoding");
var file = System.IO.File.ReadAllText("compressed.gz");
await context.Response.WriteAsync(file);
});
If I just send the file, firefox and chrome try to download it, but do not decompress it.
Proof of that:
$ cat compressed.gz | hexdump | head -1
0000000 8b1f 0808 3768 5bc6 0302 6966 656c 742e
# that's the magic number: 8b 1f 08
$ curl -X GET http://localhost:5000/ -H 'Accept-Encoding: gzip' -H 'Cache-Control: no-cache' --output - | hexdump | head -1
0000000 ef1f bdbf 0808 3768 bfef 5bbd 0302 6966
How do I prevent encoding problems?

As #t.niese pointed out, the problem is reading the text by lines.
The proper way to do this task is to return a FileStreamResult
Example:
Response.Headers["Content-Encoding"] = "gzip";
return new FileStreamResult(new FileStream(fileName, FileMode.Open), "text/plain");

Related

Why is Firebase Cloud Messaging web client side messaging.getToken() yield a different token sometimes? (Same browser, same tab)

I've found that the FCM web client side occasionally delivers a different token to me.
My code is entirely based on the official docs.
// Get registration token. Initially this makes a network call, once retrieved
// subsequent calls to getToken will return from cache.
messaging.getToken({ vapidKey: '<YOUR_PUBLIC_VAPID_KEY_HERE>' }).then((currentToken) => {
if (currentToken) {
// Send the token to your server and update the UI if necessary
// Occasionally, I received a different token here.
} else {
// Show permission request UI
console.log('No registration token available. Request permission to generate one.');
// ...
}
}).catch((err) => {
console.log('An error occurred while retrieving token. ', err);
// ...
});
Occasionally, I receive a different currentToken; this occurs regularly!
I did not change my web browser (Chrome), nor did I change the vapidKey, and I continue to use the same tab for localhost development.
It seemed to have changed every couple of hours.
And if the token changes, my service worker will record an error in the console, as well as a 404 API request failure.
The console log error is as follows:
FirebaseError: Messaging: A problem occurred while unsubscribing the user from FCM: FirebaseError: Messaging: A problem occurred while unsubscribing the user from FCM: Requested entity was not found. (messaging/token-unsubscribe-failed). (messaging/token-unsubscribe-failed).
at _callee8$ (eval at <anonymous> (app.js:2982), <anonymous>:572:45)
at tryCatch (eval at <anonymous> (app.js:6025), <anonymous>:62:40)
at Generator.invoke [as _invoke] (eval at <anonymous> (app.js:6025), <anonymous>:296:22)
at Generator.prototype.<computed> [as next] (eval at <anonymous> (app.js:6025), <anonymous>:114:21)
at step (eval at <anonymous> (app.js:636), <anonymous>:17:30)
at eval (eval at <anonymous> (app.js:636), <anonymous>:28:13)
And here is the failed API request I believe FCM was attempting to make; I did not make the request, and I copied it as curl for greater visibility.
curl 'https://fcmregistrations.googleapis.com/v1/projects/chatisfy-d2721/registrations/cD0VOZLBLdaVymfaUbQyE4:APA91bHj2qCU02_Sib6gEPw3VuPTDkjpj0ZVpgmWYaaHESpTjpH-uwY5JX5mn_W7YhJ1AOMp4dNnwpUffs7SQkBs1UYGGie0o4u_i-OjYY5Q5uRSl3pZQoRGVzwNXxe0lDrQIHD4SN5A' \
-X 'DELETE' \
-H 'authority: fcmregistrations.googleapis.com' \
-H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36' \
-H 'content-type: application/json' \
-H 'accept: application/json' \
-H 'x-goog-api-key: AIzaSyDFP12b-P9JwiDvJuqsWVz6k2Z8ww6_2-E' \
-H 'x-goog-firebase-installations-auth: FIS eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhcHBJZCI6IjE6NTI4MTcxMTk2MzYxOndlYjowMWQyMzQ4ODNiYWQ3NWU5MmYxMjE4IiwiZXhwIjoxNjQwMjUwMDc2LCJmaWQiOiJjRDBWT1pMQkxkYVZ5bWZhVWJReUU0IiwicHJvamVjdE51bWJlciI6NTI4MTcxMTk2MzYxfQ.AB2LPV8wRQIhAL4F96JV_fSn2LHzpBiYDWnOVcpA7zBT35lvWz0WqS8fAiAlC28Un0hO2uD6_DuRPnHdqnO_5wIr-byID127niFXRg' \
-H 'sec-ch-ua-platform: "macOS"' \
-H 'origin: http://localhost:8080' \
-H 'x-client-data: CJO2yQEIpbbJAQjBtskBCKmdygEInvnLAQjmhMwBCLWFzAEIy4nMAQjSj8wBGI6eywE=' \
-H 'sec-fetch-site: cross-site' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-dest: empty' \
-H 'referer: http://localhost:8080/' \
-H 'accept-language: en-US,en;q=0.9,zh-TW;q=0.8,zh;q=0.7,zh-CN;q=0.6' \
--compressed
And here is the response to the aforementioned request:
{
"error": {
"code": 404,
"message": "Requested entity was not found.",
"status": "NOT_FOUND"
}
}
What is causing this?
Some security models expire and reissue tokens at intervals to prevent long-term token validity. Or the token may contain data that updates based on request time.

Translate a working curl request into HTTP

I'm using curl to perform a POST request, but I can't assume my target platform to have curl available, so I'm trying to rewrite my curl request in HTTP (which is guaranteed to be available). My knowledge of both curl and HTTP is very limited, so I'm hoping someone can point out what I'm doing wrong.
My curl request (command line):
curl.exe POST https://xxxxxx.ingest.sentry.io/api/xxxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxx -F upload_file_minidump=#"C:\path\Minidump.dmp" -F upload_file_log=#"C:\path\program.log"
A relevant part of curl's output is shown below. This is after connecting to the server and sending it the POST request. The server now lets the client know the first file can be sent, and curl responds first by sending the file's own header, and then the data (clamped here)
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
=> Send data, 175 bytes (0xaf)
0000: --------------------------f2a4a742c08bf427
002c: Content-Disposition: form-data; name="upload_file_minidump"; fil
006c: ename="UE4Minidump.dmp"
0085: Content-Type: application/octet-stream
00ad:
=> Send data, 16384 bytes (0x4000)
0000: MDMP..a..... .......m/S`.........................;..............
0040: 8Z......T...=...........`.......8...........T................[..
0080: .........\...........]..........= ..............................
00c0: ....................................aJ.......`......Lw..........
0100: ............T........?..i/S`........ ... ... ............ ......
0140: ............G.M.T. .S.t.a.n.d.a.r.d. .T.i.m.e...................
0180: ................................G.M.T. .D.a.y.l.i.g.h.t. .T.i.m.
01c0: e...................................................1.9.0.4.1...
..etc..
By reading the verbose output of curl, I've created a HTTP request looking like this (c++ code using unreal engine 4 libraries):
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> httpRequest = FHttpModule::Get().CreateRequest();
httpRequest->SetURL(TEXT("https://xxxxxx.ingest.sentry.io/api/xxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxx"));
httpRequest->SetVerb(TEXT("POST"));
const FString boundary(TEXT("------------------------f2a4a742c08bf427"));
httpRequest->SetHeader(TEXT("Content-Type"), TEXT("multipart/form-data; boundary=") + boundary);
const FString fileName(FPaths::Combine(path, crashToReport.folderName, TEXT("UE4Minidump.dmp")));
ensure(FPaths::FileExists(fileName));
const FString prefixBoundary(TEXT("\r\n--") + boundary + TEXT("\r\n"));
const FString fileHeader(TEXT("Content-Disposition: form-data; name=\"upload_file_minidump\"; filename=\"UE4Minidump.dmp\"\r\nContent-Type: application/octet-stream\r\n\r\n"));
FString fileContents;
FFileHelper::LoadFileToString(fileContents, *fileName);
const FString suffixBoundary(TEXT("\r\n--") + boundary + TEXT("--\r\n"));
const FString content(prefixBoundary + fileHeader + fileContents + suffixBoundary);
httpRequest->SetContentAsString(content);
This works to a degree, the server now accepts this, and will receive the file - however the file ends up being unreadable server-side, leading me to think I'm not sending it in the right format.
What kind of data is expected in a multipart/form-data request?
A thing I notice is that the curl request sends the file's header separately (the first chunk of 175 bytes). I would love some information on how to achieve that!
I finally figured it out. I'm not sure how to report what I was doing wrong, but I think it had to do with what happens under the hood in:
httpRequest->SetContentAsString(..)
, which reliably caused the backend to fail to interpret the binary file I was trying to send. I ended up reading the binary file .. as a binary file:
TArray<uint8> dumpFileData;
FFileHelper::LoadFileToArray(dumpFileData, *FPaths::Combine(path,crashToReport.folderName, TEXT("UE4Minidump.dmp")));
Then send it via the POST request, similarly as before, but adding the complete form data part as binary data:
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> httpRequest = FHttpModule::Get().CreateRequest();
httpRequest->SetURL(TEXT("https://xxxxx.ingest.sentry.io/api/xxxxxx/minidump/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxxxxx"));
httpRequest->SetVerb(TEXT("POST"));
const FString boundary(TEXT("------------------------bb33b671b1212234"));
httpRequest->SetHeader(TEXT("Content-Type"), TEXT("multipart/form-data; boundary=") + boundary);
httpRequest->SetHeader(TEXT("Accept"), TEXT("*/*"));
httpRequest->SetHeader(TEXT("Expect"), TEXT("100-continue"));
{
const FString prefixBoundary(TEXT("--") + boundary + TEXT("\r\n"));
const FString fileHeader(TEXT("Content-Disposition: form-data; name=\"upload_file_minidump\"; filename=\"UE4Minidump.dmp\"\nContent-Type: application/octet-stream\r\n\r\n"));
const FString suffixBoundary(TEXT("\r\n--") + boundary + TEXT("--\r\n"));
TArray<uint8> CombinedContent;
CombinedContent.Append(FStringToUint8(prefixBoundary + fileHeader));
CombinedContent.Append(dumpFileData);
CombinedContent.Append(FStringToUint8(suffixBoundary));
httpRequest->SetContent(CombinedContent);
}
httpRequest->ProcessRequest();
For completeness, FStringToUint8 is defined as follows:
// Convert FString to UTF8 and put it in a TArray
TArray<uint8> FStringToUint8(const FString& InString)
{
TArray<uint8> OutBytes;
// Handle empty strings
if (InString.Len() > 0)
{
FTCHARToUTF8 Converted(*InString); // Convert to UTF8
OutBytes.Append(reinterpret_cast<const uint8*>(Converted.Get()), Converted.Length());
}
return OutBytes;
}

Box API bad request when uploading image

According to box API to upload a file this is the curl command:
curl https://upload.box.com/api/2.0/files/content \
-H "Authorization: Bearer ACCESS_TOKEN" -X POST \
-F attributes='{"name":"tigers.jpeg", "parent":{"id":"11446498"}}' \
-F file=#myfile.jpg
I am trying to upload an image using QT, the following is my code:
QUrl url(QString::fromLatin1("https://upload.box.com/api/2.0/files/content"));
auto* multipart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
QHttpPart attributes;
attributes.setHeader(QNetworkRequest::ContentDispositionHeader,
QLatin1String("form-data; name=\"attributes\""));
QByteArray postData = QString::fromUtf8("{\"name\":\"%1\",\"parent\":{\"id\":\"%2\"}}").arg(QFileInfo(imgPath).fileName()).arg(id).toUtf8();
attributes.setBody(postData);
multipart->append(attributes);
QFile* file = new QFile(imgPath);
file->open(QIODevice::ReadOnly);
file->setParent(multipart);
QHttpPart imagepart;
imagepart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("image/jpeg"));
imagepart.setHeader(QNetworkRequest::ContentDispositionHeader,QVariant(QString::fromLatin1("form-data; name=\"image\";filename=\"%1\"").arg(QLatin1String(QFileInfo(imgPath).fileName().toUtf8()))));
imagepart.setHeader(QNetworkRequest::ContentLengthHeader, file->size());
imagepart.setBodyDevice(file);
multipart->append(imagepart);
QNetworkRequest netRequest(url);
netRequest.setHeader(QNetworkRequest::ContentTypeHeader, QLatin1String("multipart/form-data"));
netRequest.setRawHeader("Authorization", QString::fromLatin1("Bearer {%1}").arg(d->o2->token()).toUtf8());
d->reply = d->netMngr->post(netRequest, multipart);
multipart->setParent(d->reply);
When I try to upload the image I get an error saying
Error transferring https://upload.box.com/api/2.0/files/content -
server replied: Bad Request
What am I doing wrong?

How to resolve octave machine learning submission error?

How to resolve submission error:
curl: (6) Couldn't resolve host 'www-origin.coursera.org'
m = 15
error] submission with curl() was not successful
!! Submission failed: Grader sent no response
Function: submitWithConfiguration>validateResponse
FileName: C:\Users\admin\Desktop\ex7\lib\submitWithConfiguration.m
LineNumber: 158
This error is probably caused because your computer is not able to connect to the internet at the moment.
This is the function working behind collecting the data from the online grader.
function response = validateResponse(resp)
% test if the response is json or an HTML page
isJson = length(resp) > 0 && resp(1) == '{';
isHtml = findstr(lower(resp), '<html');
if (isJson)
response = resp;
elseif (isHtml)
% the response is html, so its probably an error message
printHTMLContents(resp);
error('Grader response is an HTML message');
else
error('Grader sent no response');
end
end
Now the statement: "Grader sent no response" is printed when the response is null.
And the response can be null when the computer is not connected.
Hope this is the reason behind your error, if not then let me know.
Matlab?
This steps may helpful:
open file: submitWithConfiguration.m ; and: goto line 131 and 134;
then change:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -d #- %s', body, url);
to:
line131: json_command = sprintf('echo jsonBody=%s | curl -k -X POST -s -d #- %s', body, url);
line134: json_command = sprintf('echo ''jsonBody=%s'' | curl -k -X POST -s -d #- %s', body, url);
(both add -s)
it looks like the code is fine and the problem is that your computer can't connect to the internet.
You can simply solve this by using a VPN.
Good luck!

Access the OpenTimestamp api through R

I am working on a simple R package to submit hashes for trusted timestamping and to get timestamp info back through Origin Timestamps. I manage to get the information, but I do not manage to POST it OpenTimestamp post hash.
I am using the http package in R. My package ROriginStamp is on github, and the function which I do not get to work is store_hash_info().
Whenever I execute it, I get:
> store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e")
Error in store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e") :
Bad Request (HTTP 400).
3.
stop(http_condition(x, "error", task = task, call = call))
2.
httr::stop_for_status(result$response) at store_hash.R#29
1.
store_hash(hash = "c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e")
>
The function is defined as follow:
store_hash <- function(
hash,
error_on_fail = TRUE,
information = NULL
) {
result <- new_OriginStampResponse()
##
url <- paste0("https://api.originstamp.org/api/", hash)
request_body_json <- jsonlite::toJSON( information, auto_unbox = TRUE )
result$response <- httr::POST(
url,
httr::add_headers(
Authorization = get_option("api_key"),
body = request_body_json
),
httr::content_type_json()
)
if (error_on_fail) {
httr::stop_for_status(result$response)
}
##
try(
{
result$content <- httr::content(
x = result$response,
as = "text"
)
result$content <- jsonlite::fromJSON( result$content )
},
silent = TRUE
)
##
return(result)
}
The function get_option("api_key") just returns my api key.
Any suggestions what I am doing wrong?
Edits
Thanks to Thomas Hepp, Here is a curl command which does work:
curl 'https://api.originstamp.org/api/ff55d7bc3fe6cb2958e4bdda3d4a4a8e528fb67d9194991e9539d97a55cda2a3' \
-H 'authorization: YOUR API KEY' \
-H 'content-type: application/json' \
-H 'accept: application/json' \
-H 'user-agent: OriginStamp cURL Test' \
--data-binary '{"url":null,"email":null,"comment":"this is a test","submit_ops":["multi_seed"]}'
I’m not familiar with R. But, it’s possible to timestamp a file using opentimestamps.org, by posting a hash of the file to one of their calendar servers, using the following methodology. There is no need for an API key, and this procedure can be used to prove the existence of the file at a point in time, via a reference to a value stored in the OP_RETURN field of a bitcoin transaction, in a block in the bitcoin blockchain.
As an example, first, create a test file:
$ echo -n 'this is a test... this is only a test...' > file.txt
Now, take a sha256 hash of the file:
$ sha256sum file.txt
This produces:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca
Now, to create a timestamp of the file, post the raw bytes of the hash to one of the calendar servers (e.g. https://a.pool.opentimestamps.org/). This can be done using curl, like so:
$ echo -n 'c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca' | xxd -r -p | curl -X POST --data-binary - https://a.pool.opentimestamps.org/digest > out.ots
[Optional: If you don’t want to disclose the hash of the file that you are timestamping to Opentimestamps, you can add a random salt to the file hash, then do sha256(original file hash + salt) and post the result of this.]
The response from the above request is redirected to a file out.ots. To get the status of the timestamp, we need to parse the raw bytes of out.ots.
First, view the raw bytes of the file using a hex editor, or xxd:
$ xxd out.ots
00000000: f010 95ee b35a b002 5b8b 5e76 3522 6970 .....Z..[.^v5"ip
00000010: 886c 08f1 0462 be3e 32f0 081a ff1c ae94 .l...b.>2.......
00000020: 4f00 4600 83df e30d 2ef9 0c8e 2e2d 6874 O.F..........-ht
00000030: 7470 733a 2f2f 616c 6963 652e 6274 632e tps://alice.btc.
00000040: 6361 6c65 6e64 6172 2e6f 7065 6e74 696d calendar.opentim
00000050: 6573 7461 6d70 732e 6f72 67 estamps.org
Some of the bytes represent instructions, as follows:
f0 xx: append xx bytes
f1 xx: prepend xx bytes
80: sha256 hash
00: stop
Start with the hash that we timestamped:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca
then, proceed by parsing the response of the POST request. The first two bytes are f0 10. This means append the next 16 bytes (10 in hex is 16 in decimal). This produces:
c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca95eeb35ab0025b8b5e7635226970886c
Continuing parsing the POST response, the next byte is 80. This means take the sha256 hash of the above.
echo -n ‘c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca95eeb35ab0025b8b5e7635226970886c’ | xxd -p -r | sha256sum
produces:
f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef35090
Continuing, we have f1 04. This means prepend the next 4 bytes to the above. This produces:
62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef35090
Next, we have f0 08. Append the next 8 bytes. This produces:
62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
Finally, we have 00. This means stop. At this point, skip the next 10 bytes, then extract starting from this point to get the URL that we’ll need to get the status of the timestamp:
https://alice.btc.calendar.opentimestamps.org
then, concatenate ‘/timestamp/’ followed by the result above:
https://alice.btc.calendar.opentimestamps.org/timestamp/62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
The status of the timestamp can be accessed by making a GET request to the above URL:
curl https://alice.btc.calendar.opentimestamps.org/timestamp/62be3e32f7d0917a163a8df26066cd669eb12e2d0d59bb5f454aaee338dcc0694ef350901aff1cae944f0046
returns:
Pending confirmation in Bitcoin blockchain
If you wait a few hours, the confirmation will be written to the bitcoin blockchain, and the above GET request will return a much longer proof like the one above, eventually chaining-up to a value that is written to the OP_RETURN field of a bitcoin transaction, in a block in the bitcoin blockchain. By saving this proof, you can verify the existence of the file at the point in time that the block was written to the blockchain, without the need to query the Opentimestamps servers.
The following python script automates the above procedure:
import hashlib
import requests
filehash=bytes.fromhex('c16d7c8e23baf68525cf0a42fff6b394fdba1791db9817fd601b3f73e2f5fbca')
server='https://a.pool.opentimestamps.org/digest'
print('posting ' + filehash.hex() + ' to ' + server)
response=requests.post(url=server, data=filehash)
bytearray=response.content
print('saving response to ./out.ots')
f=open('./out.ots', 'wb')
f.write(bytearray)
f.close()
print('analysing response')
i=0
ptr=0x00
result=filehash
while(True):
print(i)
print('ptr:', hex(ptr))
nextinstruction=bytearray[ptr]
if(nextinstruction==0xf0):
#append
ptr+=1
numberofbytes=bytearray[ptr]
ptr+=1
bytesegment=bytearray[ptr:ptr+numberofbytes]
print('append ', bytesegment.hex())
result=result+bytesegment
ptr+=numberofbytes
elif(nextinstruction==0xf1):
#prepend
ptr+=1
numberofbytes=bytearray[ptr]
ptr+=1
bytesegment=bytearray[ptr:ptr+numberofbytes]
print('prepend ', bytesegment.hex())
result=bytesegment+result
ptr+=numberofbytes
elif(nextinstruction==0x08):
#sha256
print('sha256')
result=hashlib.sha256(result).digest()
ptr+=1
elif(nextinstruction==0xff):
#fork
print('fork')
ptr+=1
elif(nextinstruction==0x00):
#stop
print('stop')
ptr+=11
url=bytearray[ptr:].decode()
url=url + '/timestamp/' + result.hex()
print('url: ', url)
else:
print('invalid ots file format')
quit()
print('result:', result.hex())
print('-----')
i+=1
if(nextinstruction==0x00): break
print('to get status of timestamp, make a GET request to ' + url)
print('GET ' + url)
response=requests.get(url)
print(response.text)

Resources