Streaming mp4 requests via http with range header in grails - http

I'm on an old grails 2.5.1 app and I noticed mp4 video files served from the server don't play in Safari. I looked up the issue on SO and got some hints that it has to do with the range header. But I suspect the way I'm handling the range header isn't quite right.
So far, what I've found is Mac OS Safari 11.0 (11604.1.38.1.7) (I don't care about ios Safari right now) sends two GET requests. Firstly, it sends one with:
host: localhost:8080
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38
accept-language: en-us
accept-encoding: gzip, deflate
x-request-time: t=****
x-forwarded-for: *.*.*.*
x-forwarded-host: *.com
x-forwarded-server: *.com
connection: Keep-Alive
cookie: ...TOO BIG TO SHOW HERE
<- "GET /.../videos/lol.mp4 HTTP/1.1" 200 186ms
Subsequently, it sends second GET request:
host: localhost:8080
language: en-us
playback-session-id: 03F1B4E6-F97E-****
bytes=0-1
accept: */*
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Safari/604.1.38
https://.../videos/lol.mp4
encoding: identity
request-time: t=****
forwarded-for: *.*.*.*
forwarded-host: *.com
forwarded-server: *.com
connection: Keep-Alive
cookie: ...TOO BIG TO SHOW HERE
<- "GET /uiv2/videos/lol.mp4 HTTP/1.1" 206 149ms
Debugging this is hard because Safari web inspector doesn't show you much. In fact, it doesn't even show you all the headers it sends so I had to get this from the back end.
As can be seen, the difference between request 1 and 2 is the 2nd has playback-session-id and range.
The hard part is finding out how to please Safari in how range and playback-session-id are handled.
I've made a controller to return the range of bytes requested, if they're requested. But still no luck.
import grails.compiler.GrailsTypeChecked
import grails.plugin.springsecurity.annotation.Secured
import asset.pipeline.grails.AssetResourceLocator
import grails.util.BuildSettings
import org.codehaus.groovy.grails.commons.GrailsApplication
import org.springframework.core.io.Resource
class VideoController {
GrailsApplication grailsApplication
AssetResourceLocator assetResourceLocator
public index() {
Resource mp4Resource = assetResourceLocator.findAssetForURI('/../lol.mp4');
response.addHeader("Content-type", "video/mp4")
response.addHeader( 'Accept-Ranges', 'bytes')
String range = request.getHeader('range')
if(range) {
String[] rangeKeyValue = range.split('=')
String[] rangeEnds = rangeKeyValue[1].split('-')
if(rangeEnds.length > 1) {
int startByte = Integer.parseInt(rangeEnds[0])
int endByte = Integer.parseInt(rangeEnds[1])
int contentLength = (endByte - startByte) + 1
byte[] inputBytes = new byte[contentLength]
mp4Resource.inputStream.read(inputBytes, startByte, contentLength)
response.status = 206
response.addHeader( 'Content-Length', "${contentLength}")
response.outputStream << inputBytes
} else {
response.addHeader( 'Content-Length', "${mp4Resource.contentLength()}")
response.outputStream << mp4Resource.inputStream
}
} else {
log.info 'no range, so responding with whole mp4'
response.addHeader( 'Content-Length', "${mp4Resource.contentLength()}")
response.outputStream << mp4Resource.inputStream
}
}
}
In the Safari console, I get:
Failed to load resource: Plug-in handled load
Nothing else. And sadly lots of fields in the web inspector are blank even though they're obviously set in the server.
I've tried so many things at this point that any help, pointers, tips will be appreciated. Thanks guys :) !

After trying many things and scouring many posts, this formula worked. You need all four of those headers. Don't need to return anything in the first request. This may not work for all browsers but this works for safari. Additional modifications can ensure all browsers are handled
class VideoController {
GrailsApplication grailsApplication
AssetResourceLocator assetResourceLocator
public index() {
Resource mp4Resource = assetResourceLocator.findAssetForURI('/../lol.mp4')
String range = request.getHeader('range')
if(range) {
String[] rangeKeyValue = range.split('=')
String[] rangeEnds = rangeKeyValue[1].split('-')
if(rangeEnds.length > 1) {
int startByte = Integer.parseInt(rangeEnds[0])
int endByte = Integer.parseInt(rangeEnds[1])
int contentLength = (endByte - startByte) + 1
byte[] inputBytes = new byte[contentLength]
def inputStream = mp4Resource.inputStream
inputStream.skip(startByte) // input stream always starts at the first byte, so skip bytes until you get to the start of the requested range
inputStream.read(inputBytes, 0, contentLength) // read from the first non-skipped byte
response.reset() // Clears any data that exists in the buffer as well as the status code and headers
response.status = 206
response.addHeader("Content-Type", "video/mp4")
response.addHeader( 'Accept-Ranges', 'bytes')
response.addHeader('Content-Range', "bytes ${startByte}-${endByte}/${mp4Resource.contentLength()}")
response.addHeader( 'Content-Length', "${contentLength}")
response.outputStream << inputBytes
}
}
}
}

Related

QNetworkRequest with headers

I am using Qt to interact with a GraphQL host. I am using QNetworkRequest, and the login process is done well, gives me a token in JSON format. Since then, GraphQL expects the token to be in the HTTP header:
{
"Authorization": "Bearer <token>"
}
Testing the server, I wrote a small Python code, which works well:
headers = {'Authorization': "Bearer {0}".format(token)}
url = "http://example.com:8000/graphql"
params = {'query': '{fieldQueries{fields(userId:"xxx"){fieldName fieldID}}}'}
result = requests.post(url, params=params, headers=headers)
print(result.json())
The below code is supposed to do the same operation in Qt:
QUrl url = QUrl("http://example.com:8000/graphql");
QNetworkAccessManager * mgr = new QNetworkAccessManager(this);
QNetworkRequest request(url);
QString query = QString("{fieldQueries{fields(userId:\"%1\"){fieldName fieldID}}}").arg(userId);
QUrlQuery params;
params.addQueryItem("query", query);
connect(mgr, SIGNAL(finished(QNetworkReply*)), this, SLOT(onQueryFinish(QNetworkReply*)));
connect(mgr, SIGNAL(finished(QNetworkReply*)), mgr, SLOT(deleteLater()));
auto header = QString("Bearer %1").arg(token);
request.setRawHeader(QByteArray("Authorization"), header.toUtf8());
mgr->post(request, params.query().toUtf8());
However, server gives back an internal server error (500).
As soon as I comment out the request.setRawHeader, server gives back You are not authorized to run this query.\nNot authenticated with no error.
How to make Qt to send this header correctly?
I don't know if it helps, but I checked the packets using WireShark. The Python generated packet for this request is a single packet (around 750 bytes), though Qt request is broken into two packets which the length of the first is 600 byte.
The working package:
...POST /graphql?query=%7BfieldQueries%7Bfields%28userId%3A%22xxx%22%29%7BfieldName+fieldID%7D%7D%7D
HTTP/1.1..Host: xxx:8000..
User-Agent: python-requests/2.21.0..
Accept-Encoding: gzip, deflate..
Accept: */*..
Connection: keep-alive..
Content-Type: application/json..
Authorization: Bearer <688 bytes token>..Content-Length: 0....
Qt generated packages:
....POST /graphql HTTP/1.1..Host: xxx:8000..
Authorization: Bearer <364 bytes token>..
Content-Type: application/json..
Content-Length: 113..
Connection: Keep-Alive..
Accept-Encoding: gzip, deflate..
Accept-Language: en-US,*..
User-Agent: Mozilla/5.0....
and
.e..query=%7BfieldQueries%7Bfields(useId:%22xxx%22)%7BfieldName fieldID%7D%7D%7D
I have checked other given solutions for the header, such as
Correct format for HTTP POST using QNetworkRequest,
Sending HTTP Header Info with Qt QNetworkAccessManagerand read the
QNetworkRequest Class documentation.
Graphql accepts both POST an GET requests. Therefore, instead of post, I used GET. It sends the query as a part of the URL, not the header, which from the captured packets I realized they are sent as the body.
The solution is as follows:
QNetworkAccessManager * mgr = new QNetworkAccessManager(this);
QString query = QString("{fieldQueries{fields(userId:\"%1\"){fieldName fieldID}}}").arg(userId);
QUrl url = QUrl(QString("http://example.com:8000/graphql?query=%1").arg(query));
QNetworkRequest request(url);
connect(mgr, SIGNAL(finished(QNetworkReply*)), this, SLOT(onQueryFinish(QNetworkReply*)));
connect(mgr, SIGNAL(finished(QNetworkReply*)), mgr, SLOT(deleteLater()));
auto header = QString("Bearer %1").arg(token);
request.setRawHeader(QByteArray("Authorization"), header.toUtf8());
mgr->get(request);

ArangoDB can't send request with curl

I can't unserstand what I am doing wrong, but when I am sending next request with curl, I am getting error:
echo {"id":1,"question":"aaa"},{"id":2,"question":"bbb?"} | curl -X POST --data-binary #- --dump - http://localhost:8529/_db/otest/_api/document/?collection=sitetestanswers
HTTP/1.1 100 (Continue)
HTTP/1.1 400 Bad Request
Server: ArangoDB
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Content-Length: 100
{"error":true,"errorMessage":"failed to parse json object: expecting EOF","code":400,"errorNum":600}
Any ideas? I tied wrap it's to [...]. Nothing do not help.
With [...] validator mark this as valid
Same with D. Here is my code:
void sendQuestionsToArangoDB(Json questions)
{
string collectionUrl = "http://localhost:8529/_db/otest/_api/document/?collection=sitetestanswers";
auto rq = Request();
rq.verbosity = 2;
string s = `{"id":"1","question":"foo?"},{"id":2}`;
auto rs = rq.post(collectionUrl, s, "application/json");
writeln("SENDED");
}
--
POST /_db/otest/_api/document/?collection=sitetestanswers HTTP/1.1
Content-Length: 37
Connection: Close
Host: localhost:8529
Content-Type: application/json
HTTP/1.1 400 Bad Request
Server: ArangoDB
Connection: Close
Content-Type: application/json; charset=utf-8
Content-Length: 100
100 bytes of body received
For D I use this lib: https://github.com/ikod/dlang-requests
Same issue with vibed.
ArangoDB do not understand JSON if it's come ass array like [...]. It should be passed as key-value. So if you need pass array it should have key mykey : [].
Here is working code:
import std.stdio;
import requests.http;
void main(string[] args)
{
string collectionUrl = "http://localhost:8529/_db/otest/_api/document?collection=sitetestanswers";
auto rq = Request();
rq.verbosity = 2;
string s = `{"some_data":[{"id":1, "question":"aaa"},{"id":2, "question":"bbb"}]}`;
auto rs = rq.post(collectionUrl, s, "application/json");
writeln("SENDED");
}
otest - DB name
sitetestanswers - collection name (should be created in DB)
echo '[{"id":1,"question":"aaa"},{"id":2,"question":"bbb?"}]'
should do the trick. You need to put ticks around the JSON. The array brackets are necessary otherwise this is not valid JSON.
You are trying to send multiple documents. The data in the original question separates the documents by comma ({"id":1,"question":"aaa"},{"id":2,"question":"bbb?"}) which is invalid JSON. Thus the failed to parse json object answer from ArangoDB.
Putting the documents into angular brackets ([ ... ]) as some of the commentors suggested will make the request payload valid JSON again.
However, you're sending the data to a server endpoint that handles a single document. The API for POST /_api/document/?collection=... currently accepts a single document at a time. It does not work with multiple documents in a single request. It expects a JSON object, and whenever it is sent something different it will respond with an error code.
If you're looking for batch inserts, please try the API POST /_api/import, described in the manual here: https://docs.arangodb.com/HttpBulkImports/ImportingSelfContained.html
This will work with multiple documents in a single request. ArangoDB 3.0 will also allow sending multiple documents to the POST /_api/document?collection=... API, but this version is not yet released. A technical preview will be available soon however.

SOAP UI - Save HTTP request to a GZIP file

I m using Soap UI free version for some rest mocking.
I need to persist my HTTP POST request (request received already compressed gzip) to a gzip file.
I have tried different ways to do that, however after to execute the code, when I try to decompress manually the file I have the following error: "The archive is either in unknown format or damaged".
The HTTP POST request has the following header:
Host : 127.0.0.1:8091
Content-Length : 636
User-Agent : Java/1.7.0_07
Connection : keep-alive
Content-Type : application/octet-stream
Accept : text/plain, application/json, application/*+json, */*
Pragma : no-cache
Cache-Control : no-cache
Below the solutions that I have tried:
Solution#1:
byte[] buffer = new byte[1024];
byte[] data = mockRequest.getRequestContent().getBytes();
def path="myfile.gz";
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(path));
bos.write(data);
bos.flush();
bos.close();
Solution#2
byte[] buffer = new byte[1024];
byte[] data = mockRequest.getRawRequestData();
def path="myfile.gz";
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(path));
bos.write(data);
bos.flush();
bos.close();
Can someone please help and let me know why I cannot decompress the gzip file and how I can do that?
Thanks,
This is Groovy, so you don't need all this Java clutter.
Here's some code that might work:
new File( path) << mockRequest.rawRequestData
EDIT
Ok, based on your comments, for zip files to be copied correctly, you probably need something a little different:
import java.nio.file.*
Files.copy(new ByteArrayInputStream(mockRequest.requestContent.bytes),
Paths.get( 'destination.zip' ) )
Tested this with an actual zip file's byte[] as source and it worked. If it does not work for you, then the byte array you're getting from requestContent.bytes just isn't a zip file.

Passing Parameters in Spring MVC 4.0

I have read multiple posts, and have yet to find an answer to my issue. I have the following controller using Spring 4.
Because of the examples I have found, I have gleaned how to use my WellTest POJO to automatically bind via the #RequestParam. I can tell you that many times the "Experts" assume way too much of what they themselves do easily every day when explaining this stuff, and I am not a newby by any means. However, I don't work services very much and in Spring's case, sometimes your only as good as your last version you used. Mine was around 2.5 or so.
Here is my controller method - it's just for testing the passing of Variables.
#RequestMapping(value = "/update/{WellTestID}", method = RequestMethod.POST,headers="Accept=application/json")
public String setWellTestByWellTestID( #RequestParam(defaultValue = "99999") Long wellTestID,
#RequestParam(defaultValue = "99999") Long wellFacilityID,
#RequestParam(defaultValue = "101") int operatorPersonID,
#RequestParam(defaultValue = "67845") int durationSeconds,
#RequestParam(defaultValue = "100") long grossRate,
#RequestParam(defaultValue = "400") long waterCut,
#RequestParam(defaultValue = "30") long percentGas,
#RequestParam(defaultValue = "500") long temperature,
#RequestParam(defaultValue = "60") long flowLinePressure,
#RequestParam(defaultValue = "50") long casingPressure,
#RequestParam(defaultValue = "0") int allocation,
#RequestParam(defaultValue = "3") int wellTestTypeID,
#RequestParam(defaultValue = "you") String createUser,
#RequestParam(defaultValue = "me-me") String lastModifiedUser,
#RequestParam(defaultValue = "4-4-2014") #DateTimeFormat(iso = DateTimeFormat.ISO.DATE) Date createDate,
WellTest wellTest) throws IOException
{
/* try {
wellTestService.updateWellTest(wellTest);
} catch (ParseException e) {
e.printStackTrace();
}*/
String param1 = new String(wellFacilityID.toString());
return param1;
}
The app is for retrieving and editing Well Tests for Oil Wells. I was able to get every parameter to pass, except when I used a Date datatype. I am positive it's related to conversion within the binding, but the service will not even error within debugger - The error tells me the request is not syntactically correct and it never gets to the request method.
The error is...
Blockquote
400 Bad Request Show explanation Loading time: 18
Request headers
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/37.0.2062.120 Safari/537.36
Origin: chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip,deflate
Accept-Language: en-US,en;q=0.8,de;q=0.6
Response headers
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition 4.0 Java/Oracle Corporation/1.8)
Server: GlassFish Server Open Source Edition 4.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: x-requested-with
Content-Language:
Content-Type: text/html
Date: Wed, 17 Sep 2014 21:29:47 GMT
Connection: close
Content-Length: 1105
My question is simple, and please if you are going to comment or answer, be thorough in both. Please assume i know next to nothing, and you are the one who can explain it.
How are passed parameters datatypes converted within Spring MVC? Please explain what to do as well as the concepts behind the why.

doPost not getting called by embedded Jetty, when using Context collection

I am using Jetty 6 in embedded mode. I have a number of servlets in a ContextHandlerCollection. Beyond this problem, the servlets work fine on their different URLs.
ContextHandlerCollection contexts = new ContextHandlerCollection();
server.setHandler(contexts);
HandlerContainer mainhandler = contexts;
Context qxrpc = new Context(contexts,"/api",Context.SESSIONS);
ServletHolder rpcServHolder = new ServletHolder(new FrzRpcServlet());
rpcServHolder.setInitParameter("referrerCheck", "public");
// allows cross-domain calls
qxrpc.addServlet( rpcServHolder, "*.qxrpc");
Context statscontext =new Context(contexts,"/stats",Context.SESSIONS);
ServletHolder statsHolder = new ServletHolder(new FrzStatsServlet());
statsHolder.setInitParameter("restrictToLocalhost", "false");
// allows cross-domain calls
statscontext.addServlet(statsHolder, "/*");
Context hellocontext = new Context(contexts,"/hello", Context.SESSIONS);
hellocontext.addServlet(new ServletHolder(new HelloServlet("HELLO TEST: ")),
"/*");
Context zdbcontext = new Context(contexts,"/zdb", Context.ALL);
ServletHolder zdbHolder = new ServletHolder(new FrzZdbServlet());
statsHolder.setInitParameter("restrictToLocalhost", "false");
// allows cross-domain calls
zdbcontext.addServlet(zdbHolder, "/*");
Context root = new Context(mainhandler,"/",Context.SESSIONS);
root.setResourceBase(docroot);
root.addServlet(DefaultServlet.class, "/");
I know the POST request is coming across to my server. Here is some ngrep output:
T 127.0.0.1:51634 -> 127.0.0.1:8080 [AP]
GET /zdb/test.123:1.1.local1.stringtest HTTP/1.1..Host: 127.0.0.1:8080..Connection: keep-alive..Referer: http://127.0.0.1:8888/GWT_ZDB_editor.html?gwt.codesvr=127.0.0.1:9997..Origin: http://127.0.0.1:8888..User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.71 Safari/534.24..Content-Type: text/plain; charset=utf-8..Accept: */*..Accept-Encoding: gzip,deflate,sdch..Accept-Language: en-US,en;q=0.8..Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3....
##
T 127.0.0.1:8080 -> 127.0.0.1:51634 [AP]
HTTP/1.1 200 OK..Access-Control-Allow-Origin: *..Content-Type: application/json; charset=ISO-8859-1..Content-Length: 124..Server: Jetty(6.1.15)....
##
T 127.0.0.1:8080 -> 127.0.0.1:51634 [AP]
{ "r":0,"D":"test.123:1.1.local1.stringtest","m":"OK","t":0,"p": {"ztype": "STRING", "dat" : { "cp":0, "v": "test12131" }}}
##
Unsuccessful POST - reports 200 OK - but never gets to servlet
T 127.0.0.1:51634 -> 127.0.0.1:8080 [AP]
OPTIONS /zdb/test.123:1.1.local1.stringtest/put HTTP/1.1..Host: 127.0.0.1:8080..Connection: keep-alive..Referer: http://127.0.0.1:8888/GWT_ZDB_editor.html?gwt.codesvr=127.0.0.1:9997..Access-Control-Request-Method: POST..Origin: http://127.0.0.1:8888..User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.71 Safari/534.24..Access-Control-Request-Headers: content-type..Accept: */*..Accept-Encoding: gzip,deflate,sdch..Accept-Language: en-US,en;q=0.8..Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3....
#
T 127.0.0.1:8080 -> 127.0.0.1:51634 [AP]
HTTP/1.1 200 OK..Allow: GET, HEAD, POST, TRACE, OPTIONS..Content-Length: 0..Server: Jetty(6.1.15)...
.
What I can't figure out is why the doPost() is not getting called, while the doGet() is. The servlet in question is the FrzZdbServlet.
Found a number of threads on Google, but the Jetty folks only point back to examples, which in turn only implement do doGet() for the Context examples. As in here
Also, I am posting from GWT code, and I am using content-type application/json. Could this be the issue? Any pointers would be appreciated.
My Context apparently did not accept POSTs with content-type: application/json. Removing this on my client code fixed it. If anyone else has input appreciate it.

Resources