Log the video duration of a mp4 file in NGINX access log - nginx

I am trying to log the video duration of a mp4 file in NGINX access log, which should happen when a client 'GET' a mp4 file from the server. I have configured a custom log format as follows:
log_format nginx_custom_log '$remote_addr ... $request_uri $status $sent_http_content_type $video_duration';
access_log /var/log/nginx/access.log nginx_custom_log;
I can add a custom HTTP header - video_duration pointing to the path of a video file and manually assigning the value, but this require changing nginx configuration every time a video is added and reloading nginx:
location /path/video.mp4 {
add_header video_duration 123456;
}
The following record is written to NGINX access log:
192.168.0.1 ... /path/video.mp4 206 video/mp4 123456
I also tried configuring the X-Content-Duration HTTP header (which is no longer supported by Firefox) in NGINX configuration but no value has been logged.
I found a module named ngx_http_mp4_module. It allows specifying arguments like ?start=238.88&end=555.55, which leads me to believe NGINX is capable of reading the metadata of a mp4 file.
Is there a way to log the duration of a mp4 video file in NGINX access log, similar to how a file's content-length (in bytes) and content-type (video/mp4) can be logged?

Thanks Alex for suggesting to add a metadata file and read it using Lua. Here's the implementation for reference:
nginx.conf
location ~* \.(mp4)$ {
set $directoryPrefix '/usr/local/openresty/nginx/html';
set $metaFile '$directoryPrefix$request_uri.duration';
set $mp4_header "NOT_SET";
rewrite_by_lua_block {
local f = assert(io.open(ngx.var.metaFile, "r"))
ngx.var.mp4_header = f:read("*line")
f:close()
}
add_header mp4_duration $mp4_header;
}
log_format nginxlog '$remote_addr ... "$request_uri" $status $sent_http_content_type $sent_http_mp4_duration';
access_log /usr/local/openresty/nginx/logs/access.log nginxlog;
Please note that $metaFile refers to the metadata file containing the duration:
$directoryPrefix: /usr/local/openresty/nginx/html
$request_uri: /video/VideoABC.mp4
$metaFile: /usr/local/openresty/nginx/html/video/VideoABC.mp4.duration
access.log
192.168.0.1 ... "/video/VideoABC.mp4" 206 video/mp4 1234567890
mp4 & metadata file path
root#ubuntu: cd /usr/local/openresty/nginx/html/video/
root#ubuntu: ls
VideoABC.mp4 VideoABC.mp4.duration
root#ubuntu: cat VideoABC.mp4.duration
1234567890

Related

NGINX : Using Upstream server name

I am running nginx/1.19.6 on Ubuntu.
I am struggling to get the upstream module to work without returning a 404.
My *.conf files are located in /etc/nginx/conf.d/
FILE factory_upstream.conf:
upstream factoryserver {
server factory.service.corp.com:443;
}
FILE factory_service.conf:
server
{
listen 80;
root /data/www;
proxy_cache factorycache;
proxy_cache_min_uses 1;
proxy_cache_methods GET HEAD POST;
proxy_cache_valid 200 72h;
#proxy_cache_valid any 5m;
location /factory/ {
access_log /var/log/nginx/access-factory.log main;
proxy_set_header x-apikey abcdefgh12345678;
### Works when expressed as a literal.# proxy_pass https://factory.service.corp.com/;
### 404 when using the upstream name.
proxy_pass https://factoryserver/;
}
}
I have error logging set to debug, but after reloading the configuration and attempting a call, there are no new records in the error log.
nginx -t # Scans OK
nginx -s reload # no errors
cat /var/log/nginx/error.log
...
2021/03/16 11:29:52 [notice] 26157#26157: signal process started
2021/03/16 11:38:20 [notice] 27593#27593: signal process started
The access-factory.log does record the request :
127.1.2.3 --;[16/Mar/2021:11:38:50 -0400];";GET /factory/api/manifest/get-full-manifest/PID/904227100052 HTTP/1.1" ";/factory/api/manifest/get-full-manifest/PID/904227100052" ;404; - ;/factory/api/manifest/get-full-manifest/PID/904227100052";-" ";ID="c4cfc014e3faa803c8fe136d6eae347d ";-" ";10.8.9.123:443" ";404" ";-"
To help with debugging, I cached the 404 error, "proxy_cache_valid any 5m;" commented out in the example above:
When I use the upstream name, the cache file contains the followiing:
<##$ non-printable characters $%^>
KEY: https://factoryserver/api/manifest/get-full-manifest/PID/904227100052
HTTP/1.1 404 Not Found
Server: nginx/1.17.8
Date: Tue, 16 Mar 2021 15:38:50 GMT
...
The key contains the name 'factoryserver' I don't know if that matters or not. Does it?
The server version is different than what I see when I enter the command nginx -v, which is: nginx version: nginx/1.19.6
Does the difference in version in the cache file and the command line indicate anything?
When I switch back to the literal server name in the proxy_pass, I get a 200 response with the requested data. The Key in the cache file then contains the literal upstream server name.
<##$ non-printable characters $%^>
KEY: https://factory.service.corp.com/api/manifest/get-full-manifest/PID/904227100052
HTTP/1.1 200
Server: nginx/1.17.8
Date: Tue, 16 Mar 2021 15:59:30 GMT
...
I will have multiple upstream servers, each providing different services. The configuration will be deployed to multiple factories, each with its own upstream servers.
I would like for the deployment team to only have to update the *_upstream.conf files, and keep the *_service.conf files static from deployment site to deployment site.
factory_upstream.conf
product_upstream.conf
shipping_upstream.conf
abc123_upstream.conf
Why do I get a 404 when using a named upstream server?
Based on the nginx version in the cached response not matching what you see on the command line, it seems that maybe the 404 is coming from the upstream server. I.e, your proxying is working, but the upstream server is returning a 404. To troubleshoot further, I would check the nginx logs for the upstream server and if the incoming request is what you expect.
Note that when using proxy_pass, it makes a big difference whether you have a / at the end or not. With a trailing slash, nginx treats this as the URI it should send the upstream request to and it doesn't include the URI matched by the location block (/factory/) while without, it includes the full URI as-is.
proxy_pass https://factoryserver/ results in https://factory.service.corp.com:443/
proxy_pass https://factoryserver results in https://factory.service.corp.com:443/factory/
Docs: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/
So maybe when switching between using an upstream and specifying the literal server name, you're inadvertently being inconsistent with the trailing slash. It's a really easy detail to miss, especially when you don't know it's important.
Please provide more information about your server configuration where you proxy pass the requests. Only difference I see now is that you specify the port (443) in your upstream server.

How to config nginx to tail log on web browser

My goal is to see the newest log messages on web page.
I know I could use tail -f <file> to trace the latest 10 line log messages on terminal.
But today, I want to config nginx, so that I could see the same result on web.
For example, when I access to http://192.168.1.200/nginx (My Nginx Host)
I could see the files under /var/log/nginx
Index of /nginx/
-------------------------------------------------
../
access.log 08-Aug-2019 16:43 20651
error.log 08-Aug-2019 16:43 17810
And when I access to http://192.168.1.200/nginx/access.log
I could see the same result as tail -f /var/log/nginx/access.log in terminal (and it is dynamic).
/etc/nginx/conf.d/log.conf
server {
listen 80;
root /var/log/nginx;
location /nginx {
root /var/log/;
default_type text/plain;
autoindex on;
}
}
This is my config, but there are 2 points that doesn't meet my requirements:
I want to access to log page by accessing /log/access.log not by /nginx/access.log
When I access to /log/access.log, this page is static.

How can I use nginx brotli_static with proxy_pass?

nginx is compiled with Brotli enabled. In my nginx.conf
http {
...
brotli_static on;
}
My .br files are located on a server with proxy_pass.
location / {
...
proxy_pass http://app;
}
And .br files have been generated on that app server:
$ ls -lh public/js/dist/index.js*
-rw-r--r-- 1 mike wheel 1.2M Apr 4 09:07 public/js/dist/index.js
-rw-r--r-- 1 mike wheel 201K Apr 4 09:07 public/js/dist/index.js.br
Pulling down the uncompressed file works:
wget https://example.com/js/dist/index.js
Pulls down 1,157,704 size uncompressed file.
wget -S --header="accept-encoding: gzip" https://example.com/js/dist/index.js
Pulls down a 309,360 size gzipped file.
But:
wget -S --header="accept-encoding: br" https://example.com/js/dist/index.js
Still gets the full 1,157,704 size uncompressed file.
I had hoped brotli_static would proxy the .br file requests too - sending something a GET request to the backend for the .br equivalent resource - but this doesn't seem to work.
Can brotli_static work through proxy_pass?
Based on Maxim Dounin (an nginx core engineer)'s comment on gzip_static - which I imagine brotli_static behaves similarly to - brotli_static only handles files, not HTTP resources:
That is, gzip_static is only expected to work when nginx is about to return regular files.
So it looks like brotli_static and proxy_pass isn't possible.
Your nginx config file needs a section to tell it to serve the static content folder. You don't want your app server to do that.
I believe you'll need to place it before the location / so that it takes precedence.

Adaptive bit rate streaming not working in nginx-vod-module in NGINX server

I have installed Nginx and configured VOD for adaprive streaming using nginx-vod-module. While requesting the master.m3u8 file I'm getting same ts files served for different network bandwidth.
The master.m3u8 file has the following content:
#EXTM3U
#EXT-X-STREAM-INF:PROGRAMID=1,BANDWIDTH=1914317,RESOLUTION=1280x544,CODECS="avc1.64001f,mp4a.40.2"
http://localhost/content/Input.mp4/index-v1-a1.m3u8
The Nginx configuration is:
location /content {
vod hls;
vod_mode local;
root /usr/share/nginx/html;
gzip on;
gzip_types application/vnd.apple.mpegurl;
expires 100d;
add_header Last-Modified "Sun, 19 Nov 2000 08:52:00 GMT";
}
How can I get adaptive bitrate enabled using nginx-vod-module and what's the best way to verify it ?
You encode multiple versions of your Input.mp4 with different resolutions/bitrates. The aspect ratio should be the same. Eg: Input_high.mp4, Input_low.mp4
You edit the master m3u8 playlist and add each rendition with its specific bitrate and resolution:
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=...,RESOLUTION=...,CODECS="..."
/content/Input_low.mp4.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=...,RESOLUTION=...,CODECS="..."
/content/Input_high.mp4.m3u8
When the nginx-vod-module receives a request for a filename.mp4.m3u8 it automatically segments filename.mp4 for HLS and creates the playlist for you. Eg: /content/Input_low.mp4.m3u8 for /content/Input_low.mp4

Blank POST with nginx upload module and chunked upload

I am using the nginx upload module to accept large uploads for a PHP application. I have configured nginx following this blog post (modified for my needs).
Here is (the applicable portion of) my nginx configuration:
server {
# [ ... ]
location /upload {
set $upload_field_name "file";
upload_pass /index.php;
upload_store /home/example/websites/example.com/storage/uploads 1;
upload_resumable on;
upload_max_file_size 0;
upload_set_form_field $upload_field_name[filename] "$upload_file_name";
upload_set_form_field $upload_field_name[path] "$upload_tmp_path";
upload_set_form_field $upload_field_name[content_type] "$upload_content_type";
upload_aggregate_form_field $upload_field_name[size] "$upload_file_size";
upload_pass_args on;
upload_cleanup 400-599;
client_max_body_size 200M;
}
}
In the client side JavaScript, I am using 8MB chunks.
With this configuration, I am able to upload any file that is one chunk or smaller. However, when I try to upload any file that is more than one chunk, the response I get from the server for each intermediate chunk is blank, and the final chunk triggers the call to the PHP application without any incoming POST data.
What am I missing?
It turns out that #Brandan's blog post actually leaves out one important directive:
upload_state_store /tmp;
I added that and now everything works as expected.

Resources