How to notify if a resource completely download in nginx - nginx

I need to know about full downloading a resource from server. My server is configured with NginX web server, and I want to do something if and only if the resource downloaded completely by user.

If you are using Nginx to handle downloading your files (using XSendfile), you should add a specific access_log block to your download handling block in your Nginx configs (in your "server" block). It would be something like this:
location /download_music/ {
internal;
root /usr/share/nginx/MY_SITE/media/music;
access_log /var/log/nginx/download.MY_SITE.log download;
}
The word "download" at the end of the access_log line is actually a log format which you should add it to the nginx main config file (/etc/nginx/nginx.conf) in the "http" block. It could be something like this:
http {
...
log_format download '{ "remote_addr" : "$remote_addr", "time":"$time_local", "request":"$request", '
'"traffic":$body_bytes_sent, "x_forwarded_for":"$http_x_forwarded_for" }';
...
}
You can change this format to what format you want (you will use it in your script later). If you monitor this log file (using "tail -f /var/log/nginx/download.MY_SITE.log") you will see that any time a download is paused or finished, a line of log will be added to this file.
The next step is using rsyslog and the "imfile" and "omprog" modules. You should add these configs at the end of the config file of rsyslog (/etc/rsyslog.conf):
$ModLoad imfile
$InputFileName /var/log/nginx/download.MY_SITE.log
$InputFileTag nginx-access
$InputFileStateFile state-nginx-access
$InputFileSeverity info
$InputFileFacility local3
$InputFilePollInterval 1
$InputRunFileMonitor
$ModLoad omprog
$actionomprogbinary /usr/share/nginx/MY_SITE/scripts/download.py
$template LogZillaDbInsert,"%hostname:::lowercase%\9%pri%\9%programname%\9%msg%\n"
local3.* :omprog:;RSYSLOG_TraditionalFileFormat
Pay attention to this line:
/usr/share/nginx/MY_SITE/scripts/download.py
This is the address of the script which would be called every time a log entry is added to the log file and the whole log entry will be available in this script using (in Python code):
line = sys.stdin.readline()
Then you can parse the line and find whatever you have logged including the downloaded file size (on every pause/finish event). Now, you can simply include the file size in the download URL and retrieve it in this script and compare it with the downloaded bytes. If these two numbers are equal, it means that download has been finished successfully. Moreover, you can do any other thing you want in this script (for example, expire download link, increase download count on DB, ...)

Related

Adding new files to rsyslogd with wildcards

We're got a pre-existing rsyslog config file which is working for papertrail e.g.
/etc/rsyslog.d/20-papertrail.conf which has
*.* #logs4.papertrailapp.com:44407
However we've got a couple of NGINX websites on the server so would like to have it also monitor their error logs.
The paths to them are:
/var/log/nginx/www.website-one.com-error.log
/var/log/nginx/www.website-two.com-error.log
/var/log/nginx/www.website-three.com-error.log
However this /var/log/nginx also contains a bunch of .log files which we do not want to monitor e.g.
/var/log/nginx/error.log
/var/log/nginx/access.log
/var/log/nginx/error.log1
/var/log/nginx/nginx.log
In my head we need to add something like...
/var/log/nginx/*-error.log
And make sure they pipe to the papertrail url as well.
However I'm struggling to decipher the rsyslog documentation to figure out how to do this.
Thanks!
In rsyslog documentation it seems that you can use wildcards in files.
File
The file being monitored. So far, this must be an absolute name (no macros or templates). Note that wildcards are supported at the file name level (see WildCards below for more details).
WildCards
Before Version: 8.25.0
Wildcards are only supported in the filename part, not in directory names.
/var/log/*.log works.
/var/log/*/syslog.log does not work.
Since Version: 8.25.0
Wildcards are supported in filename and paths which means these samples will work:
/var/log/*.log works.
/var/log/*/syslog.log works.
/var/log/*/*.log works.
All matching files in all matching subfolders will work. Note that this may decrease performance in imfile depending on how many directories and files are being watched dynamically.
If you want to forward your vhosts logs you can change configuration directly in NGINX vhosts configuration, you should change/add access_log and error_log policies as explained here or use custom facilities to forward your logs (using rsyslog).
HOW TO DO IT USING RSYSLOG?
Create a new custom file in /etc/rsyslog.d/nginx_custom.conf:
module(load="imfile" PollingInterval="1") #needs to be done just once
# File 1
input(type="imfile"
File="/var/log/nginx/www.website-*.com-error.log"
Tag="websites"
Facility="local0")
local0.* #logs4.papertrailapp.com:44407
#Just to test that logs are forwarded, comment the line once you've tested it
local0.* /var/log/test.log
And restart rsyslog service
NOTE: Line local0.* /var/log/test.log is just to test that you can see forwarded logs into your local server, comment this line after you've tested that everything works.

Rsyslog: imfile does not switch to inotify mode

I'm trying to send multiple nginx logs to loggly...
Config file: /etc/rsyslog.d/21-nginx.conf
$ModLoad imfile
#$InputFilePollInterval 10
$InputFileMode inotify
$WorkDirectory /var/spool/rsyslog
$PrivDropToGroup adm
# nginx access file:
$InputFileName /var/log/nginx/*access.log
$InputFileTag nginx-access:
$InputFileStateFile stat-nginx-access
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
# other stuff continues......
after restart i get this error in log syslog:
imfile: The to-be-monitored file "/var/log/nginx/*access.log" contains wildcards. This is not supported in polling mode. [v8.16.0 try http://www.rsyslog.com/e/2420 ]
activation of module imfile failed [v8.16.0 try http://www.rsyslog.com/e/-3 ]
did i make something wrong?
Are there other places in your rsyslog configuration where the file mode is changed to pulling or the file poll interval is active? The problem with using this kind of legacy syntax is that all the configuration is loaded globally, so things in other configuration files can interact. You might consider using the new action syntax so that the inotify mode is applied to the specific source. You can see an example of it here http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html

Delete cache files with lua from inside nginx

I have nginx running and it saves cache files to the local disk. I have to clear that cache from time to time manually. I thought about adding an extra location like /clear_cache where I delete the local files directly with Lua, since it can be embedded in nginx.
I did some research and found things like rewrite_by_lua or content_by_lua. Is it possible to access/modifiy the underlying fs with Lua or is that restricted?
Yeah You can remove file:
location /clear_cache {
content_by_lua_block {
//file creation
local f = assert(io.open("/newFile.txt", 'wb')) -- open in "binary" mode
f:write(body)
f:close()
//Remove file
os.remove("/newFile.txt")
}
}

run cgi script using nginx

I am trying to execute cgi script on Nginx. In nginx.conf file, I added a location directive such as below:
location /cgi-bin{
root cgi-bin;
index index.cgi;
}
When I try to connect to http://example.com/cgi-bin/index.cgi, it says file not found. In error.log, I see the request as "http://example.com/html/cgi-bin/index.cgi.
cgi-bin is not in html folder. The correct path is "http://example.com/cgi-bin/index.cgi
I tried many possibilities for location directive. Either it looks in html/cgi-bin or it does /cgi-bin/cgi-bin/index.cgi. I am not sure why it uses 'cgi-bin' twice
Any suggestions.. I have been trying for hours now!!!

nginx not serving updated static files

Switching from apache to nginx, and encountering something weird.
1) Say I have a file yo.txt in the document root of my site and it contains 'foo'.
curl http://localhost/yo.txt => 'foo'
2) then I alter the file to contain 'bar'
curl http://localhost/yo.txt => 'foo' (still!)
If I remove yo.txt, I get a 404. If I remove all the text, I correctly get an empty file when I curl the url.
I checked the last modified HTTP header after I modify the file, and it is correct, even though the contents of the file are stale.
I'm using the standard configuration from nginx after an apt-get install nginx.
what gives?
I'm using Vagrant. Setting sendfile to off in nginx.conf fixed the problem as found here, e.g."
sendfile off;
For me the following worked:
expires modified 10y;
According to the docs:
The time in the “Expires” field is computed as a sum of the current time and time specified in the directive. If the modified parameter is used (0.7.0, 0.6.32) then the time is computed as a sum of the file’s modification time and the time specified in the directive.

Resources