In order to counter a botnet attack, I am trying to analyze a nginx access.log file to find which user agents are the most frequent, so that I can find the culprits and deny them. How can I do that?
Try something like this on your access log, replace with the path to your access log, also keep in mind that some log files would get zipped and new one would be created
sudo awk -F" " '{print $1}' /var/log/nginx/access.log | sort | uniq -dc
EDIT:
Sorry I just noticed you wanted user agent instead of IP
sudo awk -F"\"" '{print $6}' /var/log/nginx/access.log | sort | uniq -dc
To sort ascending append | sort -nr and to limit to 10 append | head -10
so the final total line would be
sudo awk -F"\"" '{print $6}' /var/log/nginx/access.log | sort | uniq -dc | sort -nr | head -10
To get user agent
sudo awk -F'"' '/GET/ {print $6}' /var/log/nginx-access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn
awk(1) - selecting full User-Agent string of GET requests
cut(1) - using first word from it
sort(1) - sorting
uniq(1) - count
sort(1) - sorting by count, reversed
Related
var2=$(echo "{$1}" | grep 'Objects that are still invalid after the validation:' | cut -d : -f2 | sed 's/ //g')
echo $var2
the above commandline substitution is not working ksh, the variable is blank each time, have tried below command too
var2="$(echo "{$1}" | grep 'Objects that are still invalid after the validation:' | cut -d : -f2 | sed 's/ //g')"
var2=`echo "{$1}" | grep 'Objects that are still invalid after the validation:' | cut -d : -f2 | sed 's/ //g'`
var2=`echo "$1" | grep 'Objects that are still invalid after the validation:' | cut -d : -f2 | sed 's/ //g'`
please hep me resolve the issue. The command is being used on remote server after ssh. The commands are working on the remote server if executed directly on the server without ssh.
What is supposed to be in $1 ? First issue is that it ought to be written as either $1 or as ${1}. Writing it as {$1} is plain wrong.
Then there is the useless use of grep and cut. The following works:
var2=$(echo ${1} | sed -ne 's/ //g' -e 's/Objectsthatarestillinvalidafterthevalidation:\(.*\)/\1/p')
I am trying to fetch the actual number of HLS viewers in an Nginx with the RTMP module.
I have tried this solution but it is not giving me the real number, I think maybe because it's not counting the viewers who are pulling the stream in HLS format.
Is there any 'good' way to achieve that?
Thanks.
There is a way by counting the number of IPs that requested HLS/DASH fragments in the last 10 to 20 minutes:
#!/bin/bash
date_prefix=$(date +'%d/%b/%Y:%H')
cur_min=$(date +'%M')
cur_min_decimal=${cur_min:0:1}
if [[ $cur_min_decimal == '0' ]]
then
prev_min_decimal=5
else
prev_min_decimal=$(($cur_min_decimal - 1))
fi
cur_minuted_date="${date_prefix}:${cur_min_decimal}"
prev_minuted_date="${date_prefix}:${prev_min_decimal}"
tail -n 10000 /var/log/nginx/access.log \
| grep -E 'GET /(hls|dash)/key-' \
| grep -E "${cur_minuted_date}|${prev_minuted_date}" \
| awk '{print $1}' \
| sort \
| uniq \
| wc -l
If you want to show it on the website, you can CRON that script every minute and output it in the /var/www/html folder
I am looking to check if a random UID on one machine is present on another machine and printing if it exists. I am pretty new to awk and have hit a roadblock.Here is how I am approaching the problem:
Pick a random line in /etc/passwd, get the 3rd column which is the UID; ssh to another machine , get the /etc/passwd contents ,check if the reference UID from the first machine is present in the 3rd column of any line and print it.
I am only able to reach up until the point where I get the reference UID. How do I use this value, ssh into another machine and compare if it exists:
shuf -n 1 /etc/passwd | awk '{print $3}' <the reference UID> <ssh 10.0.0.0> cat /etc/passwd <compare if reference UID is present>
Here's one way of doing that:
awk -vuid=$(shuf -n 1 /etc/passwd | awk -F: '{print $3}') -F: '$3 == uid' <( ssh host cat /etc/passwd)
Breaking it down:
This part stores uid : -vuid=$(shuf -n 1 /etc/passwd | awk -F: '{print $3}')
This will print the line if 3rd field matches our uid: '$3 == uid'
here we're treating output of ssh command as a file: <( ssh host cat /etc/passwd)
I'm trying to check an error every hour in daily log. Garbage collection is creating a file with ".hprof" extensions and I want to write a script that will find "OutOfMemoryError" error in that extension and in that day.That script will run one time in every hour. After that i want it to mail me. How can I do that ?
Thanks
if you had Googled for cron to know how to schedule a script for every hour.
edit crontab as this
0 * * * * <script_path> >/dev/null 2>&1
and for script you can use some thing like this..
month1=`date | awk '{print $2}'`
day1=`date | awk '{print $3}'`
year1=`date | awk '{print $6}'`
grep year1 <file>.hprof | grep month1 | grep day1 | grep OutOfMemoryError | mailx -s "report" <e-mail_addess>
I'm using AWS CodeDeploy in which server running on pm2 dose not work due to explanation given here in troubleShoot documentation.
I followed the documentation and in AfterInstall script used node . > /dev/null 2> /dev/null < /dev/null & to run the node server in the background.
I've tried following ways to kill the server
fuser -k 3000/tcp
lsof -P | grep ':3000' | awk '{print $2}' | xargs kill -9
kill -9 $(lsof -t -i:3000)
but each time a new process respwans with a different PID.
How can I kill this background process and add it to the ApplicationStop script for CodeDeploy?
One of the problems with finding a pid with grep is that the grep pid will also show up as a result and can kill itself before the target, so try;
ps ax | grep node | grep -v grep
if it looks reasonable, review this;
ps ax | grep node | grep -v grep | awk '{print $1}'
then run the kill;
ps ax | grep node | grep -v grep | awk '{print $1}' | xargs kill -9
pkill is a less flexible option (no regex filtering) but if you use that be sure to use the -I flag so you don't kill anything you did not intend to.
I was able to kill using pkill node command.