I'm trying to create a fail2ban filter that is going to ban the host when it sends over 100 POST requests over 30 seconds interval.
jail.local:
[nginx-postflood]
enabled = false
filter = nginx-postflood
action = myaction
logpath = /var/log/nginx/access.log
findtime = 30
bantime = 100
maxretry = 100
nginx-postflood.conf
[Definition]
failregex = ^<HOST>.*"POST.*
ignoreregex =
Using GREP i was able to test the regular expressions and indeed it matches Host and POST requests.
Problem is that it bans any Host that performs at least one POST request. This means likely that it's not taking findttime or maxretry options into consideration. In my opinion it's timestamp issue.
Sample line of nginx log:
5.5.5.5 - user [05/Aug/2014:00:00:09 +0200] "POST /auth HTTP/1.1" 200 6714 "http://referer.com" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0"
Any help?
I guess it maybe to late for the answer but anyway...
The excerpt you have posed has the filter disabled.
enabled = false
As there is not mentioning of Fail2Ban version and syslog/fail2ban logs are missing for this jail.
I tested your Filter on fail2ban 0.9.3-1 and it works fine although I had to enable it and had to drop the line with action = myaction as you have not provided what you are expecting fail2ban to do.
Therefore this filter should work fine, based that it's enabled and the action is correct as well.
What is happening in the provided example is that Your Filter is disabled and fail2ban is using another Filter which checks the same log file and matches your regex but has more restrictive rules i.e ban after 1 request.
Related
I want to Ban a specific bot with Fail2Ban.
Can anyone tell me how?
Here's the Bot you want to ban:.
MJ12bot, AhrefsBot, DotBot, SemrushBot
server environment:.
CentOS7
Nginx
Logs are stored in:.
/var/log/nginx/domain-name/access.log
/var/log/nginx/domain-name/error.log
You should edit /etc/fail2ban/jail.conf to enable the nginx-botsearch filter
[nginx-botsearch]
enabled = true
Then you will have to add your RegEx filters to /etc/fail2ban/filter.d/nginx-botsearch.conf
# Fail2Ban filter to match web requests for selected URLs that don't exist
#
[INCLUDES]
# Load regexes for filtering
before = botsearch-common.conf
[Definition]
failregex = ^<HOST> \- \S+ \[\] \"(GET|POST|HEAD) \/<block> \S+\" 404 .+$
^ \[error\] \d+#\d+: \*\d+ (\S+ )?\"\S+\" (failed|is not found) \(2\: No such file or directory\), client\: <HOST>\, server\: \S*\, request: \"(GET|POST|HEAD) \/<block> \S+\"\, .*?$
HERE_YOUR_NEW_REGEX
ignoreregex =
# DEV Notes:
# Based on apache-botsearch filter
#
# Author: Frantisek Sumsal
and finally you have to restart the Fail2Ban service
service fail2ban restart
I am using robotframework and I would like know what can I do so that I can avoid going through the proxy? I am using Windows and I have made sure there are no proxy variables in environment variables. I am also connected via WI-FI unlike others who are on LAN for who do not see issue with the GET request.
KEYWORD ${resp} = RequestsLibrary . Get Request session, /token, params=${user info}, headers=${HEADER}
Documentation:
Send a GET request on the session object found using the
Start / End / Elapsed: 20180102 13:17:11.558 / 20180102 13:18:26.569 / 00:01:15.011
13:17:11.566 DEBUG Starting new HTTP connection (1): proxy-sanfran
13:17:11.577 DEBUG http://proxy-sanfran:80 "GET http://localhost:9000/token?userName=test_user&password=test123 HTTP/1.1" 301 None
13:17:11.586 DEBUG Resetting dropped connection: proxy-sanfran
13:18:26.566 DEBUG http://proxy-sanfran:80 "GET http://10.138.37.40 HTTP/1.1" 503 787
13:18:26.568 INFO Get Request using : alias=session, uri=/token, headers={'Content-Type': 'application/json'} json=None
13:18:26.568 INFO ${resp} = <Response [503]>
try with setting environment variable
NO_PROXY = "localhost"
Considering that Pingdom is a high reputable site, I tried to subscribe for their uptime monitoring service. However, even though I setup for 5 minutes interval, their bot Pingdom.com_bot_version_1.4 keeps accessing my site not once every second, but tens of times every second! Resulting thousands of access per minute!
Then I tried to completely cancel the service but still get bombarded by their bot. I tried to block in robots.txt, but apparently they chokes it on. Next, I tried to block in nginx.conf with this command:
if ($http_user_agent ~* Pingdom.com_bot) {
return 403;
}
It works, but I see a lot of 503 errors in access.log. How to not log this bot? Really really annoying. I regret ever subscribed to their service.
Here is a post about blocking w00tw00t which you could easily adopt.
The easiest option for you to adapt would proablably beeing the fail2ban one.
using a fail regex triggering on your 403 error.
So something like
[Definition]
failregex = ^<HOST> .* "(GET|POST|HEAD).*HTTP.*" 403 [0-9]{1,} ".+" ".+"$
ignoreregex=
in /etc/fail2ban/filter.d/nginx-pindotban.conf
and
[pingdotban]
enabled = true
port = http,https
filter = nginx-pingdotban
logpath = /path/to/nginx/access.log
maxretry = 5
bantime = 360000
in /etc/fail2ban/jail.conf
You can test the regex with
fail2ban-regex logfile /etc/fail2ban/filter.d/nging-pingdotban.conf
Also the iptable variant could be adopted with something like
iptables -A INPUT -p tcp --dport 80 -m string --algo bm --string "the useragent" -j DROP
I am a bit confused about syslog message format. I have to write a program that parses syslog messages. When I read what I get in my syslog-ng instance I get messages like this:
Jan 12 06:30:00 1.2.3.4 apache_server: 1.2.3.4 - - [12/Jan/2011:06:29:59 +0100] "GET /foo/bar.html HTTP/1.1" 301 96 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)" PID 18904 Time Taken 0
I can clearly determine the real message (which is, in this case an Apache access log message) The rest is metadata about the syslog message itself.
However when I read the RFC 5424 the message examples look like:
without structured data
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
or with structured data
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID#32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...
So now I am a bit confused. What is the correct syslog message format ? It is a matter of spec version where RFC 5424 obsoleted RFC 3164 ?
The problem in this case is that apache is logging via the standard syslog(3) or via logger. This only supports the old (RFC3164) syslog format, i.e. there is no structured data here.
In order to have the fields from the apache log show up as RFC5424 structured data, apache would need to format the log that way.
The first example is not proper RFC3164 syslog, because the priority value is stripped from the header. Proper RFC3164 format would look like this:
<34>Jan 12 06:30:00 1.2.3.4 apache_server: 1.2.3.4 - - [12/Jan/2011:06:29:59 +0100] "GET /foo/bar.html HTTP/1.1" 301 96 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)" PID 18904 Time Taken 0
Traditionally rfc3164 syslog messages are saved to files with the priority value removed.
The other two are in RFC5424 format.
If you have access to the installed syslog-daemon on the system you could configure it to write the logs (received both locally or via network) in a different format. rsyslogd for instance allows to configure your own format (just write a template) and also if I remember correctly has a built-in template to store in json format. And there are libraries in almost any language to parse json.
EDIT: You could also make rsyslogd part of your program. rsyslog is very good in reading incoming syslogs in either of the two RFC formats. You can then use rsyslog to output the message in JSON. This way rsyslog does all the decompositioning of the message for you.
I'm trying to setup a stress/load test using the WCAT toolkit included in the IIS Resources.
Using LogParser, I've processed a UBR file with configuration. It looks something like this:
[Configuration]
NumClientMachines: 1 # number of distinct client machines to use
NumClientThreads: 100 # number of threads per machine
AsynchronousWait: TRUE # asynchronous wait for think and delay
Duration: 5m # length of experiment (m = minutes, s = seconds)
MaxRecvBuffer: 8192K # suggested maximum received buffer
ThinkTime: 0s # maximum think-time before next request
WarmupTime: 5s # time to warm up before taking statistics
CooldownTime: 6s # time to cool down at the end of the experiment
[Performance]
[Script]
SET RequestHeader = "Accept: */*\r\n"
APP RequestHeader = "Accept-Language: en-us\r\n"
APP RequestHeader = "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705)\r\n"
APP RequestHeader = "Host: %HOST%\r\n"
NEW TRANSACTION
classId = 1
NEW REQUEST HTTP
ResponseStatusCode = 200
Weight = 45117
verb = "GET"
URL = "http://Url1.com"
NEW TRANSACTION
classId = 3
NEW REQUEST HTTP
ResponseStatusCode = 200
Weight = 13662
verb = "GET"
URL = "http://Url1.com/test.aspx"
Does it look OK?
I execute the controller with this command: wcctl -z StressTest.ubr -a localhost
The Client(s) is executed like this: wcclient localhost
When the client is executed, I get this error: main client thread Connect Attempt 0 Failed. Error = 10061
Has anyone in this world ever used WCAT?
I'd look at updating to WCat 6.3 - available here for x86 and here for x64
They've changed the settings/scenario file strucutures, which is a little painful, but should suit your needs.
I've just started evaluating wcat 6.3 and I'm afraid my experience has been a bit disapointing in terms of online support/community.
There is also a major bug in the wcat.wsf script - see:
http://forums.iis.net/t/1153312.aspx
I'm now struggling with getting performance counter measurement working.
I've had good success with WCAT, though I'm struggling with simulating NTLM connections.
I'm using 6.3, so my config files look very different from yours. Some gotchas I noted along the way:
+ Make sure you've got your firewall turned off, or holes punched through for WMI.
+ Each thing you set in the request header has a tremendous impact on throughput. Apples to apples must have the same request headers.
+ Remote calls with multiple clients work only after correcting the bug identified by sthorogood.
Once I crossed those hurdles, I got great results from WCAT. It tests quickly, repeatably, and aggressively.
Best of luck,
Kevin
I don't have an answer for you, but have you considered using other tools for your testing? The WCAT tools seems pretty limited and complicated to use.
OpenSTA and JMeter are good open source tools for load/stress/performance testing.
OpenSTA and JMeter looks very Apache like. I'm running IIS on Windows Server 2003.
Have you looked at the Microsoft Web Application Stress Tool?
for performance counter you can define -p .prf in the same command run for controller as:
wcctl -c config.txt -d distribution.txt -s script.txt -a localhost - p performance.prf