I manage to send all my servers logs to a central syslog server. Now What? How do i make sense of this avalanche of data? does anybody know any good opensource syslog reader tool or any paid ones such as sawmill?
You might want to check out Sagan (http://sagan.io). Basically, it takes your logs as they come in and "detect" when bad things are happening. Sagan works a lot like the Snort IDS engine, but with logs instead of packets. Things that are detected can then be forwarded to a GUI like Snorby, Sguil, etc.
It's an open source project (GNU/GPLv2) and runs on *nix platforms. More information can be found at:
http://sagan.io
I know baretail, A free real-time log file monitoring tool
http://www.baremetalsoft.com/baretail/
Never tried it, but heard good things about it:
http://www.ossec.net/
It's perhaps more than what you asked for, but your question doesn't make exactly clear what you are looking for in the logs.
For displaying of the logs on a GUI you could check LogAnalyzer (http://loganalyzer.adiscon.com/)
Logstash, elasticsearch, and kibana is a nice modern stack for analyzing logs.
Related
I am trying to write a minifilter that more or less captures everything that happens in the kernel and was wondering if I could also capture "URLs"/network information; I stumbled upon windivert which seems to be using a .sys driver and also another thread which says we cannot get URLs in driver mode which leaves me a bit confused. If it is true then how does windivert do it?
I understand there is something called network redirect under minifilters on learn.microsoft.com which uses a dll and .sys file (same as windivert), but I could not find any resources that can help make me one.
Is there a better way to capture all visited URLs in real time?
Thanks in advance for any help or directions.
You're looking for Windows Filtering Platform and Filtering Platform Callout Drivers, which WinDivert is utilizing. This gives you the data that goes out over the wire, so for plain old HTTP over port 80 you can parse the requests to obtain the URL. This won't work for HTTPS since you're getting encrypted data over the wire; you'd have to implement some kind of MITM interception technique to handle that.
Just because of curiosity, I'd like to get to know better, how the Internet works.
I have already read quite some articles in Wikipedia, but these are very theoretical. I know, which protocols are used where, and what they are good for (IP, TCP, HTTP). But I have no idea how these are actually implemented in Linux or any other OS.
So, for me to understand these things better, I'd like to write a simple implementation of the Echo-Protocol in C, preferrably for Linux, and from ground up. My client-application should be able to send a message to a given port on any other computer, and my server-application should be able to receive this message, and send it back to the client.
Where do I start best? What should I read to get more information on these topics? Has anyone of you already done something like this, and if so, what did you learn?
In addition, it would be great if someone could tell me a bout a program that shows me the raw Internet-traffic that runs trough my PC. I think this would also help me understanding things a bit better.
Thanks in advance for your advice.
http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=berkley+socket+tutorial
http://beej.us/guide/bgnet/
http://www.tenouk.com/cnlinuxsockettutorials.html
http://www.paulgriffiths.net/program/c/sockets.php
In addition, it would be great if someone could tell me a bout a program that shows me the raw Internet-traffic that runs trough my PC.
Ethereal is pretty great for this.
I would like to know if there's an open source application that can:
-Being open-source (obviously free, no cost at all)
-Check which ports are being used and check the bandwith used by each of them.
-Based on requirements above create a weekly report. With details of each prt per day and time specifications.
I have read about Ethereal for the Network Monitoring and JasperReports for the Report-creation-stage, but haven't gone much on details yet..
If my specifications cannot be met with a free app then I would like to say that I could work with Java to check which ports are being used, but I still don't know if Java could handle ALL the requirements... please, I would really like to have an answer for that.. Because I could start working on it right now but I want to be sure Java can have everything covered.
PD: If Java can't be a solution what would you suggest?
suggestions for you:
Colasoft Capsa Free: http://www.colasoft.com
Spiceworks: new user, cannot give link.
Or google: free traffic monitor
I am looking for suggestions on doing some simple monitoring of an ASP.Net web farm as close to real-time as possible. The objectives of this question are to:
Identify the best way to monitor several Windows Server production boxes during short (minutes long) period of ridiculous load
Receive near-real-time feedback on a few key metrics about each box. These are simple metrics available via WMI such as CPU, Memory and Disk Paging. I am defining my time constraints as soon as possible with 120 seconds delayed being the absolute upper limit.
Monitor whether any given box is up (with "up" being defined as responding web requests in a reasonable amount of time)
Here are more details, things I've tried, etc.
I am not interested in logging. We have logging solutions in place.
I have looked at solutions such as ELMAH which don't provide much in the way of hardware monitoring and are not visible across an entire web farm.
ASP.Net Health Monitoring is too broad, focuses too much on logging and is not acceptable for deep analysis.
We are on Amazon Web Services and we have looked into CloudWatch. It looks great but messages in the forum indicate that the metrics are often a few minutes behind, with one thread citing 2 minutes as the absolute soonest you could expect to receive the feedback. This would be good to have for later analysis but does not help us real-time
Stuff like JetBrains profiler is good for testing but again, not helpful during real-time monitoring.
The closest out-of-box solution I've seen is Nagios which is free and appears to measure key indicators on any kind of box, including Windows. However, it appears to require a Linux box to run itself on and a good deal of manual configuration. I'd prefer to not spend my time mining config files and then be up a creek when it fails in production since Linux is not my main (or even secondary) environment.
Are there any out-of-box solutions that I am missing? Obviously a windows-based solution that is easy to setup is ideal. I don't require many bells and whistles.
In the absence of an out-of-box solution, it seems easy for me to write something simple to handle what I need. I've been thinking a simple client-server setup where the server requests a few WMI metrics from each client over http and sticks them in a database. We could then monitor the metrics via a query or a dashboard or something. If the client doesn't respond, it's effectively down.
Any problems with this, best practices, or other ideas?
Thanks for any help/feedback.
UPDATE: We looked into Cloudwatch a bit more and we may focus on trying it out. This forum post is the most official thing I can find. In it, an Amazon representative says that the offical delay window for data is 4 minutes. However, the user says that 2 minute old data is always reliable and 1 minute is sometimes reliable. We're going to try it out and hope it is enough for our needs.
Used Quest software and it seemed to be a good monitoring solution. Here is a link.
http://www.quest.com/application-performance-monitoring-solutions/
Also performance monitoring of Windows may also help.
I'm bored cause my development server is down and I'm running the command prompt to ping the server indefinitely so that I'll see when they stop timing out and know that I can work again. In the meantime I wanted to make an Air app that will do this for me, so I can have it chirp or alarm or do something when it starts to be able to connect to the server.
So I need to start with having AIR do the ping, does anyone know how to do this? I can't find any help on it. I can't use the HTTPService because whatever it is that is broken (think it's the VPN), our dev web site is still up.
Thanks
Otherwise you can use the NativeProcess class of the AIR 2.0 beta, and use the ping command of your operating system.
You can find informations on how to use it here.
According to the docs, SocketMonitor, descending from ServiceMonitor will check basic availability, but not a true ping. If you want to truly "ping" the server, you would need to use a socket an implement the protocol, This link to the Wikipedia article might help.
The AIR SDK offers some classes (URLMonitor, SocketMonitor) to handle this:
http://livedocs.adobe.com/flex/3/html/help.html?content=network_connectivity_1.html