Are there any source base filtering is applied at google end as I have a problematic source IP cannot reach or receive pings from google side but other source IPs are fine
I am expecting insights and if need to reach google what is the process to get them check this
Related
I am trying to write a minifilter that more or less captures everything that happens in the kernel and was wondering if I could also capture "URLs"/network information; I stumbled upon windivert which seems to be using a .sys driver and also another thread which says we cannot get URLs in driver mode which leaves me a bit confused. If it is true then how does windivert do it?
I understand there is something called network redirect under minifilters on learn.microsoft.com which uses a dll and .sys file (same as windivert), but I could not find any resources that can help make me one.
Is there a better way to capture all visited URLs in real time?
Thanks in advance for any help or directions.
You're looking for Windows Filtering Platform and Filtering Platform Callout Drivers, which WinDivert is utilizing. This gives you the data that goes out over the wire, so for plain old HTTP over port 80 you can parse the requests to obtain the URL. This won't work for HTTPS since you're getting encrypted data over the wire; you'd have to implement some kind of MITM interception technique to handle that.
I'm facing a challenging problem here that don't know how to resolve:
Context: I have a game launcher that connects to my server and if doesn't detect any cheating software on the player computer, launches the game and tells the server to allow that IP to connect to the game server.
This has many potential issues like if there are multiple players under the same IP but I make a queue in that case so all is fine until here.
Now the main problem is that I don't have control over what information sends the game, I can only modify the launcher. For this reason all is IP based as that's the only way I have to identify that a certain player is logging in and has been authorised by launcher. It goes:
Launcher connects to Server and tells him to Allow IP A.
Server replies: ok (save IP A)
Launcher starts game.
Player tries to login.
A connection is established to the server, server checks if origin IP (IP A) is allowed to log in, if yes, go ahead.
So, the system even though far from ideal, does the job, and considering game is compiled and we cannot modify it, I couldn't think of better way.
Anyway now we come to the problem:
Certain players, when they open the launcher, all goes fine, game launches but then when player tries to login server denies connection as it comes from a different origin IP!
That broke up my mind, how can two tcp connections made within a few seconds of difference from client A to server B have a different client source IP? obviously this ruins all my system. I even tried to periodically fetch IP from sites like whatismyip to see if it was changing overtime but it wasn't the case, it seems like maybe because it goes to another port, or I don't know the reason, sometimes changes it and sometimes it doesn't.
It seems to be related to players being using tethering internet connections,as I e never seen this before on a common internet connection.
So basically, I'm not sure what could I do to identify/relate those two connections and this is a big problem as many players are unable to join my game and I cannot let them join without the launcher for obvious reasons.
My random ideas to resolve it range from bad to terrible:
open multiple connections to server on different ports and see if that gives different source iOS
let player connect and then do some kind of validation based on netstat check on client: when player is connected to game server I should see it there and could send that info to server, server would kick any client connected if there's no validation from launch, however, I think I would still have the problem to link both connections.
maybe there's another way that I'm not aware of to identify this connections. Assume I have full control in server side and in launcher, but I cannot change the game server packet that does the "login" attempt.
Based on your assumptions (IP-based only, game/server unmodifiable), it looks like we are hitting a wall indeed..
For the moment the only thing that comes to mind is performing multiple requests to the server instead of one, and until the user finally logs in.
I mean:
Periodically: launcher connects to server and tells him to Allow current IP. Server saves this IP and hopefully at some point you will have discovered all IPs.
Do this in the backgound until the player is finalizing its login (or a fixed period of time)
With some luck, if you open multiple connections during the whole period of time needed to start the game and login, you will have discovered and allowed all IPs of the user. This will mitigate the issue but not eliminate it.
I'll edit this post if I think about something else.
I have big access log (~30Gb) and I'm looking for ways for find suspicious/bot IP addresses. Of course, we can replace IP with (IP + User_Agent). So my questions are:
find average requests number that are done from any IP
find IP addresses that are making more requests than average (see previous point)
find IP addresses that are doing requests regulary (every hour for example) during day
you recommendations about how to detect bot
This log is rather big and I don't think that R lang could process it. Should I use some kind of storage behind R (hadoop or something similar)? I absolutely have no experience in processing/analyzing big data so any ideas, recommedations & tuts/articles are appreciated.
The access log probably contains a lot of data which you may not need based on your question, if you only care about the time of the request and the orginating IP you could easily reduce the data size by extracting "columns" from the input before reading it into R, some standard command line tools such as cut or awk should do the trick.
If you want to keep more details another option could be to load the access log into a database and use this for further processing, 30GB is not a lot for a database, but of course this means some additional work: design a datbase schema and a way to load the data in the database.
You can also do the following type of analysis
Getting the geo location of IP addresses and comparing access frequency based on geo_location + time at geo_location (the access frequency could be normal during day time at the geo location but not after midnight)
If you have username information, check whether multiple IP addresses are using the same username during same time period
WSO2 has done some Anomaly Detection work using their Analytics Platform which is pretty scalable for most anomaly detection scenarios. Check it out - http://wso2.com/analytics/solutions/fraud-and-anomaly-detection-solution/
This might be a better option than doing through R, since it allows you to do complex event processing (through SQL like queries) as well as machine learning.
You can also do the following type of analysis
a) If the IP address is from data center range, it is likely from a bot than normal user.
b) If the IP address is from search engine range, it is high likely from a search engine bot.
You can get the geolocation database from IP2Location which has the usage type information to detect data center or search engine.
Check goaccess.io - works for me. With logs for different websites, distributed on several servers. Allows usage of GEOiP and identifies bots out of the box.
Check out https://ipdetective.io it tracks IP addresses that originate from datacenters, vpns, proxies, tor node and bot nets. It offers a free API as well so you can test it out.
Are there any tools to trace the exact HTTP requests sent by a program?
I have an application which works as a client to a website and facilitates certain tasks (particularly it's a bot which makes automatic offers in a social lending webstite, based on some predefined criteria), and I'm interested in monitoring the actual HTTP requests which it makes.
Any tutorials on the topic?
Some popular protocol/network sniffers are:
Wireshark (previous the famous Ethereal)
Nirsoft SmartSniff (using WinPcap)
Nirsoft SocketSniff (allows you to watch the WinSock activity of the selected process and watch the content of each send or receive call, in Ascii mode or as Hex Dump)
Microsoft's Network Monitor (and a list of video-tutorials here, note video 'Advanced Filtering 2 of 2' where they specifically filter on process)
Wikipedia article 'Comparison of packet analyzers' has a nice overview of some other tools to.
Alternatively you could also look into (man-in-the-middle) proxy tools like:
Fiddler
mitmproxy
Both of the above actually record/decrypt/modify/replay HTTPS to!! You'd need to point the application you are monitoring to this proxy. If nothing else uses that proxy the log would be application/process specific and another upside to this approach is that one could also run the monitor/logger on a different machine.
Once you choose a tool, you can easily google a tutorial to go along with it.
However the core idea is usually the same: basically one sets a filter (on capture itself or display of captured data) on things like protocol, network/mac address, portno, etc. Depending on the tool, some can also filter on local application.
Hope this helps!
Take a look at HTTP Toolkit (disclaimer: it's my project).
Totally automatic HTTP & HTTPS interception, with zero setup, isolated to just the code you want to debug.
You can open a browser with it, and see all the traffic from that one window immediately (but no others), or run a terminal and automatically see all traffic only from processes started from that terminal. Built-in HTTPS decryption for everything, with no risky system-wide certificates and no manual setup. Let me know what you think!
If memory serves me correctly, google does this for the maps site. I know google's mobile maps app can determine the rough location (I assume using some kind of cell tower lookup), yet I seem to remember the site getting somewhat close to the current location when viewing on a mobile browser.
Anyone know how/if that's possible? Does the IP address change based on the tower or area (seems like they'd be using some kind of gateway common to the carrier)?
With a regular HTTP request the only option you have is GeoIP but this will only give you the country & operator no better.
Depending on the application & device you are using there are various option for geolocation, for example iPhone & Android browsers suport the HTML5 geo javascript functions so you could put some code in the page to send you the location. However these all involve some client side code.
The other option involves doing a deal with the networks, they can supply you the phone number as an additional header in the request and you can then query this number via an API from the operator to get the location of the phone from the network. However they charge a pretty penny for this function and the accuracy can vary from about 100m in an urban environment to 35Km in very rural areas.
This is Not possible.
1.) Location By querying Network Details.
Even if you get the approximate location from an installed app by querying network or cell-id details, it'd not be that reliable. You won't be able to perform anything based on that approximate location.
2.) Location By querying IP.
Secondly, in one country, a single telco will have it geo-stationed GPRS or EDGE gateways and all devices on network are connected to internet with that gateway. So basically you can't rely on location details by IP, because that would be the IP of the Gateway and not the IP of device(in case of EDGE).
Ultimately, if you need to do anything that is based on the location, the
location itself needs be precise and not approximate.
We were using Google Gears for some time and that worked on Windows Mobile version 5.0 plus devices. Google recently stopped that service. Yahoo have now started something called Yahoo Fire Eagle. Sounds good, haven't yet checked it out. I dont have a Yahoo account, yet.
You have Google API Documentation on this:
http://code.google.com/apis/gears/geolocation_network_protocol.html
Hope it helps...
A little late in the day, but another option not mentioned is the MaxMind geo IP lookup. Enter an IP address and it will provide you with a location. There is free service (with simple API) that provides city level accuracy. There is also a more accurate database available for a fee. I have noticed during testing that there are multiple towers / IP addresses in urban areas and the MaxMind database is aware of these. You'll get location, plus the telco provider.
http://www.maxmind.com/