I'm building a scraper, and in order to not get banned, I have to make my requests from different IP addresses. Right now I'm doing that with a VPN, but it has several issues.
As I was reading about the topic, everybody was advising the methods I currently use, but those have their own problems.
I noticed that connecting to the same network from different devices gives different IP address. Furthermore, the same device connected to the same network, through WiFi, or through LAN gives different IP address. I'm not too experienced in networks, and don't know if those different addresses are really different, or will reveal that the requests are coming from the same network, but may be a good idea.
And if it is, how can I take advantage of it? Is there a way to programmatically create "fake devices" from the same device, getting different IP addresses? (I'm thinking about VMs or something)
Your public IP address is the one allocated to you (or to a group of Internet users you belong to) by your Internet provider.
On your side of your Internet box, the IP addresses are local addresses. Those local addresses cannot be seen from the outside. So you can change your local IP address as much as you want, the outside world will see you with the same IP.
Your solution could be to rent a server on the cloud and run your scraper on it.
Related
I am trying to understand how exactly routing works:
if 2 computers are on the same network
if they are on different networks.
More specifically I am trying to understand this: Routing
I am also trying to understand the difference between
IP Address,
Net Address,
Mac Address.
From what I understand:
1) IP Address: is used when computers communicate on the internet only.
2) Net Address: is a local version of the IP address and each device on the network has a unique net address. It's used when devices on the same network want to communicate with each other.
3) Mac Address: is a globally unique address and no other computer in the world has the same Mac address. In reality this is not true because it can be changed. It's used when ???
When a computer wants to communicate with another on the same network, they use net address, right? If the computers are on different networks what exactly happens?
Question: Can someone please fix my mistakes if any and explain what I am missing?
Thank you very much.
There are many network types, but since the most used ones are Ethernet and IP networks (and you seem to be asking about them), I will answer shortly based on them.
IP addresses are always used. They may not be used for deciding who gets the packets directly, but they are the basis even in local networks, since it is an IP network. There can also be other network types that have their own mechanisms, but they are not that common.
In local Ethernet the machines ask via ARP protocol "who has this IP address?" and get a reply with a MAC address. After that they send and receive packets based on that MAC address. The packets still have the IP address information, otherwise the receiving machine wouldn't know what is the destination. Do note that the receiving machine might be a firewall or other middleware device, not the actual computer that has the address. Also a single machine and network card may have several IP addresses set up for it.
In IP networks the IP address is used for routing. All routing devices have a routing table that will tell where the packets should go. If it's a simple device, it usually has a local network and everything else goes via a default gw, which will know better what to do with the packets.
A home router will just push them to the operator, there another router will know what addresses go to their networks, others are pushed forward via another connection, until a bigger place is reached where there are inter-operator connections and they choose again the correct route. And then it goes to smaller and smaller pipes the other way around.
My friend wanted to connect to my computer using Remote Desktop Connection. But the problem is I am confused what my Ip address is.
My computer is connected to the internet via router via broadband internet network. My ip address is dynamic.
Here, my main purpose is not only the remote connection but also learning how dynamic ip connect to another pc.
I searched for ip address on Google. They show me an ip address. But I think it is not mine, it's related with the router or broadband network. I also find a WAN ip (it is different from that i found on google) on router settings. It did't work.
I used Team Viewer. It worked perfectly. But I want to do that manually because I am going to make a multiplayer game on GM8.
It will helpful if someone explain about ip and port forwarding.
Teamviewer is a great tool, but uses different techniques than what you plan to do. Teamviewer always uses an outgoing connection and use a mediator on the Internet to connect you and the other PC.
You should ask your Internet provider if he technically enables you to be reachable from the outside Internet. Often this is not possible at all, even if you configure your router the correct way.
When you ask this you can ask him if you have a static IP.
It seems you are not aware of basics of IP networking, so I'd strongly advise against trying this on your router as wrong settings would render it useless. But here's for your information how port forwarding and IP Address and dynamic DNS can be used to solve your problem.
Basically your ISP is likely to give you a router having an IP address. If this IP address is a global IP address, it is possible to connect to this IP from outside. How do you find out whether your IP address is global? Look for your WAN IP address setting. If it is in 10.x.x.x or 192.168.x.x range, it's unlikely to be global and in that case it might not be possible to connect to your computer from outside - without help of a third server (some kind of a registration server, where you connect and register your application). The Registration server would determine your globally visible IP address and then convey it to another Application who is interested in connecting to it. This is somewhat complicated to make it work (but if you intend to make a game - this is something you'd have to do regardless). This is mostly how software like TeamViewer would work.
If you have a global IP address - it means it can technically be reached from anywhere in the world. In that case you could use port forwarding to make things work for you. Port forwarding works basically as follows - You expose a certain port (on TCP) to external world - say 8000 and then you make a setting like following on your router.
<TCP>-<RouterIP>-8000 --> <TCP>-<Your LAN IP><Your application Port>
(You can find you lan ip using ipconfig on windows or ifconfig on Linux).
Now all connections coming to port 8000 would be directed to your application. You might want to do it on UDP as well and the protocol above would change. That is how you 'open' a few ports to be accessible from outside, configure them on your router and then run corresponding applications on your network.
There's another thing called dynamic DNS, where the IP address you use if it is dynamic (and global) can be registered with a Dynamic DNS server so that you don't have to know and remember the current WAN IP Address. But that can be for later.
Hope that helps.
can one location have more than 1 IP address? I have 2 IP addresses and need to know if they originate from the same source. Thanks
You're missing clear definitions of "location" and "source" in your question but lets go with the assumption that you mean "physical machine" for both (as it's obvious that multiple machines will have different addresses and that a single machine can change its IP address over time).
In that case, the answer is yes. The operating system may bind as many IP addresses to a physical network port (and a single MAC -- the physical addressing used by Ethernet) as it wishes.
Binding multiple IP addresses was the standard way of doing "virtual web hosting" before HTTP/1.1 arrived with the "Host" header. The provider would use DNS to map different host names to different IPs on the same network (usually the same subnet as well) and then assign all of them to the same interface. The webserver would get address information from an incoming connection and based on the local IP address would know which virtual host was being accessed.
This led to a higher-than-typical use of public IP addresses but the practice is now gone with the proliferation of HTTP/1.1.
I'm not sure how Windows presents it, but Linux will present a physical interface with multiple IP addresses as multiple logical interfaces such as "eth0", "eth0:1", "eth0:2", etc. Each logical interface has a unique IP address even though they share the same physical interface.
This is hard to tell. Especially it is hard to tell if these IP addresses are from different times. Today I may have a different IP address than tomorrow.
Even they are from the same time, a load balancer im my internal equipment might send my packets over the fail-over line if the 1st one is overloaded or broken down.
One network adapter normally has only one IP address at once and a typical end user only has one network connection active, but even then the IP address can change. The user could switch from wireless to wired and back or a power outage might reset the ISP's DHCP server (assigning everyone a new IP address).
If you want to identify the user even if his IP address changes, you need to identify the user by his session id, stored in cookies. As two users can have the same IP address (a whole company could be behind a NAT), you should never rely on IP addresses for identification.
I would like to get the user's IP locally without any invoking any service.
Looking for a purely client side solution.
In general this problem is not solvable because the client computer might not even know its own external IP address. This happens when the client is behind a NAT or some other kind of firewall.
Note that many different client machines might share the same external IP address, and also different client machines might have the same internal IP address but different external IP addresses. Or, I can even imagine a situation where two different clients have the same internal and external IP addresses.
If you would explain why you believe you need this information, another solution to your actual problem might be possible.
Is there an elegant way to make a program detect a new computer that is connected to the network?
I would like my program to "auto-sense" a new computer being connected on the network (they're on the same network). Like a USB device being connected to the computer.
What I'm doing now is to save a list of all computers in the network from time to time. Another approach is to PING all available IPs on the subnet.
Are there any other elegant approaches?
Thanks!
Listening for ARP requests is the canonical way to do this. Independent of DHCP or not, any connected computer that wishes to communicate with the outside world will have to make an ARP request for the address of the default router. This request will go out as a broadcast, and contain the source interface's MAC and IP adresses.
If the other computer uses DHCP, it will make an ARP request for it's own address as part of duplicate address detection, which is also a broadcast you can snoop on.
(This works more or less the same way for IPv6, except you need to look for neighbor discovery or router soliciation packets instead.)
Like the answer alluded to, if you have a switch to which you can telnet or use SNMP on, you can extract the MAC table. That will give you a list of MAC adresses on each port in the switch. If you want the IP addresses however, you still need to listen for ARP:s.
On the other hand, if you have access to the default gateway on the network, you can also look at the ARP table there. That will give you MAC and IP addresses for anyone that has recently (for different values of recently...) communicated with it.
If you have a managed switch of some kind, you could probably connect to that, that would be a fairly elegant method.
If you're on a domain, you can can get a list of all the machines joined to the domain from the domain controller.
Failing that, all I can think of is either a challenge/response thing (e.g. pinging them) or by detecting traffic sent from them (see this question maybe as a starting point?), neither of which strike me as an elegant approach.