For mobile Apps, it is a valid assumption that the network may be intermittent, or it may switch from one to another as the user keeps moving. For example, your device is connected to a startbucks wifi and you are using the App before you grab your coffee and walk out of the store -> Your mobile device network may switch from wifi to carrier network, 3G/4G/LTE. Even with the carrier network itself, it may switch among 3G/4G/LTE depending on their coverage at your position.
Question,
Will this intermittent network, or frequently network switch affect the http communication?
For example, an http request was sent out with Wifi, and while the server is processing the request, the device already switched to 4G. Will the device still be able to receive the response?
If Yes, how is Http or TCP designed to support this scenario?
If No, should we try to solve the problem from the application layer? and How?
Will the device still be able to receive the response?
For current practice, No. After network is switched:
Device's public IP address is changed.
TCP connection is based on IP protocol, so all current TCP connection would be destroyed.
HTTP is based on TCP connection, so it would be destroyed too.
Actually, you can make a simple experiment to verify this: Put a web page on internet and make the web server delay the page delivery for 30 seconds. Visit this page and switch network while waiting for the response.
However, this is a classic problem in mobile world, so some work is doing to give mobile device a constant IP, which will keep TCP&HTTP alive when device switches from one network to another. You can check Mobile IP in wikipedia for more information on various technologies and protocols.
If No, should we try to solve the problem from the application layer?
It depends on whether you can tolerate network interruption for your application. If it is a static web page, I think it is totally OK to leave this problem alone, and wait for Mobile IP technology improvement in future. If it is a highly network-dependent application, such as online video or stock market app, I think this problem should be solved in application layer.
and How?
There are 3 methods to fix/workaround this problem (maybe more):
Cache. Prefetch resources, so that when TCP connection is destroyed and reconnected, device can use cached resources. This works well in online audio/video apps, but it does not apply when no resource can be prefetched (realtime stock market data for example).
Take TCP re-connection as first priority. Check your code, when HTTP failed due to destroyed TCP connection, re-send HTTP request as early as possible.
Improve user experience when network interruption do happens.
Related
I am learning about Network Protocols, and see that near the lowest level is the raw ethernet protocol:
Ethernet: this is the basic protocol that sends data to another machine on your local network using your MAC address. This is the building block for all the rest as you need to send data to the router if you want to communicate with the outside world.
Above that is the Internet Protocol (IP), of which TCP and UDP are examples. On top of TCP is HTTP, or SSL on top of which you can put HTTP (so HTTPS).
It seems that the browser only lets you go down to the level of the IP (TCP for your normal message passing connections, and UDP for things like video). But I'm wondering, if it's possible to go any lower than that in the browser. If not, then wondering why not. If so, then what an example might be.
“It seems that the browser only lets you go down to the level of the
IP (TCP for your normal message passing connections, and UDP for
things like video).”
Firstly TCP and UDP are not examples of IP. They are Transport layer protocols, which is a layer above IP.
This representation of the 5 layer model should help make the distinction:
Secondly, if by 'go down to the level of Ethernet’ you mean inspect packets with your browser’s dev tools, you can’t do that. You can only see HTTP packets.
If you want to observe all traffic, you can use tools like Wireshark - or Fiddler to capture and inspect packets across the different layers.
If you are asking if the browser can use any layers other than the Application Layer. No, it cannot. That's also why you can only observe HTTP packets, because that's the layer which the browser operates on.
Information travels from layer to layer, but doesn't skip layers:
Your request for a web page uses all the layers. It starts at the application layer, which is the user-facing layer, all the way down the stack until the Network layer transports your requests to the server's Network layer, where it goes up the stack to fetch the web page.
The packet is the basic unit of information that is transferred across
a network. The packet consists, at a minimum, of a header with the
sending and receiving hosts' addresses, and a body with the data to be
transferred. As the packet travels through the TCP/IP protocol stack,
the protocols at each layer either add or remove fields from the basic
header. When a protocol on the sending host adds data to the packet
header, the process is called data encapsulation. Moreover, each layer
has a different term for the altered packet, as shown in the following
figure.
You can read more about the way packets travel across the protocol stack in this article about Data Encapsulation and the TCP/IP Protocol Stack
Browsers only allow to use a well-defined set of protocols due to security reasons. If this wouldn't be the case, Javascript on websites could send send arbitrary packets and cause malicious behavior.
Therefore the browser APIs are limited to make only HTTP, websockets and webrtc available.
In good old days (until about 3 years ago) browsers solved that problem by allowing to run a 3-rd party plugin, and there were (and there are still are) APIs for these plugins, such as NPAPI and ActiveX control. So you could create a plugin that does anything you want, send data with raw sockets or your own protocol. Flash plugin is the best example, and there are numerous less known ones, used in specific web applications, such as web conferencing, chat, secure data exchange etc... Some browsers still support these plugins - IE, Opera, Firefox, but Chrome and Edge completely disabled them. The reason is security issues - if you can load your own native code into browser then you can have a full access to OS, intercept all data in the browser, etc... As a replacement for this disabled freedom, some browsers tried to invent new more safe APIs such as PPAPI of Chrome. Mostly a failed attempt. Browser's adoption of websockets mitigated the situation, but still, lots of networking things are not possible in the browser. One notorious example is the lack of UDP Multicast support in browsers. Previously you could do it with your own NPAPI plugin; now no replacement is offered by browsers, so your browser-based multicast-involved applications are dead.
I have been writing my own version of the 802.11 protocol with network stack. This is mostly a learning experience to see more in depth on how networks work.
My question is, is there a standard for replying to client devices that a certain protocol is unsupported?
I have an android device connecting to my custom wifi device and immediately sending a TON of requests at the DNS port of my UDP protocol. Since I would like to test out other protocols I would very much like a way for my wifi device to tell the android device that DNS is not available and get it to quite down a little.
Thanks in advance!
I don't see a possibility to send a reply that a service is not available.
I can't find anything about this case in the UDP specification.
One part of the DNS specification assumes that there are multiple DNS servers and defines how to handle communication with them. This explains part of the behavior in your network, but does not provide much information how to handle it.
4.2.1 Messages - format - UDP usage
The optimal UDP retransmission policy will vary with performance of the
Internet and the needs of the client, but the following are recommended:
The client should try other servers and server addresses
before repeating a query to a specific address of a server.
The retransmission interval should be based on prior
statistics if possible. Too aggressive retransmission can
easily slow responses for the community at large. Depending
on how well connected the client is to its expected servers,
the minimum retransmission interval should be 2-5 seconds.
7.2 Resolver Implementation - sending the queries
If a resolver gets a server error or other bizarre response
from a name server, it should remove it from SLIST, and may
wish to schedule an immediate transmission to the next
candidate server address.
According to this you could try to send garbage back to the client, but this is rather a hack, or an error, but how does an error look like? Such a solution assumes that you have knowledge about the service that you don't support.
I believe that the DNS - requests can be avoided by using DHCP. DHCP allows to specify DNS-servers as listed in the linked page. This is the usual way that I know for a DNS-resolver in a LAN to get initial DNS servers although I don't find anything about this in the DNS specification. You can give the Android - device a DNS-server with DHCP so that it does to need to try to query your device. Querying your device could be a fallback.
Additionally to DNS there is mDNS which uses multicasts in the network to send queries. This seems not to be the protocol you have to do with because it uses the special port 5353.
Not possible to stop DNS in the way you intend. However, only for your tests you can check the UDP messages and find out the names the device is looking for. Then you update the hosts file (google how to do it: http://www.howtogeek.com/140576/how-to-edit-the-hosts-file-on-android-and-block-web-sites/) and add those names with some localoop IP address. That might work for your test.
Other possibility is to change DNS server to some localloop IP address: http://xslab.com/2013/08/how-to-change-dns-settings-on-android/
Again, this is only to avoid having all the DNS messages through the wifi connection.
Reading about Webrtc i get the feeling that "it will drop server bandwidth usage dramatically" except for "a few corner enterprise-firewall cases" where one needs a TURN server which relays the whole traffic between the peers.
For example, altough not webrtc related but the idea is similar, the wikipedia article of Chatroulette states: The website uses Adobe Flash to display video and access the user's webcam. Flash's peer-to-peer network capabilities (via RTMFP) allow almost all video and audio streams to travel directly between user computers, without using server bandwidth. However, certain combinations of routers will not allow UDP traffic to flow between them, and then it is necessary to fall back to RTMP.
Also similar articles on Webrtc focus on "yeah there might be problems with firewalls so you need a TURN server but ignore this and look at my awesome PeerConnection javascript code".
What i don't understand:
A Connection between two peers requires a server socket to be open so the peers can connect to it. Even UDP requires the concept of a udp server socket. Since nearly all not-server internet connected peers are behind some kind of router. E.g. every smartphone uses a wifi router, desktop PC's use the router of the service provider, ...
It shouldnt be possible to connect to a server socket hosted on a smartphone (browser webrtc server socket) or desktop cause of the router/firewall.
Thus my understanding is practically no two peers which need to send their traffic through the internet will be able to use a direct P2P connection, right?
So the only useful case to use Webrtc is in a LAN like environment, right?
Furtherly in case of a video chat service like chatroulette based on webrtc would need to use a bunch of TURN servers to relay nearly ALL traffic. Which makes Webrtc equally costly regarding server bandwidth like hosting my own solution.
So my question is: Am i right? If not what is the technical detail that allows a PeerConnection to be used without a TURN server but for two nodes separated by the Internet? How is the connection established on Layer 4 the TCP/UDP Transport Layer? Is it using UDP and all wifi routers allow hosting UDP server sockets or such? Which wouldnt make much sense cause of NAT and security.
UPDATE 1:
Digging a bit further i found what "symmetric nat" means and what it has to do with enterprises: In most enterprises it seems that the device connected to the internet has symmetric nat implemented. This means that the routing table which maps internal "internal-ip:internal-port" tuples to "internet-ip:internet-port" also stores "destination-ip:destination-port". So such routes/nats store a table for every (tcp?) connection having 6 columns "internal-ip:internal-port:internet-ip:internet-port:destination-ip:destination-port". This means no one else but the destination is allowed to communicate with internal-ip:internal-port.
Whereas non-enterprise-routers seem to only store the "internal-ip:internal-port:internet-ip:internet-port" combination. Thats also what is meant as "poke a hole in the firewall".
You're not right. All peers have IP addresses in order to communicate, and can be reached on those same addresses, provided a firewall allows it.
NATs tend to be optimized for client-initiated client-server traffic only. That typically means they initially allow outbound traffic only, and only allow inbound traffic on the same line after outbound traffic has happened. Perfect for servers. See this WebRTCHacks article for an intro to the problem.
This is where ICE comes in to attempt to poke holes in the firewall from the inside (client-side), in order to establish a line of communication directly between two peers, without needing any "server" socket, whatever that means.
How ICE works is quite complicated, and is explained in detail in the RFC.
But in broad terms it works in a number of steps:
Each peer (e.g. browser) has an "ICE agent" that collects candidates. Candidates are addresses (IP:port numbers) at which this peer can be reached, including:
Host candidates: e.g. immediate LAN/wifi/VPN IPs of the machine.
Server-reflexive candidates: public (outside-NAT) addresses of the machine, obtained by bouncing requests off mirroring (STUN) servers on the internet.
Relay candidates: addresses to a shared TURN server to forward data if all else fails.
Once discovered, candidates are inserted into the local SDP, and trickled over the signaling channel to the other peer, where they are inserted in it's remote description, where the other agent sees them.
Once an ICE agent has both local and remote candidates, it starts pairing local and remote candidates, and checks them for connectivity by sending STUN requests on them (effectively attempts at reaching the peer).
Successful pairs are ones both ICE agents have gotten a response back on (a 4-way handshake if you will).
If there's more than one successful pair, they're sorted by some metric, and the best pair becomes selected.
The selected pair is then used to send media over. One pair is needed for each track of (video or audio) media.
If a better pair is found later, the selected pair may change, affecting what address media is sent on.
TURN should only be needed in cases either where both clients are behind symmetrical NATs, or UDP traffic is blocked entirely.
I'm not sure if this is the correct place to ask, so forgive me if it isn't.
I'm writing computer monitoring software that needs to connect to a server. The server may send out relatively urgent messages, such as sound or cancel an alarm, and the client may send out data about the computer, such as screenshots. The data that the client sends isn't too critical on timing, but shouldn't be more than a two minutes late.
It is essential to the software that portforwarding need not be set up, and it is assumed that the internet connection will be done through a wireless router that has NAT almost all the time.
My idea is to have a TCP connection initiated from the client, and use that to transfer data. Ideally, I would have no data being sent when it is not needed, but I believe this to be impossible. Would sending the equivalent of a ping every now and again keep the connection alive, and what sort of bandwidth would it use if this program was running all the time on the computer? In addition, would it be possible to reduce the header size for these keep-alives?
Before I start designing the communication and programming, is this plan for connection flawed? Are there better alternatives?
Thanks!
1) You do not need to send 'ping' data to keep the connection alive, the TCP stack does this automatically; one reason for sending 'ping' data would be to detect a connection close on the client side - typically you only find out something has gone wrong when you try and read/write from the socket. There may be a way to change various time-outs so you can detect this condition faster.
2) In general while TCP provides a stream-oriented error free channel, it makes no guarantees about timeliness, if you are using it on the internet it is even more unpredictable.
3) For applications such as this (I hope you are making it for ethical purposes) - I would tend to use TCP, since you don't want a situation where the client receives a packet to raise an alarm but misses that one that turns it off again.
A friend of mine made a small LAN-playable game, and ask me to change it, so it could be playable over the Internet. I don't want to make huge changes on the client application.
When a game is created, the server keep sending UDP BROADCAST packets to tell everyone that a game has been created. Now, I just need to change this BROADCAST in order to send these packets to a group of internet IP-adresses.
Can you tell me if the following solution is a good one: I would create a room server, lets call it 'room-broadcast-server', that contains IP adresses of everyone who joined the room. Then, the clients, instead of sending that BROADCAST packet, would send the packet to the room-broadcast-server, which would broadcast this packet to everyone who joined the room.
The problem is: The clients would receive these packets from the 'room-broacast-server' and they would try to comunicate with the room-broadcast-server, instead of comunicating with the machine that created the game. I would like to fool the clients, so that they think the packet came from the game server, and not from the room-broadcast-server. How can I make it?
Are you modifying only the server, or both the server and the clients? I would think that it would be simpler to just remove the broadcasts altogether and have the clients explicitly choose the server they want to connect to, rather than relying on the server broadcasting.
As a version 1, you could simply require players to type in the IP address/DNS name of the server in the client in order to connect.
For version 2, you could add support for a "lobby" where you have a (known) central lobby server that both clients and server connect to in order to find each other (so servers connect to the lobby to announce their presence and then clients connect to the lobby in order to browse servers that they want to connect to).
In a game that I have been writing (but on hold currently due to lack of free time :p) I had written the "lobby" server as a simple PHP+MySQL web application and had the clients and servers use HTTP requests to poll it for updates and so on. That way, I could host the central lobby server on a cheap web host and anybody could host games (the drawback is that cheap web hosts don't allow arbitrary socket connections, so I wasn't able to implement NAT punch-through on it, but if/when the game became popular my plan was to move the lobby server to a more expensive host that did allow arbitrary socket connections...)
Spoofing the source address isn't a suitable mechanism for your normal application protocol. It requires special permissions on client machines, it will be dropped by some networking filters, and is just generally gross and anti-social.
Since you're modifying the clients anyway (to send the messages to the game server instead of to the broadcast address), you can simply have the game server append the "true source" of the packet to each packet it sends out, and have the clients expect and process this information in the packets that they recieve from the game server.
The basic concept (one centric lobby server + multiple game servers) of the 2nd option from Dean is good and very common even in retail online games. One thing I don't understand is why you want to fake IP addresses. Clients should not care what server ip address is as long as it gets valid packets from it. Also, imo, you don't need to create a separate room(?) server since a game server can / should manage a list of clients as clients are connected via a lobby server.