Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
I'm about to release a MMORPG. In my game, every 1 second, each player sends 30 TCP messages and gets back from the server 30. Every message is not really long. Like 20~ chars.
the point is that I never got my hands with multiplayer games. I have programmed all the server and client, but I don't know what server I'm gonna need. I mean, RAM, CPU, etc... I still don't know what to be ready for, but let's say for 15K same-time clients. As said, every 1 second every client need to get and send 30 TCP messages, and in the most cases I need also to update my non-SQL DB with the data.
Update: It's a multiplayer game, I must have 30 msgs/sec. Most of the msgs are for the current position of the player. Also I'm using C++.
It depends on what your (already implemented) server requires. You'll never know until you try some particular hardware. Rent a powerful dedicated server for a month and profile your game server. At least just check CPU usage. You'll need multithreaded asynchronous networking.
Details you provided help only to calculate how much bandwidth you need:
~94 bytes (TCP + IP + Ethernet headers) + ~20 bytes (your data) = 114 bytes every packet * 30 per second * 15000 users = ~50MBps * 8 bits = ~400Mbps of both incoming and outgoing traffic. Seems you're in troubles here. Consider something wiser than sending your every packet in separate TCP frame. E.g., implement buffer that collects data ready to be sent and is filled by your game logic threads and separate thread that sends this data to network. In this case your several soft packets can be combined into one TCP packet greatly reducing your bandwidth requirements.
But even after this you're still in troubles. I'd recommend to wait for users first before investing into complicated solution. When you'll have them you'll need to implement some kind of clustering. It's separate story, much more complicated than simple networking.
Try to handle at least 1K users by single server. This can bring you some money to hire somebody experienced in game networking/clustering.
If you know you sending 30 messages every second, why not bundle them into 1 request, every second? Makes a lot of difference in terms in server resources...
And in which language are you going to run your server? I hope you write something dedicated to process/manage these connections? If so: do some profiling and just measure what you need...
And what is your processor doing every second, to process those 30*15K messages?
There is no generic answer to your question. It all depends on your software.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
If Chris and Pat want to exchange a text message, they send and receive via their network providers, which charge them for a connection.
If Chris and Pat are both located in New York City, and there are enough wireless devices between Chris and Pat all close enough to each other to form a continuous chain, is it possible for all those devices to be programmed to cooperatively forward packets amongst each other, bypassing the need for network providers?
It would seem the "address" of each device would have to include current geographic coordinates, and devices would have to report their movements frequently enough so routing attempts could still find them, but the speed and capacity of devices nowadays could handle that, right?
Would such a network be viable? Does it already exist or has it been attempted? Is there some kind of inherent programming problem that is difficult to overcome?
There are a few interesting things here:
Reachability. At least you need to use a technology that can do ad-hoc and peer-to-peer networking. Of those technologies only bluetooth, NFC and WiFi are more or less often implemented. Of those again only wifi currently may have the strength to connect to devices in other houses or to the street, but even there typical ranges are 30-60m (and that's for APs, it might be lower for UEs).
Mobility. ANY short-range wireless communication protocol has difficulties with fast-moving devices. It's simple math, suppose your coverage is 50m in diameter, if you move at about 20km/h or 5.5m/s, you have less than 10s to actually detect, connect and send data while passing this link. Oh, but then we did not consider receiving traffic, you actually have to let all devices know that for the next 10s you want to receive data now via this access network. To give an example, wifi connectivity times with decent authentication (which you need for something like this) alone takes a few seconds. 10s might be doable, but as soon we talk about cars, trains, ... it's becoming almost impossible with current technology. But then again, if you can't connect to those, what are the odds you will cross some huge boulevards with your limited reachability?
Hop to hop delays. You need a lot of those. We can fairly assume that you need at least a hop each 20-30m, let's average at 40 hops/km. So to send a packet over lets say 5km you'd need 200 hops. Each hop needs to take in a packet (L2 processing), route it (L3 processing) and send it out again (L2 processing). While mobile devices are relatively powerful these days I wouldn't assume they can handle that in the microseconds routers do. Next to that in a wireless network you have to wait for a transmission slot, which can actually take in the order of ms (each hop!). So all in all, odds are huge this would be a terribly slow network.
Loss. Well, this depends a bit on the wireless protocol, either it has its own reliable delivery protocol (which will make the previous point worse) or it doesn't. In that last case, suppose your wireless link has about .1% loss, or 99.9% no-loss, this would actually end up with an 18.1% loss rate for the 200 hops considered previously ( (1-0.999**200)*100) This is nearly impossible to work with in day-to-day communications.
Routing. lets say you need a few millions of devices and thus routes. For traditional routing this usually takes some very heavy multicore routers with loads of processing power. Let's just say mobile devices (today) can't cut that yet. A purely geographically based routing mechanism might work, but I can't personally think of any (even theoretical) system for this that works today. You still have to distribute those routes, deal with (VERY) frequent route updates, avoid routing loops, and so on. So even with that I'd guess you'd hit the same scale issues as with for example OSPF. But all-in-all I think this is something that mobile devices will be able to handle somewhere in the not-so-far future, we're just talking about computing capacity here.
There are some other points why such a network is very hard today, but these are the major ones I know of. Is it impossible? No, of course not, but I just wanted to show why I think it is almost impossible with the current technologies and would require some very significant improvements, not just building the network.
If everyone has a device with sufficient receive/process/send capabilities, then backbones (ISP's) aren't really necessary. Start at mesh networking to find the huge web of implementations, devices, projects, etc., that have already been in development. The early arpanet was essentially true peer-to-peer, but the number of net nodes grew faster than the nodes' individual capabilities, hence the growth of backbones and those damn fees everyone's paying to phone and cable companies.
Eventually someone will realize there are a million teenagers in NYC that would be happy to text and email each other for free. They'll create a 99-cent download to let everyone turn their phones and laptops and discarded devices into routers and repeaters, and it'll go viral.
Someday household rooftop repeaters might become as common as TV antennas used to be.
Please check: Wireless sensor network
A wireless sensor network (WSN) of spatially distributed autonomous sensors to monitor physical or environmental conditions, such as temperature, sound, pressure, etc. and to cooperatively pass their data through the network to a main location
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
Firstly, thank you for taking the time out to read this post.
I'm looking to develope a TCP/IP enabled device using the Microchip PIC18 or PIC32 family of embedded microcontrollers with Microchip's TCP/IP Stack. However, my knowledge of networking is pretty basic at the moment, thus the reason for this post.
Can anyone recommend the best protocol to use for my TPC/IP embedded device so that it can communicate with a server in a data centre? My intention is to have the embedded device located at a remote location somewhere over the internet, where the server can communicate with the device and download data such as thermometer probe readings to be stored in a database. I would also like the ability for the server in the data centre to be able to reconfigure settings and variables on the remote device should I need to.
My research on protocols so far has lead me to the following options:
SNMP v3 (version 3 due to encryption and authentication)
UDP (though I read this can be unreliable but is fast)
TCP (I'm not too clued up on this yet)
Can anyone offer me advice on the best route to go down? I'm not expecting a detailed answer from you, but I would really like an idea of what topics/protocols to look into and research.
My intent is to deploy many of these embedded devices over the internet where they all send their data back to the server.
I assume that the remote embedded device will have to connect to the server rather than vice versa as the server will have a static IP address or DNS name, whereas the remote device addresses will be unknown.
Any advice on this would be greatly appreciate. Please don't hesitate to ask if I've missed out any key information in this post.
Many thanks.
Rob
* UPDATE *
It was pointed out that I'm probably misusing the term Web Server, so I've amended my post to mention Server in a Data Centre instead. Thank you for pointing this out to me.
If the target is a Web server you don't have any choice. You have to use HTTP, which runs over TCP.
Or else you are misusing the term 'Web server'.
In many ways this depends on your specific requirements. TCP/IP is able to provide quite reliable connections because it provides a means to determine if the client is connected, when they connected and when they disconnected. UDP is connectionless, so the server opens a port and listens for data, but has no automatic connection management, so clients need to explicitly 'tell' the server when they have arrived or are going (this also means you will need to make your own timeout facility).
Also, if you have very limited memory/processing resources, it is worth bearing in mind that UDP is a less 'costly' protocol as it avoids a lot of the overheads TCP incurs due to its inbuilt connection management.
While these are all protocols, they really just handle the connections themselves. You will probably still need to create your own protocol for the management of the data itself. For instance, when you send data over either TCP or UDP, the bytes you send may not all arrive to the server at the same time. This means you need a way of validating each packet you receive to ensure you have it all. This is often achieved with a combination of a checksum and a byte representing the total size of the data sent.
You might also consider mqtt (http://mqtt.org). It is a lightweight messaging protocol. For encoding your messages, you might consider protobuf (https://code.google.com/p/protobuf/)
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
Assuming the speed of light ~ 186000 mi/sec, and the farthest from anywhere on earth you can be without leaving earth ~ 16,000 mi, that means the time it takes for light to reach any point on earth and come back <= ~172 msec. So why can ping times exceed this?
A few reasons
Your assumption about speed is wrong, electronic communications through a wire travel is about 2/3 the speed of light.
You are not traveling a strait line from Point A to Point B, so it could be longer.
Your assumption about leaving earth is wrong, satellite links can often be used for intercontinental network links
(the biggest culprit) You need to pass through many computers (run the program tracert and you can see), the computer does not instantainouly forward the packet on from the time it received to to the time it sends it to the next person. If the computer doing the forwarding is under a very heavy load it could take a while for the packet to be forwarded on while it sits in a queue waiting to be processed.
That is a completely wrong comparison. for some reasons:
Electrons are involved in the pinging not light. So you can't compare light to electrons. Thats wrong.
Servers that your ping request hops on them are not processing them in zero sec. It actually takes time to process the ping packet and send it where it's supposed to go.
Your link to the internet is not a direct link. You have to pass through a DNS server (If you ran ping with a hostname not an IP), many routers and different types of links(satellite, wired, fiber optic). So it's not like emitting from this side of planet to the another side
I am programming a toy example to do NAT traversal. Interested in how a widely used desktop application does that, I used wireshark to try to analyze its traffic. After a some study of the output I realized that server notifications (e. g., "new file added to your xxx folder") worked using some kind of Comet mechanism, with long lived HTTP connections. But the thing that amazed me the most was that, despite the low traffic (1 HTTP GET and its response every minute) the TCP connection was never closed. I can assure that the connection was not closed during at least 20 minutes.
So far, my understanding is that having a lot of long lived TCP connections opened at the same time consumes quickly the resources of the server (mainly in terms of memory). So my question is, how do this kind of applications manage to efficiently keep such a huge number of TCP and HTTP connections opened at the same time during long periods? Do they use some special kind of server? Or is it only matter of adding hardware to scale horizontally?
I googled a lot trying to find an answer, with no luck. Perhaps I am missing something pretty obvious.
Maybe you can take a look at epoll (Linux), kqueue(freebsd), libev, and libevent to get some idea.
From epoll's wikipedia page : "where the number of watched file descriptors is large". You can replace the 'watched file descriptors' with TCP socket.
This question is the result of two other questions I've asked in the last few days.
I'm creating a new question because I think it's related to the "next step" in my understanding of how to control the flow of my send/receive, something I didn't get a full answer to yet.
The other related questions are:
An IOCP documentation interpretation question - buffer ownership ambiguity
Non-blocking TCP buffer issues
In summary, I'm using Windows I/O Completion Ports.
I have several threads that process notifications from the completion port.
I believe the question is platform-independent and would have the same answer as if to do the same thing on a *nix, *BSD, Solaris system.
So, I need to have my own flow control system. Fine.
So I send send and send, a lot. How do I know when to start queueing the sends, as the receiver side is limited to X amount?
Let's take an example (closest thing to my question): FTP protocol.
I have two servers; One is on a 100Mb link and the other is on a 10Mb link.
I order the 100Mb one to send to the other one (the 10Mb linked one) a 1GB file. It finishes with an average transfer rate of 1.25MB/s.
How did the sender (the 100Mb linked one) knew when to hold the sending, so the slower one wouldn't be flooded? (In this case the "to-be-sent" queue is the actual file on the hard-disk).
Another way to ask this:
Can I get a "hold-your-sendings" notification from the remote side? Is it built-in in TCP or the so called "reliable network protocol" needs me to do so?
I could of course limit my sendings to a fixed number of bytes but that simply doesn't sound right to me.
Again, I have a loop with many sends to a remote server, and at some point, within that loop I'll have to determine if I should queue that send or I can pass it on to the transport layer (TCP).
How do I do that? What would you do? Of course that when I get a completion notification from IOCP that the send was done I'll issue other pending sends, that's clear.
Another design question related to this:
Since I am to use a custom buffers with a send queue, and these buffers are being freed to be reused (thus not using the "delete" keyword) when a "send-done" notification has been arrived, I'll have to use a mutual exlusion on that buffer pool.
Using a mutex slows things down, so I've been thinking; Why not have each thread have its own buffers pool, thus accessing it , at least when getting the required buffers for a send operation, will require no mutex, because it belongs to that thread only.
The buffers pool is located at the thread local storage (TLS) level.
No mutual pool implies no lock needed, implies faster operations BUT also implies more memory used by the app, because even if one thread already allocated 1000 buffers, the other one that is sending right now and need 1000 buffers to send something will need to allocated these to its own.
Another issue:
Say I have buffers A, B, C in the "to-be-sent" queue.
Then I get a completion notification that tells me that the receiver got 10 out of 15 bytes. Should I re-send from the relative offset of the buffer, or will TCP handle it for me, i.e complete the sending? And if I should, can I be assured that this buffer is the "next-to-be-sent" one in the queue or could it be buffer B for example?
This is a long question and I hope none got hurt (:
I'd loveeee to see someone takes the time to answer here. I promise I'll double-vote for him! (:
Thank you all!
Firstly: I'd ask this as separate questions. You're more likely to get answers that way.
I've spoken about most of this on my blog: http://www.lenholgate.com but then since you've already emailed me to say that you read my blog you know that...
The TCP flow control issue is such that since you are posting asynchronous writes and these each use resources until they complete (see here). During the time that the write is pending there are various resource usage issues to be aware of and the use of your data buffer is the least important of them; you'll also use up some non-paged pool which is a finite resource (though there is much more available in Vista and later than previous operating systems), you'll also be locking pages in memory for the duration of the write and there's a limit to the total number of pages that the OS can lock. Note that both the non-paged pool usage and page locking issues aren't something that's documented very well anywhere, but you'll start seeing writes fail with ENOBUFS once you hit them.
Due to these issues it's not wise to have an uncontrolled number of writes pending. If you are sending a large amount of data and you have a no application level flow control then you need to be aware that if you send data faster than it can be processed by the other end of the connection, or faster than the link speed, then you will begin to use up lots and lots of the above resources as your writes take longer to complete due to TCP flow control and windowing issues. You don't get these problems with blocking socket code as the write calls simply block when the TCP stack can't write any more due to flow control issues; with async writes the writes complete and are then pending. With blocking code the blocking deals with your flow control for you; with async writes you could continue to loop and more and more data which is all just waiting to be sent by the TCP stack...
Anyway, because of this, with async I/O on Windows you should ALWAYS have some form of explicit flow control. So, you either add application level flow control to your protocol, using an ACK, perhaps, so that you know when the data has reached the other side and only allow a certain amount to be outstanding at any one time OR if you cant add to the application level protocol, you can drive things by using your write completions. The trick is to allow a certain number of outstanding write completions per connection and to queue the data (or just don't generate it) once you have reached your limit. Then as each write completes you can generate a new write....
Your question about pooling the data buffers is, IMHO, premature optimisation on your part right now. Get to the point where your system is working properly and you have profiled your system and found that the contention on your buffer pool is the most important hot spot and THEN address it. I found that per thread buffer pools didn't work so well as the distribution of allocations and frees across threads tends not to be as balanced as you'd need to that to work. I've spoken about this more on my blog: http://www.lenholgate.com/blog/2010/05/performance-comparisons-for-recent-code-changes.html
Your question about partial write completions (you send 100 bytes and the completion comes back and says that you have only sent 95) isn't really a problem in practice IMHO. If you get to this position and have more than the one outstanding write then there's nothing you can do, the subsequent writes may well work and you'll have bytes missing from what you expected to send; BUT a) I've never seen this happen unless you have already hit the resource problems that I detail above and b) there's nothing you can do if you have already posted more writes on that connection so simply abort the connection - note that this is why I always profile my networking systems on the hardware that they will run on and I tend to place limits in MY code to prevent the OS resource limits ever being reached (bad drivers on pre Vista operating systems often blue screen the box if they can't get non paged pool so you can bring a box down if you don't pay careful attention to these details).
Separate questions next time, please.
Q1. Most APIs will give you "write is possible" event, after you last wrote and writing is available again (can happen immediately if you failed to fill major part of send buffer with the last send).
With completion port, it will arrive just as "new data" event. Think of new data as "read Ok", so there's also a "write ok" event. Names differ between the APIs.
Q2. If a kernel mode transition for mutex acquisition per chunk of data hurts you, I recommend rethinking what you are doing. It takes 3 microseconds at most, while your thread scheduler slice may be as big as 60 milliseconds on windows.
It may hurt in extreme cases. If you think you are programming extreme communications, please ask again, and I promise to tell you all about it.
To address your question about when it knew to slow down, you seem to lack an understanding of TCP congestion mechanisms. "Slow start" is what you're talking about, but it's not quite how you've worded it. Slow start is exactly that -- starts off slow, and gets faster, up to as fast as the other end is willing to go, wire line speed, whatever.
With respect to the rest of your question, Pavel's answer should suffice.