I am developing a SNMP poller which will poll around 40K devices every hour for CPU,Memory,Bandwidth and Connection Count related information. I am currently using snmp4j API. I am performing a snmpwalk separately for CPU, Memory, Bandwidth and Connection Count, but given the number of devices, this is taking huge amount of time. I am thinking of using SNMP getbulk request to get all the information at once, but this is restricted by the maximum response PDU packet size of the queried device. I wanted to know is there a way to know the maximum PDU response size of the remote system so that I can break up my request PDU accordingly. I have around 2500 OIDs to poll in one request. And also, I am not allowed to modify the response packet size of the remote system.
This has been a problem for 30 years (SNMP is that old): part of device discovery is to determine max response size (in addition to response time, supported versions, etc) of each device.
It's basically a trade-off of discovery time vs. just assuming some minimal capabilities.
Related
I am designing the software controlling several serial ports, operating system is OpenWrt. The device application is running in is single core ARM9 # 450 MHz. Protocol for serial ports is Modbus.
The problem is with Modbus slave implementation. I designed it in real-time manner, looping reading data from serial port (port is open in non-blocking mode with 0 characters to wait for and no timeouts). The sequence/data stream timeout is about 4 milliseconds # 9600/8/N/1 (3.5 characters as advised). Timeout is checked if application does not see anything in the buffer, therefore if application is slower that incoming stream of characters, timeouts mechanism will not take place - until all characters are removed from the buffer and bus is quiet.
But I see that CPU switches between threads and this thread is missed for about 40-60 milliseconds, which is a lot to measure timeouts. While I guess serial port buffer still receives the data (how long this buffer is?), I am unable to assess how much time passed between chars, and may treat next message as continuation of previous and miss Modbus request targeted for my device.
Therefore, I guess, something must be redesigned (for the slave - master is different story). First idea coming to my mind is to forget about timeouts, and just parse the incoming data after being synchronized with the whole stream (by finding initial timeout). However, there're several problems - I must know everything about all types of Modbus messages to parse them correctly and find out their ends and where next message starts, and I do not see the way how to differentiate Modbus request with Modbus response from the device. If developers of the Modbus protocol would put special bit in command field identifying if message is request or response... but it is not the case, and I do not see the right way to identify if message I am getting is request or response without getting following bytes and checking CRC16 at would-be byte counts, it will cost time while I am getting bytes, and may miss window for the response to request targeted for me.
Another idea would be using blocking method with VTIME timeout setting, but this value may be set to tenths of seconds only (therefore minimal is 100 ms), and this is too much given another +50 ms for possible CPU switching between threads, I think something like timeout of 10 ms is needed here. It is a good question if VTIME is hardware time, or software/driver also subject to CPU thread interruptions; and the size of FIFO in the chip/driver how many bytes it can accumulate.
Any help/idea/advice will be greatly appreciated.
Update per #sawdust comments:
non-blocking use has really appeared not a good way because I do not poll hardware, I poll software, and the whole design is again subject to CPU switching and execution scheduling;
"using the built-in event-based blocking mode" would work in blocking mode if I would be able to configure UART timeout (VTIME) in 100 us periods, plus if again I would be sure I am not interrupted during the executing - only then I will be able to have timing properly, and restart reading in time to get next character's timing assessed properly.
I came across the concept of window size when browsing gRPC's dial options. Because gRPC uses HTTP/2 underneath, I dug this article up, which describes:
Flow control window is just nothing more than an integer value indicating the buffering capacity of the receiver. Each sender maintains a separate flow control window for each stream and for the overall connection.
If this is the window size gRPC is talking about and I understand this correctly. This is for HTTP/2 to maintain multiple concurrent stream within the same connection. Basically a number that's advertised to the sender about how much data the receiver wants the sender to send next. For control flow reasons, the connection puts different stream's data among different windows in serial.
My question is/are: is the window all or nothing? Meaning if my window size is n bytes, the stream won't send any data until it's accumulated at least n bytes? More generally, how do I maximize the performance of my stream if I maintain only one stream? I assume a bigger window size would help avoid overheads but increase risk for data loss?
Meaning if my window size is n bytes, the stream won't send any data until it's accumulated at least n bytes?
No.
The sender can send any number of bytes less than or equal to n.
More generally, how do I maximize the performance of my stream if I maintain only one stream?
For just one stream, just use the max possible value, 2^31-1.
Furthermore, you want to configure the receiver to send WINDOW_UPDATE frames soon enough, so that the sender always has a large enough flow control window that allows it to never stop sending.
One important thing to note is that the configuration of the max flow control window is related to the memory capacity of the receiver.
Since HTTP/2 is multiplexed, the implementation must continue to read data until the flow control window is exhausted.
Using the max flow control window, 2 GiB, means that the receiver needs to be prepared to buffer at least up to 2 GiB of data, until the application decides to consume that data.
In other words: reading the data from the network by the implementation, and consuming that data by the application may happen at different speeds; if reading is faster than consuming, the implementation must read the data and accumulate it aside until the application can consume it.
When the application consumes the data, it tells the implementation how many bytes were consumed, and the implementation may send a WINDOW_UPDATE frame to the sender, to enlarge the flow control window again, so the sender can continue to send.
Note that implementations really want to apply backpressure, i.e. wait for applications to consume the data before sending WINDOW_UPDATEs back to the sender.
If the implementation (wrongly) acknowledges consumption of data before passing it to the application, then it is open to memory blow-up, as the sender will continue to send, but the receiver is forced to accumulate it aside until the host memory of the receiver is exhausted (assuming the application is slower to consume data than the implementation to read data from the network).
Given the above, a single connection, for the max flow control window, may require up to 2 GiB of memory.
Imagine 1024 connections (not that many for a server), and you need 2 TiB of memory.
Also consider that for such large flow control windows, you may hit TCP congestion (head of line blocking) before the flow control window is exhausted.
If this happens, you are basically back to the TCP connection capacity, meaning that HTTP/2 flow control limits never trigger because the TCP limits trigger before (or you are otherwise limited by bandwidth, etc.).
Another consideration to make is that you want to avoid that the sender exhausts the flow control window and therefore is forced to stall and stop sending.
For a flow control window of 1 MiB, you don't want to receive 1 MiB of data, consume it and then send back a WINDOW_UPDATE of 1 MiB, because otherwise the client will send 1 MiB, stall, receive the WINDOW_UPDATE, send another 1 MiB, stall again, etc. (see also how to use Multiplexing http2 feature when uploading).
Historically, small flow control windows (as the one suggested in the specification of 64 KiB) were causing super-slow downloads in browsers, that quickly realized that they needed to tell servers that their flow control window was large enough so that the server would not stall the downloads.
Currently, Firefox and Chrome set it at 16 MiB.
You want to feed the sender with WINDOW_UPDATEs so it never stalls.
This is a combination of how fast the application consumes the received data, how much you want to "accumulate" the number of consumed bytes before sending the WINDOW_UPDATE (to avoid sending WINDOW_UPDATE too frequently), and how long it takes for the WINDOW_UPDATE to go from receiver to sender.
I read somewhere (but cannot find the source anymore) that there is a certain maximum number of bytes that can be sent in the first TCP window. Sending more data requires ACK from the receiver, hence another round-trip. To reduce website latency, all above-the-fold content, including HTTP reply headers, should be less than this number of bytes.
Can anybody remember what the maximum number of bytes in the first TCP window is and how it is calculated?
This is regulated by initial tcp congestion window (initcwnd). This parameter determines how many segments (MSS) could be sent without waiting for ACK at first phase of slow start. Currently recommended value for most workloads is 10, but some old systems still using 4. Also note, used window size depends on clients receive window too, so if some client will advertise receive window lower than your initial congestion window, it will be used this receive window as limit.
For more info, refer to this page.
The GATT architecture of BLE lends itself to small fixed pieces of data (20 bytes max per characteristic). But in some cases, you end up wanting to “stream” some arbitrary length of data, that is greater than 20 bytes. For example, a firmware upgrade, even if you know its slow.
I’m curious what scheme others have used if any, to “stream” data (even if small and slow) over BLE characteristics.
I’ve used two different schemes to date:
One was to use a control characteristic, where the receiving device notify the sending device how much data it had received, and the sending device then used that to trigger the next write (I did both with_response, and without_response) on a different characteristic.
Another scheme I did recently, was to basically chunk the data into 19 byte segments, where the first byte indicates the number of packets to follow, when it hits 0, that clues the receiver that all of the recent updates can be concatenated and processed as a single packet.
The kind of answer I'm looking for, is an overview of how someone with experience has implemented a decent schema for doing this. And can justify why what they did is the best (or at least better) solution.
After some review of existing protocols, I ended up designing a protocol for over-the-air update of my BLE peripherals.
Design assumptions
we cannot predict stack behavior (protocol will be used with all our products, whatever the chip used and the vendor stack, either on peripheral side or on central side, potentially unknown yet),
use standard GATT service,
avoid L2CAP fragmentation,
assume packets get queued before TX,
assume there may be some dropped packets (even if stacks should not),
avoid unnecessary packet round-trips,
put code complexity on central side,
assume 4.2 enhancements are unavailable.
1 implies 2-5, 6 is a performance requirement, 7 is optimization, 8 is portability.
Overall design
After discovery of service and reading a few read-only characteristics to check compatibility of device with image to be uploaded, all upload takes place between two characteristics:
payload (write only, without response),
status (notifiable).
The whole firmware image is sent in chunks through the payload characteristic.
Payload is a 20-byte characteristic: 4-byte chunk offset, plus 16-byte data chunk.
Status notifications tell whether there is an error condition or not, and next expected payload chunk offset. This way, uploader can tell whether it may go on speculatively, sending its chunks from its own offset, or if it should resume from offset found in status notification.
Status updates are sent for two main reasons:
when all goes well (payloads flying in, in order), at a given rate (like 4Hz, not on every packet),
on error (out of order, after some time without payload received, etc.), with the same given rate (not on every erroneous packet either).
Receiver expects all chunks in order, it does no reordering. If a chunk is out of order, it gets dropped, and an error status notification is pushed.
When a status comes in, it acknowledges all chunks with smaller offsets implicitly.
Lastly, there is a transmit window on the sender side, where many successful acknowledges flying allow sender to enlarge its window (send more chunks ahead of matching acknowledge). Window is reduced if errors happen, dropped chunks probably are because of a queue overflow somewhere.
Discussion
Using "one way" PDUs (write without response and notification) is to avoid 6. above, as ATT protocol explicitly tells acknowledged PDUs (write, indications) must not be pipelined (i.e. you may not send next PDU until you received response).
Status, containing the last received chunk, palliates 5.
To abide 2. and 3., payload is a 20-byte characteristic write. 4+16 has numerous advantages, one being the offset validation with a 16-byte chunk only involves shifts, another is that chunks are always page-aligned in target flash (better for 7.).
To cope with 4., more than one chunk is sent before receiving status update, speculating it will be correctly received.
This protocol has the following features:
it adapts to radio conditions,
it adapts to queues on sender side,
there is no status flooding from target,
queues are kept filled, this allows the whole central stack to use every possible TX opportunity.
Some parameters are out of this protocol:
central should enforce short connection interval (try to enforce it in the updater app);
slave PHY should be well-behaved with slave latency (YMMV, test your vendor's stack);
you should probably compress your payload to reduce transfer time.
Numbers
With:
15% compression,
a device connected with connectionInterval = 10ms,
a master PHY limiting every connection event to 4-5 TX packets,
average radio conditions.
I get 3.8 packets per connection event on average, i.e. ~6 kB/s of useful payload after packet loss, protocol overhead, etc.
This way, upload of a 60 kB image is done in less than 10 seconds, the whole process (connection, discovery, transfer, image verification, decompression, flashing, reboot) under 20 seconds.
It depends a bit on what kind of central device you have.
Generally, Write Without Response is the way to stream data over BLE.
Packets being received out-of-order should not happen since BLE's link layer never sends the next packet before it the previous one has been acknowledged.
For Android it's very easy: just use Write Without Response to send all packets, one after another. Once you get the onCharacteristicWrite you send the next packet. That way Android automatically queues up the packets and it also has its own mechanism for flow control. When all its buffers are filled up, the onCharacteristicWrite will be called when there is space again.
iOS is not that smart however. If you send a lot of Write Without Response packets and the internal buffers are full, iOS will silently drop new packets. There are two ways around this, either implement some (maybe complex) protocol for the peripheral notifying the status of the transmission, like Nipos answer. An easier way however is to send each 10th packet or so as a Write With Response, the rest as Write Without Response. That way iOS will queue up all packets for you and not drop the Write Without Response packets. The only downside is that the Write With Response packets require one round-trip. This scheme should nevertheless give you high throughput.
I have several clients that constantly post data to a REST service. REST service is put behind a network load balancer. Each client sends 100 - 500 MB a day and I need to support 500+ clients.
I can POST either very large packets, this will reduce overhead for TCP/IP session set up and HTTP headers. This will, however, firmly tie one client to a particular server and limit my scalability options. Alternatively, I can send small HTTP packets, which I can load balance well, but I will get more overhead for TCP/IP session set up and HTTP headers.
What is the recommended packet size for HTTP POST? Or how can I calculate one for my environment?
There is no recommended size.
While HTTP POST size is not constrained by the RFCs, since HTTP is a commodity protocol implementing request / response type messaging, most of the infrastructure is configured around the idea that TCP connections are not particularly long lasting / does not carry significant amounts of data. i.e. there will be factors outside your control which may impact the service - although HTTP supports range requests for responses, there is no corollary for requests.
You can get around a lot of these (although not all) by using HTTPS. However you still need to think about how you detect/manage outages - are you happy to wait for a TCP timeout?
With 500+ clients presumably using the system quite heavily, the congestion avoidance limits shouldn't be a problem - whether TCP window scaling is likely to be an issue depends on how the system is used. HTTP handshakes should not be an issue unless you restrict the request size to something silly.
If the service is highly dependant on clients pushing lots of data on to your server, then I'd encourage you to look at parsing the data on the client (given the volume, presumably it's coming from files - implying a signed java applet or javascript with UniversalBrowserRead privilege) then sending it over a bi-directional communication channel (e.g. websocket).
Leaving that aside for now, the only way you can find out what the route between your clients and your server will support is to measure it - and monitor it. I would expect that a 2Mb upload size would work pretty much anywhere, while a 10Mb size would work most of the time within the US or Europe - and that you could probably increase this to 50Mb as long as there's no mobile clients.
But if you want to maintain the effectiveness of the service you'll need to monitor bandwidth, packet loss and lost connections.