I want to create an application that works as a "man in the middle" to analyze a protocol (ISO 8583) sent over TCP/IP.
A client connects to the application and sends some binary data (average length 500 bytes).
The application receives the message and then sends it to the real server.
The server responds to the request to the application.
The application sends the response to the original client.
Some context: The main idea is to get the raw binary data and convert it to a string for parsing and decoding the protocol.
There are two parts to the project:
The gateway part (man in the middle).
Parsing and decoding of the data.
I am expending too much time on the first part. So, if there is a mock-up that I can use to get me started, it will be nice. It doesn't have to be with Indy, but I prefer C++Builder.
This is my first time with Indy, and although I have experience working with TCP/IP, I have always used it as something that is already there, never at the low-level implementation.
I am testing with Hercules, and so far I can see the connections.
When I connect to a server in Hercules, I can see that my application is connecting. But, when my application disconnects, I don't see a message that says so, which means (I think) that my app is not disconnecting correctly (but I can reconnect as many times as I want).
I am sending data to my application using Hercules (a "Hello" string). It is working apparently, but I am having a hard time getting the actual data.
The documentation sometimes gets me into dead links, there are no samples or they are available on Delphi.
I am working with the following:
Windows 11 Home
Embarcadero® C++Builder 10.4 Version 27.0.40680.4203
Delphi and C++ Builder 10.4 Update 2
Indy 10.6.2.0
Have a look at Indy's TIdMappedPortTCP component. It is a TCP server that acts as a MITM proxy between clients and a specified server, giving you events when either party sends raw data.
Use the Bindings collection, or DefaultPort property, to specify the local IP/Port(s) that you want the server to listen for clients on.
Use the MappedHost and MappedPort properties to specify the remote server that you want TIdMappedPortTCP to connect to.
The OnBeforeConnect event is fired when a client has connected to TIdMappePortTCP.
The OnConnect event is fired just before TIdMappedPortTCP attempts to connect to the remote server.
The OnOutboundClientConnect event is fired when TIdMappedPortTCP has connected to the remote server.
The OnExecute event is fired when a client send bytes to TIdMappedPortTCP, and before the bytes are sent to the remote server. The event can alter the bytes, if desired.
The OnOutboundData event is fired when the remote server send bytes to TIdMappedPortTCP, and before the bytes are sent to the client. The event can alter the bytes, if desired.
Related
SSEs are advertised as a unidirectional communication tool to be used from server to client. I have a requirement to broadcast data to all clients and so i was wondering how SSEs behave on a low level. I cannot seem to find any low level information about SSEs online.
Primarily i would like to know if, after sending the data, does the server wait for a response from the client to confirm it has received the data before finishing the "send". That would mean that doing a broadcast using a for loop would be quiet dangerous and slow in which case websockets might be the better options.
Perhaps the implementation depends entirely on the language and framework? Is it not standardized?
Broadcast usually uses UDP which does not wait for a response. - - Broadcasting ip:port by socket server
.. says
UDP Packet: First four bytes as a magic number, next four bytes an IPv4 address (and you might want to add other things like a server name).
The magic number is just in case there is a collision with another application using the same port. Check both the length of the packet and the magic number.
Server would broadcast the packet at something like 30 second time intervals. (Alternatively you could have the server send a response only when a client sends a request via broadcast.)
So the client app would have to send a request back to the server app.
Different protocols would get different responses according the the underlying technology. eg HTTP uses responses extnsivly.
SSE and WebSockets are both over TCP, so there could be a wait before the socket could be used to send further data.
However, each client is a dedicated socket. So server-side you would be using threads or async coding (depending on the server-side language and its conventions). So looping through all the sockets to send a message to each client would be fine and quick.
I would want to send a message from the server actively, such as using UDP/TCPIP to a client using an arduino. It is known that this is possible if the user has port forward the specific port to the device on local network. However I wouldn't want to have the user to port forward manually, perhaps using another protocol, will this be possible?
1 Arduino Side
I think the closest you can get to this is opening a connection to the server from the arduino, then use available to wait for the server to stream some data to the arduino. Your code will be polling the open connection, but you are avoiding all the back and forth communications to open and close the connection, passing headers back and forth etc.
2 Server Side
This means the bulk of the work will be on the server side, where you will need to manage open connections so you can instantly write to them when a user triggers some event which requires a message to be pushed to the arduino. How to do this varies a bit depending on what type of server application you are running.
2.1 Node.js "walk-through" of main issues
In Node.js for example, you can res.write() on a connection, without closing it - this should give a similar effect as having an open serial connection to the arduino. That leaves you with the issue of managing the connection - should the server periodically check a database for messages for the arduino? That simply removes one link from the arduino -> server -> database polling link, so we should be able to do better.
We can attach a function triggered by the event of a message being added to the database. Node-orm2 is a database Object Relational Model driver for node.js, and it offers hooks such as afterSave and afterCreate which you can utilize for this type of thing. Depending on your application, you may be better off not using a database at all and simply using javascript objects.
The only remaining issue then, is: once the hook is activated, how do we get the correct connection into scope so we can write to it? Well you can save all the relevant data you have on the request to some global data structure, maybe a dictionary with an arduino ID as index, and in the triggered function you fetch all the data, i.e. the request context and you write to it!
See this blog post for a great example, including node.js code which manages open connections, closing them properly and clearing from memory on timeout etc.
3 Conclusion
I haven't tested this myself - but I plan to since I already have an existing application using arduino and node.js which is currently implemented using normal polling. Hopefully I will get around to it soon and return here with results.
Typically in long-polling (from what I've read) the connection is closed once data is sent back to the client (arduino), although I don't see why this would be necessary. I plan to try keeping the same connection open for multiple messages, only closing after a fixed time interval to re-establish the connection - and I hope to set this interval fairly high, 5-15 minutes maybe.
We use Pubnub to send notifications to a client web browser so a user can know immediately when they have received a "message" and stuff like that. It works great.
This seems to have the same constraints that you are looking at: No static IP, no port forwarding. User can theoretically just plug the thing in...
It looks like Pubnub has an Arduino library:
https://github.com/pubnub/arduino
I need to create an application that:
Has one server
With a client that connects to the server and sends 8 longs (data from 8 sensors: rain, air humidity, wind speed...) 1 sensor data / long (sensor data is acquired from a custom USB device)
User clients. The end user runs this type of client to connect to the server for data retrieval from the sensors.
I used Qt before, creating Client-server applications with just one type of client. And I managed to create this application too, just at a smaller scale (used 5 words, and clients were connected simultaneously to the server). I used the Qt network examples fortune threaded server and http://goo.gl/srypT and blocking fortune client example.
How can i identify which client is which? (since they have different ip everytime they connect to internet). On my small scale application, I created some kind of protocol, but there must be a more efficient way to do this.
I assume that you want to identify the client type ("sensor client" vs. "user client"), not individual client instances.
The straightforward way to do this is to implement a protocol, as mentioned in the question. For your use case, this could be very simple:
let the "sensor client" send a "write" command (one character like "w" would be sufficient) followed by your sensor data. The server then receives the "w" command and knows that he needs to read sensor data from the client.
let the "user client" send a "read" command (e.g. the character "r"). When the server receives the "r" command it knows that it needs to send data to the client.
If, for whatever reason, you do not want to implement even such a simple protocol, you could also set up two separate QTcpServer instances which listen at different ports, lets say 8192 and 8193. Your "sensor client" would then connect to port 8192, and the server knows by the port number that the client will send data. Your "user clients" would connect to port 8193, and the server knows that the clients expect data and will send the required data.
In any case, you should be aware that there is no authentication and authorization involved, and any client who knows the simple protocol and/or the port numbers can send and receive data.
To identify a client, you have to use some kind of client ID. Usually, some kind of hash (a MD5 digest, a UUID or a GUID) is used as the client ID. This client ID have to be sent from the client to the server when the client connects to the server.
What happens after the client has been identified and accepted, depends on the type of connection (protocol). If you use a stateful protocol, the same connection will be kept open as long as the client uses it so there is no need to re-identify the client. If you use a stateless connection (HTTP, for example), you will have to re-send the same ID from the client to the server every time the client requires data (that is: a document, a page, etc.) to the server.
A simpler and more efficent way to deal with a client/server architecture like this consists in using an existing, proven server of some kind. For example, you could use a RESTful web server like Wt (http://www.webtoolkit.eu/wt/blog), given that you are already using C++.
Even better, I would use a Ruby- or a Python-based RESTful web service framework like:
http://www.sinatrarb.com/
http://bottlepy.org/docs/dev/
http://flask.pocoo.org/
Or the new Ruby-on-Rails API:
http://blog.steveklabnik.com/posts/2012-11-22-introducing-the-rails-api-project
https://github.com/rails-api/rails-api
Developing the server in Ruby or Python is much faster and easier. The client can developed in any way (C++ with Qt, Javascript in a web browser and many other ways)
I have a AJAX client which must receive messages from server. Some messages from server not like request-response type. For example, imaging game in which players can enter. Server must send to client info about player entering. But how can server send message to client via http without request from client?
Only decision that i can invent: client send request to server (getNewPlayerEnter request) with big timeout, server checks state of player set, if in set new players are, then server send info to client, if not server 'sleeps' on some time, and after 'sleeping' server checks players set again.
I think my desicion a little stupid(maybe not little). How implement it right?
p.s. sorry for my english
What you are referring to is "polling" where your client side code will make a request to the server every X seconds to check for updates.
It's an easy to implement solution, which is why you see it so often. However, a better solution is to use Comet.
In 2006, Alex Russell coined the term "Comet" to identify a technique whereby the client makes and HTTP request to the server and the server keeps that connection open indefinitely. With this open connection, the server can then push updates to the client in the response when there are updates.
When the response is received, the client immediately opens a new long-lived HTTP connection.
I encourage you to do a search for "Comet" and whatever server-side language you are using to find a framework that will get you started.
Comet Daily has some really great demos, from chat applications to games, that use Comet. If you have a tool like Firebug or Chrome Debugger, you can observe the response/request cycle of these real-time applications.
http://cometdaily.com/2009/10/23/concrete-comet-examples/
The way this typically would be done is for the client to ping the server on a regular interval (say, every 10-seconds). If there has been a change on the server, the change is sent back to the client. If there has been no change, either a blank response is sent, or some sort of "no-change" code that you specify.
Manytimes clients ask for features like instant messaging (IM) and other client-to-client (P2P) communication for their web apps. Typically how is this done in normal web browsers? For example I've seen demos of Google Wave (and Gmail) that are able to IM from a regular browser. Is this via HTTP? Or does XmlHttpRequest (AJAX) provide the necessary backend for such communication?
More than anything I wonder how can a server "wake up" the remote client, lets say for sending an IM? Or does the client have to keep "polling" the message server for new IMs?
Typically the browser will poll the server for new messages. One approach that is often done to make this more efficient is the 'long poll' (see also this link) - the server responds immediately if it has anything; otherwise, it waits, keeping the connection open for a while. If a message comes in, it immediately wakes up and sends it, otherwise it comes back with a 'nope, check back' after a few tens of seconds. The client them immediately redials to go back into the long-polling state.