Suppose I have two ec2 instances on aws ec2_A and ec2_B. I have two Security groups attached to it to sg_A and sg_B respectively. now suppose I have added an inbund rule on sg_B that sg_A can access tcp protocol on port 3456 on ec2_B. But when the handshake will happen:
ec2_A:5547 -> ec2_B:3456 (this is allowed as I have allowd sg_A on sg_B right?)
in return when ec2_B will send response to ec2_A on port 5547 will it accept the response as I have not open the port 5547 in sg_A for ec2_A? If yes then HOW and "WHY SPECIALLY"?
Yes it will accept the response. The reason why is because security groups are stateful. The security group knows that this is a response for a request it has made therefore it is allowed regardless of the inbound rules.
More information on how this is achieved can be found in the official docs here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking
Related
I have a scenario where I need to route TCP traffic to a dynamic set of backend servers (Kubernetes pods to be exact but we can ignore that for purposes of this post) through a proxy like HAProxy or nginx. The traffic needs to be routed based on a key (call it the routing_key) provided by the client in the TCP payload.
I see that both nginx and HAProxy support consistent hashing. However, from what I can tell based on HAProxy's manual (see "balance" section), there's no way to perform consistent hashing based on a TCP payload. Payload-based load balancing seems to be limited to L7 HTTP parameters like Header and URI params. This post outlines a method for statically balancing based on string matching a TCP payload, but my case is more dynamic so a true consistent hashing approach is much preferred.
Nginx appears to offer a bit more flexibility in that you can set the hashing value to an arbitrary variable as shown here. This appears to work for both L7 (the "backend" stanza) and L4 (the "stream" stanza). However, I'm a bit hazy on what you are and aren't allowed to do for variables. Does anyone have an example of setting a variable to be a value extracted from the TCP payload and using that for consistent hashing?
Final bonus question: the routing_key value is actually an AES-GCM encrypted value. The proxy server would have access to the key used to decrypt this value. Is it possible to have nginx grab the routing key value from the TCP payload, decrypt it using the known key, and then use that for consistent hashing? Would that involve creating an nginscript module?
In the HAProxy 2.1 can you use aes_gcm_dec(...) in combination with req.payload(...) for such a requirement.
My idea, untested.
listen tcp-in
bind :443 ssl cert
tcp-request inspect-delay 10s
tcp-request session set-var(sess.routingkey) req.payload(0,500)
# for consistent hashing try this
hash-type consistent wt6
use_backend %[var(sess.routingkey),aes_gcm_dec(128,txn.nonce,Zm9vb2Zvb29mb29wZm9vbw==,txn.aead_tag)]
Here are also the links to the html Documentation.
aes_gcm_dec
req.payload
hash-type consistent ... is described at hash-type
I have a question about distinguishing user in HTTP.
I opened wireshark and web browser, started wandering in web pages on some known website. and noticed that my PC opened several TCP connections and on each connection there are several HTTP request\response.
My main goal is to identify user that wandering in my website (for example)
at first, I thought of finding matching response for each request but it is not trivial as I have done alot of reading about it. specially if there are several TCP connections for one user.
I also thought of identify user by its source port\ip- but there are several TCP connection for one user and there are more than one user (some of them might be beyond NAT).
So my question is how can I identify\isolate all the http requests\responses from one user given:
there are more that one user
some of them might connected to NAT
each user open several TCP connections for the "main" HTTP request
cookies might change during session
Is there some sniffer or library that already has this ability of distinguish between users (this http request\response belongs to that user)?
This process suppose to run on the fly.
Thanks alot.
I have been writing my own version of the 802.11 protocol with network stack. This is mostly a learning experience to see more in depth on how networks work.
My question is, is there a standard for replying to client devices that a certain protocol is unsupported?
I have an android device connecting to my custom wifi device and immediately sending a TON of requests at the DNS port of my UDP protocol. Since I would like to test out other protocols I would very much like a way for my wifi device to tell the android device that DNS is not available and get it to quite down a little.
Thanks in advance!
I don't see a possibility to send a reply that a service is not available.
I can't find anything about this case in the UDP specification.
One part of the DNS specification assumes that there are multiple DNS servers and defines how to handle communication with them. This explains part of the behavior in your network, but does not provide much information how to handle it.
4.2.1 Messages - format - UDP usage
The optimal UDP retransmission policy will vary with performance of the
Internet and the needs of the client, but the following are recommended:
The client should try other servers and server addresses
before repeating a query to a specific address of a server.
The retransmission interval should be based on prior
statistics if possible. Too aggressive retransmission can
easily slow responses for the community at large. Depending
on how well connected the client is to its expected servers,
the minimum retransmission interval should be 2-5 seconds.
7.2 Resolver Implementation - sending the queries
If a resolver gets a server error or other bizarre response
from a name server, it should remove it from SLIST, and may
wish to schedule an immediate transmission to the next
candidate server address.
According to this you could try to send garbage back to the client, but this is rather a hack, or an error, but how does an error look like? Such a solution assumes that you have knowledge about the service that you don't support.
I believe that the DNS - requests can be avoided by using DHCP. DHCP allows to specify DNS-servers as listed in the linked page. This is the usual way that I know for a DNS-resolver in a LAN to get initial DNS servers although I don't find anything about this in the DNS specification. You can give the Android - device a DNS-server with DHCP so that it does to need to try to query your device. Querying your device could be a fallback.
Additionally to DNS there is mDNS which uses multicasts in the network to send queries. This seems not to be the protocol you have to do with because it uses the special port 5353.
Not possible to stop DNS in the way you intend. However, only for your tests you can check the UDP messages and find out the names the device is looking for. Then you update the hosts file (google how to do it: http://www.howtogeek.com/140576/how-to-edit-the-hosts-file-on-android-and-block-web-sites/) and add those names with some localoop IP address. That might work for your test.
Other possibility is to change DNS server to some localloop IP address: http://xslab.com/2013/08/how-to-change-dns-settings-on-android/
Again, this is only to avoid having all the DNS messages through the wifi connection.
I want to develop an application where all traffic from network segment gets mirrored onto a windows station in order to be able to see all tcp-ip request/response data (filtering).
I know that it should be possible using WinPcap to capture all packets but problem in this case would be that I would have to implement all the processing needed to be able to distinguish tcp data streams (e.g. handshaking, closing, retransmissions, reordering, maybe others ?). I need the stream of data because I will be doing application level (e.g. http) filtering.
I wonder if there is a driver/solution somewhere that provides me tcp data stream, solution that could be used on a gateway machine or using port mirroring.
For starters, in WinPCap, you can define something called filter.
That filter filters out all the traffic except the type that you specify, so if you want to capture HTTP traffic only, I'd suggest you make a filter on TCP Port 80 or any other port you're using for HTTP.
Once you've captured these packets, you can check the payload of the TCP, parse the HTTP header and do whatever you wish according to your system's policy.
Check this link if you want to familiarize yourself with how to use WinPCap and how to use filters(in this example they're capturing TCP traffic in general, you should add to their filter "port 80").
Is a http end point suppose to respond to requests from a particular client in order that they are received?
What about if it doesn't make sense to in the case of requests handled by cluster behind a proxy or in requests handled with NIO where one request is finished faster than the other?
Is there a standard way of associating a unique id with each http request to associate with the response? How is this handled in clients like http componenets httpclient or curl?
The question comes down to the following case:
Suppose, I am downloading a file from a server and the request is not finished. Is a client capable of completing other requests on the same keep-alive connection?
Whenever a TCP connection is opened, the connection is recognized by the source and destination ports and IP addresses. So if I connect to www.google.com on destination port 80 (default for HTTP), I need a free source port which the OS will generate.
The reply of the web server is then sent to the source port (and IP). This is also how NAT works, remembering which source port belongs to which internal IP address (and vice versa for incoming connections).
As for your edit: no, a single http connection can execute one command (GET/POST/etc) at the same time. If you send another command while you are retreiving data from a previously issued command, the results may vary per client and server implementation. I guess that Apache, for example, will transmit the result of the second request after the data of the first request is sent.
I won't re-write CodeCaster's answer because it is very well worded.
In response to your edit - no. It is not. A single persistent HTTP connection can only be used for one request at once, or it would get very confusing. Because HTTP does not define any form of request/response tracking mechanism, it simply would not be possible.
It should be noted that there are other protocols which use a similar message format (conforming to RFC822), which do allow for this (using mechanisms such as SIP's cSeq header), and it would be possible to implement this in a custom HTTP app, but HTTP does not define any standard mechanism for doing this, and therefore nothing can be done that could be assumed to work everywhere. It would also present a problem with the response for the second message - do you wait for the first response to finish before sending the second response, or try and pause the first response while you send the second response? How will you communicate this in a way that guarantees messages won't become corrupted?
Note also that SIP (usually) operates over UDP, which does not guarantee packet ordering, making the cSeq system more of a necessity.
If you want to send a request to a server while another transaction is still in progress, you will need to create a new connection to the server, and hence a new TCP stream.
Facebook did some research into this while they were building their CDN, and they concluded that you can efficiently have 2 or 3 open HTTP streams at any one time, but any more than that reduces overall transfer time because of the extra packet overhead cost. I would link to the blog entry if I could find the link...