sendto() return error code ENETDOWN - networking

I met very strange situation.
In my program, the sendto() function returns error code ENETDOWN(Network is down) even though network is up and ping tryout success.
It's happened only when UDP stream connects to other network through several gateways. It's not always and happened sometimes.
If i run same code under same sub network, there is no error like ENETDOWN.
So, i trace sendto() function to Kernel area.
The neigh_hh_output() function in ip_finish_output2() of iop_output.c calls hh->hh_output() and it returns ENETDOWN error code.
Under normal operation, hh->hh_output() function is assigned to dev_queue_xmit() of dev.c and packet's sent to network.
When issue was happened, it seems assigned to neigh_blackhole() function in neigh_destroy() of neighbour.c. The neigh_blackhole() returns -ENETDOWN code.
But, i don't know when the neigh_destroy() is called and why that function is called.
I'm struggling with this problem for several weeks.

My test machine is placed like below description.
Test machine --- gateway(1.1.1.1) --- firewall(1.1.1.2) --- network ---- Destination.
First time, UDP connection establish between my test machine and Destination and gateway address of my test machine is 1.1.1.1.
Traffic has no problem between test machine destination. After some time or right after, suddenly, transmit traffic fail with "Network is Down" error(Error number 100, ENETDOWN).
At this time, if i tryout ping to destination in my test machine, ping response OK.
When i capture packet front of my test machine, ICMP redirect message comes from gateway(1.1.1.1). Its information is "Redirect for Host" and New gate address is "1.1.1.2".
When my test machine's OS(Linux 3.0.35) received ICMP redirect message, it changes virtual function pointer of hh->hh_output() from ev_queue_xmit() to neigh_blackhole(). Eventually, neigh_blackhole() return -ENETDOWN code.
So, change gate address of my test machine to 1.1.1.2. After that, "Network is down" error is not happened again.
I think that it's strange operation. sendto() function doesn't return ENETDOWN code as the man page. But, it's return ENETDOWN code.
Anyway, if the sendto() function returns -ENETDOWN even though network interface is update, how to overcome this error? Do i re-connect UDP stream?
I wonder this issue is bug of Linux kernel 3.0.35.
If i will know or find thing about this issue, i will update in here.
Please reference my case if anyone has similar issue with me.

It is claimed that a neighbor will be deleted for a variety of reasons including the host changed its layer 2 address while retaining its layer 3 address or is no longer reachable. See this. It also can be deleted if the gateway for the neighbor sends an ICMP redirect and the processing of redirects is enabled in the kernel.
If the neighbor is in the process of being deleted, then the packet is dispatched to neigh_blackhole which unconditionally returns -ENETDOWN. See the code here.
The man page for sendto() would lead you to believe that you shouldn't get -ENETDOWN under such circumstances, but this appears to be incorrect.
I would try to get a network capture when this occurs and look for ICMP messages indicating your destination is not reachable or for a change in the MAC address for the destination (or possibly a duplicate IP address) via ARP packets or the MAC addresses on the arriving packets from the destination.

Related

Connecting Between Multiple IP Address in TCP

when i ask from DNS Server about somewhere.com it resolve 4 IP for me in this order:
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
and my question is what happen if first IP(192.168.1.1) doesn't work?
which service or protocol switch to next IP?
and what about other services like telnet,... (telnet somewhere.com 443)?
First, when the lookup is made, the returned list is not in any kind of order, it is somewhat random. This helps ensure one system is not overloaded.
What happens if the first address in list fails, will depend on how well the application is written.
What is supposed to happen, is that if the first IP address fails, then the application should then try the next and iterate though the list until the list is exhausted or a connection is made. You can see this with many pieces of software. However, many developers are lazy, and only try the first returned. I can think of quite a few applications where this happens.
It's not fun when you see applications request ANY expecting an IPv4 address and the returned list contains an IPv6 address, only for the software to barf.

What is the difference between a 'shallow' and 'deep' ping?

So as a fairly new (out-in-the-world-working) developer I'm not unfamiliar to the concept of pinging as it was taught to me in uni and I've used it in various small assignments I've encountered before.
My understanding of it is determining whether another host is present and responsive on a network.
I've implemented in Java (the language i'm most comfortable in) some code similar to the one below when wanting to achieve pinging:
try {
String ip = "127.192.199.01";
InetAddress inet = InetAddress.getByName(ip);
System.out.println("Send ping-request to: " + ip);
if (inet.isReachable(5000)){
System.out.println(ip + " is reachable.");
} else {
System.out.println(ip + " NOT reachable.");
}
} catch ( Exception e ) {
System.out.println("Exception:" + e.getMessage());
}
However I pretty new to the concept of 'shallow' and 'deep' pinging which is something I've encountered at my current workplace.
So far though I've been able to figure out that deep pinging is basically about testing that all connections are alive and working while testing functionality through as much of a webstack as possible.
Got that from: what is deep_ping
My question is:
What does a shallow ping do then?
And what is then the main difference (if any) between a shallow and deep ping?
Shallow ping works by sending one or more (ICMP/ICMP6) Echo Request packets to the host and only tests if the network is working i.e. if the host is reachable from your machine.
Deep pinging works by testing the application. How the deep ping is implemented is up to the developer but it could be via a "status" endpoint which responds with the current application status, or another similar interface (as described in the link you already mentioned).
As per your link, deep pinging can also mean testing an entire subnet or testing other protocols than the standard Echo Request.
This sounds like you are performing host availability test on different OSI layers (https://en.wikipedia.org/wiki/OSI_model)
A ping how you know it is performed on the transport layer, where a single packet is sent to the other machine and the OS responds with another packet.
In that case you do not know if there is ANY application running on that node other than the clean OS.
Going further up in the OSI layers you can then perform tests on you business logic like calling APIs of your HTTP based REST API
My gut feeling is that "shallow ping" means a ping performed using the standard ICMP request (the well known ping command we know about), a "deep ping" is trying to access the actual remote service by opening a socket on a given port and see if you get any answer
How I can interpret it:
Shallow ping : The host is available, port on which service is exposed is accepting. (more like telnet)
Deep ping : You are actually hitting one of the service and getting an expected response back.
Again, these are more like custom terms, not an actually defined standard.

Traceroute: Can it trace a path from A to B correctly?

Traceroute is an application to trace the path from A to B. (A is your location and B is the server you want to trace). In Windows, you can type tracert. The main algorithm is:
send UDP with TTL = 1
Server A1 received, and return ICMP packet to A because TTL is expired.
--> know first machine between. For example A1.
send UDP with TTL = 2
Server A1 received, and send this UDP to server A2.
Server A2 received, and return ICMP packet to A because TTL is expired
--> know second machine between. In this example is A2.
Do it until to B. we can track down: A -> A1 -> A2 -> ... ->B
Does this algorithm work correctly? Because at different time, an intermediate server can send a message to different server. For example, at first time, UDP message is sent to A1, but at a later time, it can send to another server, for example, B1. So, trace route will not work properly.
Did I misunderstand something?
From the man page :
traceroute tracks the route packets take from an IP network on
their
way to a given host
So if you are trying to find one of the possible paths your packet may take, you'll find a friend in traceroute .
Now because routing tables do not change every minute, the packets that you send will most probably take the same path as traced by traceroute.
Another important point that cannot be missed is the record route option in the IP v4 header.
Once you specify that you want to use this option, every router in the path will add it's ip address to the options in the header. You can read more about it here. The catch being that the destination gets to know about the intermediate hops , not the source.
I see that you missed the role of icmp echo request and reply messages in the description of traceroute. In case this was not intentional , take a look.
Update : You can see the record route option in action by doing a ping -R
ping -R Turns on route recording for the Echo Request packets, and
display the route buffer on returned packets (ignored by many
routers).
The algorithm works properly. Indeed, routing may change due to considerations of different servers along the way, such as server load or availability. Let's say you want to send message from A to B. If the route is not changeable, what will happen if some server on the route is down? If the routing couldn't be adjusted dynamically, that would result in inability to deliver the message to the destination in this example. Here is a different example: let's say you have a server that is used for some heavy computation during the day but it's idle during the night. It's possible to allow it to pass traffic only during the night, so any routing using it will need to be changed at day.
To conclude all this we can definitely say that without dynamic routing the internet couldn't have existed in its' present form.
Addition:
Tracert sends message from A to B. It shows hops along the way. These hops constitute a valid route from A to B at the time of the execution. There is no guarantee that connection between 2 adjacent points along the way is valid after the hop has been completed. The only thing guaranteed is that for each hop there was a link between it's 2 endpoints when the message sent by tracert passed there.

What is a SNMP ping?

I know what SNMP is and I know what ping is. What is meant by a 'SNMP ping'? SNMP can be used to see if devices on a network are still alive.. what does it use to do this? I wouldn't have thought an SNMP ping is the same as an ICMP ping?
As you correctly suspected SNMP "ping" is definitely not the same as ICMP ping. What it does is to try to retrieve some basic information through SNMP like DNS name, system name, location, system type, system description etc. and if successful the "ping" is deemed to be successful too.
But this is not any kind of standard the way ICMP Ping (echo) is. There is no special "ping" command in SNMP - it's just a name for a tool used to scan whether SNMP is alive at some target device (by retrieving some common MIB values). So as you would expect the implementation differs too as a consequence of it being a useful tool rather than a standard.
But that has little effect in practice as there is a set of 'mandatory' SNMP records so if a device does not respond to those, you can be pretty sure it doesn't run SNMP. For an SNMP "ping" to work SNMP MUST be enabled on the target device of course ... which isn't the case by default most of the time in general so that's a big difference to ICMP Ping which can be used almost universally.
I hope I answered your question
Ping was "Packetized Internet Node Groper", originally a tool that implemented an ICMP echo response. "Ping" is now commonly used to convey an abstraction of checking whether a device is online, available, responding.
There are MIB options to ask a device to ping or trace route something -- which seem to be asking a device to ICMP to a third party and indicate success/failure -- but the colloquial use of "snmp ping" is to check responsiveness by asking a simple SNMP question of a static data point and get a response.
Most devices offer some response to requests in the 1.3.6.1.2 subtree, but it's not a hard/fast requirement. For example, on a Unix (-like) command line, one may try "snmpget -v 1 -c public 192.168.0.1 1.3.6.1.2.1.1.1.0" to ask "192.168.0.1" what its name is. The device may respond; it typically will not NAK if the access (version 1, community string "public", in this case) is incorrect. The switch(es) in between may choose to alert the requestor "unreachable", but may not. In this case as others, if there is no response, the messages or hints aside from "no response" may be helpful.
It is typical for an SNMP ping to be equivalent to an ICMP echo. IBM states, for example, that it issues a single ICMP echo (to the SNMPD on the receiving end) and returns the minRTT. If a response times out, then it sends another raw echo packet after 1 second, and another again after 2 seconds. If there is yet no reponse, -1 is returned.
Cisco also has a sort of similar implementation (SNMP GET/SET/PING-MIB). It is definitely implementation-specific, however.
Source
"ping" is just a generic term for sending a message for the purpose of seeing whether you get back a response.
ICMP is the most common method of "pinging" a host on an IP network. If you get an ICMP echo response from an IP address, then you know that something out there is able and willing to receive and reply to at least some network traffic for that address. Receiving a reply to an ICMP ping does not tell you anything about the state of any other network services. Not receiving a reply does not tell you anything at all: there are dozens of reasons why you might not receive a response to an ICMP ping at any particular moment.
The concept of a "ping" applies to any mechanism of communication. Some network protocols have specific standards for performing a "ping". For most, "pinging" just means trying to perform some basic operation to see if it works. For example, you could "ping" an HTTP server by just connecting to port 80 and doing "HEAD / HTTP/1.0". If you get back an HTTP response, then you know there's an HTTP server operational. The same idea applies to SNMP or any another network application.
In most contexts I have heard 'SNMP Ping' used, it was referring to a DISMAN PING. This method allows you to use SNMP to control a device or host remotely and tell it to 'ping' another device. Typically, this would be a traditional ICMP ping.
For example, imagine you are on your laptop, and you have a webserver which seems to be taking a long time to load a page. You can't quickly tell if it is a problem with the network between you and the webserver, the network between the webserver and the database, or something with one of the servers themselves. You want to eliminate network first, so you first ping the webserver from your laptop. No loss, and the latency looks reasonable. You ping the database server from your laptop, and it looks good too. The problem with the laptop-database ping test is it doesn't tell you what is going on between the webserver and the database. Ideally, you'd log into the webserver and ping the database, but you don't have a shell account. You do, however, have a read/write SNMP access. So, you decide to use 'snmpset' from your laptop to the webserver to create a table (for ping results) and specify a target (database server) to ping. The webserver snmpd initiates an ICMP ping from the webserver to the database and stores the results in a table. You then do a 'snmpget' from the laptop to pull the webserver-database ping results.
The other contexts you might see 'snmp ping' may be simply a snmpget from a SNMP client to a SNMP daemon to confirm SNMP is working.
In my understanding, an SNMP ping is the one defined in RFC 2925
https://www.rfc-editor.org/rfc/rfc2925
If the SNMP agent vendor indicates that a device supports this RFC, then you can use SNMP ping to monitor it. Otherwise, it won't work.
This is completely different from the "normal ping" (ICMP based).
Like ICMP ping, SNMP ping, just a form of a SNMP getrequest, is used to check the aliveness of any equipment that has standard SNMP agent running for monitoring. It is useful when the management systems that manage these equipments to query for the auto discovery. Any sysOid can be used to query the equipment as part of the request.

QUdpSocket problem

I try to send data using UDP protocol. Is it possible to understand when UDP dont send data?
Thanks a lot.
I try to a servis which run into client. And they send their IP an port number in one second. Server listen them and if they dont send this message it understand that client is not connected. I do this but I cant understand when they dont send? Do you have any suggestion
You can check the result of writeDatagram
Sends the datagram at data of size size to the host address address at port port. Returns the number of bytes sent on success; otherwise returns -1.
Then just check the return number to make sure the number of bytes sent was what you expected
Of course it's possible, but it might be hard.
I would recommend:
Verify that you don't get errors from your calls to send data (perhaps you're specifying a bad address, or the socket is in a bad state or something).
Try sending more seldom, perhaps your packets are getting dropped by your local network stack.
Make sure you really listen properly at the receiving end, perhaps the packets make it but you fail to read them properly.
Consider firewall/NAT issues, as usual with UDP. Protocol-wise, never include connection information as application data in packets, since then it's invisible to NAT-machines.
The next step might be digging down and trying to get some feedback from the local network stack, or maybe sniffing the network to see if the packets make it some way at least.

Resources