Unable to start secondarynamenode, datanode, nodemanager while starting hadoop - bigdata

I am using Hadoop hadoop-2.6.0, while starting hadoop services unable to start secondarynamenode, datanode, nodemanager services.
Getting Java.net.bind exceptions.
NodeManager:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [0.0.0.0:8040] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139)
at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.createServer(ResourceLocalizationService.java:356)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStar
NameNode:
2017-10-10 23:58:07,872 INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException
java.net.BindException: Port in use: 0.0.0.0:50090
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:891)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:827)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:276)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:192)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:671)
DataNode:
java.net.BindException: Problem binding to [0.0.0.0:50010] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
On trying netstat -ntpl command the below ports are in already in use
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN -
tcp6 0 0 :::8040 :::* LISTEN -
Please someone suggest me how to kill these ports to resolve this issue.
:~/hadoopinstall/hadoop-2.6.0$ jps
18255 org.eclipse.equinox.launcher_1.3.0.dist.jar
27492 RunJar
12387 Jps
11951 ResourceManager
11469 NameNode

the netstat command that you have provided also shows the PID of the process which is listening on that port.
For example
[root#dn1 ~]# netstat -ntpl | grep 50010
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 1093/java
In the above example, 1093 is the PID of the java process which binds the 50010 port.
You can easily check what process is running and if you have the correct permissions, you can kill the process too using the command.
kill -9 1093

After frequent searches/tries i have found the solution to kill the process(port) without pid.
*******#127:~$ sudo fuser -k 50010/tcp
[sudo] password for muralee1857:
50010/tcp: 1514
*******#127:~$ sudo kill -9 $(lsof -t -i:50010)
*******#127:~$ sudo fuser -k 50010/tcp
*******#127:~$ sudo kill -9 $(lsof -t -i:50090)
*******#127:~$ sudo fuser -k 50090/tcp
50090/tcp: 2110
*******#127:~$ sudo kill -9 $(lsof -t -i:50090)
*******#127:~$ sudo fuser -k 50090/tcp
*******#127:~$ sudo fuser -k 8040/tcp
8040/tcp: 2304
*******#127:~$ sudo kill -9 $(lsof -t -i:8040)
*******#127:~$ sudo fuser -k 8040/tcp
Now i am able start all hadoop services.
/hadoopinstall/hadoop-2.6.0$ jps
6844 NodeManager
7150 Jps
6547 SecondaryNameNode
6202 NameNode
6702 ResourceManager
6358 DataNode

Related

Suricata dont drop packets

I have a server with Suricata (169.69.1.11) installed and a specific rule:
drop ICMP any any -> 169.69.1.11 any (msg: "ping dropped";sid:10001;)
In other VM I execute:
ping 169.69.1.11 -c 5
so at this point, everything is bad because the pings reach, and nothing is registered on fast.log so I execute on the Suricata machine
sudo suricata -i enp0s8
and I ping another time with the same command ( 5 pings )
In my other machine every seems okay, the 5 pings seems they reach, but I look at the logs on Suricata /var/log/suricata/fast.log it drops that line
03/25/2022-11:11:05.231735 [wDrop] [**] [1:10001:0] ping dropped [**] [Classification: (null)] [Priority: 3] {ICMP} 169.69.1.10:8 -> 169.69.1.11:0
Why the pings are hitting and don't get blocked?
Why do I ping 5 times but only 1 time is logged?
My first problem is I didn't have Suricata IPS, first delete ur iptables rules with
sudo iptables -F
sudo iptables -I INPUT -j NFQUEUE
sudo iptables -I OUTPUT -j NFQUEUE
sudo iptables -I FORWARD -j NFQUEUE
and execute the Suricata with -D to let as bg
sudo Suricata -q 0 -D

Docker: unexpected error (Failure EADDRINUSE)

I'm really new to Docker. I'm trying to run Wordpress, and I've run into an error.
$ docker-compose up -d
testpublichtml_mariadb_1 is up-to-date
Starting 00b4dc8e3264_testpublichtml_wordpress_1
ERROR: for wordpress Cannot start service wordpress: driver failed programming external connectivity on endpoint
00b4dc8e3264_testpublichtml_wordpress_1 (63165c221c0b2b11d513e97d35afa39146790086115029b9bb229212d0c8c06a): Error starting userland proxy: Bind for 0.0.0.0:80: unexpected error (Failure EADDRINUSE)
ERROR: Encountered errors while bringing up the project.
$
My guess is to try and check if something is on port 80, though I'm not sure how to check that.
When I enter netstat -tulnp | grep ':80', I get:
$ netstat -tulnp | grep ':80'
netstat: option requires an argument -- p
Usage: netstat [-AaLlnW] [-f address_family | -p protocol]
netstat [-gilns] [-f address_family]
netstat -i | -I interface [-w wait] [-abdgRtS]
netstat -s [-s] [-f address_family | -p protocol] [-w wait]
netstat -i | -I interface -s [-f address_family | -p protocol]
netstat -m [-m]
netstat -r [-Aaln] [-f address_family]
netstat -rs [-s]
Probably you have some service running on port 80. To check this, execute the following command.
netstat -tulnp | grep ':80'
The last column is PID/Program name of your process. If you want to kill it, use the following command.
kill PID
After that, you should be able to start your container.

Grunt Task - freeing up the port

I setup this.
http://www.zell-weekeat.com/grunt-sass-with-susy/
It works fine for one run then, if I exit out of the sass watch I can't free up the port.
Fatal error: Port 35729 is already in use by another process.
Then
sudo lsof -i :35729
gives
\COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
grunt 26305 author 12u IPv6 7064373 0t0 TCP *:35729 (LISTEN)
How do I get this port to free up
Just for organization sake, as mentioned in comments:
You should stop a process with Ctrl+C in the terminal.
Ctrl+Z will keep it running.
If you need to kill it, find the PID with sudo lsof -i :35729 and then kill -9 PID.
Just sudo netstat -tulpn | grep -i : <port>. kill it with kill -9 PID

proxy_pass isn't working when SELinux is enabled, why?

I'm having an application listening on port 8081 and Nginx running on port 8080. The proxy pass statement looks like:
$ cat /var/etc/opt/lj/output/services/abc.servicemanager.conf
location /api/abc.servicemanager/1.0 { proxy_pass http://localhost:8081;}
In nginx.conf, I include this file as:
include /etc/nginx/conf.d/services/*.conf;
The /etc/nginx/conf.d/service is a symlink:
# ll /etc/nginx/conf.d/
lrwxrwxrwx. 1 root root 39 Dec 10 00:19 services -> ../../../var/etc/opt/lj/output/services
This is a CentOS 7.0 SELinux Enabled system. If I setenforce 0, and make it Permissive, I don't see any issues. So the file is in right place and no issues with paths. If SELinux is enforcing, I see the following in audit log:
type=AVC msg=audit(1418348761.372:100930): avc: denied { getattr } for pid=3936 comm="nginx" path="/var/etc/opt/lj/output/services/abc.servicemanager.conf" dev="xvda1" ino=11063393 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=file
I want to know how to enable Nginx to find the conf file without having to disable SELinux.
Worth noting for beginners in SELinux that if your proxied service is running on 8080, you can use the command below without compiling a policy.
$ sudo setsebool httpd_can_network_connect 1 -P
Read about audit2allow and used it to create a policy to allow access to the denied requests for Nginx.
Step 1 involves running audit2allow targeting nginxlocalconf:
$ sudo grep nginx /var/log/audit/audit.log | \
grep denied | audit2allow -m nginxlocalconf > nginxlocalconf.te
Step 2, review results:
$ cat nginxlocalconf.te
module nginxlocalconf 1.0;
require {
type httpd_t;
type var_t;
type transproxy_port_t;
class tcp_socket name_connect;
class file { read getattr open };
}
#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'httpd_can_network_connect'
allow httpd_t transproxy_port_t:tcp_socket name_connect;
allow httpd_t var_t:file { read getattr open };
Review steps to activate:
$ sudo grep nginx /var/log/audit/audit.log | grep denied | \
audit2allow -M nginxlocalconf
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i nginxlocalconf.pp
Step 3, active:
$ sudo semodule -i nginxlocalconf.pp
If you have another port or custom port allow it:
Show allow port in http:
semanage port -l | grep http
This is output in my localhost:
http_cache_port_t tcp 8080, 8118, 8123, 10001-10010
http_cache_port_t udp 3130
http_port_t tcp 80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t tcp 5988
pegasus_https_port_t tcp 5989
And allow 8081:
semanage port -a -t http_port_t -p tcp 8081
Always prefer changing types to creating custom policies. In this case, Nginx will serve files with the httpd_sys_content_t type. Assuming your files are located in /var/www:
semanage fcontext -a -t httpd_sys_content_t /var/www/*
restorecon -R -v /var/www

Determine the process pid listening on a certain port

As the title says, I'm running multiple game servers, and every of them has the same name but different PID and the port number. I would like to match the PID of the server which is listening on certain port, and then I would like to kill this process. I need that in order to complete my bash script.
Is that even possible? Because it didn't find yet any solutions on the web.
Short version which you can pass to kill command:
lsof -i:80 -t
The -p flag of netstat gives you PID of the process:
netstat -l -p
*use sudo if showing - instead of PID
Edit: The command that is needed to get PIDs of socket users in FreeBSD is sockstat.
As we worked out during the discussion with #Cyclone, the line that does the job is:
sockstat -4 -l | grep :80 | awk '{print $3}' | head -1
netstat -p -l | grep $PORT and lsof -i :$PORT solutions are good but I prefer fuser $PORT/tcp extension syntax to POSIX (which work for coreutils) as with pipe:
pid=`fuser $PORT/tcp`
it prints pure pid so you can drop sed magic out.
One thing that makes fuser my lover tools is ability to send signal to that process directly (this syntax is also extension to POSIX):
$ fuser -k $port/tcp # with SIGKILL
$ fuser -k -15 $port/tcp # with SIGTERM
$ fuser -k -TERM $port/tcp # with SIGTERM
Also -k is supported by FreeBSD: http://www.freebsd.org/cgi/man.cgi?query=fuser
netstat -nlp should tell you the PID of what's listening on which port.
Syntax:
kill -9 $(lsof -t -i:portnumber)
Example:
To kill the process running at port 4200, run following command
kill -9 $(lsof -t -i:4200)
Tested in Ubuntu.
Since sockstat wasn't natively installed on my machine I hacked up stanwise's answer to use netstat instead..
netstat -nlp | grep -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\:2000" | awk '{print $7}' | sed -e "s/\/.*//g""
I wanted to programmatically -- using only Bash -- kill the process listening on a given port.
Let's say the port is 8089, then here is how I did it:
badPid=$(netstat --listening --program --numeric --tcp | grep "::8089" | awk '{print $7}' | awk -F/ '{print $1}' | head -1)
kill -9 $badPid
I hope this helps someone else! I know it is going to help my team.
on windows, the netstat option to get the pid's is -o and -p selects a protocol filter, ex.:
netstat -a -p tcp -o

Resources