I am running a map-reduce job which crashes with a ClassNotFoundException for class HTable. This is weird because I am specifying all hbase jars. So I checked whether HTable is defined in any of the jars. In the folder where all jars were stored, I ran the following shell script.
for j in `ls *.jar`
do
jar tf $j | grep HTable
done
The result was blank. So it did not find HTable in these jars. Is it possible that the HTable definition might be in a jar within a jar and my method does not look into nested jars? I am surprised because these jars reside in the hbase-<version>/lib folder and so must have a definition of HTable.
How do I check whether a class exist in a (possibly) nested jar file?
I found it in hbase-client-<version>.jar. I am using CDH 5, so that JAR can be found here. Hope this helps!
Related
I have read here that the log4j library can be "nested" within other files that are deployed with an application.
I can find files with 'log4j' in the filename but don't know how to find log4j in these "nested jars". Is there a way to do this from the command line?
update
This question has moved to SuperUser here.
Run the following command search for log4j jar files in an application:
dir /s /b <application_root>\*log4j*.jar
If any files are displayed, check the version number that is part of the file name. For example Tomcat\webapps\ROOT\WEB-INF\lib\log4j-core-2.15.0.jar is version 2.15.
If the version is between 2 and 2.14 (2.15 is not vulnerable), the application is vulnerable to CVE-2021-44228 and one of the following mitigations must be applied.
I am trying to place an updated jar under lib path and removing the old jar. Unfortunately , I see the old logs in oozie console which were present in old jar. For confidential purpose I am unable to show logs here. But I am doing the below steps:
Replacing a jar (mycode.jar) under lib folder which is mentioned in workkflow.xml
Submitted the oozie job using oozie job -oozie http://host -config job.properties -run
When I see logs in console, I could see old jar(older version of mycode.jar) logs even if jar is replaced.
If you are talking about the lib directory in the oozie workflow application then you need not to do anything. The next execution of the workflow will automatically pick the new (updated) jar.
For updating the jars into share lib /user/oozie/share/lib/lib_*/* then after replacing the jar, you need to execute the following command to update the share lib into oozie server.
oozie admin -sharelibupdate
Hope this will help. Thanks.
To make sure issue is same I'll narrate what I was facing:
created a MapReduce JAR and placed it in lib folder.
Ran oozie(MapReduce action) job and picked the JAR as expected and ran fine.
I had some functionality changes in my code(JAR) so I added new log statements to make sure new JAR is being picked. Built the JAR and replaced the old JAR with newly built JAR in lib folder(hdfs)
Ran oozie job again, code from old JAR was executed because new log statements did not show up.
After few search I found following tips:
Clear the Yarn Cache: found this in HortonWorks site(https://community.hortonworks.com/articles/92339/how-to-clear-local-file-cache-and-user-cache-for-y.html) - pasting content below for reference
Short Description:
To use different version jar file with same name, clear cache on all NodeManager hosts to prevent the application using old jar
a. Find out the cache location by checking the value of the yarn.nodemanager.local-dirs property
< property >
< name >yarn.nodemanager.local-dirs< /name>
< value>/hadoop/yarn/local</value>
< /property>
b. Remove filecache and usercache folder located inside the folders that is specified in yarn.nodemanager.local-dirs.
[yarn#node2 ~]$ cd /hadoop/yarn/local/
[yarn#node2 local]$ ls filecache nmPrivate spark_shuffle usercache
[yarn#node2 local]$ rm -rf filecache/ usercache/
c. Restart YARN service.
I was unable to clear cache because I did not have the necessary access. Thus I followed below workaround
Rename the Package or class, since this package/class was written by me, I had the liberty to simply rename the class, thus in oozie when new Class name was looked up, automatically the new functionality was executed.
Option 2 may not be viable for many and the question remains open as to why oozie does not pick New JAR/Class.
I want to create one executable JAR file which contains other JAR libraries. But when I execute the JAR, it gives an error of class path problem.
How do I solve the class-path problem?
I think you can try it like this;
Here is a simple example for you question. First, we assume we have a project directory like D:\javademo. In this working directory we then create a main class HelloWorld.java and thtat contains our other JAR files, like commons-lang.jar. Now, we must archive our main classes HelloWorld and commons-lang.jar into test.jar file.
First we must edit our manifest file so that we can specify our class-path and main-class
like this:
Manifest-Version: 1.0
Created-By: tony example
Class-Path: test.jar commons-lang.jar
Main-Class: org.tony.java.HelloWorld
We named this file test.mf. Now we use the jar command to generate our JAR file like this:
jar -cvfm test.jar test.mf -C ./ .
Then it will generate the JAR file test.jar. You can use this command to run this main class using java command:
java -jar test.jar
That is my solution. I hope it give you something helpful...
You should use third-party libraries for it. For example, OneJar. You'll have to build the final JAR file using the OneJar tool (like Ant task) instead of standard JRE's tools.
On running such a JAR file, OneJar's service class is launched instead of yours. This class then loads JAR files packed inside, as well as your classes, and run your main class.
There are similar questions and answers already. For example: Stack Overflow question Easiest way to merge a release into one JAR file.
I created simple Java Servlet: WelcomeServlet.java.
Than, I tried compile this file via:
javac WelcomeServlet.java
In result I see compile error:
package javax.servlet doesn't exit
I try find solution for this error with Google. And I find first part of answer: java compiler doesnt see servlet-api.jar file.
I know, that Apache Tomcat in it lib folder contains servlet-api.jar file.
So, I have this file, but where I must copy this file??
I try different folders:
echo %JAVA_HOME%
C:\Program Files\Java\jdk1.6.0_26
%PATH% contains this line: C:\Program Files\Java\jdk1.6.0_26\bin
So, I copy in:
%JAVA_HOME%\bin
%JAVA_HOME%\lib
%JAVA_HOME%\jre\lib
And in result same error.
And only after I copy servlet-api.jar in directory:
%JAVA_HOME%\jre\lib\ext
compilation complite sucessful.
My question: Why? Why I must copy in folder %JAVA_HOME%\jre\lib\ext ??
Where This moment describe in documentation?
And other question we have some official docs or specifications that describe folder structure for jdk folder??
You'll need to specify the directory or directories you want the compiler to search by using the -classpath command line option when running javac. The reason the compiler found your .jar in %JAVA_HOME%\jre\lib\ext is because it searches the extension directories by default.
This is for Java 1.5, but I believe it is more or less still correct:
http://docs.oracle.com/javase/1.5.0/docs/tooldocs/findingclasses.html
The link Shaun provides is a more complete answer. But in short, using the classpath is the best way to introduce 3rd party or external (to the JDK/JRE) libraries. The classpath is a concept much like the %PATH% or the $PATH variables, but specifies locations for java to use for lookup rather than the shell to use for lookup of executables.
The classpath provides the java compiler or java virtual machine a list of items to use when searching for resources. This "path" may include directories or files. It will typically include jar files and sometimes locations of configuration files. Many Java based lookup schemes for files configuration or otherwise use some variant of what is accomplished by [Class#getResourceAsStream()][1]'s use of walking the Classpath.
I have rarely seen an incident where putting a jar file in the lib/ext location was preferred to utilizing the Classpath.
The classpath is typically an environment variable (%CLASSPATH% or $CLASSPATH) or specified on the command line when running java or javac (e.g. -cp or -classpath see the help from the executable you are running).
Build tools such as Ant and Maven will also provide abstractions to defining the list of jars to be utilized by your applications and are highly recommended to be used for any length of repetitive change code, build, test, run cycles.
I have a bunch of JAR files (from a maven2 project) and maven reports some package could not be found (org.openanzo.client.jena to be exact). I want to dig into the JAR files downloaded as the result of maven dependency resolution and find what packages are thus available from these JAR files. Insights?
UPDATE: Apparently, the only good solution to inspect insides of a jar file is the "jar" utility or one can use the facilities of their IDE to do so.
jar tvf filename.jar will show you the contents of a jar file without requiring you to extract it.
But I think that maybe what you are really trying to do is find the right coordinates for the dependency that you are missing, since obviously none of the ones you have right now are supplying the package you are looking for (in other words, checking their contents is not likely to help you).
I confess that the first place I would suggest to check is Sonatype's public Nexus instance. A search for your example turns up nothing, though. Usually that means the project is not trying to get their stuff into Maven Central or other major repositories (which is okay), so you have to resort to a web search. Usually the first two sections of the package tell you where to look (openanzo.org in your case).
If you are on Linux or a Mac, you could go to the terminal at the root of the folder containing your JARs and type:
# grep -ri "org.openanzo.client.jena" *
It will return a recursive list of all JAR files that contain that package name. If it returns 0 results, then none of those JARS contain that package.
If you wanted to do a more exhaustive search, you could unJAR the JAR files. The directory structure and .class files will be organized by packages in folders.
# jar xvf filename.jar
If you are on Windows, you can unJAR a JAR file using a tool such as 7Zip.
#Carsten
you do not have to rename a .jar file to .zip. You can directly open the jar file in winzip/or other zip utility (assuming windows OS)
#ashy_32bit
try using "jar class finder" eclipse plugin from IBM. Simple plugin for finding classes (if you know the class name)
OR
as carsten suggested... set the jar files as lib files and manually look it up
OR
create a batch file called a.bat (where you have all your jar files directly under a single folder) and paste the following 4 lines
#ECHO OFF
dir /b *.jar > allJarFilesList.txt
FOR /F %%A IN (allJarFilesList.txt) DO jar -tf %%A > list_of_packages.txt
FOR %%B IN (list_of_packages.txt) DO FIND /I "com/sun" %%B
NOTE the "com/sun" in the last line.. it is hard coded, you can pass as argument as well...
I know this is very basic form and can be improved "a lot" like looking up in various sub directories.
hope this helps :-)
.jar files are just ZIP compressed archives, rename it to zip, open it with your favourite unzip programm, and traverse through the directory.
If you add the jar file to a eclipse project, you can traverse through the lib in th project explorer.
HTH
Assuming maven downloaded the jar files,the files will be loaded in to a local repository.
You could use maven browser that comes packaged with Eclipse to browse and search for artifacts in your repository.(usually in userdir/.m2/repository)
Note:You can explore your repository directly if you want. You will understand the packages that were downloaded. But I suggest using the plugin.
If you are using Intellij IDEA, each project contains a tree called External Library that allows you to search and explore your libraries.