Oozie: Is there anything that needs to be done after placing an updated jar under lib folder? - oozie

I am trying to place an updated jar under lib path and removing the old jar. Unfortunately , I see the old logs in oozie console which were present in old jar. For confidential purpose I am unable to show logs here. But I am doing the below steps:
Replacing a jar (mycode.jar) under lib folder which is mentioned in workkflow.xml
Submitted the oozie job using oozie job -oozie http://host -config job.properties -run
When I see logs in console, I could see old jar(older version of mycode.jar) logs even if jar is replaced.

If you are talking about the lib directory in the oozie workflow application then you need not to do anything. The next execution of the workflow will automatically pick the new (updated) jar.
For updating the jars into share lib /user/oozie/share/lib/lib_*/* then after replacing the jar, you need to execute the following command to update the share lib into oozie server.
oozie admin -sharelibupdate
Hope this will help. Thanks.

To make sure issue is same I'll narrate what I was facing:
created a MapReduce JAR and placed it in lib folder.
Ran oozie(MapReduce action) job and picked the JAR as expected and ran fine.
I had some functionality changes in my code(JAR) so I added new log statements to make sure new JAR is being picked. Built the JAR and replaced the old JAR with newly built JAR in lib folder(hdfs)
Ran oozie job again, code from old JAR was executed because new log statements did not show up.
After few search I found following tips:
Clear the Yarn Cache: found this in HortonWorks site(https://community.hortonworks.com/articles/92339/how-to-clear-local-file-cache-and-user-cache-for-y.html) - pasting content below for reference
Short Description:
To use different version jar file with same name, clear cache on all NodeManager hosts to prevent the application using old jar
a. Find out the cache location by checking the value of the yarn.nodemanager.local-dirs property
< property >
< name >yarn.nodemanager.local-dirs< /name>
< value>/hadoop/yarn/local</value>
< /property>
b. Remove filecache and usercache folder located inside the folders that is specified in yarn.nodemanager.local-dirs.
[yarn#node2 ~]$ cd /hadoop/yarn/local/
[yarn#node2 local]$ ls filecache nmPrivate spark_shuffle usercache
[yarn#node2 local]$ rm -rf filecache/ usercache/
c. Restart YARN service.
I was unable to clear cache because I did not have the necessary access. Thus I followed below workaround
Rename the Package or class, since this package/class was written by me, I had the liberty to simply rename the class, thus in oozie when new Class name was looked up, automatically the new functionality was executed.
Option 2 may not be viable for many and the question remains open as to why oozie does not pick New JAR/Class.

Related

A .jar file does not run after building it

Actually, I'm trying to add new language to Streama Media Server. I downloaded source code, added new language file (as guided here). After that, I want to build a jar with that project, I build it with IntelliJ Idea 2021.1 (here is how I did). So, When I run that jar file (in Ubunt 20.04), it fails and gives this error: Screenshot
When you have made adjustments to the source code, it is likely that you will want to create a new .jar file and deploy it on your server. For this, you can use a simple command:
# for unix based systems
**./gradlew assemble**
# for windows
**./gradlew.bat assemble**
This will create 2 new .jar files under build/libs,
streama-{version}.jar
streama-{version}.jar.original
all you will need is the streama-{version}.jar.
This file is an executable, so you can just copy it into your deployment directory / your server and start it as usual.

'firebase init' always leads to C:/Users/PC

When I run the command:
PS C:\Users\PC\Desktop\Project> firebase init
it says:
You're about to initialize a Firebase project in this directory:
C:\Users\PC
I have tried running it in C:\Users and it worked properly, it won't go anywhere farther than C:\Users\PC
Can anyone help me with this?
When you run firebase init, then first thing is does is check parent directories for the presence of a hidden ".firebaserc" file to see if there is a project already created. If the file isn't found, it will create a new project in the current directory. If the file is found, it assumes you are working with an existing project in the stated folder.
You have a .firebaserc file in C:\Users\PC. If this is not what you expect, and you don't actually have a project there, d then you should probably just delete that file and start over.

Unable to push Repo from "react-native-firebase-starter" template

I cloned this repo to start my own project
https://github.com/invertase/react-native-firebase-starter
I have made some modifications and got it setup for Firebase, however I cannot push or rename the Repository.
I ran npm run rename and renamed the directory. GitHub still seems to think I am trying to push the orginal repository as my own.
When I try to push I get:
Authentication failed. You may not have permission to access the repository or the repository may have been archived...
How can I keep this template/starter and push a copy of it as my own repository?
I have tried removing all of the inessential files from the Repo and pushing that way. I get the following error:
I expected to be able to use the starter as a starter to get a project up and running... Maybe I am missing something super obvious.
I don't see a .git folder within the root of the react-native-firebase-starter template, perhaps this is causing issues with pushing this template since git needs to know where to point to upstream.
Maybe you could try initializing the template to your personal git repository and seeing if this resolves your authentication issue:
Create a new repository on GitHub. To avoid errors, do not initialize the new repository with README, license, or gitignore files.
Initialize the local directory containing the template as a Git repository:
git init
Add the files in your new local repository. This stages them for the first commit:
git add .
Commit the files that you've staged in your local repository:
git commit -m "Initial commit"
At the top of your GitHub repository ,created in step 1, copy the remote repository URL.
Add the URL for the remote repository where your local repository will be pushed:
git remote add origin <remote_repository_url>
Push the changes in your local repository to your upstream repository contained in GitHub:
git push -u origin master
You should now be able to push this starter template into your own GitHub repository and use it as your own project.
As for the npm run rename command: this is a custom npm run script created by the author of this starter template and it simply runs the rename.js file contained within the .bin directory of the template's root directory. All this command does is recursively rename the files contained within this template project to the new name specified by your input, so I don't think this is causing the issue. I suspect once your project has been initialized properly with git the authentication issue will disappear as it will now point upstream to your personal repository.
Hopefully that helps!

how to add a jar file stored outside of the oozie project's ./lib folder?

I'm writing an oozie java action which has my custom code in a jar file in the job ./lib folder.
I would also like to add to the classpath a jar in a folder external to my job (i.e. /home/me/otherjars/spark-assembly.jar).
The ./lib folder gets added to the classpath automatically. How can I get oozie to also add the external jar?
The oozie.libpath property is definitely what you need. Please check...
the Oozie documentation
this Oozie JIRA about global/local scope for that property
this orphan thread about precedence order (search for that
phrase)
this post and this other post, for example
The Bestway to use any custom Jars in Oozie useing, Once Oozie Sharedlib Installed in Cluster, you can mention place the Jar, in Sub Folder and pass the parameter
oozie.use.system.libpath = true
These will call Jar when every the Jobs are getting started.
Another option you can use, is adding Custom Path with UDF jar in hadoop_env.sh file under Hadoop ClassPath, These required your Hadoop restart to take effect, along with it also required you Custom JAR Path should be available in all the Nodes of Hadoop Cluster.

Symfony Warning : rename (../app/cache/dev , ../app/cache/dev_old ) : Access Denied . (Code : 5)

I am working on Symfony project.
When i try to do :
php app/console cache:clear
i get the following ErrorException:
Warning : rename (../app/cache/dev , ../app/cache/dev_old ) : Access Denied . (Code : 5) in ../vendors/Symfony/src/Symfony/Bundle/FrameworkBundle/Command/CacheClearCommand.php on line 76
What is the problem here? I have given all permissions to the user on my machine (Windows 7 OS). Any ideas why it is happening?
Thank You.
Be sure that the files are not in use (as meze pointed out). If you're using something like TortoiseGit or Netbeans, etc - be sure to mark the cache folder as ignored so that they are not accessed.
If all else fails, download a free program like Unlocker that will allow you to quickly and easily detach running processes from the files/folders you are trying to modify.
To expand on leek's post, Symfony 2 cache-clearing operations shuffle the cached items across different folders during the cleanup. Part of this process includes creating cache/dev-new/ and cache/dev-old/ folders.
If you are using Eclipse or another IDE that dynamically monitors subfolders within your project, the IDE will nearly instantly spot the new folder creation and look in those folders for new files (in Eclipse, I noticed the DLTK module constantly doing this in the Progress View). This may unfortunately get in the way of Symfony, which wants to rename and/or delete these folders.
Specifically with Eclipse Indigo on Windows 7 64-bit, you can remove the cache/, cache/dev/, cache/dev_old/ and cache/dev_new/ folders from the build path by right-clicking your project and selecting "Build Path > Configure Build Path...". This originally had no effect for me; I kept seeing the DLTK module trying to index the cache folders. I ended up uninstalling the Aptana Studio plug-in, closing all Editor documents, shutting down Eclipse, manually deleting the sub-folders in the cache/ folder, running Symfony cache:clear, then starting up Eclipse and reinstalling Aptana. Seems to have worked thus far.
It's an issue with Symfony 2.0.x and Symfony 2.1.x. These a workaround for this:
Open the file: src\Symfony\Bundle\FrameworkBundle\Command\CacheClearCommand.php
and add the statement sleep(1); where the directory creation is failing, in the execute() function:
//...
rename($realCacheDir, $oldCacheDir);
sleep(1);
rename($warmupDir, $realCacheDir);
//...
You might have to re-open the CLI twice and run cache:clear, but it will fix the problem after that
If you're using text editors as Sublime Text try to ignore path of cached files
Go to Preferences/Setting
edit config file
{
...
"folder_exclude_patterns": ["var","node_modules", ".git"],
...
}
in my case
Cache folders containers.
Symfony 4
/var
Symfony 2
/app/cache
Nice coding!
I had the same issue with Symfony 4.1.13 and the root cause was that the VS Code was using the file "var/cache/dev/srcDevDebugProjectContainer.xml4QSKuA".
The issue was Cannot rename "var/cache/dev/srcDevDebugProjectContainer.xml4QSKuA".
I fix it adding **/var on the fields "Files to Exclude"
See the following steps:
1º Step:
2º Step search for "exclude" word, then add **/var/cache as the following image:
It's worked for me.
To have tranquility once and for all I create clear.bat file in MyProject folder under Windows 7 .
rmdir d:\symfony\framework-standard-edition\app\cache\ /S /Q
I guarantee 100% effectiveness for cleaning cache. Some minor problems can occurred after first time refresh website: Symfony must recreate some folders they need.
SYMFONY AND ANTIVIRUS
Multiple symfony calling in local windows machine get cross with some Antivirus software such as NOD in my case. Exclude symfony cache folder from real time protection

Resources