I have few Jar files/packages in the DBFS and I want an init script (so that I can place that in the automated cluster) to install the Jar package everytime the cluster starts.
I also want to install maven packages from maven using an init script.
I can do all of these using databricks UI. But the requirement is to install libraries using an init script.
To install jar files, just put files onto DBFS, in some location, and in the init script do:
cp /dbfs/<some-location>/*.jar /databricks/jars/
Installation of the maven dependencies is more tricky, because you also will need to fetch dependencies. But it's doable - from the init script:
Download and unpack Maven
Execute:
mvn dependency:get -Dartifact=<maven_coordinates>
move downloaded jars:
find ~/.m2/repository/ -name \*.jar -print0|xargs -0 mv -t /databricks/jars/
(optional) remove not necessary directory:
rm -rf ~/.m2/
P.S. But really, I recommend to automate such stuff via Databricks Terraform Provider.
Related
I am trying to run API Manager 4.0.0 from source code, I download product-am and carbon-apimgt from github. How can i debug source code in idea or eclipse ?
First of all, you have to build the product. Follow these steps in order to build the product locally.
Make sure you have installed Java and Maven in your machine.
Download or clone carbon-apimgt repository from
https://github.com/wso2/carbon-apimgt.
Go to carbon-apimgt directory and run mvn clean install command in the terminal. (You can ignore unit tests by running mvn clean install -Dmaven.test.skip=true)
Copy the build version to the clipboard (ex:- 9.12.3-SNAPSHOT)
Download or clone product-apim from https://github.com/wso2/product-apim
Replace the value of carbon.apimgt.version in pom.xml file with the value you copied. (ex:- <carbon.apimgt.version>9.12.3-SNAPSHOT</carbon.apimgt.version>)
Go to product-apim directory and run mvn clean install command in the terminal. (You can ignore integration tests by running mvn clean install -Dmaven.test.skip=true which will save your time)
The built pack can be found in product-apim/modules/distribution/product/target directory.
After building the pack, extract the content in the zip file and run sh bin/api-manager.sh --debug 5005 command.
I recommend JetBrains Intellij IDEA to debug the code easily. So, open the carbon-apimgt project in IDEA. Then Add Configuration > Add new... > Remote JVM Debug > OK. After adding the configuration, you can click on the debug button and start debugging.
I'm trying to create a multi-stage build for an R application based on rocker/r-ubuntu:20.04 image.
The reason that I'm based on that image is to install binary packages via apt-get as suggested in order to improve building time.
If I build the image without multi-stage build, the final image size is 2.32GB, so i need to decrease the final size with multi-stage build.
The approach that I follow is create an alpine:latest image and copy the app folder from builder, and the R libraries installed (/usr/local/lib/R/site-library/) and the binaries packages located in /usr/share/doc.
The final image doesn't work, because no commands to execute the app is installed.
The Dockerfile is the following:
FROM rocker/r-ubuntu:20.04 as builder
# # system libraries of general use
RUN apt-get update && apt-get install -y \
pandoc \
...
libxml2-dev
RUN apt-get update && \
apt-get install -y -qq \
r-cran-config \
...
r-cran-tidyverse
RUN R -e "install.packages(c('other-packages'), dependencies=T)"
# copy the app to the image
RUN mkdir -p /root/bloomapp/tmp
COPY . /root/bloomapp
COPY .Renviron Rprofile.site /usr/lib/R/etc/
FROM alpine:latest
#Copy app to alpine
COPY --from=builder /root/bloomapp /root/bloomapp
COPY --from=builder /usr/local/lib/R/site-library/ /usr/local/lib/R/site-library/
COPY --from=builder /usr/share/doc /usr/share/doc
WORKDIR /root/bloomapp/
EXPOSE 3838
Is reasonable this approach? or exist any other better way to do multi-stage build for a R app image?
Thanks.
Alternative to starting with a rocker build, use a slim image & install R from source in the builder stage of the multistage build.
See R Installation docs here for reference: https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-R-under-Unix_002dalikes
Choose a directory to install the R tree (R is not just a binary, but has additional data sets, help files, font metrics etc). Let us call this place R_HOME. Untar the source code. This should create directories src, doc, and several more under a top-level directory... If the configure and make commands execute successfully, a shell-script front-end called R will be created and copied to R_HOME/bin.
When I build the corda project locally, I find a corda-4.1-corda.jar. Is that basically the same as the corda.jar file produced when node directories are generated by the cordformation plugin? Can I replace the latter by the former, after renaming it, in a deployment?
Once your dependencies are set correctly, you can build your CorDapp JAR(s) using the Gradle jar task
Unix/Mac OSX: ./gradlew jar
Windows: gradlew.bat jar
Run ./gradlew clean build -x test from the root directory. Once the process finishes, the jar is produced in the corda/node/capsule/build/libs/ directory as corda-<version>.jar.
At this point I'm thinking about calling bash command pip install fabric2 each time my operator executed, but this does not looke like a good idea.
Create a requirements.txt file similar and pass that as a variable while creating the cloud composer enviroment.
Sample requirements.txt file:
scipy>=0.13.3
scikit-learn
nltk[machine_learning]
Pass the requirements.txt file to the environments.set-python-dependencies command to set your installation dependencies.
gcloud beta composer environments update ENVIRONMENT_NAME \
--update-pypi-packages-from-file requirements.txt \
--location LOCATION
Turns out you can use: PythonVirtualenvOperator it supports pip deps.
Another option that is available for the Composer users is to install deps via the composer itself: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies
I built the meteor package as follow:
meteor build /tmp
It built the jar.gz file.
Now How do install this on different server?
Thanks
upload the tar file using ftp or scp on the terminal.
untar it on the server
cd bundle/programs/server
npm install
node bundle/main.js
Additional info:
use pm2 on the server so you can manage the node process seamlessly.
OR
use forever to run your app even if you quit the terminal.