So I've been trying to do some AWS lambda with scala and sbt. And one of the recommendations is
Minimize your deployment package size to its runtime necessities. This
will reduce the amount of time that it takes for your deployment
package to be downloaded and unpacked ahead of invocation. For
functions authored in Java or .NET Core, avoid uploading the entire
AWS SDK library as part of your deployment package.
I've been using sbt-assembly to create fat jars for my code and dependencies, but it seems like sbt-assembly will package all library dependencies when I only use like 10% of the aws-core library which adds a lot of content. Was wondering if there is something I can do to cut down on the number of dependencies to what is actually imported in my code (and their dependencies).
As far as I know, there is no direct and safe way to selectively contains the dependent classes in a fat jar generated by the sbt-compile plugin.
First of all, you should understand that sbt plugins just provide a settings and jar files required to invoke methods in your project. It means that the dependent plugin is brought to your project with the pre-compiled jar file, which is determined by the version that you specified on your build setting (e.g., plugins.sbt in your project dir).
For example, the jars of sbt-assembly are brought to your project from this link when you specify that you want to use the sbt-assembly (although it is brought to your project when you use recent sbt version by default).
Therefore, at least you may have two choices to shrink your jar files.
Compile the jar file from the scratch
For the sbt-aws, its source code is provided on this link, so you may selectively compile the source codes to get the classes that your program is going to use.
Use the tool for shrinking jar file
There are several tools to shrink your jar file based on the dependencies. The most popular tool is proguard; it seems that there is a proguard support for sbt.
Warning
As mentioned in another stack overflow answer, selectively choosing some classes from the jar may cause your program crash depending on the input value and several other conditions. You've said that only 10 percent of the jar file is used, but you cannot ensure whether other classes are required from your code and library that your project depends on. When you use the tool for helping you to shrink the jar file, be careful when the program is security critical one.
Related
I have a jar that works on Java 8.
I would like to create a new jar, that is going to be Multi-Release JAR but empty, just with 'patched' classes in META-INF/versions.
I would like to have a separate jar, so people can include it on Java9, otherwise, they use the default one. Why? Because so many tools are not yet prepared for Java9 MR-Jars.
Would this be possible? Would Java9 MR-Jar override classes from others jars?
Why?
The idea behind Multi-Release jars is that they provide simple patching. In my humble opinion, the way MR jars works is not satisfying.
There are two reasons why I can't make 2 separate Jars:
try to make cross-compile source base that works with Java8 and Java9. You would end up with folders like java, java8 and java9... and then have the build produce two jars, two poms... Yeah, good luck.
Imagine that I even build a library for java9. What about transient dependencies? That would mean that all other libraries that uses mine, would need to have jre8 version that depends on my jre8 version. Just because there is Java9 version!
Here is the story:
My A is a Java library built on Java8 but packaged as Multi-Release Jar which means it contains additional classes for when jar is run on Java9. Additional classes are built separately on JDK9 and I copied them manually (yeah, I know, but it works for now).
Unfortunately, some tools and servers (Jetty) are not aware of MR Jars and this makes them NOT working.
For that reason, I have A-jre8 version of my library, that comes without any extra classes, so servers can use it.
However, if user is using library B that depends on my A, he will still get the MRJar version of A and this will fail again. I want to be able to prevent this somehow. And I can't say to B: hey, could you make B-jre8?
Possible solution
JAR is just about packaging!
Allow the separate jar to patch existing jar.
In my case, I would just include A.jar9 and Java would consider A.jar and A.jar9 together as a package. No need for META-INF/versions. Very clean. And, best of all, it would help in situations like above! If run on Java8, the jar9 jar would make no difference; if run on Java9 the jar9 jar would patch the jar with the same name. Simple as that. No transitive dependency hell.
Rename classes in META-INF/versions.
Common Oracle, have you ever heard about the classpath scanning? Could you at least rename the classes in versions to e.g. *.class9 so not to be caught by existing classpath scanners.
As it is today (Java v9.0.4) - no.
We have a multi module project consisting of two modules, modA and modB.
modA depends on modB.
modB in turn depends on a list of libraries (libA and libB) where we also have the source code. This sources have already been adapted by us.
At last, libB and libC are independend from each other, but depend on a third library, libC.
What I want to have is a setup, where the three libraries (which are in principle also a multi-module SBT project) can just be "included" in the top level project.
The point here is also that these libraries can be re-used for other projects, too, so the changed sources should not belong to this super project only.
Currently I tried to solve it by including the library as GIT submodule.
Unfortunately SBT does not (seem to) support hierarchical sub modules, so I cannot really just have a second, also multi-module SBT file for all libraries which just gets included in the "super-super" project.
This current setup is clearly not the SBT way.
What is the intended method of solving this?
Just adapting the library separately and re-using it just as JAR file in the super project is possible, but clumsy, because the using project(s) are the main reason to hack the library, so it would be nice if this works in a smooth way.
I am trying to achieve what a resourceGenerator in Runtime would do: create a resource that is available on the classpath during runtime, however that would not be packaged under the main configuration.
In my specific case, I am trying to create an sbt plugin that facilitates dealing with JNI native libraries. The above mentioned resource would be a "fat" jar containing a shared library, thus it is not required for compilation but only during runtime.
My goal in the end is to publish the standard jar (in the Compile configuration) and publish the fat jar as an extra artifact (in the Runtime configuration). However, during local testing, I would like the shared libraries to be available on the classpath when simply calling run from sbt.
I tried implementing a resourceGenerator in Runtime, however with no success. An alternative approach I could imagine would be to modify runtime:exportedProducts or alter runtime:managedClasspath directly, however I first wanted to know if there is already a way to include resources only in the runtime configuration?
I am working at a library needing some dependencies.
For ease of deployment, I want to create a JAR file containing everything, including the dependencies.
I have tried sbt-assembly - this works, but it may be inadvisable due to legal reasons, so I'm looking for a solution where the resulting JAR file has the original JAR files inside, and where the classpath entry in MANIFEST.MF is set up such that client classes may just add this "nested JAR file" into their classpaths.
Is something like this even possible? sbt-one-jar nearly does, what I want, but only for executables - my product will result in a library, so this is not a perfect fit.
As I've used SBT so far, an SBT plugin would be easiest to use, as it is rather too much work to convert everyting to maven or gradle or ... now.
After thinking a bit more about how class lookup works, we dediced to abandon this experiment.
Basically classes are loaded by ClassLoader instances, and the standard class loaders for applications use a fixed strategy of how to find classes in JAR files or directories.
It seems that to allow a library to be located in a hierarchical JAR file, we must also provide the user of this library (i.e. the library client) with a special classloader so that our client may load all needed classes from the hierarchical JAR.
This is too much work to be worth it - the whole idea of a hierarchical JAR was enteratained only to simplify deployment, and having to juggle own classloaders would nullify this simplification.
In short - possible, but probably not worth the effort.
I just started working with ant a few days ago. Right now I have a general buildall.xml which should call each project's build.xml. Because some projects depend on each other, I need to rebuild some other projects which depend on it. This isn't a problem--I'm just setting the depends property of the target. However, ant is always building the dependencies, even when the files haven't changed.
Let's say project1 has no dependencies; project2 depends on project1; project3 depends on project1, 2; project4 depends on project1, 2, and 3; and so on.
I could hack a solution which looks at project K, and checks if project 1 .. project K have updated files using uptodate. If so, then run the target. This is messy and appears unnecessary.
What is the cleanest way to implement this?
EDIT: So I decided to just hack in a bunch of targets, "check_projectK" where it does the uptodate checks on all of its source files, its build file, and the build files of the 1 .. K-1 projects. Due to dependencies, this is always handled correctly. However, this is still a large amount of copy and paste for a large workspace. I will leave this open.
Short answer, ANT can't do it, not unless you have some kind of way to connect to your version control system and check if anything has changed (you are using source control right?). Ant doesn't know about when what the last time a file changed and then see if it matches with what was built; it doesn't have the concept of a dependency repository. The whole purpose of Ant is that it just builds.
The solution to your problem isn't Ant, it's Maven. Maven HAS a dependency repository. There's also a very nifty plugin for Maven used specifically with Flex appropriately called FlexMojos. By using this, Maven can know when something was last built because it's uploaded to the repository. Then your other projects can add it's dependencies and download the SWC needed.
On top of that, it mixes great with a continuous integration engine like Hudson, Bamboo and Teamcity, which builds a project every time a file has been committed to your source control system, and then updates all dependent projects automatically!