I have a multi-project build that makes extensive use of sub-projects in order to make dependencies between them explicit (by means of dependsOn - the sub-projects act as layers).
If it comes to publishing I want to roll up all the sub-project artifacts into the artifact of the main project (jar), and collect all their libraryDependencies. In other words, I want to publish as if there would be no sub-projects, just a single root project containing all my code (i.e. no ueber-jar). The sub-projects are only there to break up the compilation into smaller scopes with enforced dependency structure, they are not supposed to be distributed separately.
What is the best way to achieve that?
Related
So I've been trying to do some AWS lambda with scala and sbt. And one of the recommendations is
Minimize your deployment package size to its runtime necessities. This
will reduce the amount of time that it takes for your deployment
package to be downloaded and unpacked ahead of invocation. For
functions authored in Java or .NET Core, avoid uploading the entire
AWS SDK library as part of your deployment package.
I've been using sbt-assembly to create fat jars for my code and dependencies, but it seems like sbt-assembly will package all library dependencies when I only use like 10% of the aws-core library which adds a lot of content. Was wondering if there is something I can do to cut down on the number of dependencies to what is actually imported in my code (and their dependencies).
As far as I know, there is no direct and safe way to selectively contains the dependent classes in a fat jar generated by the sbt-compile plugin.
First of all, you should understand that sbt plugins just provide a settings and jar files required to invoke methods in your project. It means that the dependent plugin is brought to your project with the pre-compiled jar file, which is determined by the version that you specified on your build setting (e.g., plugins.sbt in your project dir).
For example, the jars of sbt-assembly are brought to your project from this link when you specify that you want to use the sbt-assembly (although it is brought to your project when you use recent sbt version by default).
Therefore, at least you may have two choices to shrink your jar files.
Compile the jar file from the scratch
For the sbt-aws, its source code is provided on this link, so you may selectively compile the source codes to get the classes that your program is going to use.
Use the tool for shrinking jar file
There are several tools to shrink your jar file based on the dependencies. The most popular tool is proguard; it seems that there is a proguard support for sbt.
Warning
As mentioned in another stack overflow answer, selectively choosing some classes from the jar may cause your program crash depending on the input value and several other conditions. You've said that only 10 percent of the jar file is used, but you cannot ensure whether other classes are required from your code and library that your project depends on. When you use the tool for helping you to shrink the jar file, be careful when the program is security critical one.
We have a multi module project consisting of two modules, modA and modB.
modA depends on modB.
modB in turn depends on a list of libraries (libA and libB) where we also have the source code. This sources have already been adapted by us.
At last, libB and libC are independend from each other, but depend on a third library, libC.
What I want to have is a setup, where the three libraries (which are in principle also a multi-module SBT project) can just be "included" in the top level project.
The point here is also that these libraries can be re-used for other projects, too, so the changed sources should not belong to this super project only.
Currently I tried to solve it by including the library as GIT submodule.
Unfortunately SBT does not (seem to) support hierarchical sub modules, so I cannot really just have a second, also multi-module SBT file for all libraries which just gets included in the "super-super" project.
This current setup is clearly not the SBT way.
What is the intended method of solving this?
Just adapting the library separately and re-using it just as JAR file in the super project is possible, but clumsy, because the using project(s) are the main reason to hack the library, so it would be nice if this works in a smooth way.
A question about composing sbt projects:
I forked a library project (https://github.com/allenai/pipeline) and wish to hack on it vis-a-vis my project that uses it. If I put them in the same multi-project build ― really my preferred option over publishing locally and consuming ― sbt will not read the project dir of the nested fork (where .scala files and plugin definitions originally reside).
E.g. I try, in build.sbt:
lazy val pipeline = (project in file("pipeline"))
lazy val githubCruncher = (project in file("."))
.dependsOn(pipeline)
And I get errors indicating directory pipeline/project is being ignored; the build.sbt itself of the nested project is being read, but sbt does not find what is defined alongside to it in its sibling project directory (and hence fails to load build definition).
So, must I squash the two projects into a single build.sbt or can I somehow elegantly compose the two without that? I am asking because this has also been a problem on other projects I work on where I ended up squashing each and every build definition minutia into one big monolith ― even though I'd prefer distributing some of the sub-projects as independent projects aside my own dev configuration.
According to my experience looks like everything inside the project directory of a project being pulled into a multi-project build ― needs to move to project under the root of the multi-project build, whereas the build.sbt of the pulled-in project does not have to be squashed ― a reference to it from from the top level build.sbt will work (unless that reference is followed by definitions for it right there in the top level build.sbt).
Sbt plugins coming from the pulled-in project then become applied to the unified multi-project build, and need to be disabled with disablePlugins for other sub-projects, if or where appropriate or desireable.
I am trying to achieve what a resourceGenerator in Runtime would do: create a resource that is available on the classpath during runtime, however that would not be packaged under the main configuration.
In my specific case, I am trying to create an sbt plugin that facilitates dealing with JNI native libraries. The above mentioned resource would be a "fat" jar containing a shared library, thus it is not required for compilation but only during runtime.
My goal in the end is to publish the standard jar (in the Compile configuration) and publish the fat jar as an extra artifact (in the Runtime configuration). However, during local testing, I would like the shared libraries to be available on the classpath when simply calling run from sbt.
I tried implementing a resourceGenerator in Runtime, however with no success. An alternative approach I could imagine would be to modify runtime:exportedProducts or alter runtime:managedClasspath directly, however I first wanted to know if there is already a way to include resources only in the runtime configuration?
I just started working with ant a few days ago. Right now I have a general buildall.xml which should call each project's build.xml. Because some projects depend on each other, I need to rebuild some other projects which depend on it. This isn't a problem--I'm just setting the depends property of the target. However, ant is always building the dependencies, even when the files haven't changed.
Let's say project1 has no dependencies; project2 depends on project1; project3 depends on project1, 2; project4 depends on project1, 2, and 3; and so on.
I could hack a solution which looks at project K, and checks if project 1 .. project K have updated files using uptodate. If so, then run the target. This is messy and appears unnecessary.
What is the cleanest way to implement this?
EDIT: So I decided to just hack in a bunch of targets, "check_projectK" where it does the uptodate checks on all of its source files, its build file, and the build files of the 1 .. K-1 projects. Due to dependencies, this is always handled correctly. However, this is still a large amount of copy and paste for a large workspace. I will leave this open.
Short answer, ANT can't do it, not unless you have some kind of way to connect to your version control system and check if anything has changed (you are using source control right?). Ant doesn't know about when what the last time a file changed and then see if it matches with what was built; it doesn't have the concept of a dependency repository. The whole purpose of Ant is that it just builds.
The solution to your problem isn't Ant, it's Maven. Maven HAS a dependency repository. There's also a very nifty plugin for Maven used specifically with Flex appropriately called FlexMojos. By using this, Maven can know when something was last built because it's uploaded to the repository. Then your other projects can add it's dependencies and download the SWC needed.
On top of that, it mixes great with a continuous integration engine like Hudson, Bamboo and Teamcity, which builds a project every time a file has been committed to your source control system, and then updates all dependent projects automatically!