We have a pipeline with 3 stages:
Building a lib
Building an application that depends on the lib.
Packinging the app into a distro (the app inlines the lib, so the distro is only directly dependent on the app).
If we make a commit to the lib, it should be built, then trigger the app to be built with the new lib, and finally, the distro to be repackaged. The job-plan for the distro includes this:
- name: build-distro
serial_groups: [grp]
plan:
- get: app
passed: [build-app]
trigger: true
- get: distro
trigger: true
All the jobs are members of the same serial group, but the distro stage does not run if the pipeline was triggered by a change to the lib. Only with a change to the app will the distro step run.
In order to get the distro to build when a commit is made to the lib, another resource-dependency must be added to the plan for the distro:
- get: lib
passed: [build-lib]
trigger: true
In this simplified setup, this is not such a bad deal, but our real situation has more than ten libs and five apps with various dependencies on the libs. These apps should then be packaged together. If the distro must depend on all the libs in addition to the apps, in order to be built for all changes, the setup becomes very complicated, both in the YAML-files and the graphical view of the pipeline. We also want to add a fourth stage to do UI-testing of the completed distro, but that is almost unmanageable with all the required dependencies.
Is there some kind of dependency I can set up from distro to app, in order for the distro to build every time the app is built, without the dependency on the lib?
So, I think I finally found out what is going on here. It is actually quite logical:
The dependency triggers are depending on the resource, not the conclusion of a build. So, if the build-app job is run, because it depends on the lib resource, this does not change the app resource, which the build-distro depends on (with the restriction that build-app passed).
In order to make this work, there must be output resources from build-app that are put to a resource that the next stage can discover a change to and trigger on. We didn't do this, because our builds are done with gradle, and the build-scripts already had functionality to upload the output resources to our local repository where it could be easily downloaded in the next stage.
Related
I added Flow to my React/Redux/Webpack project and initially, it was great, I loved the type annotations. Over time, I noticed that the start up time of the project became really slow - around 120s; it used to be around 20s. Upon investigation, I realized that it was because Flow was scanning through all the JS files in node_modules.
I attempted to ignore node_modules in .flowconfig by adding:
[ignore]
.*node_modules/.*
The start up became fast again but Flow would complain Required module not found in places where I import external libraries from my code.
A workaround suggested in this Github issue was to flowignore the node_modules and manually add interfaces for external libraries. This seems to work but is a hassle to maintain when new libraries are added into the project.
It's frustrating to have to wait almost 2 minutes each time I start the project, are there any better ideas?
One way to avoid the Required module not found errors would be to run flow-typed install which will fetch existing libdefs for popular libraries from the flow-typed repository. It also generates stubs for the libraries that cannot be found in it. This works great for many projects but in some rare cases, the stubs for certain libraries like Immutable.js were not generated.
I eventually came up with cli command flow-scripts to automatically generate the libdef stub interfaces such that I could flowignore node_modules but not get the Required module not found errors.
Simply run
$ flow-scripts stub
in the root of the repository and the libdef stubs will be automatically generated in flow-typed directory.
I have several Scala modules that I'm building with SBT. Some of them (I'll call them dependant modules) are being published to Artifactory and then used by top-level modules.
All changes to the code are done in separate git branches. When the feature (or bugfix) is done, that branch is compiled in Jenkins and then deployed to the testing instance and handed off to the QA team.
So it's possible that there will be several git branches with different code in the dependant modules.
The problem is that Ivy is caching these modules locally, so it's possible that the top-level module will be built with the dependant module from a different branch (taken from the local cache).
I've tried adding changing() directive to the dependency specification in build.sbt.
In that case Ivy is ignoring the local cache and goes to Artifactory to download the POM file every time. Then it parses the POM file, but concludes that it has the jar file with that version in the local cache and fetches the jar file from the local cache and not from Artifactory. Which isn't what I want.
Since the code in branches hasn't been integrated into the master branch at this point, it's perfectly valid that different feature branches have the same version number, but different code.
Is there a way to tell Ivy (via SBT) to ignore local cache for a certain groupid? Or a single dependency at least?
If you are using versioning for your dependant modules then each codebase change must produce a different version. Ivy and maven expect that once the artifact has been published with a specific version it will stay unchanged forever. That is why they are using cached files. If you want to download a fresh version from repository on every compile you should add -SNAPSHOT suffix to the dependant module version number (eg: dep-module-1.1.1-SNAPSHOT)
I am using Spring Source Tool Suite 2.8.1 to implement Spring applications.
I frequently get build errors because references are lost for no apparent reason. In Right-click project in Package Explorer->Properties->Java Build Path->Order and Export, I find projects sometimes are deselected. And often packages are gone in Right-click project in Package Explorer->Properties->Java Build Path->Deployment Assembly.
Having to reset these settings frequently is frustrating. Is there some way I can work around these problems?
I have tried to update STS to the latest version, but the upgrade process fails with incomprehensible error messages. I want to avoid a clean install because setting up the environment again would probably be a nightmare.
Now that I know this is a maven project and you are adding references yourself, this is making sense to me. STS 2.8.x was the last STS to ship with the legacy m2e (maven plugin for Eclipse). It did not recognize build path entries added manually (it likes to have complete control over the classpath). So, what is likely happening is that you are adding these classpath entries and then an update project operation gets kicked off automatically. This will have the effect of removing all of your extra classpath entries.
You are best off doing the following:
Updgrading STS
Or just upgrading your m2e component (you will have to first uninstall the old m2e, but this should be taken care of automatically from the discovery update page).
Or, just accept the fact that you can't manually change your classpath with the legacy m2e.
If you have a project that builds one project before building the next, but the next needs to know the 'path' of the first build, is it possible to get this?
For example:
Project A has Build Configuration A and Build Configuration B.
Build Configuration B has a dependency on Build Configuration A. From without the Build Configuration B it will need access to the path of Build Configuration A. Is there are a way to obtain this?
Most simple approach would be to define a custom checkout directory in the A and use the same hard-coded value in B.
If you use TeamCity snapshot or artifact dependencies, you can use %dep.btXXX.teamcity.build.checkoutDir% to get checkout directory of the dependency build. However, this will not work in 6.5.0-6.5.5 TeamCity versions, see details and workaround in the issue TW-18715.
However, you should really avoid accessing checkout directory of one build from another. If you need sources of A, you can checkout them in B; if you output of the A's build, then publishing the output as build's artifacts and then using TeamCity artifact dependencies is the way to go. In both cases additionally using TeamCity snapshot dependencies will ensure both builds use the same sources snapshot which is probably what you need.
If you have one agent, and only ever one agent then you could try and use the path from a previous build.
I wouldn't recommend doing this however because if you had two agents, or scaled up in the future to two agents, then it is possible your projects will be built on different agents; this would mean your dependency working directory won't be on the same machine, or it will be outdated as the latest was built elsewhere.
I assume you're after the path of the first build to get its output?
If so, the method we use to share dependencies between projects is to checkin the output from each project into our source control, then every project that requires the output simply has to check them out.
I just started working with ant a few days ago. Right now I have a general buildall.xml which should call each project's build.xml. Because some projects depend on each other, I need to rebuild some other projects which depend on it. This isn't a problem--I'm just setting the depends property of the target. However, ant is always building the dependencies, even when the files haven't changed.
Let's say project1 has no dependencies; project2 depends on project1; project3 depends on project1, 2; project4 depends on project1, 2, and 3; and so on.
I could hack a solution which looks at project K, and checks if project 1 .. project K have updated files using uptodate. If so, then run the target. This is messy and appears unnecessary.
What is the cleanest way to implement this?
EDIT: So I decided to just hack in a bunch of targets, "check_projectK" where it does the uptodate checks on all of its source files, its build file, and the build files of the 1 .. K-1 projects. Due to dependencies, this is always handled correctly. However, this is still a large amount of copy and paste for a large workspace. I will leave this open.
Short answer, ANT can't do it, not unless you have some kind of way to connect to your version control system and check if anything has changed (you are using source control right?). Ant doesn't know about when what the last time a file changed and then see if it matches with what was built; it doesn't have the concept of a dependency repository. The whole purpose of Ant is that it just builds.
The solution to your problem isn't Ant, it's Maven. Maven HAS a dependency repository. There's also a very nifty plugin for Maven used specifically with Flex appropriately called FlexMojos. By using this, Maven can know when something was last built because it's uploaded to the repository. Then your other projects can add it's dependencies and download the SWC needed.
On top of that, it mixes great with a continuous integration engine like Hudson, Bamboo and Teamcity, which builds a project every time a file has been committed to your source control system, and then updates all dependent projects automatically!