An sbt build configuration can get quite complex. When it does, it's hard to tell exactly what sbt will do because you're not sure you did the right thing in the build definition. And if you have a large project, running full build and test cycles to test changes is really painful and slow.
For example, I'm configuring sbt to build and test my project in parallel. It would be great if I could ask sbt to report:
exactly how many JVMs will be forked and when
exactly what options each JVM will be assigned
exactly what tasks will be assigned to each JVM
exactly how tests will be grouped in the JVMs
Perhaps some of this information can only be determined by doing some work, perhaps even by performing the full build and test. But it would nonetheless be great to have some kind of neat report that clearly lays out what's going on.
Does sbt offer such a facility?
I don't think it could easily be achieved in sbt.
There's inspect command that could tell you quite a bit what you need, but it's just a small piece comparing to what you're after and lots is missing to have a complete plan.
> help inspect
inspect <key>
For a plain setting, the value bound to the key argument is displayed using its toString method.
Otherwise, the type of task ("Task" or "Input task") is displayed.
"Dependencies" shows the settings that this setting depends on.
"Reverse dependencies" shows the settings that depend on this setting.
When a key is resolved to a value, it may not actually be defined in the requested scope.
In this case, there is a defined search sequence.
"Delegates" shows the scopes that are searched for the key.
"Provided by" shows the scope that contained the value returned for the key.
"Related" shows all of the scopes in which the key is defined.
inspect tree <key>
Displays `key` and its dependencies in a tree structure.
For settings, the value bound to the setting is displayed and for tasks, the type of the task is shown.
inspect uses <key>
Displays the settings and tasks that directly depend on `key`.
inspect definitions <key>
Displays the scopes in which `key` is defined.
Read Inspect the build to learn about the command.
You may also want to use show with some settings like fork or javaOptions:
> help fork
If true, forks a new JVM when running. If false, runs in the same JVM as the build.
> help javaOptions
Options passed to a new JVM when forking.
See Forking in the official documentation.
I would like to have such a tool, too.
Related
Is it possible to have do_package() before do_compile() in a bitbake recipe?
In this case, binaries from the previous bitbake run would be deployed. Would bitbake have warning about this situation?
I don't know for sure if it is possible to change the order of some built-in tasks, but given your sentence "in this case, binaries from the previous bitbake run would be deployed", I think it is not the proper way to archive that.
Yocto/Bitbake supports incremental build and package dependencies so that if a local artifact is still valid it won't be rebuild.
In Yocto, you can also setup a shared folder (on a server) to store build artifacts so that even when building "from scratch" from a known state, will actually results in downloading the artifacts instead of building them.
You can look at https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#shared-state-cache to get more info on that.
Note: if you want/need to try it anyway, maybe adding a custom task using add task custompackage before do_compile would be a way to investigate it.
Otherwise, to use built-in names, I would try to deltask those, and recreate them with custom ordering, but once again, that looks risky to me.
The Artifactory REST API has a resource for build promotion, which takes a build name and build number. One of my projects has been posting three builds with identical build names and build numbers. When viewing the projects in the repo they are sufficiently distinct:
Myproject
Myproject-foo
Myproject-no-bar
I am looking to promote just one of these (Myproject-foo). I can get the build runs for the build name, but as the URIs are matching there is no way to examine the different builds further. Is there anything I can do to automate the promotion of a specific project from each of these build numbers, or is manually clicking through, examining each one and promoting the appropriate images the only way?
Although it's possible to end up with the same build name and build number, you should avoid this situation because it makes it harder to distinguish between runs in UI and REST API. From what I can see in that case the promotion REST API will promote the latest build run (not all of them).
If you still want to do that, you can check out this plugin https://github.com/jfrog/artifactory-user-plugins/tree/master/build/promoteWithDeps
(which has a wider scope but it accepts an additional parameter buildStartTime to distinguish between build runs)
You shouldn't be able to publish three builds with the same build.name and build.number. Artifactory will normally throw a 500 error if this is attempted. Check the artifactory.log to see if all three builds are actually making it into the server.
I'm adapting an existing program to use Autotools for its build, but the resulting process depends on an environment variable. Is there a way to permanently set this environment variable during the build or installation process?
The program is intended to be used by Unix users and I could try to concatenate an export command directly to the .bashrc file and warn the user in case it fails because most of them will actually just use Ubuntu to run it (it's a relatively simple program that targets students), but I'd like to know if there's a more portable way to do this.
That's what I wouldn't like to do:
export VAR=/my/totally/not/hardcoded/path >> $HOME/.bashrc
Sorry to come to this late, but all of the answers to date are shockingly ... incomplete.
Building and installing software are both core use cases for the Autotools, and the installation part can absolutely involve adding or modifying files that affect user environments. If the software is installed by a user with sufficient privilege, then such effects can absolutely be applied to all system users, though the details may vary a bit from system to system (and the Autotools can help with that, too!).
For example, on RedHat-family Linuxes such as RedHat Enterprise, Fedora, Oracle Linux, and various others, you can drop an appropriately named file in /etc/profile.d, and the commands in it will automatically be read and executed by every login shell. Setting environment variables for all users is one of the common uses of this feature. I'm uncertain about Debian-family Linuxes such as Ubuntu, but it is always possible to modify file /etc/profile instead to have the same effect, and you absolutely can write an Automake install hook to do that.
Or for an altogether different approach, you can always provide a wrapper script around your program that sets the needed environment variables (supposing that the point is other than to add a directory to the PATH so as to find the program in the first place). In that case, you can even install the main program in a location that is not ordinarily in the path, so that users don't accidentally run it directly. This mechanism has the advantage that the environment variables are scoped to a run of the program, not a whole login session, but the disadvantage that users cannot override them.
I guess, no.
Autotools are about building your program, not about environment setup for the program to run. That's what users/admins are supposed to do. (Well I can imagine doing this, but I really don't want to try to figure it out, because the idea itself seems broken to me)
If your program REALLY needs some environment variable during run-time, then you should patch your sources for your application to test if the variable exists, and set one to default desired value, if it doesn't. Another idea is to enforce usage of an obligatory command line switch to pass the value in.
It's not clear what this has to do with autotools (or any other build system). No build system, by itself, can arrange for an env var to be present when the program it builds is run at a later tiem.
One solution is for your program to have a hardcoded default value for the var which is used if the environment var isn't present when the program starts running. Another frequently used solution is to name your binary something like myprog.bin and install a shell script named myprog which sets up the environment before doing exec myprog.bin.
I'm adapting an existing program to use Autotools for its build, but the resulting process depends on an environment variable. Is there a way to permanently set this environment variable during the build or installation process?
You've not been very concrete about what the program is (e.g. is the program a daemon? A user program?) or the nature of the environment variable dependency (e.g. is it another program? A mount point? A URL? A DB connection string?). Being more specific might give a better answer for you.
Anyway, autotools is not likely to offer any feature to help: It's a build system. Depending on the nature of your environment variable dependency, you're likely going to need package management (if you package it) or system administration level setup.
Since you think your primary user base is on Ubuntu this help page might give you some ideas.
I'm trying to embed git describe-generated version info into AssemblyInfo.cs plus some label within ASP.NET website.
I already tried using git-vs-versionino but this assumes Git executable on PATH. However default install of msysgit on Windows does not set this up; it uses git bash. This caused problems.
Now I am looking for a way to utilize libgit2sharp library (for zero external dependencies) to use as build number generator. However this library has no describe command...
Thanks!
git-describe is a UI feature that nobody has implemented in the library or bindings yet (or at least nobody's contributed it), but you can do it yourself fairly easily.
You get a list of the tags and what commits they point to, do a walk down the commits and count how many steps it took to get to a commit that you have in the list you built. This already gives you the information you need. If the steps were zero, then your description would be the tag name only; otherwise you append the number of steps and the current commit's id to it.
There's a work in progress libgit2 pull request that proposes an implementation of git-describe functionalities.
See #1066 for more information.
It's not finished yet. Make sure to subscribe to it in order to be notified of its future progress.
Once it's done, it should be quite easy to bind it and make it available through LibGit2Sharp.
In the project I'm working for we're having a continuous deployment setup. The goal is to always install the latest working build to production, unless someone manually overrides this functionality.
In order to make this working we
Run static code analysis
Run unit tests
Run integration tests
Run automatic UI tests, to the extent this is feasible
If any of the above steps fail, the build process is halted, and the build marked as failed. If the installation package is created it is then in steps installed to
CI --> staging --> production
At each step we run a integration and UI tests for the environment, to make sure we didn't introduce some new things which fail on on the subsequent environments. If none of the tests fail, and N minutes pass without anyone pressing the panic button, the build gets promoted to the next env. If the tests fail, we want to delete the package, and discard it completely. The installation packages are, however, delivered to other servers, so we need to run a bunch of remote (shell) scripts to make this step happen.
The problem is, that there are a big set of failure cases which we cannot reliably test in the normal automated cycle, e.g. page layout, or some integrations fail only production and so on.
So the actual question: How shall I demote/delete builds, once they've been promoted? Is it possible to either run a remote script when doing delete build or use any of the promotion plugins to achieve this functionality? Is there some think-outside-the-box solution for this that I might not have thought about?
Instead of deleting builds manually, you may write a Jenkins job that accepts the build number as a parameter, deletes it, and then does the rest of the housekeeping. You can configure Jenkins access privileges so that people do not delete builds manually by accident.
This might be a very particular case, but we decided against creating a separate job for removing the builds, for the very simple reason of keeping all the logging related to a specific build number in one single place. The setup was the following:
Promotion here means make the installation package (RPM) available to the given server, where auto-update handles the actual upgrade of the package.
We have one main build, that builds every time a new commit is available. We had some fine-tuning related to quiet times etc. but basically every new pushed set of commits resulted in a new build. The build contains all the relevant and available testing, which is far from being complete, and probably never will be.
Every hour a separate promotion step handles promotion from staging to production. This build kicks off a separate promotion which takes the latest accepted build from CI to staging. There is a 30min delay before builds were promoted CI-->staging, to prevent accidental promotions for last second commits. Delays were achieved with some bash find scripting. The order of promotions is this, to make sure a build is available in staging for (at least) one hour before going to promotion.
The actual answer:
The promotion steps were done as separate builds. In order to do a real promotion, rather than a separate build with a separate log, the build kicks off an actual promotion in the main build, using curl and calling the remote HTTP API. This leaves a relevan promotions star in the main build log. Using different colors, the promotions are visible with one look.
To demote builds I decided to create a separate "demote build" promotion step. This would then issue a purple star as a sign of the build being defective, and thus removed. The demotion is done by accessing the correct build in the UI, and pressing the "Remove build" button. No automation has been added to this step, but by creating a separate test step, it would be fairly easy to automate the demotion as well. We, however, have not gotten quite this far yet.
The benefits of this approach include
A build is deleted by accessing the failed build, not by providing parameters. Makes it much easier to document, and get right under pressure
Having a "panic button" like this available for anyone to press, builds trust and ownership for the process not only amongst the developers but also managers and DevOps.
It's dead simple to spot dead builds, as the log is available besides the other promotion logs
Having all the relevant promotion calls in the same build makes further scripting easier
Acute things we still have to improve include automating the testing on the later stages of the build pipeline, and also a suitable way of downgrading builds after demotion. E.g. in production a defective build and a demotion must always lead to installing the last good build, which has turned out to be fairly hard to achieve. Production data centers are rarely allowed to be accessible to this level from the development DC where the CI system sits. Also stopping and starting the build pipeline must be automated, as else there is the chance of slipping back to the manual state.
Naturally, in the spirit of continuous improvement, there are always things to improve. The whole setup is something of a bash/perl scripting mess, but since it's scripted and repeatable, there is always the option of improving one small piece at a time. The most important thing is the automation, as it allows for incremental steps, which any manual steps more or less prevent.
For anyone looking for an easy way to delete a build with custom steps:
Create a 'defective' promotion.
Make it manually triggered.
Force it run on the master.
Add a choice parameter DELETE with choices NO and YES.
Add action Execute Shell.
_
if [ "${DELETE}" == "YES" ]; then
# TODO: my custom steps
curl -X POST ${PROMOTED_URL}/doDelete"
fi
To delete a build now, just go to promotions, flip the choice to YES and click approve.