wanted to get the opinion of the people for this one. Currently I am working on setting retention policies inside Artifacts in DevOps, I want to get some clarification on how exactly that works. For example: Lets say I set my Max number of version per package to 10 and my Days to keep recently downloaded packages 14. I currently have these versions for my builds:
3.0.1.1.4
3.0.2.1.2 - 3.0.2.1.8
3.1.2.0.1 - 3.1.2.0.4
Is it possible with azure retention policies to keep versions 3.0.1, even if its not the latest version of my build? I assume it will just save the 10 latest versions of the build which would be 3.0.2.1.2 - 3.1.2.0.4. Let me know if I am assuming correctly. Assume no packages have been downloaded for this scenario.
Another alternative I was thinking about using is maybe Artifactory, who to me seems fairly similar? Not really too sure about it. I want to know which one is best for retention policies overall to reduce cost. Any information help, I know its not the clear thing in the world so if you have any questions just let me know.
Related
We have 2 solutions.
SolutionA is an internal solution where we put reusable code through our products
For the sake of the question, it has only two projects NugetProjectA and NugetProjectB which has a project reference to NugetProjectA.
SolutionB its a solution that has package references towards SolutionA via nuget.
The thing that troubles is:
add new method added in NugetProjectA
add new method in NugetProjectB project that uses previous method
publish new version of NugetProjectB
update nuget reference on Project of SolutionA
execute in Project newly added method of NugetProjectB
Since we didn't publish the NugetProjectA updated version, last step described will fail.
This seems to be a easy problem to solution. But imagine this with many more projects in SolutionB and many more in SolutionA.
Your question is vague and open-ended, so there isn't a simple, concise answer. Honestly it could easily be a multi-post blog series or even a short book. I'll try to give some general suggestions, but I'm not going to go into a lot of details to avoid making this answer too long.
Mono-repo vs multiple repos
Just like the microservice craze from a few years ago, you should first ask yourself if you need it. Having all your source code in a single repository and solution might make it feel "legacy", and it sure seems nicer to have a 3 minute build of a component rather than a 30 minute build of a whole system when checking in a 1-line bug fix. But are the problems you listed in your question worth the benefit of a shorter CI build?
On the other extreme, Google may famously run almost everything out of a single repository, but they have teams of people doing nothing more than managing their mono-repo and build system because a large number of developers working on a single repo have a different set of problems. These are engineers not working on customer applications. If their work make other teams more productive, then it can be worthwhile.
You need to figure out what's best for you, but given the problem you described in the question, maybe you have too many repos/solutions and your processes could be faster with fewer mistakes if you consolidate a little.
Automation
The next best thing is to just use good engineering practises. Automate as much as possible to reduce the risk of human mistakes, including automating processes that validate that manual processes are followed correctly.
Have the a CI or CD pipeline push packages, so you can't forgot to push dependencies, like in your example.
There are tools that will automatically generate a version number from the git history based on the number of commits since the versioning config file was checked in. This prevents you from forgetting to increment the package version when you check in a change to the package.
Have tests to make sure the package works before pushing it. You could make it as simple as making sure that dependencies exist, but you could also go further and have test programs run that use the package to ensure that backwards compatibility isn't broken and that new features actually work as expected. Have these tests run before pushing the package, so if the tests fail, you don't push a bad package.
You can have a pipeline that will automatically create pull requests on solutions that use a package when a new version of one of your packages is published. You can even automatically merge it, but you'll want to make sure you have real good tests to avoid a buggy package cascading into a huge mess, particularly if you do automatic deployments on successful builds.
Be creative and think about other ways you can automate your processes to make things easier for yourself and your team.
But be pragmatic about what you automate. There's no point spending a week full time to automate something that only takes you 5 minutes once a month to do manually. But if the manual process sometimes goes wrong and causes hours or days of effort to fix, then that makes automating the process more worthwhile.
Use modern features
The .NET ecosystem has changed a lot in the last 2-3 years since .NET Core was announced. Now packing with MSBuild (either dotnet pack or msbuild -t:pack) is easier to create packages than creating .nuspec files and making sure you do the right things to get project dependencies packed as nuget dependencies, getting all files in the right places, etc. If your class library uses SDK style projects, then there's nothing extra to do. If your project is a traditional project, you'll need to use PackageReference for your NuGet dependencies (or specify <ProjectStyle>PackageReference</ProjectStyle> as an MSBuild property in your project file), and then reference the NuGet.Build.Tasks.Pack package.
Version the application, not each package
Like the mono-repo vs multiple repos point, considering versioning all packages in the application with a single version number, rather than versioning each package individually. Yes, this means you'll sometimes (or maybe often) publish a new version of a package that doesn't have any code changes to the previous version, but it simplifies the release considerably. Coupled with packing with MSBuild in the section above, you can create a Directory.Build.props file in your repository root, and set the <Version> property to your app version, and all projects in the repo will have the same version. So, when you're ready to release, bump the version in a single file and every project and every NuGet packages will have the same version.
Summary
In an ideal world each component would be reusable in different applications, in a separate source code repository, each package individually versioned using semantic versioning. But in the real world this adds a lot of development time complexity. Your customers may be happier to get bug fixes and new features more quickly, even if the version number of packages are less meaningful. So, make data driven decisions. If you're frequently having dependency version problems, reduce your dependencies so there are fewer things that can go wrong.
Don't get me wrong, there are many good reasons to have multiple projects, multiple solutions, multiple repositories. Just make sure that the reason you're doing them is because it helps your team/company be more productive, not for idealistic reasons that are slowing you down or causing bugs.
I am working as a Data Scientist for a small start up and we are using R as part of our platform for analysis, dashboards etc. Therefore, I need to ensure that we maintain security with each package we use and load.
I have looked around and done extensive searching and have come across the following links:
This is the official R Studio Blog Security update page.
This blog post shows how you can implement rJava to help with those packages that require it, though it does state that '...the integrity & safety of the R package ecosystem is still in the “trust me, everything’s 👍!!”'
This post gives some good advice for package security, but basically boils down to: if you get it from CRAN or another trusted source then it should be ok.
The CVE site lists vulnerabilities, though the last one was back in 2017.
However, all the above links essentially say the same thing, which is "if its from CRAN (or similar), then it is probably fine". Now this might indeed be the case, but I was hoping for something a bit more rigorous. Has anyone else come across this issue with production R deployment?
If possible, if someone could direct to where I might be able to find out more information on checking for security updates, breaches and changes for R packages, or how to go about testing the security myself, I would be very grateful.
Thanks!
I've downloaded package TideSDK-1.2.1.RC1-0be9cd89-windows-7-x86-64.zip from tidesdk.org. Installer was found in tidesdk\sdk\win32\1.2.0.RC4\installer. This installer doesn't work: "Installation failed. The installer could not determine the application path."
What's wrong with it? Is it the only (right) way to install TideSDK? No docs on topic were found in resources and links on tidesdk.org.
It looks like you picked up an artefact from our Continuous Integration System from some point in time in development. We will eventually expose a Nightly Build site for developmental releases. That said, we will do this once the 1.3.0-beta is ready so we can properly support you with developmental artefacts.
Please use the legacy 1.2.0.RC4 in the interim that can be downloaded from front page of tidesdk.org until TideSDK 1.3.0-beta is available (which will be along very soon). This will serve your development needs in short term as we continue the work to get the beta prepared.
A new 'Getting Started Guide' will be up later today for the legacy 1.2.0.RC4 as there have been many requests for this help.
We appreciate how much attention our project is getting and have been working hard as a team to produce great documentation. Despite this, our efforts were primarily targeted on the API documentation at the outset. We experienced a surge of interest prior to getting the new guides in place. Our apologies to anyone new that has experienced any difficulty getting started. We appreciate your patience while we fill these gaps.
The new documentation is being prepared in anticipation of the 1.3.0 release so that we have great API docs, guides, and example apps when the time arrives. It is targeted for the end of September. We hope to also have our Tide Builder app available at that time to provide a nice app to help create, run and package your apps. There will also be an enhanced tidebuilder CLI since a tool with a UI will be strictly an option. For those that appreciate minimalism, this will get you going with no more than the SDK and a text editor.
You need to download Titanium Studio first.
Once that's done, you can install the package : Help menu > Install Sepcific Titanium SDK.
I am attempting to convince my colleagues to start using a build server and automated building for our Silverlight application. I have justified it on the grounds that we will catch integration errors more quickly, and will also always have a working dev copy of the system with the latest changes. But some still don't get it.
What are the most significant advantages of using a Build Server for your project?
There are more advantages than just finding compile errors earlier (which is significant):
Produce a full clean build for each check-in (or daily or however it's configured)
Produce consistent builds that are less likely to have just worked due to left-over artifacts from a previous build
Provide a history of which change actually broke a build
Provide a good mechanism for automating other related processes (like deploy to test computers)
Continuous integration reveals any problems in the big picture, as different teams/developers work in different parts of the code/application/system
Unit and integration tests ran with the each build go even deeper and expose problems that would maybe not be seen on the developer's workstation
Free coffees/candy/beer. When someone breaks the build, he/she makes it up for the other team members...
I think if you can convince your team members that there WILL be errors and integration problems that are not exposed during the development time, that should be enough.
And of course, you can tell them that the team will look ancient in the modern world if you don't run continuous builds :)
See Continuous Integration: Benefits of Continuous Integration :
On the whole I think the greatest and most wide ranging benefit of Continuous Integration is reduced risk. My mind still floats back to that early software project I mentioned in my first paragraph. There they were at the end (they hoped) of a long project, yet with no real idea of how long it would be before they were done.
...
As a result projects with Continuous Integration tend to have dramatically less bugs, both in production and in process. However I should stress that the degree of this benefit is directly tied to how good your test suite is. You should find that it's not too difficult to build a test suite that makes a noticeable difference. Usually, however, it takes a while before a team really gets to the low level of bugs that they have the potential to reach. Getting there means constantly working on and improving your tests.
If you have continuous integration, it removes one of the biggest barriers to frequent deployment. Frequent deployment is valuable because it allows your users to get new features more rapidly, to give more rapid feedback on those features, and generally become more collaborative in the development cycle. This helps break down the barriers between customers and development - barriers which I believe are the biggest barriers to successful software development.
From my personal experience, setting up a build server and implementing CI process, really changes the way the project is conducted. The act of producing a build becomes an uneventful everyday thing, because you literally do it every day. This allows you to catch things earlier and be more agile.
Also note that setting build server is only a part of the CI process, which includes setting up tests and ultimately automating the deployment (very useful).
Another side-effect benefit that often doen't get mentioned is that CI tools like CruiseControl.NET becomes the central issuer of all version numbers for all branches, including internal RCs. You could then enforce your team to always ship a build that came out of the CI tool, even if it's a custom version of the product.
Early warning of broken or incompatible code means that all conflicts are identified asap, thereby avoiding last minute chaos on the release date.
When your boss says "I need a copy of the latest code ASAP" you can get it to them in < 5 minutes.
You can make the build available to internal testers easily, and when they report a bug they can easily tell you "it was the April 01 nightly build" so that you can work with the same version of the source code.
You'll be sure that you have an automated way to build the code that doesn't rely on libraries / environment variables / scripts / etc. that are set up in developers' environments but hard to replicate by others who want to work with the code.
We have found the automatic VCS tagging of the exact code that produce a version very helpful in going back to a specific version to replicate an issue.
Integration is a blind spot
Integration often doesn't get any respect - "we just throw the binaries into an installer thingie". If ithis doesn't work, it's the installers fault.
Stable Build Environment
Prevents excuses such as "This error sometimes occurs when built on Joe's machine". Prevents using old dependent libraries accidentally when building on Mikes machine.
True dogfooding
You inhouse testers and users have a true customer experience. Your developers have a clear reference for reproducing errors.
My manager told us we needed to set them up for two major reasons. None were really to do with the final project but to make sure what is checked in or worked on is correct.
First to clean up DLL Hell. When someone builds on their local machine they can be pointing at any reference folder. Lots of projects were getting built with the wrong versions of dlls from someone not updating their local folder. In the build server it will always be built of the same source. All you have to do is get latest to get the latest references.
The second major thing for us was a way to support projects with little knowledge of them. Any developer can go grab the source and do a minor fix if required. They don't have to mess with hours of set up or finding references. We have an overseas team that works primarily on a project but if there is a rush fix we need to do during US hours we can grab latest and be able to build not have to worry about broken source or what didn't get checked in. Gated checkins save everyone else on your team time.
OS (usually security-based) patches and hotfixes that Microsoft releases to the community normally consist of, in my understanding, a series of updated DLLs or other binaries.
How does Microsoft, and other companies like it, ensure that that hotfixes don't clash with each other? Do they always go for a cumulative patch approach, where a single hotfix will includes all of the fixes in previous hotfixes? This doesn't seem to be the case, because many hotfixes seem to be focused on fixing specific problems. If they are focused hotfixes, how do they prevent one hotfix from trashing another one (e.g. incompatible DLLs being installed with each other).
I have always admired Microsoft's ability to manage this process. The company I work for is much smaller, and when I worked on the patch process a few years ago, we always went for the cumulative approach, where a single patch immediately superseded all previous patches based on that release. This meant that the patches got progressively larger in size, until the next "official" release came out.
What are some good practices for managing patch dependencies?
First off, Microsoft Windows Installer has the ability to patch binaries directly. Given known earlier states of a file, it can bring them to a known current state. We used to do this for our Large Commercial Product, but after a couple of releases, it was taking upwards of 24 hours for our four-way systems to produce a patch - which isn't good when you have (or want to have) nightly builds.
After a while, we opted for cumulative fixes where we merely allowed upgrades. We check that you're at a lower level, and then basically replace the entire product. (We also had the case whereby the second or third "delta" was basically everything anyway.)
On Unix/Linux, we can't use MSWI, obviously, so we provide another installer which basically does the same thing: move all the files out of the way, install as if brand new, and then delete the backup. The reality is, for us in our business, this is sufficient. We haven't gotten any complaints that I'm aware of (and those complaints would hit me pretty quickly based on my current job) with people unhappy enough to actually call in and complain. Mostly, they want to get the newer level with the patches so they can get on with their real business. Oddly enough, their business isn't installing patches.