Details:
I had already read several articles regarding a security-specific regression test, which could not be more different. Some advocate a strict mixture of CI pipelines and manual pentest procedures, others swear by no CI automated test procedures and manual test procedures.
I myself would prefer a clear mixture of several procedures for the security regression test. Here just about a connection of Owasp Zap + Docker and Jenkins as a pipeline integration.
The questions:
What are your experiences regarding security regression?
How do you handle the corresponding workflow in the Agile area and in the definition of Ready and Done?
Do you do more manual security regression or do you swear by corresponding CI tooling and automation?
Thank you for your interest!
We actually have a Google Summer of Code project for retesting vulnerabilities found by ZAP scans.
The blog https://pranavsaxena17.github.io/GSoC-with-ZAP/ is a bit light but hopefully the student will update it soon. In any case the project is progressing well.
Related
I'm trying to use the google-analytics framework to create predictive analysis tools. For example I would like to cluster my webpage visitors, etc.
In general, is there any list of machine learning algorithms implemented by this framework? for example: regression, clustering, classification, feature selection, etc.
Thank you for any help
Depending upon your language of choice, you might want to export your Google Analytics Metrics to flat files or a database and then start experimenting with ML models. Two popular languages with stable ML Implementations are Python and R. R's caret package includes tools for building a predictive model pipeline. Python's scikit-learn also contains implementations of all major classes of ML algorithms.
When you say GA framework I'll assume you're referring to the set of Google Analytics APIs listed here. The framework by itself doesn't provide machine learning capabilities. It merely provides access to the processed and aggregated GA data stored in Google's servers. You can use the API and feed the data to a machine learning application/system/program that does all of the stuff you mentioned.
Currently there has been movement to start continuous integration of Plone addons on Travis CI service. I have seen a lot of packages doing their own this and that testing, from running unit tests to do source code linting of merge requests. I see this healthy development as it takes the pain of setting up your own test runner and makes it easier for less Python-sawy community members to have tests run for their patches.
However, I have not found a definite guide of best practices how one should test his/her addon on Travis CI. At least in my case this has resulted to cargo cult programming and copy-pasting code from other random addons, because I don't know what's available and how one should do it.
Could authors, who have insight how Travis CI and Plone testing works clarify
What kind of tests you should run on Travis CI
How you should run these tests, in particulary in the context of collective package
What are the best examples you should look into
Where these practices are or should be documented*
How one should test against multiple Plone versions, or is it possbile
Where to pipe the result? Personal email?
*) Traditional Finnish carrot provided for those who open their text editor and spend five minutes to dump the contents of their brain
What kind of tests you should run on Travis CI
All collective or other packages that are not in the Plone core and not already tested on jenkins.plone.org.
How you should run these tests, in particulary in the context of collective package
http://datakurre.pandala.org/2012/09/speed-up-your-plone-add-on-tests-on.html
See also Nejc's talk:
http://de.slideshare.net/zupo/travis-ci-fun-and-easy-ci-for-your-plone-packages
What are the best examples you should look into
These packages all use the approach described in Asko's blogpost:
https://github.com/collective/collective.mailchimp
https://github.com/plone/plone.act
https://github.com/plone/plone.app.collection
Where these practices are or should be documented*
The testing team is currently working on a testing documentation that will include travis ci best practices.
How one should test against multiple Plone versions, or is it possbile
This should be possible. I'm not sure anybody tried that yet though. Maybe somebody else is able to answer that question.
Where to pipe the result? Personal email?
I guess that depends on the package. I currently use my personal email address for collective packages I'm working on.
How one should test against multiple Plone versions, or is it possbile
plone.api tests against different Python (2.6 & 2.7) and Plone (4.2 & 4.3) versions: https://github.com/plone/plone.api/blob/master/.travis.yml
Where to pipe the result? Personal email?
If you leave the email notification empty, it will email the person who triggered the build (aka. the last commiter). I find this a sane default.
Apart from that I always post a message to IRC channel: #niteoweb for packages we maintain, #client for client work we do, #sprint for plone.api (this one needs to change, Plone needs a CI channel where we can post such notifications).
We are working in a small team. We often had problems like developer1 did some changes in stored procedure or function and it affected work of developer2. Such issues are traced out by chance later. Please guide me how such issues can be stopped. Is there a free tool that we can run to test such issues?
Slowly introduce unit tests, focused integration tests and full system tests.
For all of those use a .net unit test framework to do it. It'll be what you do in the test what makes it be any of the above scenarios. Make sure to keep each of those 3 type of tests separately, as those will have a big difference on the speed it takes to execute them.
For the unit test framework I suggest NUnit but there are others, one that I've found interesting but never made the jump is xUnit.net.
For full system tests I suggest to run them in the unit test framework using WatiN. You could also go with Selenium RC.
We often had problems like developer1 did some changes in stored procedure or function and it affected work of developer2. Such issues are traced out by chance later.
For that specific type of scenario I strongly suggest focused integration tests. Full system tests might catch such scenario, but it will still left you to figure out why it broke.
Instead focus the test in the very specific db access code that makes the call to the procedure. By adding scenarios in there that reveal all the expectations developer2 had from said procedure when (s)he wrote the related .net code, regression issues with that integration code can be revealed very quickly and be dealt with very effectively. Also note that developer1 can easily run the focused integration tests that involve that procedure or area of the database many times / which is a lot more likely to happen than doing the same with full system tests.
You can do either automated unit testing using tools such as NUnit or automated black-box testing using tools such as Selenium. Note that both options (even with free tools) may need significant investment in terms of time and efforts. Typically, unit test cases are created by developers them selves while for automated black box testing, a separate team of QA is utilized - this is mostly because unit test cases are generally written in languages such as C#, VB.NET while automated black-box testing tools typically utilize scripting languages.
I thought I understood that a smoke server was used for continous integration to measure the performance and test coverage of a project. Is this basically correct? What does it mean to smoke-test a program? Does it simply mean to continously apply the tests defined on the smoke server to be able to pinpoint performance degradation to a particular time frame?
This other question does describe the difference between sanity testing and smoke testing but does not ask to define smoke testing. This question is many years older than the reference question.
Smoke testing is a set of basic cheap to run tests that precede actual testing. It aims to verify that the build is deployed successfully and that all test env. aspects are running and ready for the actual test process. It saves you bringing the full extent of your testing wrath down a faulty build and just realizing that you have been testing on a bad env. or erroneously deployed build possibly too late.
Turn it on, see if any smoke comes out.
The term originates in hardware repair and has been applied to software. It's intended to be a quick test to see if the application "catches on fire" when run for the first time. As stated above it's just to make sure you don't waste a bunch of folks time by setting them loose on something that's obviously broken.
Smoke testing isn't performance testing or regression prevention testing.
Smoke testing is a set of your highest priority fully automated tests. Even if your regular build succeeds, it's possible for your product to be broken in very fundamental ways that make it 100% unusable. Smoke tests are designed to test that level of core functionality. A build that doesn't pass smoke tests is very likely to fail a majority of all of your automated tests.
If a smoke test fails that generally means that you're losing productivity because the build is not usable, can't be tested, etc. Smoke test failures tend to require immediate fixes, on the order of hours not days.
The important aspect about smoke tests is that they can be run quickly. You should ALWAYS be running smoke tests of some sort. Many groups have sufficient resources to run a larger suite of tests on their daily or continuous integration builds (which is goodness), but smoke tests should be considered the bare minimum.
The wikipedia page on smoke testing is actually quite good.
It includes a pointer to Steve McConnel's ("Code Complete") paper in the best practices section of IEEE Software 13(4), 1996 titled "Daily Build and Smoke Test".
I'd say smoke testing is a precondition to unit and other forms of testing: if the smoke test fails there's no point in even starting up a unit test.
We use the term smoke test to refer to a testing that does quick basic tests to an application or product to make it qualify for further testing like integration testing or functionality testing. It is of little use to do integration testing or other adavanced testing if the application fails to do some of the elementary things required for these advanced testing. For instance we have a web based application and we do smoke test to ensure we can load all the web pages, lets users to login/logout etc. These are the basic things that should work with the application to make it testable. If the pages do not even load or if the users could not even login, we cannot do much with functionality or integration testing. We do smoke testing with every build followed by functionality and integration testing.
Smoke testing is the first step of integration testing. As a developer, you've put all the parts together and you're making sure your app will run before giving it to your QA team for "real" testing.
I wanted to write more, but MahdeTo beat me to a correct answer... :)
*checking the basic and critical feature of an application before proceeding the testing is known as smoke testing.
*In smoke testing we check the positive flow of basic and critical feature to check if the build is testable or not.
I agree with some of the previous answers. To me, Smoke Testing is simply a check if all my X thousands of sub-pages and redirects are still working.
Working means that the status code of each sub-page is "200 OK".
I run Smoke Tests as late as possible. After I deployed my code to the server and before I take this server active.
Today there is even an online service (SaaS) doing the job for me. https://www.smest.it helps me with Smoke Testing. I just entered once my private domain on their homepage and ever since I get my website checked daily. No stress at all.
Smoke testing can also be called build verification testing or build acceptance testing, if smoke testing fails it means build is not usable and can not be accepted , this defect needs to be fixed immediately on high priority
From Wikipedia
a smoke test may address basic questions like "does the program run?",
"does the user interface open?", or "does clicking the main button do
anything?" The process of smoke testing aims to determine whether the
application is so badly broken as to make further immediate testing
unnecessary.
What constitutes a good CI build-process?
We use CI, but is deployment to production even a realistic CI goal when you have dependencies on several services that should be deployed too and other apps may depend on these too.
Is a good good CI build process good enough when its automated to QA and manual from there?
Well "it depends" :)
We use our CI system to:
build & unit test
deploy to single box, run intergration tests and code analisys
deploy to lab environment
run acceptance tests in prod-like system
drop builds that pass to code drop for prod deployment
This is for a greenfield project of about a dozen services and databases deployed to 20+ servers, that also had dependencies on half a dozen other 'external' services.
Using a CI tool to deploy your product to a production environment as a realistic goal? again... "it depends"
Why would you want to do this?
if you have the process you can roll changes (and roll back) faster and more often
less chance for human error
you can test the same deployment strategy in a test environment before going to production and catch issues earlier
Some technical things you have to address before you can answer this:
what is the uptime requirements for your system -- Are you allowed to have downtime or does it need to be up 24/7?
do you have change control processes in place that require human intervention/approval?
is your deployment robust enough for any component to roll back to a known-good state if a deployment fails?
is your system designed to handle different versions of services or clients in case one or several component deployments fails (and you have the above rollback to last known good)?
does the process have the smarts to handle a partial deployment where a component cannot handle mixed versions of its dependencies/clients?
how are you handing database deployment/upgrades?
do you have monitoring in place so you know when something goes wrong?
Here are a couple of recent related links about automation and building the tools you need.
When it comes down to it the more complex your system the more difficult it is do automate everything, but that does not mean it is not a worthy goal, it just takes a lot more effort and willpower to get it done -- everything from knowing the difficulties you're going to face, the problems you have to account for (failure will happen), the political challenges of building infrastructure (vs. more product features).
Now heres the big secret... the technical challenges are challenging but not impossible... the political challenges may be insurmountable. Everything about this costs money whether its dev time or buying 3rd party solutions. So really, can you build the $1K, $10K, $100K, or $1M solution?
Whatever solution you go for make sure the automation is robust first, complete second... i.e. make sure you have as robust a solution as you can for getting deployment to a test environment rather than a fragile solution that deploys to production.
CI is not intended as a deployment mechanism. It is good to have your CI execute any automated deployment to a QA/Test server, to ensure those aspects of your build work, but I would not use a CI system like Cruise Control or Bamboo as the means of deployment.
CI is for building the codebase periodically to automate execution of automated tests, verification of the codebase via static analysis and other checks of that nature.
Be sure you understand the idea behind a CI build. CI stands for Continuous Integration and CI builds are really intended to be throw-away builds that are performed when a developer checks code in to the source control system (or at some specified interval) to ensure that the newest changes do not break the code base (hence the idea of continuously integrating the changes to the code base).
To that end, the technology used for the actual build server process is largely irrelevant compared to what actually happens during the build. As #pdavis mentioned, the CI build should compile the code base, execute some code analysis (FxCop, StyleCop, Lint, etc.), execute unit tests and code coverage, and execute any other custom analysis you want performed that should impact the concept of a "successful" or "failed" build.
Having a CI build automatically deploy to an environment really doesn't fall under the control of a build server. That being said, you can always create a separate project that runs on the build server that handles the deployment when it detects certain conditions (such as a build completes successfuly), but that should always be done as a completely independent thing.
I am starting on a new project at work that I am really looking forward to. We are still in the initial design stage and have just recently completed the Logical System Architecture. We have ordered new servers for the testing and staging environments and are setting up a Continuous Integration (CI) build system based on Cruise Control (http://cruisecontrol.sourceforge.net/) and MSBuild (http://msdn2.microsoft.com/en-us/library/wea2sca5.aspx) which is basically an improved port of ANT. It appears that Visual Studio 2005 project and solution files are all now in MSBuild format. Cruise Control will be automatically pulling the source from Visual Source Safe (ok, it isn't Subversion but we can deal), compiling it, and then running it through fxCop (http://www.gotdotnet.com/Team/FxCop/), nUnit (http://www.nunit.org/), nCover (http://ncover.org/site/), and last but not lease Simian (http://www.redhillconsulting.com.au/products/simian/). Cruise Control has a pretty good website interface for displaying all of the compiled results from the various tools and can even display code changes from one build to the next. It also keeps track of all builds in a build history. I'm looking forward to the test driven development and think that this type of approach combined with nUnit/nCover should give us a pretty good idea before we roll out changes that we haven't broken anything. There are also plans to incorporate some type of automated user interface testing once we are far enough along in the project. Depending on the tool, this should be just a matter of installing the tool on the build server and calling it from Cruise Control. Sweet.
A good CI process will have full or nearly-full unit test coverage. Unit tests test classes and methods, vs. integration tests, which test multiple parts of the system. When you set up your CI builds, have them automate the unit tests. That way, the CI builds can run multiple times per day. We have ours set to run every 2 hours.
You can have longer running builds that run once per day. These can use other services and run integration tests.
I was watching a ThoughtWorks presentation (creators of Cruise Control) and they actually addressed this issue. Their answer is that NO deployment is too complex to test. Why? Because otherwise, your customers become your testers, which is exactly where you don't want to be.
If you have a complex deployment structure, set up a visualization server. Have it pretend to be all the systems you need to talk to. They can always start in a known good state, because you can reset to a clean image.
To answer your initial question, a good process is one which enables communication between the repository and the developers. If the repository is in a bad state (non-compiling code, failed tests, etc.), the developers know about it as soon as possible, so that they can correct it.
The later a bug is discovered, the costlier it is to fix. So bugs should be discovered as early as possible. This is the motivation behind CI.
A good CI should ensure catching as many bugs as possible. The whole application comprises of code (often in multiple languages), Database schema, deployment files etc. Errors in any of these can cause bugs - so the CI should try to exercise as many of them as possible.
CI does not replace a proper QA discipline. Also, CI need not be very comprehensive on day one of the project. One can start with a simple CI process that does basic compilation & unit testing initially. As you discover more classes of bugs in QA, you should adapt the CI process to try to catch future occurrences of those bugs. It can also involve static code-analysis checks, so that you can implement consistent coding and design ideals across the codebase.