What is the preferred (or just a good) pattern for a multi-part Codecept.js scenario, such as this:
Select file to upload.
Clear selection.
Select file to upload after having cleared selection.
I can do this in a single scenario and use I.say to delineate the parts, but it feels like I should be able to write these as independent scenarios and use .only on part 2, for example, and have part 1 run prior to part 2, because it depends on it.
I would also want to skip parts 2 and 3 if part 1 fails when running the whole suite.
I like thinking about behaviour in terms of capabilities. I can see that you have a couple here:
Uploading files
Correcting mistakes while uploading files.
So I would expect these to be in two scenarios:
One where you actually upload the file
One where you correct mistakes you make.
A lot of people say there should only be one "When" in scenarios, but that doesn't take into account interactions with people (including your past mistaken self) or time. In this situation, it's the whole process of correcting the file upload that provides the value. I can't see any value in the intermediate steps, so I'd leave them all in one scenario.
If there's any different behaviour associated with different contexts (eg: you've already got too many files uploaded) or outcomes (eg: your file system doesn't have room) or rules (eg: your status means you qualify for super-fast upload) then I would expect those to be new scenarios. If you start getting to the point where there are a lot of scenarios associated with file uploads and different things that happen to them, that might be a good time to separate this scenario out. Right now I can't see any reason to do that.
Re failing the first part: if you're doing BDD right, you'll be talking through not just the behaviour of your system, but the behaviour of individual bits of code too. That should help produce a good design which minimizes the chances of having bugs. Really good BDD teams produce scenarios that hardly ever catch bugs!
The scenarios act as living documentation, rather than regression tests; helping future devs understand the value of the code and get it right, rather than nailing it down to stop them getting it wrong.
So I wouldn't worry about it failing. If it's doing that a lot, you've got a different problem. Code it as if it's going to be passing most of the time, and make sure it's readable and comprehensible. As long as you can see when it fails and work it out, even if that takes a little bit of time, it'll be fine.
Having said that, I'd be surprised if Codecept doesn't have at least an option to stop on failure. Most BDD tools don't continue a scenario after a failed step; it would be an odd thing to do.
As far as I know, you're not be able to set priority for execution in codeceptjs. Better make one test. it is also will be more flexible if you will need to add or delete some part.
Related
I recently reached this limit. The error that the cli gives is very unobvious simply stating Request contains an invalid argument and it took me quite a while to realize I had reached the maximum limit of 64KB for the firestore.rules file. Would be great if that error was a bit more obvious as it would have saved me a bunch of time.
The limits are documented here.
https://firebase.google.com/docs/firestore/security/rules-structure#security_rule_limits
After doing a bunch of searching around solutions to the 64KB limit and not finding anything I contacted support. Their guidance was somewhat helpful but in other cases a bit shocking.
I'm adding these details here for anyone else that is struggling with this issue as not much else comes up when searching on Google.
Here was their response
The given limit on the ruleset size is fixed, and it cannot be increased. However, we do offer ways to reduce the size of your ruleset, or 'lines of code' in particular.
Here are some suggestions:
1) You can define your custom functions, which can be reusable throughout the ruleset
- This will definitely save you a lot of data space, and makes it look organized, as overused conditions can be called in one place
2) If possible, reconsider the database structure by making it efficient
- This means the less the number of collections and subcollections, the less rules are written, which makes the ruleset smaller in size
- Refactor your database structure and security rules as much as possible by removing unnecessary or redundant parts
3) Minimize the use of data validation rules, and put it on application-level instead
- Not only it can reduce lines of code, but it can reduce the number of expressions evaluated to avoid reaching the given limit of 1000 expressions per request
- As much as possible, use your app logic to ensure your data is on the right character length, correct data type, meets the patter criteria, etc. You may also use your app's advanced UI elements like a password textbox, a textbox that you can limit the character length, among others
Some of these suggestions such as using functions are definitely helpful. However, i'm a bit surprised by the suggestions to restructure my database and to put validation on the application level.
In my case i've already used functions quite a lot in my database rules and have removed most of the redundancy.
Asking me to reconsider the database structure and having to overhaul my application for the sake of reducing my rules file size is a huge ask for little gain.
Minimize the use of data validation rules seems completely against the design of the Firestore database. The database is designed in such a way that you can directly connect your client to your database which removes the middle application logic. Suggesting to do this seems to go directly against the architectural nature of Firebase. I would prefer to keep building my application in this way (and want to maintain my security), so this seems like a non-option.
Does anyone else have any suggestions on how to handle this issue?
Technically, this problem does not seem too dissimilar from trying to keep javascript file size down on the web. A lot of the same approaches taken by javascript minifiers could be used to reduce the size of the firebase rules file. It would be great if Firebase provided a tool like this for us.
Out of desperation I made a simple minifier that removes whitespace and comments. In case anyone else finds themselves in a similar situation and needs a quick fix. https://github.com/brianneisler/firemin
I've got a bit of an issue. I'm writing and continually developing an ASP.NET MVC application. The problem is, everytime I update one small part of our site (not even anything to do with our database), it seems to break about 4 other things in other parts of the site. I've been doing my best to anticipate it, but in the end, I know there are better ways out there to test, and I'm just wondering what the general consensus is for best practices?
Thanks!
In short - not organized answer!
Have a plan.
Know what the change is.
Document what you are going to test and why.
Maintain a test suite (Regression) to execute after all major changes.
Keep improving your test cases and adding the ones you missed
(there should be a test case associated with all your bugs.
This will ensure the same bug doesn't get introduced due to regression.
And like you said have a proper structure, ensure process implementation, have better code reviews (catch the bugs before they get submitted) and keep reading a lot about all this.
This answer is open for editing to make it better.
The other day i stumbled onto a rather old usenet post by Linus Torwalds. It is the infamous "You are full of bull****" post when he defends his choice of using plain C for Git over something more modern.
In particular this post made me think about the enormous amount of abstraction layers that accumulate one over the other where I work. Mine is a Windows .Net environment. I must say that I like C# and the .Net environment, it really makes most things easy.
Now, I come from a very different background made of Unix technologies like C and a plethora or scripting languages; to me, also, OOP is just one, and not always the best, programming paradigm.. I often struggle (in a working kind of way, of course!) with my colleagues (one in particular), because they appear to be of the "any problem can be solved with an additional level of abstraction" church, while I'm more of the "keeping it simple" school. I think that there is a very different mental approach to the problems that maybe comes from the exposure to different cultures.
As a very simple example, for the first project I did here I needed some configuration for an application. I made a 10 rows class to load and parse a txt file to be located in the program's root dir containing colon separated key / value pairs, one per row. It worked.
In the end, to standardize the approach to the configuration problem, we now have a library to be located on every machine running each configured program that calls a service that, at startup, loads up an xml that contains the references to other xmls, one per application, that contain the configurations themselves.
Now, it is extensible and made up of fancy reusable abstractions, providers and all, but I still think that, if we one day really happen to reuse part of it, with the time taken to make it up, we can make the needed code from start or copy / past the old code and modify it.
What are your thoughts about it? Can you point out some interesting reference dealing with the problem?
Thanks
Abstraction makes it easier to construct software and understand how it is put together, but it complicates fully understanding certain issues around performance and security, because the abstraction layers introduce certain kinds of complexity.
Torvalds' position is not absurd, but he is an extremist.
Simple answer: programming languages provide data structures and ways to combine them. Use these directly at first, do not abstract. If you find you have representation invariants to maintain that are at a high risk of being broken due to a large number of usage sites possibly outside your control, then consider abstraction.
To implement this, first provide functions and convert the call sites to use them without hiding the representation. Hide the data representation only when you're satisfied your functional representation is sufficient. Make sure at this time to document the invariant being protected.
An "extreme programming" version of this: do not abstract until you have test cases that break your program. If you think the invariant can be breached, write the case that breaks it first.
Here's a similar question: https://stackoverflow.com/questions/1992279/abstraction-in-todays-languages-excited-or-sad.
I agree with #Steve Emmerson - 'Coders at Work' would give you some excellent perspective on this issue.
During software development, there may be bugs in the codebase which are known issues. These bugs will cause the regression/unit tests to fail, if the tests have been written well.
There is constant debate in our teams about how failing tests should be managed:
Comment out failing test cases with a REVISIT or TODO comment.
Advantage: We will always know when a new defect has been introduced, and not one we are already aware of.
Disadvantage: May forget to REVISIT the commented-out test case, meaning that the defect could slip through the cracks.
Leave the test cases failing.
Advantage: Will not forget to fix the defects, as the script failures will constantly reminding you that a defect is present.
Disadvantage: Difficult to detect when a new defect is introduced, due to failure noise.
I'd like to explore what the best practices are in this regard. Personally, I think a tri-state solution is the best for determining whether a script is passing. For example when you run a script, you could see the following:
Percentage passed: 75%
Percentage failed (expected): 20%
Percentage failed (unexpected): 5%
You would basically mark any test cases which you expect to fail (due to some defect) with some metadata. This ensures you still see the failure result at the end of the test, but immediately know if there is a new failure which you weren't expecting. This appears to take the best parts of the 2 proposals above.
Does anyone have any best practices for managing this?
I would leave your test cases in. In my experience, commenting out code with something like
// TODO: fix test case
is akin to doing:
// HAHA: you'll never revisit me
In all seriousness, as you get closer to shipping, the desire to revisit TODO's in code tends to fade, especially with things like unit tests because you are concentrating on fixing other parts of the code.
Leave the tests in perhaps with your "tri-state" solution. Howeveer, I would strongly encourage fixing those cases ASAP. My problem with constant reminders is that after people see them, they tend to gloss over them and say "oh yeah, we get those errors all the time..."
Case in point -- in some of our code, we have introduced the idea of "skippable asserts" -- asserts which are there to let you know there is a problem, but allow our testers to move past them on into the rest of the code. We've come to find out that QA started saying things like "oh yeah, we get that assert all the time and we were told it was skippable" and bugs didn't get reported.
I guess what I'm suggesting is that there is another alternative, which is to fix the bugs that your test cases find immediately. There may be practical reasons not to do so, but getting in that habit now could be more beneficial in the long run.
Fix the bug right away.
If it's too complex to do right away, it's probably too large a unit for unit testing.
Lose the unit test, and put the defect in your bug database. That way it has visibility, can be prioritized, etc.
I generally work in Perl and Perl's Test::* modules allow you to insert TODO blocks:
TODO: {
local $TODO = "This has not been implemented yet."
# Tests expected to fail go here
}
In the detailed output of the test run, the message in $TODO is appended to the pass/fail report for each test in the TODO block, so as to explain why it was expected to fail. For the summary of test results, all TODO tests are treated as having succeeded, but, if any actually return a successful result, the summary will also count those up and report the number of tests which unexpectedly succeeded.
My recommendation, then, would be to find a testing tool which has similar capabilities. (Or just use Perl for your testing, even if the code being tested is in another language...)
We did the following: Put a hierarchy on the tests.
Example: You have to test 3 things.
Test the login (login, retrieve the user name, get the "last login date" or something familiar etc.)
Test the database retrieval (search for a given "schnitzelmitkartoffelsalat" - tag, search the latest tags)
Test web services (connect, get the version number, retrieve simple data, retrieve detailed data, change data)
Every testing point has subpoints, as stated in brackets. We split these hierarchical. Take the last example:
3. Connect to a web service
...
3.1. Get the version number
...
3.2. Data:
3.2.1. Get the version number
3.2.2. Retrieve simple data
3.2.3. Retrieve detailed data
3.2.4. Change data
If a point fails (while developing) give one exact error message. I.e. 3.2.2. failed. Then the testing unit will not execute the tests for 3.2.3. and 3.2.4. . This way you get one (exact) error message: "3.2.2 failed". Thus leaving the programmer to solve that problem (first) and not handle 3.2.3. and 3.2.4. because this would not work out.
That helped a lot to clarify the problem and to make clear what has to be done at first.
I tend to leave these in, with an Ignore attribute (this is using NUnit) - the test is mentioned in the test run output, so it's visible, hopefully meaning we won't forget it. Consider adding the issue/ticket ID in the "ignore" message. That way it will be resolved when the underlying problem is considered to be ripe - it'd be nice to fix failing tests right away, but sometimes small bugs have to wait until the time is right.
I've considered the Explicit attribute, which has the advantage of being able to be run without a recompile, but it doesn't take a "reason" argument, and in the version of NUnit we run, the test doesn't show up in the output as unrun.
I think you need a TODO watcher that produces the "TODO" comments from the code base. The TODO is your test metadata. It's one line in front of the known failure message and very easy to correlate.
TODO's are good. Use them. Actively management them by actually putting them into the backlog on a regular basis.
#5 on Joel's "12 Steps to Better Code" is fixing bugs before you write new code:
When you have a bug in your code that you see the first time you try to run it, you will be able to fix it in no time at all, because all the code is still fresh in your mind.
If you find a bug in some code that you wrote a few days ago, it will take you a while to hunt it down, but when you reread the code you wrote, you'll remember everything and you'll be able to fix the bug in a reasonable amount of time.
But if you find a bug in code that you wrote a few months ago, you'll probably have forgotten a lot of things about that code, and it's much harder to fix. By that time you may be fixing somebody else's code, and they may be in Aruba on vacation, in which case, fixing the bug is like science: you have to be slow, methodical, and meticulous, and you can't be sure how long it will take to discover the cure.
And if you find a bug in code that has already shipped, you're going to incur incredible expense getting it fixed.
But if you really want to ignore failing tests, use the [Ignore] attribute or its equivalent in whatever test framework you use. In MbUnit's HTML output, ignored tests are displayed in yellow, compared to the red of failing tests. This lets you easily notice a newly-failing test, but you won't lose track of the known-failing tests.
I try to externalize all strings (and other constants) used in any application I write, for many reasons that are probably second-nature to most stack-overflowers, but one thing I would like to have is the ability to automate spell checking of any user-visible strings. This poses a couple problems:
Not all strings are user-visible, and it's non-trivial to spearate them, and keep that separation in place (but it is possible)
Most, if not all, string externalization methods I've used involve significant text that will not pass a spell checker such as aspell/ispell (eg: theStrName="some string." and comments)
Many spellcheckers (once again, aspell/ispell) don't handle many words out of the box (generally technical terms, proper nouns, or just 'new' terminology, like metadata).
How do you incorporate something like this into your build procedures/test suites? It is not feasible to have someone manually spell check all the strings in an application each time they are changed -- and there is no chance that they will all be spelled correctly the first time.
We do it manually, if errors aren't picked up during testing then they're picked up by the QA team, or during localization by the translators, or during localization QA. Then we lodge a bug.
Most of our developers are not native English speakers, so it's not an uncommon problem for us. The number that slip through the cracks is so small that this is a satisfactory solution for us.
Nothing over a few hundred lines is ever 100% bug-free (well... maybe the odd piece of embedded code), just think of spelling mistakes as bugs and don't waste too much time on it.
As soon as your application matures, over 90% of strings won't change between releases and it would be a reasonably trivial exercise to compare two versions of your resources, figure out what'ts new (check them first), what's changed/updated (check next) and what hasn't changed (no need to check these)
So think of it more like I need to check ALL of these manually the first time, and I'm only going to have to check 10% of them next time. Now ask yourself if you still really need to automate spell checking.
I can think of two ways to approach this semi-automatically:
Have the compiler help you differentiate between strings used in the UI and strings used elsewhere. Overload different variants of the string datatype depending on it's purpose, and overload the output methods to only accept that type - that way you can create a fake UI that just outputs the UI strings, and do the spell checking on that.
If this is doable of course depends on the platform and the overall architecture of the application.
Another approach could be to simply update the spell checkers database with all the strings that appear in the code - comments, xpaths, table names, you name it - and regard them as perfectly cromulent. This will of course reduce the precision of the spell checking.
First thing, regarding string externalization - GNU GetText (if used properly) creates string files that are contain almost no text other then the actual content of the strings (there are some headers but its easy to cause a spell checker to ignore them).
Second thing, what I would do is to run the spell checker in a continuous integration environment and have the errors fed externally, probably through a web interface but email will also work. Developers can then review the errors and either fix them in the code or use some easy interface to let the spell check know that a misspelling should be ignored (a web interface can integrate both the error view and the spell checker interface).
If you're using java and are storing your localized strings in resource bundles then you could check the Bundle.properties files and validate the bundle strings. You could also add a special comment annotation that your processor could use to determine if an entry should be skipped.
This method will allow you to give a hint as to the locale and provide a way of checking multiple languages within the one build process.
I can't answer how you would perform the actual spell checking itself, though I think what I've presented will guid you as for the method of performing the spell checking.
Use aspell. It's a programme, it's available for unixoids and cygwin, it can be run over lots of kinds of source code. Use it.
First point, please don't put it into you build process. I would be a vengeful coder if I (meaning my computer) had to spell check all the content on the site every time I tried to debug or build a new feature. I don't even think this kind of operation belongs as a unit test (you're testing a human interface, not a computerised one).
Second point, don't write a script. You're going to have so many false positives fall through the cracks that people will stop reading the reports and you are no better off than when you started.
Third point, this is probably most easily solved by having humans do it: QA team, copy writers, beta testers, translators, etc. All the big sites with internationalised content that I've built had the same process: we took the copy from the copy writers, sent it to the translating service/agency, put it into the persistence layer, and deployed it. Testers (QA, developers, PMs, designers, etc.) would find spelling or grammatical mistakes and lodge bug reports. There is just too much red tape and pairs of eyes for that many spelling/grammar errors to slip through.
Fourth point, there will always be spelling and grammar mistakes on your page. Even major newspaper web sites haven't gotten around this and they have whole office buildings filled with editors.