So I already know the difference between white box and black box testing, but I am looking for fairly straightforward examples of situations where White box says "everything is OK" and Black box testing says "there is a problem here" - as well as vice versa
These examples would occur in rather popular software products. i.e - Microsoft Windows, Word, or an Equation solver (as shown below)
Something of similar degree to this: (Where black box test fails and white box passes)
Example: Solving of quadratic equations;
Input data: numeric values of coefficients A, B, and C;
Output data: numeric values of roots X1 and X2
BLACK BOX TEST
Black-Box testing (environment, users): NO (there is a MISTAKE in
internal module, it works incorrectly)
I’ve been asked to provide numeric values for coefficients A, B and C – I did provide requested values.
the internal SQRT module (solving of quadratic equations) did NOT deliver required numeric values of roots X1 and X2.
as result, I made a conclusion that there is a mistake inside an internal module.
WHITE BOX TEST
White-Box testing (an internal module): YES (I work CORRECTLY)
The input combination of values for A, B and C is illegal because the
value the Discriminant D is negative; therefore, I cannot calculate
roots X1 and X2.
This is a kind of the Exception handling in the code
Where the Developer can see the exception as (The input combination of values for A, B and C is illegal) but failed to capture and display exception to user
as a result Black box test fails, where the User is expects output whereas code throws exception and has to display it, so that Black box tester will correct input and gets required output
Related
Can somebody tell me why this author has used the following code in their normalisation.
The first line appears fine to me they have standardised the training set by the following formula;
(x - mean(x)) / std(x)
However the second line and third line (validation and test) they have used the train mean (trainme) and train standard deviation (trainstd). Should they not have used the validation mean (validationme) and validation standard deviation (validationstd) along with the test mean and test standard deviation?
You can also view the page from the book at the following link (page 173)
What the authors are doing is reasonable and it's what is conventionally done. The idea is that the same normalization is applied to all inputs. This is essentially allocating some new parameters (offset and scale) and estimating them from the training data. In that scheme, if the value 100 is input, then the normalized value is (100 - offset)/scale, no matter where (training, testing, whatever) that 100 came from.
I guess one can also make an argument that the offset and scale should be context dependent in the sense that if you are given a set of data and for some reason the offset and scale are very different from the original training data, maybe what's important is how big each value is relative to the others in the same data set. E.g. maybe you should treat 200 the same as 100, if the scale is twice as big in the data set containing 200.
Whether that data-dependent scaling is reasonable would have to be decided case by case. I don't remember ever having seen it, but it's plausible that it could be the right thing to do in some cases.
By the way, you'll get more interest in general statistical questions at stats.stackexchange.com and/or datascience.stackexchange.com.
When I was in high school, I figured out how to program my TI-84 Plus calculator to do quadratic equations for me. Like the goody-two-shoes I was, I deleted the program before the final exam. I am trying to recreate the program now, but it's not working well. Here's my code:
:Prompt A, B, C
:(-B+√(B²-4AC))/2A→Y
:(-B-√(B²-4AC))/2A→Z
:Disp Y
:Disp Z
(→ corresponds to the STO> (store) button on the calculator, which allows a user to set a value for a given letter variable.)
As far as I can tell, this should work. The math and the parentheses seem to be in order, the Prompt function works (after the program finishes, asking the calculator to print A, B, and C match the values stored from the last time the program was run).
When I ask it to calculate quadratic equations that I already know the answers to, it gives me funny numbers. Entering A=1, B=-3, C=2, which should return x-intercept values of 1 and 2, returns 2 and 0 instead. The x-intercepts of 0=3x²-10x+7 are 1 and 7/3, but the calculator returns 21 and 0. I can't reproduce it right now, but the this program has also returned some imaginary numbers where there shouldn't have been.
What's wrong with this code? The math works (inputting the second and third lines of code into the calculator to calculate, as opposed to lines of code in a program, after storing values in the variables does return the correct value), the Prompt and Disp functions work; what's wrong here?
Order of operations strikes again. The expression
(-B+√(B²-4AC))/2A
is being parsed as
((-B+√(B²-4AC))/2)*A
Add parentheses to /(2A) to fix this.
So I have built a CNN and now I am trying to get the training of my network to work effectively despite my lack of a formal education on the topic. I have decided to use stochastic gradient descent with a standard mean squared error cost function. As stated in the title, the problem seems to lie within the cost function.
When I use a couple of training examples, I calculate the mean squared error for each, and get the mean, and use that as the full error. There are two output neurons, one for face, and one for not a face; which ever is higher is the class that is yielded. Essentially, if a training example yields the wrong classification, I calculate the error (with the desired value being the value of the class that was yielded).
Example:
Input an image of a face--->>>
Face: 500
Not face: 1000
So in this case, the network says that the image isn't a face, when in fact it is. The error comes out to:
500 - 1000 = -500
-500^2 = 250000 <<--error
(correct me if i'm doing anything wrong)
As you can see the desired value is set to the value of the incorrect class that was selected.
Now this is all good (from what I can tell), but here is my issue:
As I perform b-prop on the network multiple times, the mean cost of the entire training set falls to 0, but this is only because all of the weights in the network are becoming 0, so all classes always become 0.
After training:
Input not face->
Face: 0
Not face: 0
--note that if the classes are the same, the first one is selected
(0-0)^0 = 0 <<--error
So the error is being minimized to 0 (which is good I guess), but obviously not the way we want.
So my question is this:
How do I minimize the space between the classes when the class is wrong, but also get it to overshoot the incorrect class so that the correct class is yielded.
//example
Had this: (for input of face)
Face: 100
Not face: 200
Got this:
Face: 0
Notface: 0
Want this: (or something similar)
Face: 300
Not face: 100
I hope this question wasn't too vague...
But any help would be much appreciated!!!
The way you're computing the error doesn't correspond to the standard 'mean squared error'. But, even if you were to fix it, it makes more sense to use a different type of outputs and error that are specifically designed for classification problems.
One option is to use a single output unit with a sigmoid activation function. This neuron would output the probability that the input image is a face. The probability that it's not a face is given by 1 minus this value. This approach will work for binary classification problems. Softmax outputs are another option. You'd have two output units: the first outputs the probability that the input image is a face, and the second outputs the probability that it's not a face. This approach will also work for multi-class problems, with one output unit for each class.
In either case, use the cross entropy loss (also called log loss). Here, you have a target value (face or no face), which is the true class of the input image. The error is the negative log probability that the network assigns to the target.
Most neural nets that perform classification work this way. You can find many good tutorials here, and read this online book.
I need to demonstrate the Bayesian spam filter in school.
To do this I want to write a small Java application with GUI (this isn't a problem).
I just want to make sure that I really grasped the concept of the filter, before starting to write my code. So i will describe what I am going to build and how I will program it and would be really grateful if you could give a "thumbs up" or "thumbs down".
Remember: It is for a short presentation, just to demonstrate. It does not have to be performant or something else ;)
I imagine the program having 2 textareas.
In the first I want to enter a text, for example
"The quick brown fox jumps over the lazy dog"
I then want to have two buttons under this field with "good" or "bad".
When I hit one of the buttons, the program counts the appearances of each word in each section.
So for example, when I enter following texts:
"hello you viagra" | bad
"hello how are you" | good
"hello drugs viagra" | bad
For words I do not know I assume a probability of 0.5
My "database" then looks like this:
<word>, <# times word appeared in bad message>
hello, 2
you, 1
viagra, 2
how, 0
are, 0
drugs, 1
In the second textarea I then want to enter a text to evaluate if it is "good" or "bad".
So for example:
"hello how is the viagra selling"
The algorithm then takes the whole text apart and looks up for every word it's probability to appear in a "bad" message.
This is now where I'm stuck:
If I calculate the probability of a word to appear in a bad message by # times it appeared in bad messages / # times it appeared in all messages, the above text would have 0 probability to be in any category, because:
how never appeared in a bad message, so probability is 0
viagra never appeared in a good message, so probability also 0
When I now multiply the single probabilities, this would give 0 in both cases.
Could you please explain, how I calculate the probability for a single word to be "good" or "bad"?
Best regards and many thanks in advance
me
For unseen words you would like to do Laplace smoothing. What does that mean: having a zero for some word count is counterintuitive since it implies that probability of this word is 0, which is false for any word you can imagine :-) Thus you want to add a little, but positive probability to every word.
Also, consider using logarithms. Long messages will have many words with probability < 1. When you multiply lots of small floating numbers on a computer you can easily run into numerical issues. In order to overcome it, you may note that:
log (p1 * ... * pn) = log p1 + ... + log pn
So we traded n multiplications of small numbers for n additions of relatively big (and negative) ones. Then you can exponentiate result to obtain a probability estimate.
UPD: Actually, it's an interesting subtopic for your demo. It shows a drawback of NB of outputting zero probabilities and a way one can fix it. And it's not an ad-hoc patch, but a result of applying Bayesian approach (it's equivalent to adding a prior)
UPD 2: Didn't notice it first time, but it looks like you got Naive Bayes concept wrong. Especially, bayesian part of it.
Essentially, NB consists of 2 components:
We use Bayes' rule for a posterior distribution over class labels. This gives us p(class|X) = p(X|class) p(class) / p(X) where p(X) is the same for all classes, so it doesn't have any influence on the order of probabilities. Or, another way to say the same, is to say that p(class|X) is proportional to p(X|class) p(class) (up to a constant). As you may guessed already, that's where Bayes comes from.
The formula above does not have any model assumptions, it's a probability theory law. However, it's too hard to apply it directly since p(X|class) denotes probability of encountering message X in a class. No way we would have enough data to estimate probability of every single message possible. So here goes our model assumption: we say that words of a message are independent (which is, obviously, wrong and incorrect, thus method is Naive). This leads us to p(X|class) = p(x1|class) * ... * p(xn|class) where n is amount of words in X.
Now we need somehow estimate probabilities p(x|class). x here is not a whole message, but just a (one) word. Intuitively, probability of getting some word from a given a class is equal to the number of occurrences of that word in that class divided by the total size of the class: #(word, class) / #(class) (or, we could use Bayes' rule once again: p(x|class) = p(x, class) / p(class)).
Accordingly, since p(x|class) is a distribution over xs, we need it to sum to 1. Thus, if we apply Laplace smoothing by saying p(x|class) = (#(x, class) + a) / Z where Z is a normalizing constant, we need to enforce the following constraint: sum_x p(x|class) = 1, or, equivalently, sum_x(#(x, class) + a) = Z. It gives us Z = #(class) + a * N where N is number of all words (just number of words, not their occurrences!)
I am new to WinBUGS/OpenBUGS and having difficulty debugging my code.
Does anyone know of a list of potential error messages for BUGS models and their meanings in plain English?
The WinBUGS manual has a list of some common error. I have added some additional notes from my own experience:
expected variable name indicates an inappropriate variable name. I occasionally get this error in providing the data, might have used 1.02e04 instead of 1.02E04.
undefined variable - variables in a data file must be defined in a model (just put them in as constants or with vague priors). If a logical node is reported undefined, the problem may be with a node on the 'right hand side'. I occasionally get this error when I have removed a variable from the model but not from the data or missed a comma in the data.
invalid or unexpected token scanned - check that the value field of a logical node in a Doodle has been completed.
index out of range - usually indicates that a loop-index goes beyond the size of a vector (or matrix dimension); sometimes, however, appears if the # has been omitted from the beginning of a comment line
linear predictor in probit regression too large indicates numerical overflow. See possible solutions below for Trap 'undefined real result'.
logical expression too complex - a logical node is defined in terms of too many parameters/constants or too many operators: try introducing further logical nodes to represent parts of the overall calculation; for example, a1 + a2 + a3 + b1 + b2 + b3 could be written as A + B where A and B are the simpler logical expressions a1 + a2 + a3 and b1 + b2 + b3, respectively. Note that linear predictors with many terms should be formulated by 'vectorizing' parameters and covariates and by then using the inprod(.,.) function
unable to choose update method indicates that a restriction in the program has been violated
You might also hit a trap at the start or during the MCMC. The BUGS manual list the following common traps (I always get the first two, never met the last two):
undefined real result indicates numerical overflow. Possible reasons include:
initial values generated from a 'vague' prior distribution may be numerically extreme - specify appropriate initial values;
numerically impossible values such as log of a non-positive number - check, for example, that no zero expectations have been given when Poisson modelling;
numerical difficulties in sampling. Possible solutions include:
better initial values;
more informative priors - uniform priors might still be used but with their range restricted to plausible values;
better parameterisation to improve orthogonality;
standardisation of covariates to have mean 0 and standard deviation 1.
can happen if all initial values are equal.Probit models are particularly susceptible to this problem, i.e. generating undefined real results. If a probit is a stochastic node, it may help to put reasonable bounds on its distribution, e.g.
probit(p[i]) <- delta[i]
delta[i] ~ dnorm(mu[i], tau)I(-5, 5)
This trap can sometimes be escaped from by simply clicking on the update button. The equivalent construction
p[i] <- phi(delta[i])
may be more forgiving.
index array out of range
possible reasons include:
attempting to assign values beyond the declared length of an array;
if a logical expression is too long to evaluate break it down into smaller components.
stack overflow can occur if there is a recursive definition of a logical node.
NIL dereference (read) can occur at compilation in some circumstances when an inappropriate transformation is made, for example an array into a scalar.
Trap messages referring to DFreeARS indicate numerical problems with the derivative-free adaptive rejection algorithm used for log-concave distributions. One possibility is to change to "Slice" sampling
This WinBUGS User Manual might be of some use.