Strange behavior of Luis.ai - microsoft-cognitive
In my Luis app i have an intent "goodbyeIntent" with examples "goodbye", "thanks" and "ciao". There is another intent containing lots of examples and, of course, the None-intent.
When i test the app i get the following predictions:
goodbye: 0.97 (goodbyeIntent)
ciao: 0.97 (goodbyeIntent)
thanks: 0.97 (goodbyeIntent)
Although i would have expected scores of 1.00 these results seems to be ok.
Now when i test the app with a word or sentence that is different from all examples of goodbyeIntent, let's say "this is a test", luis nevertheless predicts goodbyIntent with a very high score.
In this case i would have expected the None-Intent to be predicted.
Can someone explain this behavior?
Related
What is the difference between mAP from Mask R-CNN utils.py and mAP from (MS)COCO?
I trained my Mask R-CNN Network with my own data, which i transformed into COCO Style for my Thesis and now i want to evaluate my results. I found two methods to do that. One method is the evaluation from COCO itself. Mask R-CNN itself shows how to evaluate with COCO Metric in their coco.py file: https://github.com/matterport/Mask_RCNN/blob/master/samples/coco/coco.py I basically use the evaluate_coco(...) function in line 342. As a result i get the result from COCO metric with Average Precisions and Average Recall for different metrics, see the images below. For the parameter eval_type i use eval_type="segm". For me the mAP is interesting. I know that the mAP50 uses 0.5 for the IuO (Intersection over Union) and their standard mAP is from IuO = 0.5 until 0.95 in 0.05 steps. The second method is from Mask R-CNN itself in their utils.py: https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/utils.py The name of the function is compute_ap(...) in line 715 and there is written that the IoU is 0.5. The function returns a mAP value. This raises the question of what type compute_ap() evaluates for. With COCO you can choose between "bbox" and "segm". Also i want to know the difference between the mAP value from the compute_ap(...) function and the mAP50 from COCO, because with the same data i get different results. Unfortunately i can't upload a better image now, because i can only go on Monday-Friday to my university and friday i was in a hurry and took pictures without checking them, but the mean Value over all AP from the compute_ap() is 0.91 and i am pretty sure that the AP50 from COCO is 0.81. Do someone know the difference or is there no differnce? Is it because of the maxDets=100 Parameter?: Understanding COCO evaluation "maximum detections" My pictures have only 4 categories and a maximum of 4 instances per picture. Or is the way how the mAP is calculated different? EDIT: Now i have a better image with the whole COCO Metric for Type "Segm" and Type "BBox" and compere it with the result from compute_ap(), which is "mAP: 0,41416..." By the way, if there is an "-1" in the COCO Result, does this mean there were no from this typ?
Understanding Event Coincidence Analysis from CoinCalc R package
I have two binarized events (eventA and eventB), I want to know if there is any coincidence in these two events. So I'll use the new Package CoinCalc to investigate the potential relation between these two. library(CoinCalc) #note that the package is not visible (at least for) me in CRAN. I got it from GitHub https://github.com/JonatanSiegmund/CoinCalc two binary events eventA= c(0,1,0,0,1,1,0,0,1,1,0,0,1,0,1,0,1,0,1,1,1,1,0,0,0,1,1,1,0,0,1,1,0,1,1,0,1,0,0,0,1,1,0,0,0,1,1,0,1,1,1,1,1,1,0,1,0,0,0,1,1,0,0,0,0,0,1,1,0,0,1,1,1,0,0,1,0,1,1,1,0,0,0,1,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,1,0,0,1,1,0,0,1,1,0,1,1,0,0,0,1,0,0,0,0,0,1,1,0,1,0,1,1,0,1,0,0,0,1,0,0,1,0,1) eventB = c(0,1,0,0,0,0,1,0,0,0,1,1,1,1,1,0,0,0,0,1,0,1,0,0,0,1,0,1,0,1,0,1,0,1,1,0,1,1,1,0,0,1,1,1,0,0,0,1,1,0,1,1,1,1,1,1,0,1,1,1,0,0,1,0,1,1,1,1,1,1,0,0,1,1,1,0,1,1,1,1,0,1,1,0,0,0,0,0,0,1,1,1,1,0,0,0,1,1,1,0,1,0,0,0,0,0,1,0,1,1,1,1,0,0,0,0,1,0,0,0,1,0,0,0,0,1,1,0,1,0,0,1,0,1,0,1,1,0,0,0) run ECA analysis ca.out <- CC.eca.ts(eventA, eventB,delT=2,tau=2) this yields: $NH precursor 1 TRUE $NH trigger 1 FALSE $p-value precursor 1 0.2544052 $p-value trigger 1 0.003287963 $precursor coincidence rate 1 0.8243243 $trigger coincidence rate 1 0.9285714 I want to make sure I'm understanding this properly. Based on the results, the null hypothesis can only be rejected for the trigger, which is statistically significant at the 0.003 level, and the coincidence rate is 0.92 (very high, is this equivalent to R2?). Can this be interpreted that eventB has a strong influence on eventA, but not the opposite? Then I can plot these two events using the CC.plot function: CC.plot(eventA,eventB,dates=c(1900:2040),delT=2, tau=2, seriesAname = 'EventA', seriesBname = 'EventB') Which yields: Is there any way to modify the graphical parameters in CC.plot? The dummy years are not visible in this plot. I'd like to change fonts, size, colours, etc. Is there any way to draw the same figure by calling the model output (ca.out)? Thanks in advance!
I'll try to answer your questions: Question #1: The most important problem that I see in your example is that your events are not "rare". Therefore the most important pre-condition of the analytical significance test that you used by default (sigtest="poisson") in not fulfilled. Another "problem" is, that the events in both series seem to be clustered (may also be an effect of the high number of events). I would recommend to use sigtest="shuffle.surrogate" which is more appropriate for this case. More information about the significance test can be found at Siegmund et al. 2017 (http://www.sciencedirect.com/science/article/pii/S0098300416305489) Executing this reveals that both coincidence rates are not significant. By the way: with such a high number of events it is extremely unlikely that you would ever get a 'significant coincidence rate', because the chance that simultaneities occur by random is very very high. Nevertheless, if the trigger coincidence rate would be significant and the precursor not, your interpretation is a possible one. Question #2: The problem with the plot is again, that there are too many events (compared to what the method was originally designed for). This is why everything looks so messy. The function was ment to be more like a help to explain how the method works and what you have done. If you e.g. only plot e.g. 20 years of your data CC.plot(eventA[120:140],eventB[120:140],dates=c(2020:2040),delT=2, tau=2, seriesAname = 'EventA', seriesBname = 'EventB') you will get a much better image, that yet, due to the high event-density of almost 50%, is not very nice. CoinCalc plot For now, there are no options to change the plot parameters. This might come for a future version of the package. I hope that this helps you a bit!
The ± 2 window in Word similarity of NLP
There is a question illustrate below: //--------question start--------------------- Consider the following small corpus consisting of three sentences: The judge struck the gavel to silence the court. Buying the cheap saw is false economy. The nail was driven in when the hammer struck it hard. Use distributional similarity to determine whether the word gavel is more similar in mean- ing to hammer or saw . To compute distributional similarity you must (1) use bag-of-words in a ± 2 window around the target as features, (2) not alter the context words in any way (e.g. by stemming or removing stop words) and (3) use the Dice measure to compare the feature vectors. Make sure to show all stages of your working. //--------question end--------------------- I don't understand what is a ± 2 window in (1). Would someone explain for me? Thank you guys very much.
A ± 2 window means 2 words to the left and 2 words to the right of the target word. For target word "silence", the window would be ["gavel", "to", "the", "court"], and for "hammer", it would be ["when", "the", "struck", "it"].
Computer Adaptive Testing 1PL Ability Calculation Math: How to implement?
Preamble: I have been implementing my own CAT system. The resources that have helped me most are these: An On-line, Interactive, Computer Adaptive Testing Tutorial, 11/98 -- A good explanation of how to pick a test question based on which one would return the most information. Fascinating idea, really. The equations are not illustrated with examples, however... but there is a simulation to play with. Unfortunately the simulation is down! Computer-Adaptive Testing: A Methodology Whose Time Has Come -- This has similar equations, although it does not use IRT or the Newton-Raphson Method. It is also Rasch, not 3PL. It does, however, have a BASIC program that is far more explicit than the usual equations that are cited. I have converted portions of the program in order to get my own system to experiment with, but I would prefer to use 1PL and/or 3PL. Rasch Dichotomous Model vs. One-parameter Logistic Model -- This clears some stuff up, but perhaps only makes me more dangerous at this stage. Now, the question. I want to be able to measure someone's ability level based on a series of questions that are rated at a 1PL difficulty level and of course the person's answers and whether or not they are correct. I have to first have a function that calculates the probably of a given item. This equation gives the probability function for 1PL. Probability correct = e^(ability - difficulty) / (1+ e^(ability - difficulty)) I'll go with this one arbitrarily for now. Using an ability estimate of 0, we get the following probabilities: -0.3 --> 0.574442516811659 -0.2 --> 0.549833997312478 -0.1 --> 0.52497918747894 0 --> 0.5 0.1 --> 0.47502081252106 0.2 --> 0.450166002687522 0.3 --> 0.425557483188341 This makes sense. A problem targeting their level is 50/50... and the questions are harder or easier depending on which direction you go. The harder questions have a smaller chance of coming out correct. Now... consider a test taker that has done five questions at this difficulty: -.1, 0, .1, .2, .1. Assume they got them all correct except the one that's at difficulty .2. Assuming an ability level of 0... I would want some equations to indicate that this person is slightly above average. So... how to calculate that with 1PL? This is where it gets hard. Looking at the equations on the various pages... I will start with an assumed ability level... and then gradually adjust it with each question after more or less like the following. Starting Ability: B0 = 0 Ability after problem 1: B1 = B0 + [summations and function evaluated for item 1 at ability B0] Ability after problem 2: B2 = B1 + [summations and functions evaluated for items 1-2 at ability B1] Ability after problem 3: B3 = B2 + [summations and functions evaluated for items 1-3 at ability B2] Ability after problem 4: B4 = B3 + [summations and functions evaluated for items 1-4 at ability B3] Ability after problem 5: B5 = B4 + [summations and functions evaluated for items 1-5 at ability B4] And so on. Just reading papers on this, this is the gist of what the algorithm should be doing. But there are so many different ways to do this. The behaviour of my code is clearly wrong as I get division by zero errors... so this is where I get lost. I've messed with information functions and done derivatives, but my college level math is not cutting it. Can someone explain to me how to do this part? The literature I've read is short on examples and the descriptions of the math appears incomplete to me. I suppose I'm asking for how to do this with a 3PL model that assumes that c is always zero and a is always 1.7 (or maybe -1.7-- whatever works.) I was trying to get to 1PL somehow anyway. Edit: A visual guide to item response theory is the best explanation of how to do this I've seen so far, but the text gets confusing at the most critical point. I'm closer to getting this, but I'm still not understanding something. Also... the pattern of summations and functions isn't in this text like I expected.
How to do this: This is an inefficient solution, but it works and is reasonably inituitive. The last link I mentioned in the edit explains this. Given a probability function, set of question difficulties, and corresponding set of evaluations-- ie, whether or not they got it correct. With that, I can get a series of functions that will tell you the chance of their giving that exact response. Now... multiply all of those functions together. We now have a big mess! But it's a single function in terms of the unknown ability variable that we want to find. Next... run a slew of numbers through this function. Whatever returns the maximum value is the test taker's ability level. This can be used to either determine the standard error or to pick the next question for computer adaptive testing.
Trying to use the Naive Bayes Learner in R but predict() giving different results than model would suggest
I'm trying to use the Naive Bayes Learner from e1071 to do spam analysis. This is the code I use to set up the model.
library(e1071)
emails=read.csv("emails.csv")
emailstrain=read.csv("emailstrain.csv")
model<-naiveBayes(type ~.,data=emailstrain)
there a two sets of emails that both have a 'statement' and a type. One is for training and one is for testing. when I run
model
and just read the raw output it seems that it gives a higher then zero percent chance to a statement being spam when it is indeed spam and the same is true for when the statement is not. However when I try to use the model to predict the testing data with
table(predict(model,emails),emails$type)
I get that
ham spam
ham 2086 321
spam 2 0
which seems wrong. I also tried using the training set to test the data on as well, and in this case it should give quite good results, or at least as good as what was observed in the model. However it gave
ham spam
ham 2735 420
spam 0 6
which is only slightly better then with the testing set. I think it must be something wrong with how the predict function is working.
how the data files are set up and some examples of whats inside:
type,statement
ham,How much did ur hdd casing cost.
ham,Mystery solved! Just opened my email and he's sent me another batch! Isn't he a sweetie
ham,I can't describe how lucky you are that I'm actually awake by noon
spam,This is the 2nd time we have tried to contact u. U have won the £1450 prize to claim just call 09053750005 b4 310303. T&Cs/stop SMS 08718725756. 140ppm
ham,"TODAY is Sorry day.! If ever i was angry with you, if ever i misbehaved or hurt you? plz plz JUST SLAP URSELF Bcoz, Its ur fault, I'm basically GOOD"
ham,Cheers for the card ... Is it that time of year already?
spam,"HOT LIVE FANTASIES call now 08707509020 Just 20p per min NTT Ltd, PO Box 1327 Croydon CR9 5WB 0870..k"
ham,"When people see my msgs, They think Iam addicted to msging... They are wrong, Bcoz They don\'t know that Iam addicted to my sweet Friends..!! BSLVYL"
ham,Ugh hopefully the asus ppl dont randomly do a reformat.
ham,"Haven't seen my facebook, huh? Lol!"
ham,"Mah b, I'll pick it up tomorrow"
ham,Still otside le..u come 2morrow maga..
ham,Do u still have plumbers tape and a wrench we could borrow?
spam,"Dear Voucher Holder, To claim this weeks offer, at you PC please go to http://www.e-tlp.co.uk/reward. Ts&Cs apply."
ham,It vl bcum more difficult..
spam,UR GOING 2 BAHAMAS! CallFREEFONE 08081560665 and speak to a live operator to claim either Bahamas cruise of£2000 CASH 18+only. To opt out txt X to 07786200117
I would really love suggestions. Thank you so much for your help
Actually predict function works just fine. Don't get me wrong but problem is in what you are doing. You are building the model using this formula: type ~ ., right? It is clear what we have on the left-hand side of the formula so lets look at the right-hand side. In your data you have only to variables - type and statement and because type is dependent variable only thing that counts as independent variable is statement. So far everything is clear. Let's take a look at Bayesian Classifier. A priori probabilities are obvious, right? What about conditional probabilities? From the classifier point of view you have only one categorical Variable (your sentences). For the classifier point it is only some list of labels. All of them are unique so a posteriori probabilities will be close to the the a priori. In other words only thing we can tell when we get a new observation is that probability of it being spam is equal to probability of message being spam in your train set. If you want to use any method of machine learning to work with natural language you have to pre-process your data first. Depending on you problem it could for example mean stemming, lemmatization, computing n-gram statistics, tf-idf. Training classifier is the last step.