I have an asp.net application deployed in azure. This generates plenty of logs, some of which are exceptions. I do have a query in Log Analytics Workspace that picks up exceptions from logs.
I would like to know what is the best and/or cheapest way to detect anomalies in the exception count over a time period.
For example, if the average number of exceptions for every hour is N (based on information collected over the past 1 month or so), and if average goes > N+20 at any time (checked every 1 hour or so), then I need to be notified.
N would be dynamically changing based on trend.
I would like to know what is the best and/or cheapest way to detect anomalies in the exception count over a time period.
Yes, we can achieve this by following steps:
Store the average value in a Stored Query Result in Azure.
Using stored query result
.set stored_query_result
These are some limitations to keep the result. Refer MSDOC for detailed information.
Note: The stored query result will be available only 24 hours.
Workaround Follows
Set the Stored query result
# here i am using Stored query result to store the average value of trace message count for 5 hours
.set stored_query_result average <|
traces
| summarize events = count() by bin(timestamp, 5h)
| summarize avg(events)
2. Once Query Result Set you can use the Stored Query Result value in another KQL Query (The stored value was available till 24 hours)
# Retrieve the stored Query Result
stored_query_result(<StoredQueryResultName>) |
Query follows as per your need
Schedule the alert.
I'm new to DynamoDB and would like some help on how to best structure things, and whether it's the right tool for the job.
Let's say I have thousands of users signed up to receive messages. They can choose to receive messages every half hour, hour, couple of hours or every 4 hours. So essentially there is a schedule attribute for each user's message. Users can also specify a time window for when they receive these messages, e.g. 09:00 - 17:00 and also toggle an active state.
I want to be able to easily get the messages to send to the various users at the right time.
If done in SQL this would be really easy, with something like:
Select * from UserMessageSchedules
where
now() > startTime
and now() < endTime
and userIsActive
and schedule = 'hourly'
But I'm struggling to do something similar in DynamoDB. At first I thought I'd have the following schema:
userId (partion Key)
messageId (sort key)
schedule (one of half_hour, hour, two_hours, four_hours)
startTime_userId
endTime
I'd create a Global Secondary Index with the 'schedule' attribute being the partition key, and startTime + userId being the sort key.
I could then easily query for messages that need sending after a startTime.
But I'd still have to check endTime > now() within my lambda. Also, i'd be reading in most of the table, which seems inefficient and may lead to throughput issues?
And with the limited number of schedules, would I get hot partitions on the GSI?
So I then thought rather than sending messages from a table designed to store users preferences, I could process this table when an entry is made/edited and populate a toSend table, which would look like this:
timeSlot (pk) timeSlot_messageId (sk)
00:30 00:30_Message1_Id
00:30 00:30_Message2_Id
01:00 01:00_Message1_Id
Finding the messages to send at certain time would be nice and fast as I'd just query on the timeSlot
But again I'm worried about hot spots and throughput. Is it ok for each partition to have 1000's rows and for just that partition to be read? Are there any other problems with this approach?
Another possibility would be to have different tables (rather than partitions) for each half hour when something could be sent
e.g, toSendAt_00:30, toSendAt_01:00, toSendAt_01:30
and these would have the messageId as the primary key and would contain the data needing to be sent. I'd just scan the table. Is this overkill?
Rather than do big reads of data every half an hour, would I be better duplicating the data into Elastic Search and querying this?
Thanks!
There are several date/time fields in this view and I can never wrap my head around which column to order by (and which secondary column) in order to retrieve a list of SQL statements in the order in which they were executed on the server.
StartTime - The timestamp associated with when the query was submitted to Teradata for parsing.
FirstStepTime - The timestamp associated with when the first step of the query was executed.
FirstRespTime - The timestamp associated with when the first row was returned to the client.
The gap in time between the StartTime and FirstStep time include parsing time and any workload throttle delay that was enforced by Teradata's Dynamic Workload Manager. For the sake for keeping things simple here I will defer to an excellent article written by Teradata's Carrie Ballinger on dealing with delay time here.
I'm having a hard time wrapping my head around what seems to be a somewhat simple issue. Let's say that I have a business whose hours are 12PM - 3AM daily. Each customer gets a bonus once per day based on their initial purchase for that day. So, let's say they spend twenty bucks on their first transaction that day -- they might get a twenty percent discount on that transaction, and that's it for the day.
I'm trying to figure out the most accurate way to check the last bonus that was given and make sure that the customer is eligible for one. I can't do a simple 24-hour check, obviously, because if a customer comes in at 11 PM Monday, for instance, and again at noon Tuesday, they will not get their second bonus.
We are using a VB6 frontend for our POS, with a SQL Server 2008 R2 database. Each time a bonus is applied, it is audited on the database side, so I can easily query the last time the bonus was applied.
EDIT: I should note that, for various reasons, the solution cannot include making any changes to the structure of the database.
I'm not sure on which side (VB or SQL) you want to apply the biz logic but in either case the process should be the same: You need to persist each customer's daily hours of operation with two attributes:
Time (the time of day that they open for business)
TimeSpan (number of hours of operation)
You then check if a transaction's time is between Time and Time + TimeSpan to calculate your business logic and the customer's bonus. Both calculations are fairly trivial in VB and SQL. You just need to make sure you persist the data logically and use it consistently.
I think your answer would be cleaner if you modified it to something like:
IF #LastBonus BETWEEN #store_open AND #store_close
BEGIN
SET #BonusDue = 0
END
ELSE
BEGIN
SET #BonusDue = 1
END
where you figure the store open and close dates based on a fixed times that are added to the date part of the last bonus. Something like
Set #openTime = '12:00'
Convert(date, #LastBonus) + #openTime
And then adding the timespan (as was suggested) to get the close time. It might be a little tricky because if it's after midnight, the open time would need to be added to the previous date instead, but you could probably work this out with a CASE statement. I'd try it out myself if my baby didn't have an ear infection. Hope that is useful to you.
How about:
IF (DATEPART(dayofyear, [lastBonusTime]) <> DATEPART(dayofyear, GetDate())) ...
Where lastBonusTime is the time of the last bonus transaction ?
You can look at the problem a bit differently. If a customer is here now (GETDATE()), has it been over 24 hours since their last purchase?
So, something like
SELECT *
FROM Customers c
INNER JOIN CustomerVisits cv
ON c.CustomerId=cv.CustomerId
INNER JOIN CustomerBonus cb
ON cv.VisitId=cb.VisitId
WHERE c.CustomerId=#CustomerId
AND LastVisitDt BETWEEN
(
DATEADD(hh,12,convert(DATE, LastVisitDt))
)
AND
(
DATEADD(hh,27,convert(DATE, LastVisitDt))
)
AND DATEADD(hh,24,LastVisitDT)<=GETDATE()
I would also consider the specifics of the data--the above is NOT TUNED FOR PERFORMANCE AT ALL. I just wanted to explain my thought process.
In the interest of separating your concerns, I would add a new table, like CUSTOMER_BONUS, with these columns:
BonusStart datetime
BonusEnd datetime
CustomerID int/uniqueidentifier/whatever
TransactionID int/whatever (points to what qualified for the bonus)
When you apply a bonus for a customer for a day, write a new record into this table for the period that it applies to. Presence of a record in this table indicates that the customer is not eligible for another bonus between BonusStart and BonusEnd. When you create a new sale, look in this table. If the record exists, no bonus, but if not, apply the bonus and create a new record here.
I came up with an answer that I'm content with but it's a little kludgy and I would be more than happy to accept a more elegant answer if one is provided. Also, I haven't thoroughly tested this since it's getting late in the day, but if there are flaws in my logic, I will happily revise or accept an answer with revisions.
Basically, I'm just going to determine that the day of the week in terms of a business day is whatever day started four hours ago. This means that all the way up through 3:59 AM, "today" will be considered the day before, which is correct for these hours of operation (I'm overshooting the 3 AM closing time to account for a site deciding to stay open a little later). I then compare this span of time to the most recent time a bonus was applied to that customer's account, using the same rules. If the two match, the bonus has been applied this business day. If they are different, it has not, and the customer is eligible.
DECLARE #CustID AS int
DECLARE #LastBonus AS date
DECLARE #BonusDue AS bit
SET #LastBonus = (SELECT TOP 1 [DateTime] FROM Audit WHERE CustomerID = #CustID AND TransactionType = 'BONUS' ORDER BY [DateTime] DESC)
IF (SELECT DATEADD(hh, -4, CURRENT_TIMESTAMP)) <>
(SELECT DATEADD(hh, -4, #LastBonus))
BEGIN
SET #BonusDue = 1
END
ELSE
BEGIN
SET #BonusDue = 0
END
If I throw this in a stored procedure, I can simply throw a customer ID at it and have it spit out a bit that will show me 1 if the customer is eligible, 0 otherwise. What I don't like about it is that if a customer's hours of operation end up getting much earlier, I'll be sunk (I guess at about 7:00 AM, when simply subtracting four hours will overlap into the previous business day, but subtracting less will not be enough to reach the previous business day). So it will work for the time being, but I'd love to see a better solution.
I've got a query where I'm trying to get the hours in duration (eg 6.5 hours) between two different times.
In my database, time and date are held in different fields so I can efficiently query on just a startDate, or endDate as I never query specifically on time.
My query looks like this
SELECT COUNT(*), IFNULL(SUM(TIMEDIFF(endTime,startTime)),0) FROM events WHERE user=18
Sometimes an event will go overnight, so the difference between times needs to take into account the differences between the dates as well.
I've been trying
SELECT COUNT(*), IFNULL(SUM(TIMEDIFF(CONCAT(endDate,' ',endTime),CONCAT(startDate,' ',startTime))),0) FROM events WHERE user=18
Unfortunately I only get errors when I do this, and I can't seem to combine the two fields into a single timestamp.
Pretty sure your problem is that your concatenated values are being sent to TIMEDIFF() as strings rather than DATETIMEs. Try calling the DATETIME function on them:
SELECT COUNT(*), IFNULL(SUM(TIMEDIFF(DATETIME(CONCAT(endDate,' ',endTime)),DATETIME(CONCAT(startDate,' ',startTime)))),0) FROM events WHERE user=18
I don't have a MySQL DB in front of my to test that, but I think that or some similar form of it is what you are looking for. There's an example of it in the MySQL docs involving MICROSECOND:
http://dev.mysql.com/doc/refman/5.0/en/datetime.html
Edit: Hmm... looks like TIMEDIFF is supposed to work with strings. Worth trying anyway.
TIMEDIFF(endDate,startDate) + TIMEDIFF(endTime,startTime)