I want to monitor that a pod in kubernetes is running correctly as cronjob twice a day using stackdriver.
In order to do it I want to send start msg and end msg logs in the pod and I want to create an alert metric in stack driver that if not receiving these msgs after 24 hours, send Email.
Is it possible to do this alerting in stack driver ?
There are several ways of accomplishing this.
In order to generate the event, I think the easiest way is to check on a log-based metric based on the CRON itself. If you are running a kind:CronJob, you can either use the Metrics Explorer to find Resource type:GKE Container Metric: Log entries, and then filter by container_name (which will be your CronJob spec.containers.name)
You could also create a log based metric on something like
logName="projects/[PROJECT-ID]/logs/[CONTAINER-NAME]"
...and maybe add a string to the spec.containers.args section to make filtering easier.
You could also publish to a pub/sub topic and do your alerting on publish message operations.
Once you decide on the metric, you just need to alert if Any time series is absent[1] for 13 hours. Add a notification channel type=email[2], and you will receive an alert whenever the cron does not run at least once a day.
[1] https://cloud.google.com/monitoring/alerts/concepts-indepth#condition-types
[2] https://cloud.google.com/monitoring/support/notification-options#email
Related
I want to schedule a telegram bot message to be sent at a specific unixtime.
As from telegrams official api (https://core.telegram.org/api/scheduled-messages) that should be possible by setting the schedule_date flag.
To schedule a message, simply provide a future unixtime in the schedule_date flag of messages.sendMessage or messages.sendMedia.
However I was not able to set that flag. To be more precisely, I do not even know how to set a flag, or if I am using the correct api.
What I have tried is to use the api directly via the browser (could use curl as well) like so: https://api.telegram.org/botBOT:TOKEN/sendMessage?chat_id=ID&text=Test&schedule_date=1653503351
I also did not find any way to access this flag via https://pypi.org/project/pyTelegramBotAPI/#description https://telepot.readthedocs.io/en/latest/#send-a-message, nor https://github.com/nickoala/telepot.
I want to implement this feature in a python environment, but any working suggestion would be much appreciated.
EDIT:
I decided to save the intention to send a telegram bot message at a certain unixtime in a database. I then create an infinite loop that checks if there are any unsent messages before the current timestamp. If the loop detects such a message it sends the message and sets a flag, that that message has been sent.
And as promised, here is a fully dockerized example of that behaviour in action: https://github.com/Sokrates1989/nameTheCountDown-lightweight
It creates a bot that you can pass a name and the duration. Once the duration has passed it sends a message with the passed name. Basically a simple countdown that you can give several names, that run simltaniously. As it is a telegram chat, you can modify the way you are informed about the end of a countdown by modifying the notificaiton of that chat.
And here is the Bot in action: http://t.me/NameTheCountdownBot
We can't do this by bot API itself, and there's no schedule_date parameter in sendMessage method:
https://core.telegram.org/bots/api#sendmessage
And what you've read is for Telegram clients, not bot API consumers.
If you don't really need unixtime, you can simply create a table for scheduled messages with a text, chat_id and a publish_time column (like 22:15), and run a command every minute to look if there's a message for current time to send. Then send the message and delete the record.
Note that the python-telegram-bot library has a built-in solution for scheduling tasks: The JobQueue. This feature is based on the APScheduler library, which you can ofc also use without python-telegram-bot.
Disclaimer: I'm currently the maintainer of python-telegram-bot.
https://core.telegram.org/method/messages.sendScheduledMessages
Now you can send scheduled messages right away
I need to actively receive crash notifications for firebase functions.
Is there any way to set up Slack webhooks to receive a message when Firebase Functions throw an Error, functions crash, or something like that?
I would love to receive issue messages by velocity ie: Firebase Functions crash 50 times a day.
Thank you so much.
First you have to create a log based (counter) metric that will be counting specific error occurencies and second - you create alerting policy with Slack notification channel.
Let's start from finding corresponding logs that appear when the function throws an error. Since I didn't have one that would crash I used logs that indicated that it was started.
Next you have to create a log based metric. Ignore the next screen and go to Monitoring > Alerting. Click on "Create new policy", find your metric and select "Rolling Window" to whatever time period you need. For testing I used 1 minute. Then set "Rollind windows function" to "mean".
Now configure when the alert has to be triggered - I chose over 3 (within 1 minute window).
On the next screen you select notification channel. In case of Slack it has to be configured first in "Notification Channels".
You can save policy the policy now.
After a few minutes I gathered enough data to generate two incidents:
And here's some alerting related documentation that may help you understand how to use them.
Lets say i have a job that runs for 10mins(lets say the time as 10.00). i set max_run_alarm as 3. i should get an email at 10.03 where i can goahead and see why the job is running more than the max_run_alarm. if i use max_run_alarm i am able to see in the logs triggering that alarm, but I cannot spend all day monitoring the logs to see which job is taking long as i have many jobs. my question is am i using max_run_alarm in the correct way or is there something else i am missing or is there entirely different way for the emails to generate.
There is an added functionality where you can configure the email id where all Autosys notification to be sent out.
Incase of any event such as Job Failure / Termination / Run Alarms etc are notified to that email address.
This is configured by the Autosys Admin or the Scheduling Team which i call them as.
I am trying to call an api every minute for ski lift status and check for changes. I am going to store the value of if the lift is open or closed in firebase (Real Time Database) and read to see if value from api is different and only update/ write to that node when it's a different value. Then I can set up a cloud function that will listen for database changes and send push notifications to the list of FCM tokens from that channel. I am not sure if this is the most efficient way, but I was going to set up scheduled functions to call the third party api.
I have been using these docs:
https://firebase.google.com/docs/functions/schedule-functions
I was planning to do something like this:
exports.scheduledFunction = functions.pubsub.schedule('every 5 minutes').onRun((context) => {
CALL MY API IN HERE AND UPDATE DATABASE IF SNAPSHOT BACK IS DIFFERENT
});
I was wondering how would I run only between set times- say 8am-6pm EST. I am struggling to find anything about times to run. Should I just run the function every minute and then pause and resume by checking the time? In which case how does it know to keep checking the time when it is paused?
Firebase scheduled functions use Cloud Scheduler to implement the schedule. It accepts cron style time specifiers to indicate when a job should be run. The full spec for that can be found here. You will have to use ranges of numbers to indicate the valid times and frequency of the schedule. For example, you might use "8-18" in the hour field to limit the hours of execution.
Currenty I have requirement in my enviroment for the autosys email notifition.
Requirement: If the job runs more than the specified time it should trigger an email.
What I am trying is using max_run_alam, but I am not successful.
Lets say i have a job that runs for 10mins(lets say the time as 10.00). i set max_run_alarm as 3. i should get an email at 10.03 where i can goahead and see why the job is running more than the max_run_alarm. if i use max_run_alarm i am able to see in the logs triggering that alarm, but I cannot spend all day monitoring the logs to see which job is taking long as i have many jobs. my question is am i using max_run_alarm in the correct way or is there something else i am missing or is there entirely different way for the emails to generate.
pls advise.
We are using autosys R11 at work. I believe the triggering of emails is already automated in higher versions of autosys, but, in our version, to send automatic emails after a certain time, we create two extra autosys jobs. One autosys job starts at the same time as the job you want to "monitor". This job contains a 'sleep' command. (in your example, the command would be "sleep 180" for the job to run for 3 minutes until completion). The second extra job is the sending of the email and only starts after successful completion of the sleep-job.
To prevent the mail from being send every time the autosys box starts, you have to add your first job as BOX_SUCCESS condition. The sleep-job will run to completion, but the mail-job went from the "ACTIVATED" state to the "INACTIVE" state because the autosys box isn't RUNNING anymore.