I'm creating an app where a user can create a piece of data that could be presented in the ui at a later date. I'm trying to find a way to create cron entries dynamically using either java code (for android devices) or node.js code (firebase cloud function generates a cron job). I haven't been able to find a way to do it dynamically and based on what I read it may not be possible. Does anyone out there know a way?
Presently the only way to create GAE cron jobs is via deployment of a cron configuration file (by itself or, in some cases, together with the app code). Which today can only be done via CLI tools (from the GAE or Cloud SDKs).
I'm unsure if you'd consider programmatically invoking such CLI tools qualifying to 'create cron entries dynamically'. If you do - generation the cron config file and scripting the desired CLI-based deployment would be an approach.
But creating jobs via an API is still a feature request, see How to schedule repeated jobs or tasks from user parameters in Google App Engine?. It also contains another potentially acceptable approach (along the same lines as the one mentioned in #ceejayoz's comment))
Related
I have 2 functions on Google Cloud Functions, using python, that use the same library.
The file organization I have is:
/libs/libCommon.py
/funcA/main.py
/funcB/main.py
Both function A and function B use libCommon.
Through the docs I only see ways for including subdirectories.
There is no clear way to include a parent directory.
What's the best way to organize the code?
Thanks
You can't share code between functions. However you have several solution to achieve this:
Create a package, deploy on PiPy and add this as dependency in your requirements.txt file. Issue -> PiPy is public without souscription
Create a deployment script which copy the source where they should be, and then run the gcloud command -> I'm not fan of scripting, especially if your project becomes complex
Use Cloud Run instead of Function. You can create 2 different containers or only one with 2 entry points. Cloud Run has many advantages.
If your request can be processed in parallel on the same instance, you can save money.
If not, set the concurrency param to 1 (same behavior as function).
Your code can be shared between several endpoints.
Your code is portable
Your service has always 1vCPU for processing, memory is customizable. You can also save money compare to function
Of course, I'm Cloud Run fan, but I think it's the best solution. About Storage event, it's not an issue. Set up a notification to publish storage event to PubSub and then set up a Push Subscription to your service
I'm trying to create a function to delete all users from the Database and write it on index.ts file and deploy it. Now, how can I run it?? I don't want my clients to run it from their mobile app, I need some admin tool to run the management functions. When I see cloud functions on Firebase console there's no option to run the functions, just to view their logs.
Cloud Functions isn't for running one-time scripts. You should do that on your local machine. Cloud Functions is for running code that responds to HTTP requests, or events that occur within other products in your project.
You may want to read this article: https://firebase.googleblog.com/2017/03/how-to-schedule-cron-jobs-with-cloud.html - it suggests using App Engine cron job for that purpose. Relevant project on github: https://github.com/FirebaseExtended/functions-cron
Also there's an example on similar topic in Firebase docs: https://github.com/firebase/functions-samples/tree/master/delete-unused-accounts-cron
I have an R script that I run every day that scrapes data from a couple of different websites, and then writes the data scraped to a couple of different CSV files. Each day, at a specific time (that changes daily) I open RStudio, open the file, and run the script. I check that it runs correctly each time, and then I save the output to a CSV file. It is often a pain to have to do this everyday (takes ~10-15 minutes a day). I would love it if someway I could have this script run automatically at a pre-defined specific time, and a buddy of mine said AWS is capable of doing this?
Is this true? If so, what is the specific feature / aspect of AWS that is able to do this, this way I can look more into it?
Thanks!
Two options come to mind thinking about this:
Host a EC2 Instance with R on it and configure a CRON-Job to execute your R-Script regularly.
One easy way to get started: Use this AMI.
To execute the script R offers a CLI rscript. See e.g. here on how to set this up
Go Serverless: AWS Lambda is a hosted microservice. Currently R is not natively supported but on the official AWS Blog here they offer a step by step guid on how to run R. Basically you execute R from Python using the rpy2-Package.
Once you have this setup schedule the function via CloudWatch Events (~hosted cron-job). Here you can find a step by step guide on how to do that.
One more thing: You say that your function outputs CSV files: To save them properly you will need to put them to a file-storage like AWS-S3. You can do this i R via the aws.s3-package. Another option would be to use the AWS SDK for python which is preinstalled in the lambda-function. You could e.g. write a csv file to the /tmp/-dir and after the R script is done move the file to S3 via boto3's S3 upload_file function.
IMHO the first option is easier to setup but the second-one is more robust.
It's a bit counterintuitive but you'd use Cloudwatch with an event rule to run periodically. It can run a Lambda or send a message to an SNS topic or SQS queue. The challenge you'll have is that a Lambda doesn't support R so you'd either have to have a Lambda kick off something else or have something waiting on the SNS topic or SQS queue to run the script for you. It isn't a perfect solution as there are, potentially, quite a few moving parts.
#stdunbar is right about using CloudWatch Events to trigger a lambda function. You can set a frequency of the trigger or use a Cron. But as he mentioned, Lambda does not natively support R.
This may help you to use R with Lambda: R Statistics ready to run in AWS Lambda and x86_64 Linux VMs
If you are running windows, one of the easier solution is to write a .BAT script to run your R-script and then use Window's task scheduler to run as desired.
To call your R-script from your batch file use the following syntax:
C:\Program Files\R\R-3.2.4\bin\Rscript.exe" C:\rscripts\hello.R
Just verify the path to the "RScript" application and your R code is correct.
Dockerize your script (write a Dockerfile, build an image)
Push the image to AWS ECR
Create an AWS ECS cluster and AWS ECS task definition within the cluster that will run the image from AWS ECR every time it's spun-up
Use EventBridge to create a time-based trigger that will run the AWS ECS task definition
I recently gave a seminar walking through this at the Why R? 2022 conference.
You can check out the video here: https://www.youtube.com/watch?v=dgkm0QkWXag
And the GitHub repo here: https://github.com/mrismailt/why-r-2022-serverless-r-in-the-cloud
I am running the Quartz.Net server as a Windows service, like described in the documentation. I am trying to understand how I can create new jobs for Quartz to schedule, without the need to rebuild the Quaretz.net server application everytime.
I would like to be able to add new jobs from an exe, dll, or other options welcome. This way I can add jobs dynamically. From what I can tell it seems all jobs must be defined up front and built into the server. From there the user can pass parameters and enable triggers via XML file. I am using MS SQL Server instead of XML file for persistence layer.
My use case is I need to generate reports at particular times, but the users can create new reports after launch of my application. I am using Dev Express for my reporting (not sure if this matters).
Any guidance is very appreciated.
You should check out the work Tolis Bekiaris did on the eXpand Framework's JobScheduler. It's a module for DevExpress's XAF and Quartz.NET which should give you plenty of sample code, especially if you are already using XPO for your data.
You can get the source code here.
Or alternatively, it's on Github.
You'll find the job scheduler code in eXpand/Xpand/Xpand.ExpressApp.Modules/JobScheduler.
I use Scrum methodology and deploy functionality in builds every sprint.
There is necessity to perform different changes in the stored data (I mean data in database and on filesystem). I'd like to implement it as a PHP scripts invoked from console. But they should be executed only once, during the deployment.
Is there any way to implement it through app/console without listing it in the list of registered Console commands? Or is there any other way to implement runonce scripts?
DoctrineMigrations covers some part of my requirements, but it's hard to implement complex changes in Model. And it does not cover changes in files on the filesystem.
I don't think symfony has a facility for that, and besides, hiding the command is not the same as securing the command.
Instead, I would make the script determine if it has been run already (could be as simple as adding a version number to a file and checking that number before running) and stop if it detects it has already run before.