I've set up a hook on my gitlab server to call salt-run fileserver.update from a post-update hook.
How can I disable the schedule that does a update every 60 to reduce the load on my gitlab server?
The 60 seconds interval in which the Git filesystem is updated is defined by the loop_interval setting, which you can set in your master configuration file:
# The loop_interval option controls the seconds for the master's maintenance
# process check cycle. This process updates file server backends, cleans the
# job cache and executes the scheduler.
#loop_interval: 60
However, this interval controls not only the GitFS update schedule, but also a number of other maintenance tasks, so you should not increase this interval by too much.
From a quick reading of the source code (I'm not a core Salt developer though, so I might be mistaken), the GitFS update is hard-coded to run on the same schedule as these other maintenance tasks. There does not appear to be a way to disable or change the interval of only the GitFS update schedule.
Related
We have most, but not all of our build artifacts into a Git repo (Bitbucket).
Our current build looks like this, and takes 30+ minutes to build/deploy to Firebase, we would like to reduce the time to build.
We are not using Google Cloud Build at the moment, but before heading down that path, I want to find out if that would even be fruitful.
We have all of the code cloned from the git repo (Bitbucket), to a GCE VM.
And then 1 TB of static data is then copied into a directory under the git repo area, artifacts that are needed for the deploy.
We do not want to check in that 1TB of data into the git repo, it is from a 3rd party, it is rarely updated, and would be too heavy of a directory to pull into developer environments on their IDE's, it is pointless to do so.
We launch a build script on the GCE VM to build the code, and deploy to Firebase (bash script), it takes about 30 minutes.
We want the builds to go faster, and possible to use cloud build.
With this:
a git repo
external files that need to remain in a stateful container, not copied over each time, due to the time it would take
how do we create a stateful container that would only require a git update (pull origin master), and then to fire off a build/deploy to Firebase?
We want to avoid ingress traffic to the Firebase deploy using external build services where the 1TB of data that remains the same each and every time is sent to Firebase, where we would be billed.
Cloud Run containers are not stateful. GCE VM's are stateful, but it requires that we keep them up and going 24x7x365, so that any developer anywhere can run a build, and that may take only 30 minutes out of any day, and we don't know when that will be, so leaving it up 24x7x365 is mostly wasteful.
We want to avoid building a stateless container where the code is checked out fresh each and every time, a git pull origin master will do, and to have to copy the 1TB of artifacts into the container each and every time taking time.
We just want to do:
git pull origin master
Fire off the build as the next step in the script
spin down the container, have it save it's state for the next build, minimizing time, each and every time, saving the previous 'git pull origin master' updated artifacts, and preserving the 1TB files we copied to the container.
The ideal situation would be to have a container that is stateful, that spins down when not in use, and "spins up", or is made active for use when we need to do a build.
It would retain the previous git update (git pull origin master), and would retain all artifacts outside the git repo that we copy over. We also need shell access to the container (ssh, scp) etc.
A stateful 'Cloud Run' option would be ideal, but I don't know of such a thing (stateful containers with GCP that we can run and only be billed for runtime/compute time)
One solution is to use a VM for this. Add a startup script. In it
git pull origin master
Fire off the build as the next step in the script
Add this line which stop your compute
gcloud compute instances stop $(curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/id) --zone=$(curl -H "Metadata-Flavor: Google" http://metadata.google.internal/compu
teMetadata/v1/instance/zone)
By the way, each time that you start your VM, it will apply the startup script and shutdown automatically. You keep your persistent disk, and thus your 1TB data, and you pay few because of automatic stop.
If you have to wait an external build. 2 solutions:
Either you set a sleep timer and shutdown after in the startup script
Or customize this tutorial -> At the end of your build, publish a message in PubSub, which trigger a function which will stop your instance.
EDITED to reply to comments
Here again, 2 solutions:
You can create a custom role with only the permission required. You can see all compute role here. If you provide an access to the console, I recommend list (to view the VMs), start and stop. Else, only start and/or stop if you write a script.
You can create a private function or Cloud Run. Assign a service account as identity to this, with enough role to start the VM (even if there is more permission as required -> it's not a good practice. Prefer the least privilege with custom role) and grant the role function.invoker or run.invoker to the user (depend if you use Function or Cloud Run) for allowing it to call this private endpoint and start the VM without right on the VM (only the right to perform an HTTP call).
I have a task that's scheduled to run hourly, however it's not being triggered. When I look at theTask Instance Details it says:
All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:
- The scheduler is down or under heavy load
- The following configuration values may be limiting the number of queueable processes: parallelism, dag_concurrency, max_active_dag_runs_per_dag, non_pooled_task_slot_count
- This task instance already ran and had its state changed manually (e.g. cleared in the UI)
If this task instance does not start soon please contact your Airflow administrator for assistance.
If I clear the task in the UI I am able to execute it through terminal but it does not run when scheduled.
Why do I have to manually clear it after every run?
My level of experience with the product is basic at best, but I'm expected to be a developer; I have a basic understanding of many things.
Right now my job is to investigate canceling lines in Purchase Orders. We have a workflow set up to handle those, and I'm trying to duplicate the scenario in my dev instance. Whenever a user cancels a line, the workflow is supposed to engage, and I've found that a batch job is what triggers that workflow to work (maybe that's the case with all workflows, but I don't know that for sure).
I've set up my personal Dev AX Instance under System Configuration => System => Server Configuration to use my personal Dev AOS server that my client is also running under, but when I go to System Configuration => Batch Jobs => Batch Jobs, then find the Batch Job I've been looking for and set the status to Waiting, the Batch Job never runs.
On our Test instance, the jobs is configured exactly the same way, except they use the AOS Server allotted for it.
I did a SQL script to change the batch job to use my personal Dev AOS Server, then did a restart of the Dynamics AX Servers.
There must be something I'm doing wrong for my personal dev instance. I've been reading some things from here about what may be going on and following down the list, but I'm pretty sure the problem is even stupider => https://www.daxrunbase.com/2017/07/02/troubleshooting-batch-jobs-in-ax/
First of all, do you have all 3 workflow jobs set up?
Workflow message processing
Workflow due date processing
Workflow line-item notifications
They can be set up from System administration > Setup > Workflow > Workflow infrastructure configuration.
Secondly, it is OK for the periodic batch jobs to have status Waiting. They will be in status Executing for a short time and then they will be Waiting for the next run. If the Scheduled start date/time value in this batch job is in the past, that could be a problem. Otherwise everything is OK.
Lastly, if you have already ticked the Is batch server check-box in System administration > Setup > System > Server configuration, please also make sure to move the workflow batch group in the Batch server groups section in the same form from Remaining groups to Selected groups.
The batch jobs should start at Scheduled start date/time - or a bit later, you'd need to wait a minute and refresh the grid.
We're running Wordpress in an environment that features a multi-master DB behind a load-balancer. The error log was filling up with a deadlock error when WP tried to update the cron table in wp_options. We disabled wp-cron altogether but are still seeing the error, so, two questions:
1) What causes the cron table in wp_options to be updated?
2) It appears to run on every page load. Can this be disabled and a cronjob setup to run it periodically in crontab?
Thanks
Wordpress uses wp-cron.php as a means for running scheduled tasks when the user doesn't have access to or want to setup cronjobs via Unix. This process looks at the scheduled jobs in the cron table in wp_options and if the specified time (or more) has elapsed then the job executes.
wp-cron.php uses wp-includes/cron.php (the Wordpress Cron API) to run scheduled jobs. In cron.php you'll find a number of functions that update the cron table table, these functions are all around the scheduling of events.
Any function of Wordpress or plugin that requires a scheduled event uses the Cron API to do so. However, the action of scheduling an event (even if it already exists) updates the cron table in wp_options. Even with wp-cron.php totally disabled, these elements of Wordpress/the plugin are loading and scheduling their events, trying to update the cron table in the process.
I've not figured-out exactly why the deadlock occurs, other than knowing it must be related to the DB/site config, but I do now know that Wordpress is behaving itself.
I've run into this same issue -- the databases would go out of sync very quickly. Certain plugins made it occur faster (they scheduled lots of cron jobs), but even with them disabled, eventually the errors would block replication.
I was able to keep replication working by doing two things.
The first, in my.ini, was to add:
slave-skip-errors = 1062
This instructs MySql to skip creating entries when a duplicate key already exists. My cluster is set up as active-passive, so in theory, there should be no "real" writes to the passive MySql node unless the active node is down, in which case, there will be no "real" writes to that node. The only stuff that gets written to the passive node is as a result of wp-cron jobs, which (in theory) are also running on the active node.
The second, in each site's wp-config.ini, was to add:
/** disable cron */
define('DISABLE_WP_CRON', true);
This blocks wp-cron from running at all, so either one of these solutions should work on their own.
Another option would be to disable wp-cron, but leave the full database syncing in place, and schedule a script to call each site's wp-cron.php (you'd be accomplishing manually what the wp-cron service does automatically). That way, it will only run on the active node and the data should be synced over to the passive node with no problems.
I need to index 80.000 nodes.
The max amount of nodes I can index per each cron run is 500.
I need to run crone 80.000 / 500 times to index the entire website.
How can I automatically schedule these runs (when a run is finished, the next run automatically should start)?
I don't have SSH access so I cannot use drush.
Thanks
All cron does is visit yoursite.com/cron.php
So you could use cron/schedule task/etc on a local machine.
Did you try Poormanscron?
A module which runs the Drupal cron operation using normal browser/page requests instead of having to set up a crontab to request the cron.php script. The module inserts a small amount of JavaScript on each page of your site that when a certain amount of time has passed since the last cron run, calls an AJAX request to run the cron tasks. Your users should not notice any kind of delay or disruption when viewing your site. However, this approach requires that your site gets regular traffic/visitors in order to trigger the cron request.
Why don't you set a cronjob every 4 minutes or so? Just make sure that the interval between cronjobs is longer than the time it takes to run the cron script, so it won't overlap.
Give a try to Apache Solr Search module in drupal.
To reiterate and clarify other answers: As long as you haven't explicitly blocked it in .htaccess or Apache configuration, you can trigger Drupal's cron.php simply by visiting yoursite.com/cron.php from any browser. You can also set up your local machine (or any other machine that has web access, really) to run its own cronjob which triggers your site's cron.php. This process varies from platform to platform, but for example, on most Linux systems, you could run crontab -e and add a line like this:
0 * * * * wget -O - -q -t 1 http://www.example.com/cron.php
# Run example.com's cron tasks at the beginning of every hour.
or possibly:
*/5 * * * * wget -O - -q -t 1 http://www.example.com/cron.php
# Run example.com's cron tasks at every five minute interval.