In my organization, we are using jenkins pipeline as code (groovy script). Now i want to integrate my independent automation job with that pipeline but the condition is that it should be triggered asynchronously which means after successfully executing of the development code it will triggered my automation job but pipeline should not wait for my job to end
This should be possible using Parallel stages with declarative Pipeline as follows:
stage('run-parallel-branches') {
steps {
parallel(
executeDevCode: {
//Perform job for code execution
},
executeAutomation: {
//Perform automation job
}
)
}
}
More details can be found here
Related
Let me introduce the scenario:
I need to test an AzureFunction with a queue trigger:
[FunctionName("AFunction")]
public async Task DispatchAction([QueueTrigger("queuename")] string message)
{
await DoMyLogicAsync();
}
The test needs to be run by the "functional-test-container" in my docker-compose testing env, which is made up by:
a) functional-test-container: a .net core container running an nUnit test suite
b) azure-function-container: this container hosts the azure function
c) azurite-container: this container hosts the queue server
d) sql-server-container
e) wiremock-container
The test logic is the following:
Clear the sql database, the queue and wiremock status
Prepare the wiremock stubs
Somehow trigger the function
wait for the function to end
make assertions on what the function produced in sql server, in the queue and on what wiremock's stubs have been called
As far as I know I have 2 ways of triggering the function:
a) pushing a message in the queue
b) using azure function's admin API /admin/functions/afunction
the problem is that both of them don't give any hint on when the function ends its execution.
Here it is my question: is there a way to call the function in a "sync" way (so that I can know when the execution ends)?
I don't think it can be implemented. The queue trigger function runs as an instance in azure server, we can just trigger it to run. It doesn't response any data like HttpTrigger function. So it can't be executed synchronously in your entire process.
To solve your problem, I think you can just add some code to do an operation at the end of your function. The operation is used to let you know the function execution is completed. Or another solution is move the steps after function into your function code.
I have an app service in Azure running the front end from my MVC5 app, and another app service for web jobs. The app has several endpoints (GET Actions) which do some processing, send some emails or other simple task. Previously when we were hosted on a VPS, we used the Windows Task Scheduler to call each URL on a custom schedule. In Azure, the way we're currently doing this is with a Powershell script which uses CURL to fetch the URL and trigger the processing.
It seems messy though - as each powershell script has to be uploaded individually, and can't be viewed or changed after uploading. I've found various guides on deploying a .NET Core console app, but from what I can tell each job would need it's own project, deployed with it's own pipeline.
Is there a nicer way of doing this, are Webjobs even the right tool for this job, given the seemingly simple task we're performing.
As far as I understand your use case, you can use Azure Function App Timer Trigger to accomplish it.
A timer trigger lets you run a function on a schedule.
The following example shows a C# function that is executed each time the minutes have a value divisible by five (eg if the function starts at 18:57:00, the next performance will be at 19:00:00). The TimerInfo object is passed into the function.
[FunctionName("TimerTriggerCSharp")]
public static void Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, ILogger log)
{
if (myTimer.IsPastDue)
{
log.LogInformation("Timer is running late!");
}
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
}
The attribute's constructor takes a CRON expression or a TimeSpan. You can use TimeSpan only if the function app is running on an App Service plan. TimeSpan is not supported for Consumption or Elastic Premium Functions.
CRON (NCRONTAB expressions)
I have developed .Net Core 2.2 Azure Web Job project which is having multiple timer trigger functions. One function is run on every 15 minutes and other one is run on every 10 minutes.
public async Task GetRecordsFromCosmosDBCollection_0([TimerTrigger("0 0/15 * * * *")]TimerInfo timerInfo){
//Custom Business Logic
}
public async Task GetRecordsFromCosmosDBCollection_1([TimerTrigger("0 0/10 * * * *")]TimerInfo timerInfo){
//Custom Business Logic
}
If I used the CRON expression directly in the function parameters then it works as expected. But I want to read the CRON expression information from appsettings.json file and then pass it to the above two functions.
So, can anyone suggest the right approach of reading the CRON expression information from appsettings.json in Functions.cs file in Azure WebJob project.
I assume the motivation is to get the schedule out of the compiled code where it can be changed without having to re-compile and re-deploy.
This might not be the most elegant solution - but you could put your two functions into two different scheduled WebJobs. Then you could set a separate settings.job for each one.
https://learn.microsoft.com/en-us/azure/app-service/webjobs-create#CreateScheduledCRON
The external settings.job file is deployed to the folder where the WebJob exe is - typically D:\home\site\wwwroot\App_Data\jobs\triggered\WebJobName - and you could change the schedule there.
To have different schedules - you'd have to split into different WebJobs because the settings.job schedule is kicking off Main, as opposed to a specific function like [TimerTrigger].
In a scheduled job, the code in Main would look like this:
await host.StartAsync();
await jobHost.CallAsync("ManualTrigger", inputs);
await host.StopAsync();
where "ManualTrigger" is the function in Functions.cs. The schedule in settings.jobs kicks off Main, runs the function, and then shuts down.
I am working on an asp.net mvc 5 web application , deployed inside IIS-8, and i have a method inside my application to perform a long running task which mainly scans our network for servers & VMs and update our database with the scan results. method execution might last between 30-40 minutes to complete on production environment. and i am using a schedule tool named Hangfire which will call this method 2 times a day.
here is the job definition inside the startup.cs file, which will call the method at 8:01 am & 8:01 pm:-
public void Configuration(IAppBuilder app)
{
var options = new SqlServerStorageOptions
{
PrepareSchemaIfNecessary = false
};
GlobalConfiguration.Configuration.UseSqlServerStorage("scanservice",options);
RecurringJob.AddOrUpdate(() => ss.Scan(), "01 8,20 ***");
}
and here is the method which is being called twice a day by the schedule tool:-
public void Scan()
{
Service ss = new Service();
ss.NetworkScan().Wait();
}
Finally the method which do the real scan is (i only provide a high level description of what the method will do):-
public async Task<ScanResult> NetworkScan()
{
// retrieve the server info from the DB
// loop over all servers & then execute some power shell commands to scan the network & retrieve the info for each server one by one...
// after the shell command completed for each server, i will update the related server info inside the DB
currently i did some tests on our test environment and every thing worked well ,, where the scan took around 25 seconds to scan 2 test servers.but now we are planning to move the application to production and we have around 120++ servers to scan. so i estimate the method execution to take around 30 -40 minutes to complete on the production environment. so my question is how i can make sure that this execution will never expire , and the ScanNetwork() method will complete till the end?
Instead of worrying about your task timing out, perhaps you could start a new task for each server. In this way each task will be very short lived, and any exceptions caused by scanning a single server will not effect all the others. Additionally, if your application is restarted in IIS any scans which were not yet completed will be resumed. With all scans happening in one sequential task this is not possible. You will likely also see the total time to complete a scan of your entire network plummet, as the majority of time would likely be spent waiting on remote servers.
public void Scan()
{
Service ss = new Service();
foreach (var server in ss.GetServers())
{
BackgroundJob.Enqueue<Service>(s => s.ServerScan(server));
}
}
Now your scheduled task will simply enqueue one new task for each server.
We have configured Hangfire to run as part of our web app using OWIN as provided in the tutorial.
We enqueue long running background process via an API we provide. The job we run initializes a R process in the background using the.Net Process class. The R code internally spawns a number of processes internally to finish the job faster. We get a number of Rscript process running in Task manager when the job runs.
On manually Recycling the app pool of our web app(to see how process restart works) the Rscript Processes are not killed. We have a custom kill strategy we to get rid of all the Rscript process within our code.
while (IsNotTimedOut())
{
try
{
_token.ThrowIfCancellationRequested();
Thread.Sleep(2000);
}
catch (OperationCanceledException)
{
Kill();
throw;
}
}
Within the kill method, we block using Process.WaitForExit() method.
When we do the manual recycle all the process are not killed. The current thread instead of blocking for the process to kill just dies after killing a couple of Rscript processes.
The hangfire code seems to just cancel the token and it doesn't seem to wait for the process to get killed that are listening on the cancellation token. Please, can someone suggest how do we get this working, Please let me know if more details are required?