I'm trying to test that a job is batching the correct number of another job like so
Bus::fake();
...
(new App\Jobs\ProcessChunkOfProducts($this->productsAsChunk()))->handle();
Bus::assertBatched( function (PendingBatch $batch) {
return $batch->jobs->count() === 49;
});
It passes most the time, but occasionally fails. Is it possible that I've created some sort of race condition where the jobs aren't batched before the assertion is made?
Related
Keeping a cron job pub/sub function (functions.pubsub.schedule), within a cloud function (functions.https.OnRequest) and exporting it, does not execute.
A complete example is as follows:
export const sayHelloWhen = functions.https.onRequest((request, response) => {
cors(request, response, () => {
const scheduleExpression = request.body.data.scheduleExpression;
functions.logger.log(`Called sayHelloWhen with ${scheduleExpression}`);
functions.pubsub.schedule(scheduleExpression).onRun((context) => {
functions.logger.log(`Executed sayHelloWhen with ${scheduleExpression}`)
});
response.send({
status: "success",
data: `scheduled at ${scheduleExpression}`
})
})
})
The problem is pub/sub does not trigger. Other codes are executed.
I would like to have HTTP request body scheduleExpression bring into pubsub.schedule's parameter. I don't want a static schedule expression in corn job.
In client, I would like to define a schedule expression in client side as follows:
function scheduleFunction() {
const functions = getFunctions();
const sayHello = httpsCallable(functions, "sayHelloWhen");
sayHello({ scheduleExpression: "every 1 minute" }).then((result) => {
// const data = result.data;
console.log("Result:", result);
});
}
The example below works only for a static schedule expression, meaning that a cloud function itself has a fixed schedule expression:
exports.scheduledFunction = functions.pubsub.schedule('every 5 minutes').onRun((context) => {
console.log('This will be run every 5 minutes!');
return null;
});
It can be exported as cron job trigger and it executes.
But keeping pub/sub cron job function, within onRequest cloud function, as in the first code example, does not execute.
I think it is very interesting what you are trying to do, but I would like to point out that you are missing some steps that could cause your app not to execute the way you need it to.
First, you need to Terminate your HTTP functions
Always end an HTTP function with send(), redirect(), or end(). Otherwise, your function might continue to run and be forcibly terminated by the system. See also Sync, Async and Promises.
When you do not terminate them, you might end up in a deeper level in which the following code will not execute unless the previous code has finished. I would also like to say I have not found any application with nested functions like you are doing.
In the Sync, async, and promises page you can find a very explicative video to understand the lifecycle of your functions, and in the How promises work with functions section we have:
When you return a JavaScript promise to a function, that function keeps running until the promise is resolved or rejected. To indicate that a function has completed its work successfully, the promise should be resolved. To indicate an error, the promise should be rejected. This means you only need to handle errors that you want to.
In addition to all this, I would suggest using a separate file to Organize multiple functions for a more organized code, but this is only a suggestion not really necessary.
Finally, I am concerned about the scheduledExpression parameter, I think if none of the above works, you might want to check and share what this value is.
I'm trying to solve some kind of task route assigning problem with Optaplanner. The task on a user's way can be of a different type. If there occurs a task with the type lunch_delivery then there needs to be a task lunch_pick before it (lunch_pick task always exists in the list of all the tasks).
I'm wondering if I can somehow connect these two tasks as it is necessary that the same user does both the lunch_pick and the lunch_delivery task (and not that one user would pick the lunch and the other would deliver it).
The current code that doesn't provide a good solution is that the task of type lunch_pick has a parameter linked_task with an ID of the lunch_delivery task. Code in constraint section looks like this:
Constraint lunchPickConflict(ConstraintFactory factory) {
return factory.from(Task.class)
.filter(task -> task.getVisitType().equals("lunch_pick"))
.penalize("lunch pick not scheduled good",
HardMediumSoftScore.ONE_MEDIUM,
task -> task.lunchConflictWeight());
}
The lunchConflictWeight function is simple for now and is defined as:
public int lunchConflictWeight(){
int score = 0;
// we want that lunch pick and lunch delivery are close to each other and that delivery is after pick-up
Task nextTask = getNextTask();
while (nextTask != null){
if (nextTask.visitType.equals("lunch_delivery") && nextTask.getId() == linked_task){
// this is what we want - we found a task that is connected with our lunch_pick
break;
}
else if(nextTask.visitType.equals("normal")){
score +=3;
}
nextTask = nextTask.getNextTask();
}
if (nextTask == null){
// the lunch pick was the last task or the delivery is done by some other user -> big penal
score = 100;
}
return score;
}
What would be a better solution for this problem? Can I somehow force that lunch_pick and lunch_delivery would be done by the same user? When I see my results I can easily see that just swapping two tasks between users would bring significant improvement in the score.
+extra question: Is there a way that one constraint would penalize hard in some cases and medium in others?
A) Usually, we don't assign lunch tasks. Instead, we get a shadow variable to delay the start time of the first task after noon by 1 hour. I think I cover it in the shadow vars video.
B) If lunch and lunch_pick are always assigned sequentially to the same user, why not just make them one task?
So I have a cloud function that is triggered each time a transaction is liked/unliked. This function increments/decrements the likesCount. I've used firestore transactions to achieve the same. I think the problem is the Code inside the Transaction block is getting executed multiple times, which may be correct as per the documentation.
But my Likes count are being updated incorrectly at certain times.
return firestore.runTransaction(function (transaction) {
return transaction.get(transRef).then(function (transDoc) {
let currentLikesCount = transDoc.get("likesCount");
if (event.data && !event.data.previous) {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 1 : transDoc.get("likesCount") + 1;
} else {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 0 : transDoc.get("likesCount") - 1;
}
transaction.update(transRef, { likesCount: newLikesCount });
});
});
Anyone had similar experience
Guys finally found out the cause for this unexpected behaviour.
Firestore isn't suitable for maintaining counters if your application is going to be traffic intensive. They have mentioned it in their documentation. The solution they suggest is to use a Distributed counter.
Many realtime apps have documents that act as counters. For example,
you might count 'likes' on a post, or 'favorites' of a specific item.
In Cloud Firestore, you can only update a single document about once
per second, which might be too low for some high-traffic applications.
https://cloud.google.com/firestore/docs/solutions/counters
I wasn't convinced with that approach as it's too complex for a simple use case, which is when I stumbled across the following blog
https://medium.com/evenbit/on-collision-course-with-cloud-firestore-7af26242bc2d
These guys used a combination of Firestore + Firebase thereby eliminating their weaknesses.
Cloud Firestore is sitting conveniently close to the Firebase Realtime
Database, and the two are easily available to use, mix and match
within an application. You can freely choose to store data in both
places for your project, if that serves your needs.
So, why not use the Realtime database for one of its strengths: to
manage fast data streams from distributed clients. Which is the one
problem that arises when trying to aggregate and count data in the
Firestore.
Its not correct to say that Firestore is an upgrade to the Realtime database (as it is advertised) but a different database with different purposes and both can and should coexist in a large scale application. That's my thought.
It might have something to do with what you're returning from the function, as you have
return transaction.get(transRef).then(function (transDoc) { ... })
And then another return inside that callback, but no return inside the inner-most nested callback. So it might not be executing the transaction.update. Try removing the first two return keywords and add one before transaction.update:
firestore.runTransaction(function (transaction) {
transaction.get(transRef).then(function (transDoc) {
let currentLikesCount = transDoc.get("likesCount");
if (event.data && !event.data.previous) {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 1 : transDoc.get("likesCount") + 1;
} else {
newLikesCount = currentLikesCount == 0 || isNaN(currentLikesCount) ? 0 : transDoc.get("likesCount") - 1;
}
return transaction.update(transRef, { likesCount: newLikesCount });
});
});
Timeouts
First of all, check your Cloud Functions logs to see if you get any timeout messages.
Function execution took 60087 ms, finished with status: 'timeout'
If so, sort out your function so that it returns a Promise.resolve(). And shows
Function execution took 344 ms, finished with status: 'ok'
Idempotency
Secondly, write your data so that the function is idempotent. When your function runs, write a value to the document that you are reading. You can then check if that value exists before running the function again.
See this example for ensuring that functions are only run once.
In the kotlinx.coroutines library you can start new coroutine using either launch (with join) or async (with await). What is the difference between them?
launch is used to fire and forget coroutine. It is like starting a new thread. If the code inside the launch terminates with exception, then it is treated like uncaught exception in a thread -- usually printed to stderr in backend JVM applications and crashes Android applications. join is used to wait for completion of the launched coroutine and it does not propagate its exception. However, a crashed child coroutine cancels its parent with the corresponding exception, too.
async is used to start a coroutine that computes some result. The result is represented by an instance of Deferred and you must use await on it. An uncaught exception inside the async code is stored inside the resulting Deferred and is not delivered anywhere else, it will get silently dropped unless processed. You MUST NOT forget about the coroutine you’ve started with async.
I find this guide to be useful. I will quote the essential parts.
🦄 Coroutines
Essentially, coroutines are light-weight threads.
So you can think of a coroutine as something that manages thread in a very efficient way.
🐤 launch
fun main(args: Array<String>) {
launch { // launch new coroutine in background and continue
delay(1000L) // non-blocking delay for 1 second (default time unit is ms)
println("World!") // print after delay
}
println("Hello,") // main thread continues while coroutine is delayed
Thread.sleep(2000L) // block main thread for 2 seconds to keep JVM alive
}
So launch starts a coroutine, does something, and returns a token immediately as Job. You can call join on this Job to block until this launch coroutine completes.
fun main(args: Array<String>) = runBlocking<Unit> {
val job = launch { // launch new coroutine and keep a reference to its Job
delay(1000L)
println("World!")
}
println("Hello,")
job.join() // wait until child coroutine completes
}
🦆 async
Conceptually, async is just like launch. It starts a separate coroutine which is a light-weight thread that works concurrently with all the other coroutines. The difference is that launch returns a Job and does not carry any resulting value, while async returns a Deferred -- a light-weight non-blocking future that represents a promise to provide a result later.
So async starts a background thread, does something, and returns a token immediately as Deferred.
fun main(args: Array<String>) = runBlocking<Unit> {
val time = measureTimeMillis {
val one = async { doSomethingUsefulOne() }
val two = async { doSomethingUsefulTwo() }
println("The answer is ${one.await() + two.await()}")
}
println("Completed in $time ms")
}
You can use .await() on a deferred value to get its eventual result, but Deferred is also a Job, so you can cancel it if needed.
So Deferred is actually a Job. Read this for more details.
interface Deferred<out T> : Job (source)
🦋 async is eager by default
There is a laziness option to async using an optional start parameter with a value of CoroutineStart.LAZY. It starts coroutine only when its result is needed by some await or if a start function is invoked.
launch and async are used to start new coroutines. But, they execute them in different manner.
I would like to show very basic example which will help you understand difference very easily
launch
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
btnCount.setOnClickListener {
pgBar.visibility = View.VISIBLE
CoroutineScope(Dispatchers.Main).launch {
val currentMillis = System.currentTimeMillis()
val retVal1 = downloadTask1()
val retVal2 = downloadTask2()
val retVal3 = downloadTask3()
Toast.makeText(applicationContext, "All tasks downloaded! ${retVal1}, ${retVal2}, ${retVal3} in ${(System.currentTimeMillis() - currentMillis)/1000} seconds", Toast.LENGTH_LONG).show();
pgBar.visibility = View.GONE
}
}
// Task 1 will take 5 seconds to complete download
private suspend fun downloadTask1() : String {
kotlinx.coroutines.delay(5000);
return "Complete";
}
// Task 1 will take 8 seconds to complete download
private suspend fun downloadTask2() : Int {
kotlinx.coroutines.delay(8000);
return 100;
}
// Task 1 will take 5 seconds to complete download
private suspend fun downloadTask3() : Float {
kotlinx.coroutines.delay(5000);
return 4.0f;
}
}
In this example, my code is downloading 3 data on click of btnCount button and showing pgBar progress bar until all download gets completed. There are 3 suspend functions downloadTask1(), downloadTask2() and downloadTask3() which downloads data. To simulate it, I've used delay() in these functions. These functions waits for 5 seconds, 8 seconds and 5 seconds respectively.
As we've used launch for starting these suspend functions, launch will execute them sequentially (one-by-one). This means that, downloadTask2() would start after downloadTask1() gets completed and downloadTask3() would start only after downloadTask2() gets completed.
As in output screenshot Toast, total execution time to complete all 3 downloads would lead to 5 seconds + 8 seconds + 5 seconds = 18 seconds with launch
async
As we saw that launch makes execution sequentially for all 3 tasks. The time to complete all tasks was 18 seconds.
If those tasks are independent and if they do not need other task's computation result, we can make them run concurrently. They would start at same time and run concurrently in background. This can be done with async.
async returns an instance of Deffered<T> type, where T is type of data our suspend function returns. For example,
downloadTask1() would return Deferred<String> as String is return type of function
downloadTask2() would return Deferred<Int> as Int is return type of function
downloadTask3() would return Deferred<Float> as Float is return type of function
We can use the return object from async of type Deferred<T> to get the returned value in T type. That can be done with await() call. Check below code for example
btnCount.setOnClickListener {
pgBar.visibility = View.VISIBLE
CoroutineScope(Dispatchers.Main).launch {
val currentMillis = System.currentTimeMillis()
val retVal1 = async(Dispatchers.IO) { downloadTask1() }
val retVal2 = async(Dispatchers.IO) { downloadTask2() }
val retVal3 = async(Dispatchers.IO) { downloadTask3() }
Toast.makeText(applicationContext, "All tasks downloaded! ${retVal1.await()}, ${retVal2.await()}, ${retVal3.await()} in ${(System.currentTimeMillis() - currentMillis)/1000} seconds", Toast.LENGTH_LONG).show();
pgBar.visibility = View.GONE
}
This way, we've launched all 3 tasks concurrently. So, my total execution time to complete would be only 8 seconds which is time for downloadTask2() as it is largest of all of 3 tasks. You can see this in following screenshot in Toast message
both coroutine builders namely launch and async are basically lambdas with receiver of type CoroutineScope which means their inner block is compiled as a suspend function, hence they both run in an asynchronous mode AND they both will execute their block sequentially.
The difference between launch and async is that they enable two different possibilities. The launch builder returns a Job however the async function will return a Deferred object. You can use launch to execute a block that you don't expect any returned value from it i.e writing to a database or saving a file or processing something basically just called for its side effect. On the other hand async which return a Deferred as I stated previously returns a useful value from the execution of its block, an object that wraps your data, so you can use it for mainly its result but possibly for its side effect as well. N.B: you can strip the deferred and get its value using the function await, which will block the execution of your statements until a value is returned or an exceptions is thrown! You could achieve the same thing with launch by using the function join()
both coroutine builder (launch and async) are cancelable.
anything more?: yep with launch if an exception is thrown within its block, the coroutine is automatically canceled and the exceptions is delivered. On the other hand, if that happens with async the exception is not propagated further and should be caught/handled within the returned Deferred object.
more on coroutines https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html
https://www.codementor.io/blog/kotlin-coroutines-6n53p8cbn1
Async and Launch, both are used to create coroutines that run in the background. In almost every situation one can use either of them.
tl;dr version:
When you dont care about the task's return value, and just want to run it, you may use Launch. If you need the return type from the task/coroutine you should use async.
Alternate:
However, I feel the above difference/approach is a consequence of thinking in terms of Java/one thread per request model. Coroutines are so inexpensive, that if you want to do something from the return value of some task/coroutine(lets say a service call) you are better off creating a new coroutine from that one. If you want a coroutine to wait for another coroutine to transfer some data, I would recommend using channels and not the return value from Deferred object. Using channels and creating as much number of coroutines as required, is the better way IMO
Detailed answer:
The only difference is in the return type and what functionality it provides.
Launch returns Job while Async returns Deferred. Interestingly enough, Deferred extends Job. Which implies it must be providing additional functionality on top of Job. Deferred is type parameterised over where T is the return type. Thus, Deferred object can return some response from the block of code executed by async method.
p.s. I only wrote this answer because I saw some factually incorrect answers on this question and wanted to clarify the concept for everyone. Also, while working on a pet project myself I faced similar problem because of previous Java background.
launch returns a job
async returns a result (deferred job)
launch with join is used to wait until the job gets finished. It simply suspends the coroutine calling join(), leaving the current thread free to do other work (like executing another coroutine) in the meantime.
async is used to compute some results. It creates a coroutine and returns its future result as an implementation of Deferred. The running coroutine is cancelled when the resulting deferred is cancelled.
Consider an async method that returns a string value. If the async method is used without await it will return a Deferred string but if await is used you will get a string as the result
The key difference between async and launch:
Deferred returns a particular value of type T after your Coroutine finishes executing, whereas Job doesn’t.
launch / async no result
Use when don't need the result,
Don't block the code where is called,
Run in sequential
async for result
When you need to wait for the result and can run in parallel for
efficiency,
Block the code where is called,
Run in concurrent
Alongside the other great answers, for the people familiar with Rx and getting into coroutines, async returns a Deferred which is akin to Single while launch returns a Job that is more akin to Completable. You can .await() to block and get the value of the first one, and .join() to block until the Job is completed.
We're using Firebase DB together with RxSwift and are running into problems with transactions. I don't think they're related to the combination with RxSwift but that's our context.
Im observing a data in Firebase DB for any value changes:
let child = dbReference.child(uniqueId)
let dbObserverHandle = child.observe(.value, with: { snapshot -> () in
guard snapshot.exists() else {
log.error("empty snapshot - child not found in database")
observer.onError(FirebaseDatabaseConsumerError(type: .notFound))
return
}
//more checks
...
//read the data into our object
...
//finally send the object as Rx event
observer.onNext(parsedObject)
}, withCancel: { _ in
log.error("could not read from database")
observer.onError(FirebaseDatabaseConsumerError(type: .databaseFailure))
})
No problems with this alone. Data is read and observed without any problems. Changes in data are propagated as they should.
Problems occur as soon as another part of the application modifies the data that is observer with a transaction:
dbReference.runTransactionBlock({ (currentData: FIRMutableData) -> FIRTransactionResult in
log.debug("begin transaction to modify the observed data")
guard var ourData = currentData.value as? [String : AnyObject] else {
//seems to be nil data because data is not available yet, retry as stated in the transaction example https://firebase.google.com/docs/database/ios/read-and-write
return TransactionResult.success(withValue: currentData)
}
...
//read and modify data during the transaction
...
log.debug("complete transaction")
return FIRTransactionResult.success(withValue: currentData)
}) { error, committed, _ in
if committed {
log.debug("transaction commited")
observer(.completed)
} else {
let error = error ?? FirebaseDatabaseConsumerError(type: .databaseFailure)
log.error("transaction failed - \(error)")
observer(.error(error))
}
}
The transaction receives nil data at first try (which is something you should be able to handle. We just just call
return TransactionResult.success(withValue: currentData)
in that case.
But this is propagated to the observer described above. The observer runs into the "empty snapshot - child not found in database" case because it receives an empty snapshot.
The transaction is run again, updates the data and commits successfully. And the observer receives another update with the updated data and everything is fine again.
My questions:
Is there any better way to handle the nil-data during the transaction than writing it to the database with FIRTransactionResult.success
This seems to be the only way to complete this transaction run and trigger a re-run with fresh data but maybe I'm missing something-
Why are we receiving the empty currentData at all? The data is obviously there because it's observed.
The transactions seem to be unusable with that behavior if it triggers a 'temporary delete' to all observers of that data.
Update
Gave up and restructured the data to get rid of the necessity to use transactions. With a different datastructure we were able to update the dataset concurrently without risking data corruption.