I have an array of payloads I want perform a couple async actions on each and then send to a server. I'm trying to do it like this:
Rx.Observable.from(payloads)
.flatMap(item => ... do something A something async, return modified payload)
.flatMap(item => ... do something B)
.flatMap(item => ... post to server)
.subscribe();
This works....
But it seems to do A for each payload all at once, then B for each payload all at once, then post each payload all at once.
With lists of potentially thousands of payloads. Is there any way to go through the payloads one at a time, one step at a time?
If they need to be done a->b->c for each payload:
Rx.Observable.from(payloads)
.concatMap(payload => doA())
.concatMap(resA => doB())
.concatMap(resB => doC())
.subscribe(console.log)
concatMap will wait until the previous emission completes before starting the next. This will limit your concurrency.
Related
I'm using Firebase Functions with https triggers, and I was wondering how long after sending the response to the client, the functions keeps executing. I want to send a response to the client and then perform another operation (send a mail).
Currently I'm doing this as following:
module.exports.doSomeJob = functions.https.onRequest((req, res) => {
doSomeAsyncJob()
.then(() => {
res.send("Ok");
})
.then(() => {
emailSender.sendEmail();
})
.catch(...);
});
The above code is working for me, but I'm suspecting that the code only works because sending the mail has finished before the res.send has completed, so I was wondering how exactly the termination process is working to make sure the code will not break.
You should expect that the HTTP function terminates the moment after you send the response. Any other behavior is some combination of luck or a race condition. Don't write code that depends on luck.
If you need to send a response to the client before the work is fully complete, you will need to kick off a second function to continue where the HTTP function left off. It's common to use a pub/sub function to do with. Have the HTTP function send a pub/sub message to another function, then terminate the HTTP function (by sending a response) only after the message is sent.
If the expected response is not pegged to the outcome of the execution, then you can use
module.exports.doSomeJob = functions.https.onRequest((req, res) => {
res.write('SUCCESS')
return doSomeAsyncJob()
.then(() => {
emailSender.sendEmail();
})
.then(() => {
res.end();
})
.catch(...);
});
This sends back a response a soon as a request is received, but keeps the function running until res.end() is called
Your client can end the connection as soon as a response is received back, but the cloud function will keep running in the background
Not sure if this helps, but it might be a workaround where the client needs a response within a very limited time, considering that executing pub/sub requires some extra processing on its own and takes time to execute
TL;DR
While https functions will terminate shortly after res.send(), it is not guaranteed that 0 lines of code after res.send() will be executed.
I think a fuller answer has 2 components:
as Doug pointed out, do not put any additional code you expect to be executed after res.send()
cloud functions will terminate shortly after res.send(), but don't expect that exactly 0 lines of code will be executed
I ran into a situation where for a db maintenance script, if no records met my criteria, I said goodbye with res.send() and had additional logic after it. I was expecting that piece not to be run, since I've already terminated the request.
Example producing unexpected results:
exports.someFunction = functions.https.onRequest((req, res) => {
if (exitCriteria === true) {
// we can exit the function, nothing to do
console.log('Exit criteria met')
res.status(200).send()
}
// code to handle if someCriteria was falsy
console.log('Exit criteria not met, continue executing code')
})
In the above example, I was expecting res.send() to terminate the function immediately - this is not so, the second console.log may also be hit - along with any other code you may have. This is not guaranteed, however, so execution may abruptly stop at some point.
Example producing correct results:
exports.someFunction = functions.https.onRequest((req, res) => {
if (exitCriteria === true) {
// we can exit the function, nothing to do
console.log('Exit criteria met')
res.status(200).send()
}
else {
// code to handle if someCriteria was falsy
console.log('Exit criteria not met, continue executing code')
}
})
In this version, you will see exactly 1 line of console.logs - as I was originally intending.
I have an epic which dispatches an action that is handled by a redux middleware, which returns a promise once the action is dispatched. The epic looks approximately like this:
const myEpic = action$ =>
action$.pipe(
ofType(SAVE_ACTION),
switchMap(action => [
saveData(action.payload)
])
)
When the SAVE_ACTION is dispatched, it is picked up by the epic, which dispatches an action created by the saveAction action creator.
This resulting action is intercepted by a redux middleware (specifically redux-axios-middleware) which does an HTTP request and converts the result of dispatching such action to a promise, which has the standard behavior of resolving when the HTTP request succeeds and rejecting when the HTTP request fails.
What I would need to do is do additional work once that promise is resolved, within this epic, because I need data contained both in the payload of the initial SAVE_ACTION action and the payload of the HTTP response.
Because the middleware also dispatches actions upon completion of the underlying HTTP request, it would be easy to write an additional epic to do the additional work I need to do in a distinct epic, but I wouldn't have access to the payload of the initial SAVE_ACTION anymore, so I'm trying to figure out if this can be handled all within a single epic, and I haven't found a way so far. I have found online posts like this one, which is very informative, but still doesn't address the issue of awaiting the dispatch of an action, rather than a plain observable.
One approach would be to use merge like this:
import { merge } from 'rxjs/observable/merge';
const myEpic = action$ => {
let initialPayload = null;
return merge(
action$.pipe(
ofType(SAVE_ACTION),
switchMap(action => {
initialPayload = action.payload;
return [saveData(action.payload)]
})
),
action$.pipe(
ofType(SAVE_ACTION_SUCCESS),
mergeMap(action => {
// Use initialPayload here.
})
),
)
}
I'm using Firebase Functions with https triggers, and I was wondering how long after sending the response to the client, the functions keeps executing. I want to send a response to the client and then perform another operation (send a mail).
Currently I'm doing this as following:
module.exports.doSomeJob = functions.https.onRequest((req, res) => {
doSomeAsyncJob()
.then(() => {
res.send("Ok");
})
.then(() => {
emailSender.sendEmail();
})
.catch(...);
});
The above code is working for me, but I'm suspecting that the code only works because sending the mail has finished before the res.send has completed, so I was wondering how exactly the termination process is working to make sure the code will not break.
You should expect that the HTTP function terminates the moment after you send the response. Any other behavior is some combination of luck or a race condition. Don't write code that depends on luck.
If you need to send a response to the client before the work is fully complete, you will need to kick off a second function to continue where the HTTP function left off. It's common to use a pub/sub function to do with. Have the HTTP function send a pub/sub message to another function, then terminate the HTTP function (by sending a response) only after the message is sent.
If the expected response is not pegged to the outcome of the execution, then you can use
module.exports.doSomeJob = functions.https.onRequest((req, res) => {
res.write('SUCCESS')
return doSomeAsyncJob()
.then(() => {
emailSender.sendEmail();
})
.then(() => {
res.end();
})
.catch(...);
});
This sends back a response a soon as a request is received, but keeps the function running until res.end() is called
Your client can end the connection as soon as a response is received back, but the cloud function will keep running in the background
Not sure if this helps, but it might be a workaround where the client needs a response within a very limited time, considering that executing pub/sub requires some extra processing on its own and takes time to execute
TL;DR
While https functions will terminate shortly after res.send(), it is not guaranteed that 0 lines of code after res.send() will be executed.
I think a fuller answer has 2 components:
as Doug pointed out, do not put any additional code you expect to be executed after res.send()
cloud functions will terminate shortly after res.send(), but don't expect that exactly 0 lines of code will be executed
I ran into a situation where for a db maintenance script, if no records met my criteria, I said goodbye with res.send() and had additional logic after it. I was expecting that piece not to be run, since I've already terminated the request.
Example producing unexpected results:
exports.someFunction = functions.https.onRequest((req, res) => {
if (exitCriteria === true) {
// we can exit the function, nothing to do
console.log('Exit criteria met')
res.status(200).send()
}
// code to handle if someCriteria was falsy
console.log('Exit criteria not met, continue executing code')
})
In the above example, I was expecting res.send() to terminate the function immediately - this is not so, the second console.log may also be hit - along with any other code you may have. This is not guaranteed, however, so execution may abruptly stop at some point.
Example producing correct results:
exports.someFunction = functions.https.onRequest((req, res) => {
if (exitCriteria === true) {
// we can exit the function, nothing to do
console.log('Exit criteria met')
res.status(200).send()
}
else {
// code to handle if someCriteria was falsy
console.log('Exit criteria not met, continue executing code')
}
})
In this version, you will see exactly 1 line of console.logs - as I was originally intending.
I am working on an application whereby I we periodically persist information to a server when a user navigates between pages.
We currently do this by scheduling a "persist" action, which propagates a sequenced number of events, before finishing with a "persist_end" action. Currently, if a user navigates quickly, these grouped actions can intercept each other, resulting in various problems. I thought I could buffer the starting action and wait until the ending action was executed.
I've created a similar example using the ping-pong example from Redux-Observables site: https://codepen.io/dualcyclone/pen/GOZRxW?editors=0011
const actionPauser = new BehaviorSubject(false);
const pingEpic = action$ =>
action$.ofType(PING)
.do(action => console.log(action)) // check if action caught by epic
.buffer(actionPauser.asObservable().filter(paused => !paused))
.do(eh => console.log('buffered? ',eh)) // check if buffered actions is occurring
.do(() => actionPauser.next(true)) // tell pauser to pause
.map((buf) => buf[buf.length-1])
.filter(action => action !== undefined)
.delay(1000)
.mapTo({ type: PONG });
const pauseEpic = action$ =>
action$.ofType(PONG)
.delay(1000)
.do(() => actionPauser.next(false)) // tell pauser to not pause
.mapTo({ type: PING });
The premise is similar, I am allowing the user to press the "start PING" button as often as they like, the epic that is listening to this should check to see if there is a ping action currently occuring (by the "actionPauser" BehaviorSubject), and queue any actions until a previous ping action has completed.
The epic should emit the most recent buffered action, so it filters the buffered list then passes through the latest one.
What I can't seem to understand is - the console log to indicate how many buffered actions there are fires as soon as the page loads; which may indicate a problem with the way this is built - am I missing something?
So, whilst the output of the actions isn't exactly desirable (theres not really much I can do about it given the starting action is emitted by a user event), the suggestion Cartant recommended actually does precisely what I need.
Audit:
Ignores source values for a duration determined by another Observable, then emits the most recent value from the source Observable, then repeats this process.
In essence, this allows me to ignore multiple emitted 'PING' events whilst one is currently ongoing. It will then continue the execution of the last most recent 'PING' event, so the output we see is as follows:
(click) PING (click) PING (click) PING (click) PING PONG DONE PONG DONE
The first and last 'PING' actions are the only ones propagated through the Epic, so we see two final PONG actions, both followed by a DONE action.
So, here is the answered example (as also seen on my codepen here)
const pingEpic = action$ =>
action$.ofType(PING)
.audit(() => actionPauser.filter(paused => !paused))
.do(() => actionPauser.next(true))
.delay(1000)
.mapTo({ type: PONG });
const pauseEpic = action$ =>
action$.ofType(PONG)
.delay(1000)
.mapTo({ type: DONE })
.do(() => actionPauser.next(false));
Sounds like concatMap might work well in this case.
Projects each source value to an Observable which is merged in the output Observable, in a serialized fashion waiting for each one to complete before merging the next.
So in the case below, only one timer is running at a time. Any PINGs that come in while the previous is still waiting will be buffered.
const pingEpic = action$ =>
action$.ofType(PING)
.concatMap(() =>
Observable.timer(1000)
.mapTo({ type: PONG })
);
https://jsbin.com/gocezut/edit?js,output
(rapidly click four times)
PING
PING
PING
PING
(1000ms pass)
PONG
(1000ms pass)
PONG
(1000ms pass)
PONG
(1000ms pass)
PONG
Remember that most redux-observable questions can be reframed to just be regular RxJS questions, broadening the resources and help you can find. That's the beauty of redux-observable: it's almost entirely just regular RxJS patterns.
I have a scenario, where in when a http port error occurs, the effects gets unsubscribed, and doesn't work anymore. For Example,
#Effect() newMessages$ : Observable<any> = this.actions$
.ofType(SEND_NEW_MESSAGE_ACTION)
.switchMap(action => this.threadsService.saveNewMessage(action.payload))
.catch(() => Observable.of(new ErrorOccurredAction("Error Ocurred while saving
message")) );
lets say error occurs, and it gets displayed in in the UI in the messages section. However after that, the send new message (post) button doesn't do another post, as the effect is no longer subscribed. Any idea how to handle this gracefully?
There are basically two ways (probably more) what you could do:
1. Handle the error inside the switchMap:
#Effect() newMessages$ = this.actions$
.ofType(SEND_NEW_MESSAGE_ACTION)
.switchMap(action => this.threadsService.saveNewMessage(action.payload)
.catch(() => Observable.of(new ErrorOccurredAction("Error Ocurred while saving message")))
);
Meaning, that if there is an error in saveNewMessage, it will not affect the effect, just the single run of saveNewMessage.
2. Re-subscribe the original observable (see the catch-documentation)
#Effect() newMessages$ = this.actions$
.ofType(SEND_NEW_MESSAGE_ACTION)
.switchMap(action => this.threadsService.saveNewMessage(action.payload))
.catch((err, orig) => orig.startWith(new ErrorOccurredAction("Error Ocurred while saving message")));
What the catch here does, it will return the original stream(basically a clean version of newMessages$) for the old subscription to switch the subscription to and initially emits the ErrorOccurredAction.
Both solutions are technically valid solutions, personally I prefer the first one, because it seems to be more intuitive to read for someone who it not super-familiar with the rxjs-api. (But that's just my personal taste)