My Node.js application accepts connections from the outside. Each connection handler reads a SET on Redis, eventually modifies the set itself, then moves on. The problem is that in the meanwhile another async connection can try to read the same SET and try to update it or decide its next step based on what it reads.
I know that Redis does its best to be atomic, but this is not quite sufficient for my use case. Think about this: the set is read to understand if it's FULL (there is a business rule for that). If it's FULL, then something happens. The problem is that if there is one only slot left, two semi-concurrent connections could think each one is the last one. And I get an overflow.
I there a way to keep a connection "waiting" for the very short time the other eventually needs to update the set state?
I think this is a corner case, very very unluckely... but you know :)
Using another key as the "lock" is an option, or does it stink?
How about using blpop to do locking. blpop key 5 to wait 5 seconds for key. At start put item(to identify queue is not empty) at key. The connection acquiring the lock should remove item from key. The next connect then can't acquire the lock, because empty, but blpop has the following nice property:
Multiple clients can block for the same key. They are put into a
queue, so the first to be served will be the one that started to wait
earlier, in a first-BLPOP first-served fashion.
When connection which acquired lock has finished task it should put back item back in queue, then the next connection waiting can acquire lock(item).
You may be looking for WATCH with MULTI/EXEC. Here's the pattern that both threads follow:
WATCH sentinel_key
GET value_of_interest
if (value_of_interest = FULL)
MULTI
SET sentinel_key = foo
EXEC
if (EXEC returned 1, i.e. succeeded)
do_something();
else
do_nothing();
else
UNWATCH
The way this works is that all of the commands between MULTI and EXEC are queued up but not actually executed until EXEC is called. When EXEC is called, before actually executing the queued instructions it checks to see if sentinel_key has changed at all since the WATCH was set; if it has, it returns (nil) and the queued commands are discarded. Otherwise the commands are executed atomically as a block, and it returns the number of commands executed (1 in this case), letting you know you won the race and do_something() can be called.
It's conceptually similar to the fork()/exec() Unix system calls - the return value from fork() tells you which process you are (parent or child). In this case it tells you whether you won the race or not.
Related
My painful hunt for this feature is fully described in disgustingly log question: Several last offsets aren't getting commited with reactive kafka and it shows my multiple attemps with different failures.
How would one subscribe to ReactiveKafkaConsumerTemplate<String, String>, which will process the records in synchronous way (for simplicity), and will ack/commit every 2s AND upon manual cancellation of stream? Ie. it works, ack/commits every 2s. Then via rest/jmx/whatever comes signal, the stream terminates and ack/commits the last processed kafka record.
After a lot of attempts I was able to come up with following solution. It seems to work, but it's kinda ugly, because it's very "white-box" where outer flow highly depends on stuff happening inside of other methods. Please criticise and suggest improvements. Thanks.
kafkaReceiver.receive()
.flatMapSequential(receivedKafkaRecord -> processKafkaRecord(receivedKafkaRecord), 16)
.takeWhile(e-> !stopped)
.sample(configuration.getKafkaConfiguration().getCommitInterval())
.concatMap(offset -> {
log.debug("ack/commit offset {}", offset.offset());
offset.acknowledge();
return offset.commit();
})
.doOnTerminate(()-> log.info("stopped."));
What didn't work:
A) you cannot use Disposable.dispose, since that would break the stream and your latest processed record won't be committed.
B) you cannot put take on top of stream, as that would cancel the stream and you won't be able to commit either.
C) not sure how I'd be able to intercorporate usage of errors here.
Because of what didn't work stream termination is triggered by boolean field named stopped, which can be set anyhow.
Flow explained:
flatMapSequential — because of inner parallelism and necessity to commit N only if all N-1 was processed.
processKafkaRecord returns Mono<ReceiverOffset>, ie. the offset of processed record to have something to ack/commit. When stopped the method will skip processing and return Mono.empty
take will stop stream if stopped, this has to be put here becaue of possibility of whole sample interval consisting only from "empties"
rest is simple: sample by given interval, commit in order. If sample does return empty record, commit is skipped. Finally we log that stream is cancelled.
If anyone know how to improve, please criticise.
I need some help understanding transaction scoping for procedures/programs outside the current program.
Suppose I've three program, program A, program B and program C. Inside program A, I've a procedure that has some lines in it wrapped inside a do transaction (not strongly typed) block. Within that do transaction block, it calls another Program B. Upon return from program B there is an undo, leave command. Within the same transaction block, it calls program C and has an undo, leave after this call too.
My question is, if within the transaction block, program B executes without errors, but program c returned an error, will the undo,leave after program C call will also undo transactions that happened inside program B?
Procedure do_something:
some processing....
do transaction:
error-message = "".
{run programB.p}
if error-message <> "" then undo, leave.
some further processing...
error-message = "".
{run programC.p}
if error-message <> "" then undo, leave.
end. /* end of do transaction */
end procedure.
Yes. In the example that you describe everything gets rolled back.
It is not so much that it is "extended" per se but just that the transaction includes everything that happens in that session from the point in time when it is enabled all the way until it is either committed or rolled back. Internal procedures, external procedures, user defined functions, methods of classes, trigger code etc.
"In that session" is important - if you call a procedure on an app server that activity is NOT included since it is its own process with its own distinct transaction context.
When app servers are involved things get messy. The original caller has no (built-in) capability to know what to roll back in the called app server session. The app server call could return an error that causes the caller to roll back if it encounters problems but the caller could also decide to trap and ignore that error.
Yes. Everything happening in the transaction block will be undone.
How do I create PL/SQL function which waits for update on some row for specified timeout and then returns.
What I want to accomplish is - I have long running process which will update it's status to ASYNC_PROCESS table by process_id. I need function which returns with true/false when this process has completed, but also I need this function to wait some time for this process complete, return on timeout or return imediately with true, when process has completed. I don't want to use sleep(1 sec), because in such case I will be having 1 sec lag. I don't want to use sleep(1 msec), because in such case I am spending cpu resources (and 1msec lag).
Is there a good way how experienced programmer would accomplish this?
That function will be called from .NET (So I need minimal lag between DB operation and .NET/UI)
THNX,
Beef
I think the most sensible thing to do in this case is to use update triggers on that ASYNC_PROCESS table.
You should also look into the DBMS_ALERT package. Here's an edited excerpt from that doc:
Create an alert:
DBMS_ALERT.REGISTER('emp_table_alert');
Create a trigger on your table to fire the alert:
CREATE TRIGGER emptrig AFTER INSERT ON emp
BEGIN
DBMS_ALERT.SIGNAL('emp_table_alert', 'message_text');
END;
From your .net code, you can the use something that calls this:
DBMS_ALERT.WAITONE('emp_table_alert', :message, :status, :timeout);
Make sure you read the docs for what :status and :timeout do.
You should look at Oracle Advanced Queuing. It offers the kind of functions your looking for.
You'll probably need a separate queue table where a trigger on ASYNC_PROCESS inserts messages. You then use the AQ functions to retrieve (or wait for) the next message in the queue table.
If you're doing this in C#.NET, why wouldn't you simply spawn a worker thread to do the update (via ODAC)? Why hand the responsibility over to Oracle when (it seems) you want a .NET process to make the update call (in background) and have the main process be notified of its completion.
See here and here for examples, although there are several approaches in .NET for this (delegates, events, async callbacks, thread pools, etc)
In the main thread I open a new thread that gets the number of new messages of user (takes about 5 secs) and this second thread should save the number in some place.
In the main thread I should check the "some place" and if the value exists I display it on the page.
Where can I save the value from the second thread to read it from the main one? This value is unique per user so I can't use static field.
Thank you for advance!
You can use static dictionary with user id as key and result as value. Protect dictionary access with locks. After main thread reads value, you can clear it from dictionary.
Use critical section to protect access to some data when several threads can read/write it. Use singleton instance to store data, global variable, registry pattern or whatever.
The way I do it, i have a vector od "ThreadData" elements.
Each started thread gets this element when started and it can update that data (protected by mutexes).
The main thread simply checks some flag in the element (ThreadState -- Running, Idle, Stopped, etc) and read the other data which the thread updated.
I have a system set up to lock certain content in a database table so only one user can edit that content at a time. Easy enough and that part is working fine. But now I'm at a road block of how to send a request to "unlock" the content. I have the stored procedure to unlock the content, but how/where would I call it when the user just closes their browser?
You also can't know when the user turns off his computer. You have to do it the other way around.
Require that the lock be renewed periodically. Only the web site would do the periodic renewal. If the user stops using the web site, then the lock expires.
Otherwise, require the user to explicitly unlock the content. Other users who want to edit the content can then go yell at the first user when they can't do their jobs. Not a technological solution, but still a good one. Shame works.
The best thing you can really do is add something to your Session_End in your global.asax. Unfortunately, this won't fire until the session times out.
When the user clicks the "X" in their browser, there isn't anyway to guarantee the browser will send you anything back.
A quick note on the Session_End approaches. If you use this method, then you have to ensure
That sessionstate is InProc, eg. add something like this to your Web.config
<sessionState mode="InProc" timeout="timeout_in_minutes"/>
Make sure that you've setup IIS as to not recycle worker processes during normal operation (see for instance this blog post).
Edit:
Not directly answering the question directly, but another approach would be to use Optimistic concurrency control on the data in question.
There is such event as "user closes browser".
Nevertheless, I can think of two workarounds:
Use Javascript/Ajax to permanently
(lets say every 10 seconds) call a
method in your page. The DateTime of
your last query needs to be stored
somewhere. Now you write a windows
service that checks every second
which session are timed out. Perform
your custom action there.
Use the global.asax Session_End()
-Event. (cannot be used with every SessionState, look up for which ones
it is usable)
Trying to leave a stackoverflow answer page pops up an "are you sure" dialog. Perhaps during the on-page-leave event that SO uses (or however SO does this), you can send a final request with an XmlHttpRequest object. This won't cover if the browser process closes unexpectedly (use session_onend for that), but it will at least send the "I'm closed" event earlier
I think your one stored procedure can do the locking and unlocking (used with "Select #strNewMax As NewMax")...
Here is an example from a system I have:
Declare #strNewMax Char
Select #strNewMax = 'N'
BEGIN TRANSACTION
/* Lock only the rows for this Item ID, and hold those locks throughout the transaction. */
If #BidAmount > (Select Max(AB_Bid_AMT) from AuctionBid With(updlock, holdlock) Where AB_AI_ID = #AuctionItemId)
Begin
Insert Into AuctionBid (AB_AI_ID, AB_Bid_AMT, AB_Emp_ID, AB_Entry_DTM)
Select #AuctionItemId, #BidAmount, #EmployeeId, GetDate()
Select #strNewMax = 'Y'
End
COMMIT TRANSACTION
Select #strNewMax As NewMax
This will insert a record as the next highest bid, all while locking the entire table, so no other bids are processed at the same time. It will return either a 'Y' or 'N' depending on if it worked or not.
Maybe you can take this and adjust it to fit your application.