In a mariadb table with tokuDb engine; I am ecountering the below error - either on a delete statement; whilst there is a background insert load, and vice versa.
Lock wait timeout exceeded; try restarting transaction
Does tokuDb user a setting that can be updated to determine how long it waits before it timesout a statement?
I couldn't find the answer in tokuDb documents. The maria varaible is still at its default value: 'lock_wait_timeout', '31536000' -- but my timeout is coming back in quite a bit less than a year. The timeouts are coming during a load test; and I haven't spotted a time value in the error - but it feels like a few seconds; to minutes at the most before the timeout is thrown.
Thanks,
Brent
TokuDB has its own timeout variable, tokudb_lock_timeout, it is measured in milliseconds and has the default value 4000 (4 seconds), which fits your observations. It can be modified both on the session and global levels, and can also be configured in the .cnf file.
Remember that when you set a global value for a variable which has both scopes, it only affects future sessions (connections), but not the existing ones.
-- for the current session
SET SESSION tokudb_lock_timeout = 60000;
-- for future sessions
SET GLOBAL tokudb_lock_timeout = 60000;
Related
I’ve inherited a system that uses Hangfire with sql server job storage. Usually when a job is scheduled to be run immediately we notice it takes a few seconds before it’s triggered.
Looking at SQL Profiler when running in my dev environment, the SQL run against Hangfire db looks like this -
exec sp_executesql N'delete top (1) JQ
output DELETED.Id, DELETED.JobId, DELETED.Queue
from [HangFire].JobQueue JQ with (readpast, updlock, rowlock, forceseek)
where Queue in (#queues1) and (FetchedAt is null or FetchedAt < DATEADD(second, #timeout, GETUTCDATE()))',N'#queues1 nvarchar(4000),#timeout float',#queues1=N'MYQUEUENAME_master',#timeout=-1800
-- Exactly the same SQL as above is executed about 6 times/second for about 3-4 seconds,
-- then nothing for about 2 seconds, then:
exec sp_getapplock #Resource=N'HangFire:recurring-jobs:lock',#DbPrincipal=N'public',#LockMode=N'Exclusive',#LockOwner=N'Session',#LockTimeout=5000
exec sp_getapplock #Resource=N'HangFire:locks:schedulepoller',#DbPrincipal=N'public',#LockMode=N'Exclusive',#LockOwner=N'Session',#LockTimeout=5000
exec sp_executesql N'select top (#count) Value from [HangFire].[Set] with (readcommittedlock, forceseek) where [Key] = #key and Score between #from and #to order by Score',N'#count int,#key nvarchar(4000),#from float,#to float',#count=1000,#key=N'recurring-jobs',#from=0,#to=1596053348
exec sp_executesql N'select top (#count) Value from [HangFire].[Set] with (readcommittedlock, forceseek) where [Key] = #key and Score between #from and #to order by Score',N'#count int,#key nvarchar(4000),#from float,#to float',#count=1000,#key=N'schedule',#from=0,#to=1596053348
exec sp_releaseapplock #Resource=N'HangFire:recurring-jobs:lock',#LockOwner=N'Session'
exec sp_releaseapplock #Resource=N'HangFire:locks:schedulepoller',#LockOwner=N'Session'
-- Then nothing is executed for about 8-10 seconds, then:
exec sp_executesql N'update [HangFire].Server set LastHeartbeat = #now where Id = #id',N'#now datetime,#id nvarchar(4000)',#now='2020-07-29 20:09:19.097',#id=N'ps12345:19764:fe362d1a-5ee4-4d97-b70d-134fdfab2b87'
-- Then about 500ms-2s later I get
exec sp_executesql N'delete top (1) JQ ... -- i.e. Same as first query
The update LastHeartbeat query is only there every second time (from just a brief inspection, maybe that’s not exactly right).
It looks like there’s at least 3 threads running the DELETE query against JQ, since I can see several RPC:Starting before the RPC:Completed, suggesting they’re being executed in parallel instead of sequentially.
I don’t know if that’s normal but seems weird as I thought we had just one ‘consumer’ of the jobs.
I only have one Queue in my dev environment, although in live we’d have 20-50 I’d guess.
Any suggestions on where I should look for the configuration that’s causing:
a) the 8-10s pause between checking for jobs
b) the number of threads that are checking for jobs - it seems like I have too many
After writing this I realised we were using an old version so I upgraded from 1.5.x to 1.7.12, upgraded the database, and changed the startup config to this:
app.UseHangfireDashboard();
GlobalConfiguration.Configuration
.UseSqlServerStorage(connstring, new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
UseRecommendedIsolationLevel = true,
PrepareSchemaIfNecessary = true, // Default value: true
EnableHeavyMigrations = true // Default value: false
})
.UseAutofacActivator(_container);
JobActivator.Current = new AutofacJobActivator(_container);
but if anything the problem is now worse. Or the same but faster: 20 calls to delete top (1) JQ... happen within about 1s now, then the other queries, then a 15s wait, then it starts all over again.
To be clear, the main problem is that if any jobs are added during that 15s delay then it'll take the remainder of that 15s before my job is executed. A second problem I think is it's hitting SQL Server more than needed: 20 times in a second is a bit much, for my needs at least.
(Cross-posted to hangfire forums)
If you don't set QueuePollInterval then Hangfire with sql server storage defaults to polling every 15s. So the first thing to do if you have this problem is set QueuePollInterval to something smaller, e.g. 1s.
But in my case even when I set that it wasn't having any effect. The reason for that was calling app.UseHangfireServer() before I was calling GlobalConfiguration.Configuration.UseSqlServerStorage() with the SqlServerStorageOptions.
When you call app.UseHangfireServer() it uses the current value of JobStorage.Current. My code had set that:
var storage = new SqlServerStorage(connstring);
JobStorage.Current = storage;
then later called
app.UseHangfireServer()
then later called
GlobalConfiguration.Configuration
.UseSqlServerStorage(connstring, new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
UseRecommendedIsolationLevel = true,
PrepareSchemaIfNecessary = true,
EnableHeavyMigrations = true
})
Reordering it to use SqlServerStorageOptions before app.UseHangfireServer() means the SqlServerStorageOptions take effect.
I would suggest checking the Hangfire BackgroundJobServerOptions to see what polling interval you have set up there. This will define the time before the hangfire server will check to see if there are any jobs in queue to execute.
From the documentation
Hangfire Docs
Hangfire Server periodically checks the schedule to enqueue scheduled jobs to their queues, allowing workers to
execute them. By default, check interval is equal to 15 seconds, but you can change it by setting the SchedulePollingInterval property on the options you pass to the BackgroundJobServer constructor:
var options = new BackgroundJobServerOptions
{
SchedulePollingInterval = TimeSpan.FromMinutes(1)
};
var server = new BackgroundJobServer(options);
I am using Python to download some data from bloomberg. It works most of the time, but sometimes it pops up a 'Time Out Issue`. And after that the response and request does not match anymore.
The code I use in the for loop is as follows:
result_IVM=con.bdh(option_name,'IVOL_MID',date_string,date_string,longdata=True)
volatility=result_IVM['value'].values[0]
When I set up the connection, I used following code:
con = pdblp.BCon(debug=True, port=8194, timeout=5000)
If I increase the timeout parameter (now is 5,000), will it help for this issue?
I'd suggest to increase the timeout to 5000 or even 10000 then test for few times. The default value of timeout is 500 milliseconds, which is small!
The TIMEOUT Event is triggered by the blpapi when no Event(s) arrives within milliseconds
The author of pdblp defines timeout as:
timeout: int Number of milliseconds before timeout occurs when
parsing response. See blp.Session.nextEvent() for more information.
Ref: https://github.com/matthewgilbert/pdblp/blob/master/pdblp/pdblp.py
Imagine that you click on an element using RSelenium on a page and would like to retrieve the results from the resulting page. How does one check to make sure that the resulting page has loaded? I can insert Sys.sleep() in between processing the page and clicking the element but this seems like a very ugly and slow way to do things.
Set ImplicitWaitTimeout and then search for an element on the page. From ?remoteDriver
setImplicitWaitTimeout(milliseconds = 10000)
Set the amount of time
the driver should wait when searching for elements. When searching for
a single element, the driver will poll the page until an element is
found or the timeout expires, whichever occurs first. When searching
for multiple elements, the driver should poll the page until at least
one element is found or the timeout expires, at which point it will
return an empty list. If this method is never called, the driver will
default to an implicit wait of 0ms.
In the RSelenium reference manual (http://cran.r-project.org/web/packages/RSelenium/RSelenium.pdf), you will find the method setTimeout() for the remoteDriver class:
setTimeout(type = "page load", milliseconds = 10000)
Configure the amount of time that a particular type of operation can execute for before they are aborted and a |Timeout| error is returned to the client.
type: The type of operation to set the timeout for. Valid values are: "script" for script timeouts, "implicit" for modifying the implicit wait timeout and "page load" for setting a page load timeout. Defaults to "page load"
milliseconds: The amount of time, in milliseconds, that time-limited commands are permitted to run. Defaults to 10000 milliseconds.
This seems to suggests that remDr$setTimeout() after remDr$navigate("...") would actually wait for the page to load, or return a timeout error after 10 seconds.
you can also try out this code that waits for the browser to provide whether page loaded or not.
objExecutor = (JavascriptExecutor) objDriver;
if (!objExecutor.executeScript("return document.readyState").toString()
.equalsIgnoreCase("complete")){
Thread.sleep(1000);
}
You can simply put it in your base page so you wont need to write it down in every pageobjects. I have never tried it out with any AJAX enabled sites, but this might help you and your scenario dependency will also get away.
Consider,
A query is taking more than a minute to retrieve the data (Due to larger volume of data) from the database.
I know that, we can set "timeout" attribute in the select tag (For a single query alone) or "defaultStatementTimeout" attribute in settings tag (SqlMapConfig.xml - For all the query) to forcibly terminate the query in execution.
<select id='uniqueName' parameterClass='java.util.Map' resultClass = "java.lang.String" timeout="60">
or
<settings useStatementNamespaces="false" defaultStatementTimeout="60"/>
By doing the above configuration, IBatis will throw "User cancelled request" error and terminates the execution.
Do we have any other way to terminate the execution?
My Scenario is:
When user requests for 3 years data, it takes more than a minute to fetch data from the database.
In the meantime, when the user requests for 1 day's data or sends a "cancel" request, I have to forcibly terminate the previous execution (3 years data retrieval) because it is affecting performance even with limited number of users.
NOTE
I didn't used any of the setting above.
Please provide me solution for this. Thanks in advance.
You can set a resource limit by modifying the Profile associated with the database user.
My ASP.NET intranet web application uses Windows Authentication, and I would like to record the following details:
1) Windows ID
2) Session Start Time
3) Session Stop Time
4) URL being browsed to (optional)
I've got some basic code setup in "Session_Start" method of the Global.ASAX to log session start times (seen below), but that's it so far. I have the feeling this is a primitive approach and there are "better" ways of doing this. So I really have two questions:
1) Is this the right way to go about doing this? If not what are some other options?
2) If this is the right way, do I just need to drop some code in the "Session_End" method to record the time they exit, and thats a complete solution? Does this method always get called when they close the browser tab they have the site open in, or do they have to close the entire browser (I don't have logout functionality)? Any way users can skip over this session end method (or start for that case)?
Dim connsql As New System.Data.SqlClient.SqlConnection(ConfigurationManager.ConnectionStrings("MyConnectionstring").ConnectionString)
Dim cmdsql As System.Data.SqlClient.SqlCommand = connsql.CreateCommand
cmdsql.CommandText = "BeginUserSession"
cmdsql.CommandType = Data.CommandType.StoredProcedure
Try
cmdsql.Parameters.Add("#windowsid", System.Data.SqlDbType.VarChar, 30, "windowsid")
cmdsql.Parameters("#windowsid").Value = Session("UserInfo").identity.name
If connsql.State <> System.Data.ConnectionState.Open Then connsql.Open()
cmdsql.ExecuteNonQuery()
connsql.Close()
Catch ex As Exception
Finally
If connsql.State <> Data.ConnectionState.Closed Then connsql.Close()
End Try
'Stored Proc records start time
Session_End is not reliable.
What I would suggest is on Session_Start you create a record that notes the time the Session was created, and in Session_End you update the record with the time it was ended.
To handle the majority of sessions which are passively abandoned, use Application_BeginRequest to update the record to note when the user was "last seen".
You will then need to determine a way of marking sessions that have been passively abandoned. This will be site/app specific. It could be as simple as picking a number of minutes that must pass before the session is considered abandoned - like 10 minutes.
So then you have a query:
SELECT Username,
SessionStart,
SessionEnd,
LastSeenOn,
DATEDIFF(mi, SessionStart, ISNULL(SessionEnd, LastSeenOn)) DurationMinutes
FROM SessionAudit
WHERE SessionEnd IS NOT NULL
OR DATEDIFF(mi, LastSeenOn, getdate()) > 10
Which will bring back your session audit log.
Your approach could be described as simple, but that could be totally fine - it comes down to what the requirements are. If you need to log a full suite of application errors and warnings, look at implementing something like Log4Net. Otherwise I wouldn't say there is anything wrong with what you are doing.
Sessions are ended when there has been no user activity for the amount of time specified in the timeout value, or when you explicitly call Session.Abandon() in your code. Because of the stateless nature of HTTP, there is no way to tell if a user has left your site, closed the browser or otherwise stopped being interactive with their session.
I am not sure you can catch the end of the session accurately because
The user can close their browser and that will not necessarily end the session.
They can then go back to your site and thus may have multiple sessions.
You can try messing with setting in IIS to kill the session very quickly after inactivity but its not a good idea.
Also... If the users are not all on an internal network you will have no control as to whether they have a "Windows ID" or not.