I have a .NET Core 3.1 console application that is running some background cron jobs. There are different jobs doing different things and they will start every x minute, do its thing and then stop. Most of the time they run fine, but lately these jobs have been starting to hang. They just never finish, the process is stuck.
I don't really know how to debug this or how to figure out what is causing it to hang. What I've done is:
Create a DMP file from the task manager
Load this using dotnet dump analyze myfile.DMP
Run dumpasync -stacks
This gives a list of all the stacks. I've created a few DMP files from different workers (doing different things), but they all have in common that there is one task on Npgsql.NpgsqlReadBuffer.
This is one example:
000001ed01aa18c8 00007ffca0c9e3d8 128 1 Npgsql.NpgsqlReadBuffer+<>c__DisplayClass34_0+<<Ensure>g__EnsureLong|0>d
Async "stack":
.000001ed01aa1988 (1) Npgsql.NpgsqlConnector+<>c__DisplayClass160_0+<<DoReadMessage>g__ReadMessageLong|0>d
..000001ed01aa1a40 (5) Npgsql.NpgsqlDataReader+<NextResult>d__44
...000001ed01aa1ae0 (0) Npgsql.NpgsqlCommand+<ExecuteReaderAsync>d__102
....000001ed01aa1b90 (0) Npgsql.NpgsqlCommand+<ExecuteDbDataReaderAsync>d__97
.....000001ed01aa1c10 (1) Dapper.SqlMapper+<QueryAsync>d__33`1[[System.__Canon, System.Private.CoreLib]]
......000001ed021ac5d8 (3) Acme.Common.Data.Dapper.Repositories.AccountItems.GetDapperAccountItemsHandlerSql+<GetAccountItemsAsync>d__3
.......000001ed021ac638 (0) Acme.Common.Data.Dapper.Repositories.ItemRepository`1+<GetAccountItemsHigherThanIdAsync>d__9[[Acme.Core.Db.Dapper.DapperReaderConnection, Acme.Core.Db.Dapper]]
........000001ed021ac698 (1) Acme.Common.Services.EmailReport.ReportDataService+<MakeInstantAlertDto>d__20
.........000001ed00badd90 (3) Acme.Common.Services.EmailReport.ReportDataService+<GetReportDtoAsync>d__19
..........000001ed0105f968 (2) Acme.Common.Services.EmailReport.InstantAlertReportService+<SendInstantAlertReportAsync>d__6
...........000001ed0105f9c8 (0) Acme.Common.Services.EmailReport.EmailReportWorkerService+<SendInstantAlertReportsAsync>d__10
............000001ed01b902d0 System.Threading.Tasks.TaskFactory+CompleteOnInvokePromise
I don't know if this means that npgsql is the cause of the hang, but it seems to be what is common between all of them.
The connection is created like this:
public async Task<IEnumerable<MyDto>> GetData()
{
using (var dbConnection = await _dapperConnection.OpenAsync())
{
var sql = "SELECT * FROM ....";
var result = await dbConnection.QueryAsync<MyDto>(sql);
return result;
}
}
private async Task<NpgsqlConnection> OpenAsync(CancellationToken cancellationToken = default)
{
var connection = new NpgsqlConnection(_connectionString);
await connection.OpenAsync(cancellationToken);
return connection;
}
The connection string looks like this:
User ID=<userid>;Password=<password>;Host=<host>;Port=5432;Database=<databasename>;Pooling=true;Maximum Pool Size=200;Keepalive=30;
How can I debug this further? What would help?
Further technical details
Npgsql version: 4.1.3
PostgreSQL version: 9.6
Operating system: Windows
Related
I am trying to load test a Kafka instance on one of our servers.
Here is the code that does it using NBomber:
public static void Run()
{
var testScenario = NBomber.CSharp.Step.Create("testScenario",
async context =>
{
try
{
// The testData is a string variable that reads contents from a text file in the Init method.
var kafkaObject = new KafkaObject { Topic = TestTopic, Message =testData };
SampleKafkaFlow sampleKafkaFlow = new SampleKafkaFlow();
var response = await sampleKafkaFlow.SendMessageToKafka(kafkaObject);
return Response.Ok();
}
catch (Exception ex)
{
return Response.Fail(ex.Message);
}
});
var scenario = ScenarioBuilder.CreateScenario("scenario", testScenario)
.WithoutWarmUp()
.WithInit(Init)
.WithLoadSimulations(new[]
{
Simulation.InjectPerSec(rate: 100, during: TimeSpan.FromMinutes(3))
});
NBomber.CSharp.NBomberRunner
.RegisterScenarios(scenario)
.WithReportFileName($"testScenario-Report-{DateTime.UtcNow.ToString("yyyy-dd-M--HH-mm-ss")}")
.WithReportFolder("test_reports")
.WithReportFormats(ReportFormat.Html)
.Run();
}
My laptop configuration:
Core i5 10th Gen with 16 Gb RAM, running Windows 10.
At the time of running the load test only VS 2022 was running.
Now I assumed that at 100 RPS it would generate a total of 18k requests for 3 minutes of execution time. The report says different - while it did run for a total of 3 minutes there were only 2057 total requests!
What am I missing here?
How do I get to doing a load test with a higher RPS?
Thanks in advance.
Probably there is a issue with NBomber itself https://github.com/PragmaticFlow/NBomber/issues/488
Also check nbomber log – it could contains a lot of
Error: step unhandled exception: One or more errors occurred. (Too many open files in system
or another errors that indicate that OS limits your load test
I’m having issues running realm with xUnite and Net core. Here is a very simple test that I want to run
public class UnitTest1
{
[Scenario]
public void Test1()
{
var realm = Realm.GetInstance(new InMemoryConfiguration("Test123"));
realm.Write(() =>
{
realm.Add(new Product());
});
var test = realm.All<Product>().First();
realm.Write(() => realm.RemoveAll());
}
}
I get different exceptions on different machines (Windows & Mac) on line where I try to create a Realm instace with InMemoryConfiguration.
On Mac I get the following exception
libc++abi.dylib: terminating with uncaught exception of type realm::IncorrectThreadException: Realm accessed from incorrect thread.
On Windows I get the following exception when running
ERROR Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. at
System.Net.Sockets.NetworkStream.Read(Span1 destination) at
System.Net.Sockets.NetworkStream.ReadByte() at
System.IO.BinaryReader.ReadByte() at
System.IO.BinaryReader.Read7BitEncodedInt() at
System.IO.BinaryReader.ReadString() at
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable() at
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action1 errorHandler, CancellationToken cancellationToken) Source: System.Net.Sockets HResult: -2146232800 Inner Exception: An existing connection was forcibly closed by the remote host HResult: -2147467259
I’m using Realm 3.3.0 and xUnit 2.4.1
I’ve tried downgrading to Realm 2.2.0, and it didn’t work either.
The solution to this problem was found in this Github post
The piece of code from that helped me to solve the issue
Realm GetInstanceWithoutCapturingContext(RealmConfiguration config)
{
var context = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(null);
Realm realm = null;
try
{
realm = Realm.GetInstance(config);
}
finally
{
SynchronizationContext.SetSynchronizationContext(context);
}
return realm;
}
Though it took a while for me to apply this to my solution.
First and foremost, instead of just setting the context to null I am using Nito.AsyncEx.AsyncContext. Because otherwise automatic changes will not be propagated through threads, as realm needs a non-null SynchronizationContext for that feature to work. So, in my case the method looks something like this
public class MockRealmFactory : IRealmFactory
{
private readonly SynchronizationContext _synchronizationContext;
private readonly string _defaultDatabaseId;
public MockRealmFactory()
{
_synchronizationContext = new AsyncContext().SynchronizationContext;
_defaultDatabaseId = Guid.NewGuid().ToString();
}
public Realm GetRealmWithPath(string realmDbPath)
{
var context = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(_synchronizationContext);
Realm realm;
try
{
realm = Realm.GetInstance(new InMemoryConfiguration(realmDbPath));
}
finally
{
SynchronizationContext.SetSynchronizationContext(context);
}
return realm;
}
}
Further, this fixed a lot of failing unit tests. But I was still receiving that same exception - Realm accessed from incorrect thread. And I had no clue why, cause everything was set correctly. Then I found that the tests that were failing were related to methods where I was using async realm api, in particular realm.WriteAsync. After some more digging I found the following lines in the realm documentation.
It is not a problem if you have set SynchronisationContext.Current but
it will cause WriteAsync to dispatch again on the thread pool, which
may create another worker thread. So, if you are using Current in your
threads, consider calling just Write instead of WriteAsync.
In my code there was no direct need of using the async API. I removed and replaced with sync Write and all the tests became green again! I guess if I find myself in a situation that I do need to use the async API because of some kind of bulk insertions, I'd either mock that specific API, or replace with my own background thread using Task.Run instead of using Realm's version.
I'm trying to figure out why my webservice is so slow and find ways to get it to respond faster. Current average response time without custom processing involved (i.e. apicontroller action returning a very simple object) is about 75ms.
The setup
Machine:
32GB RAM, SSD disk, 4 x 2.7Ghz CPU's, 8 logical processors, x64 Windows 10
Software:
1 asp.net mvc website running .net 4.0 on IISEXPRESS (System.Web.Mvc v5.2.7.0)
1 asp.net web api website running .net 4.0 on IISEXPRESS (System.Net.Http v4.2.0.0)
1 RabbitMQ messagebus
Asp.net Web API Code (Api Controller Action)
[Route("Send")]
[HttpPost]
[AllowAnonymous)
public PrimitiveTypeWrapper<long> Send(WebsiteNotificationMessageDTO notification)
{
_messageBus.Publish<IWebsiteNotificationCreated>(new { Notification = notification });
return new PrimitiveTypeWrapper<long>(1);
}
The body of this method takes 2ms. Stackify tells me there's a lot of overhead on the AuthenticationFilterResult.ExecuteAsync method but since it's an asp.net thing I don't think it can be optimized much.
Asp.net MVC Code (MVC Controller Action)
The RestClient implementation is shown below. The HttpClientFactory returns a new HttpClient instance with the necessary headers and basepath.
public async Task<long> Send(WebsiteNotificationMessageDTO notification)
{
var result = await _httpClientFactory.Default.PostAndReturnAsync<WebsiteNotificationMessageDTO, PrimitiveTypeWrapper<long>>("/api/WebsiteNotification/Send", notification);
if (result.Succeeded)
return result.Data.Value;
return 0;
}
Executing 100 requests as fast as possible on the backend rest service:
[HttpPost]
public async Task SendHundredNotificationsToMqtt()
{
var sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 100; i++)
{
await _notificationsRestClient.Send(new WebsiteNotificationMessageDTO()
{
Severity = WebsiteNotificationSeverity.Informational,
Message = "Test notification " + i,
Title = "Test notification " + i,
UserId = 1
});
}
sw.Stop();
Debug.WriteLine("100 messages sent, took {0} ms", sw.ElapsedMilliseconds);
}
This takes on average 7.5 seconds.
Things I've tried
Checked the number of available threads on both the REST service and the MVC website:
int workers;
int completions;
System.Threading.ThreadPool.GetMaxThreads(out workers, out completions);
which returned for both:
Workers: 8191
Completions: 1000
Removed all RabbitMQ messagebus connectivity to ensure it's not the culprit. I've also removed the messagebus publish method from the rest method _messageBus.Publish<IWebsiteNotificationCreated>(new { Notification = notification }); So all it does is return 1 inside a wrapping object.
The backend rest is using identity framework with bearer token authentication and to eliminate most of it I've also tried marking the controller action on the rest service as AllowAnonymous.
Ran the project in Release mode: No change
Ran the sample 100 requests twice to exclude service initialization cost: No change
After all these attempts, the problem remains, it will still take about +- 75ms per request. Is this as low as it goes?
Here's a stackify log for the backend with the above changes applied.
The web service remains slow, is this as fast as it can get without an expensive hardware upgrade or is there something else I can look into to figure out what's making my web service this slow?
Team,
My biztalk send port instance gets hung and stays in the active state for longer periods of time. I would like to monitor that send port active instance with the help of C#.
I intend to run a code which will check if the send port(passed as a parameter) is still in a running state or not. Can anyone help me with that piece of code ?
Use WMI MSBTS_ServiceInstance.ServiceStatus Property:
public static int GetRunningServiceInstanceCount()
{
int countofServiceInstances = 0;
try
{
ManagementObjectSearcher searcher = new ManagementObjectSearcher("root\\MicrosoftBizTalkServer", "SELECT * FROM MSBTS_ServiceInstance WHERE ServiceStatus = 1 or ServiceStatus = 2");
countofServiceInstances = searcher.Get().Count;
return countofServiceInstances;
}
catch (ManagementException exWmi)
{
throw new System.Exception("An error occurred while querying for WMI data: " + exWmi.Message);
}
}
To get to your actual problem: The SFTP adapter in BizTalk 2016 has a great way of using the most recent version of the FTP code. This might solve stability issues.
Assuming from your BizTalk 2013 tag, you're probably not using the 2016 version, in that case double check you are at least at CU3 since that one solves a few critical SFTP bugs.
Am facing a problem, while creating components through TOM API using .NET/COM Interop.
Actual Issue:
I have 550 components to be created through custom page. I am able to create between 400 - 470 components but after that it is getting failed and through an error message saying that
Error: Thread was being aborted.
Any idea / suggestion, why it is getting failed?
OR
Is there any restriction on Tridion 2009?
UPDATE 1:
As per #user978511 request, below is error on Application event log:-
Event code: 3001
Event message: The request has been aborted.
...
...
Process information:
Process ID: 1016
Process name: w3wp.exe
Account name: NT AUTHORITY\NETWORK SERVICE
Exception information:
Exception type: HttpException
Exception message: Request timed out.
...
...
...
UPDATE 2:
#Chris: This is my common function, which is called in a loop by passing list of params. Here am using Interop dll's.
public static bool CreateFareComponent(.... list of params ...)
{
TDSE mTDSE = null;
Folder mFolder = null;
Component mComponent = null;
bool flag = false;
try
{
mTDSE = TDSEInitialize();
mComponent = (Component)mTDSE.GetNewObject(ItemType.ItemTypeComponent, folderID, null);
mComponent.Schema = (Schema)mTDSE.GetObject(constants.SCHEMA_ID, EnumOpenMode.OpenModeView, null, XMLReadFilter.XMLReadAll);
mComponent.Title = compTitle;
...
...
...
...
mComponent.Save(true);
flag = true;
}
catch (Exception ex)
{
CustomLogger.Error(String.Format("Logged User: {0} \r\n Error: {1}", GetRemoteUser(), ex.Message));
}
return flag;
}
Thanks in advance.
Sounds like a timeout, most likely in IIS which is hosting your custom page.
Are you creating them all in one synchronous request? Because that is indeed likely to time out.
You could instead create them in batches - or make sure your operations are done asynchronously and then polling the status regularly.
The easiest would just be to only create say 10 Components in one request, wait for it to finish, and then create another 10 (perhaps with a nice progress bar? :))
How you call TDSE object. I would like to mention here "Marshal.ReleaseComObject" procedure. Without releasing COMs objects can lead to enormous memory leaks.
Here is code for component creating:
private Component NewComponent(string componentName, string publicationID, string parentID, string schemaID)
{
Publication publication = (Publication)mTdse.GetObject(publicationID, EnumOpenMode.OpenModeView, null, XMLReadFilter.XMLReadContext);
Folder folder = (Folder)mTdse.GetObject(parentID, EnumOpenMode.OpenModeView, null, XMLReadFilter.XMLReadContext);
Schema schema = (Schema)mTdse.GetObject(schemaID, EnumOpenMode.OpenModeView, publicationID, XMLReadFilter.XMLReadContext);
Component component = (Component)mTdse.GetNewObject(ItemType.ItemTypeComponent, folder, publication);
component.Title = componentName;
component.Schema = schema;
return component;
}
After that please not forget to release mTdse ( in my case it is previously created TDSE object). Disposing "Components" object can be useful also after finish working with them.
For large Tridion batch operations I always use a Console Application and run it directly on the server.
Use Console.WriteLine to write to the output window and Console.ReadLine as the last line of code in the app (so the window stays open). I also use Log4Net as the logger.
This is by far the best approach if you have access to a remote session on the server - or can ask an admin to run it for you and give you access to the log folder via a network share.
As per #chris suggestions and part of immediate fix I have changed my web.config execution time out to 8000 seconds.
<httpRuntime executionTimeout="8000"/>
With this change, custom page is able to handle as of now.
Any more best suggestion, please post it.