I'm deploying an ASP.NET application with PowerShell. It includes the web service, and I need to call one of its methods right after the web application is deployed. Obviously, this request will take time as it should "warm up" the website.
So, I make a plain call to New-WebServiceProxy to get the proxy object and then call a method:
$ps = New-WebServiceProxy -Uri "http://mysite/folder/myservice.asmx"
However, this operation takes some time and always fails with the timeout exception. I should say that it's all happening in the remote session. When the application is deployed locally (hence the script runs in a local session) the call succeeds.
I have verified the executionTimeout of the <httpRuntime> in the web.config - it has quite big timeout. But it is my assumption that it is not taken into consideration before the first call is made. So, I assume it is some property/switch of the New-WebServiceProxy, which should influence this behavior, but I failed to find it.
Does anyone have an idea?
P.S. I'm using PowerShell 2.0, but if the solution is there in v3.0, it is acceptable.
If the issue is indeed a timeout, then try this approach on PowerShell V3:
$wr = Invoke-WebRequest http://mysite/folder/myservice.asmx?wsdl -TimeoutSec 30
$wr.Content | Out-File -enc utf8 c:\myservice.wsdl
$ps = New-WebServiceProxy file:///c:\myservice.wsdl
BTW your problem could be the missing ?wsdl on the URI
The workaround I ended up with is warming up the website after deployment and before calling the web method. Something like this (works with PowerShell 2.0 as well):
$request = [system.net.WebRequest]::Create("http://mysite")
$request.Timeout = 2400000 # some big enough timeout goes here
$request.GetResponse() | Out-Null
After this, the call to New-WebServiceProxy succeeds and it's possible to call the web method.
Related
Note: Help with the immediate problem would be great, but mostly I'm looking for advice on troubleshooting gRPC timing issues in general (this isn't my first such issue).
I am adding a new server streaming service to a C++ module which has an existing server streaming service, and the two appear to be conflicting. Specifically, the completion queue Next() call on the server is crashing intermittently after the C# client calls Cancel() on the cancellation token for one of the services. This doesn't happen if I run each service independently.
On the client, I get this at the response stream MoveNext() call:
System.InvalidOperationException
HResult=0x80131509
Message=Shutdown has already been called
Source=Grpc.Core
StackTrace:
at Grpc.Core.Internal.CompletionQueueSafeHandle.BeginOp()
at Grpc.Core.Internal.CallSafeHandle.StartReceiveMessage(IReceivedMessageCallback callback)
at Grpc.Core.Internal.AsyncCallBase`2.ReadMessageInternalAsync()
at Grpc.Core.Internal.ClientResponseStream`2.<MoveNext>d__5.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at MyModule.Connection.<DoSubscriptionReceives>d__7.MoveNext() in C:\snip\Connection.cs:line 67
On the server, I get this at the completion queue next() call:
Exception thrown: read access violation.
core_cq_tag->**** was 0xDDDDDDDD.
The stack trace:
MyModule.exe!grpc_impl::CompletionQueue::AsyncNextInternal(void * * tag, bool * ok, gpr_timespec deadline) Line 59 C++
> MyModule.exe!grpc_impl::CompletionQueue::Next(void * * tag, bool * ok) Line 176 C++
...snip...
It appears something is being added to the queue after shutdown. The difficulty is I have little visibility into what is being added into the queue and in what order.
I'm trying to write a server-side interceptor to log all requests & responses, but there seems to be no documentation. So far, poking through the API hasn't gotten me very far. Is there any documentation available on wiring up an interceptor in C++? Or, are there other approaches for troubleshooting timing conflicts between services?
Windows 11, Grpc.Core 1.27
What I've tried:
I first played with the GRPC_TRACE & GRPC_VERBOSITY environment variables. I was able to get some unhelpful output from the client, but nothing from the server. Of course, there's been lots of debugging, stripping the client & server down to barebones, disabling keep alives, ensuring we aren't using deadlines, having the services share a cancellation token, etc.
Update: I have found that the crash only happens when the client is run from an NUnit test. In that environment, the completion queue is getting more hits on Next(), but I'm still trying to figure out where they are coming from.
Is 1.27 the version you are using? That seems pretty old.. There might have been fixes since then.
For using the C++ server interception API, I think you would find this very useful - https://github.com/grpc/grpc/blob/0f2a0f5fc9b9e9b9c98d227d16575d106f1e8d43/test/cpp/end2end/server_interceptors_end2end_test.cc#L48
One suggestion I have is to run the code another sanitizers https://github.com/google/sanitizers to make sure that we don't have a heap-use-after-free type bug.
I would also check for API misuse issues. (If you had posted the code, I could've given a look to see if anything seems weird..)
ASP.NET WebAPI2 application on .NET 4.6.2, hosted on IIS on Windows Server 2016. From time to time, there is a lot (hundreds) of requests stuck for hours (despite the fact I have request timeout 60s set) with no CPU usage. So, I took the memory dump of w3wp process, along with sos.dll, clr.dll and mscordacwks.dll and all my project's dlls and pdbs from bin directory from server and used WinDbg as described in many blogs and tutorials. But, in all of them, they are able to directly see CLR stack by calling ~*e !clrstack. I can see CLR stacktrace for some Redis and ApplicationInsights workers, but for all other managed threads I can see only:
OS Thread Id: 0x1124 (3)
Child SP IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000
!dumpstack for any of these gives just this:
0:181> !dumpstack
OS Thread Id: 0x1754 (181)
Current frame: ntdll!NtWaitForSingleObject+0x14
Child-SP RetAddr Caller, Callee
000000b942c7f6a0 00007fff33d63acf KERNELBASE!WaitForSingleObjectEx+0x8f, calling ntdll!NtWaitForSingleObject
000000b942c7f740 00007fff253377a6 clr!CLRSemaphore::Wait+0x8a, calling kernel32!WaitForSingleObjectEx
000000b942c7f7b0 00007fff25335331 clr!GCCoop::GCCoop+0xe, calling clr!GetThread
000000b942c7f800 00007fff25337916 clr!ThreadpoolMgr::UnfairSemaphore::Wait+0xf1, calling clr!CLRSemaphore::Wait
000000b942c7f840 00007fff253378b1 clr!ThreadpoolMgr::WorkerThreadStart+0x2d1, calling clr!ThreadpoolMgr::UnfairSemaphore::Wait
000000b942c7f8e0 00007fff253d952f clr!Thread::intermediateThreadProc+0x86
000000b942c7f9e0 00007fff253d950f clr!Thread::intermediateThreadProc+0x66, calling clr!_chkstk
000000b942c7fa20 00007fff37568364 kernel32!BaseThreadInitThunk+0x14, calling ntdll!LdrpDispatchUserCallTarget
000000b942c7fa50 00007fff3773e821 ntdll!RtlUserThreadStart+0x21, calling ntdll!LdrpDispatchUserCallTarget
So I have no idea, where to look for bug in my code.
(here is the full result:
https://gist.github.com/rouen-sk/eff11844557521de367fa9182cb94a82
and here is the results of !threads:
https://gist.github.com/rouen-sk/b61cba97a4d8300c08d6a8808c4bff6e)
What can I do? Google search for GetFrameContext failed gives nothing helpful.
As mentioned, this is not trivial, however you can find a case study of similar problem here: https://blogs.msdn.microsoft.com/rodneyviana/2015/03/27/the-case-of-the-non-responsive-mvc-web-application/
In a nutshell:
Download NetExt. It is the zip file here:
https://github.com/rodneyviana/netext/tree/master/Binaries
Open your dump and load NetExt
Run !windex to index the heap
Run !whttp -order -running to see a list of running requests
If the requests contains thread number you can go to the thread to see what is happening
If the requests contains --- instead of thread number, they are waiting a thread and this is a sign that some throttling is happening
If it is a WCF service, run !wservice to see the services
Run !wruntime to see runtime information
Run !wapppool to see Application Pool information
Run !wdae to list all errors
... And so it goes. When you do this again and again you will be able to spot issues easily
I''m trying to get log output (Console.WriteLine(..)) in my Docker logs, but I'm getting zero avail.
I've tried:
Console.WriteLine(..)
Trace.WriteLine(..)
Flushing the console, flushing the trace.
I can see these outputs in a VS output window when I'm debugging, so they go somoewhere.
I'm on windows Container, using microsoft/aspnet:4.7.1-windowsservercore-1709 and net4.7
These are the logs I get on container start
docker logs -f exportapi
ERROR ( message:Cannot find requested collection element. )
Applied configuration changes to section "system.applicationHost/applicationPools" for "MACHINE/WEBROOT/APPHOST" at configuration commit path "MACHINE/WEBROOT/APPHOST"
You have many good lateral options, like self-contained/server-contained executables (eg. Dotnet Core using microsoft/dotnet:runtime would proxy Console.WriteLine by default off the dotnet new web scaffold). Zero-configuration STDOUT logging has never been a common approach on IIS, but these modern options adopt it as best practice (logging should be a transparent backing service).
If you want or need a chain of three programs/assemblies to get your web service up (ServiceMonitor, W3SVC, and finally your assembly), then you need something like this: https://blog.sixeyed.com/relay-iis-log-entries-to-read-them-in-docker/
Overriding the entrypoint to tail more logs than the image does by default is unfortunately a common hack (not just in Microsoft land). So, in your case, I believe you need at least a trace listener config to emit Trace.WriteLine, and then the above approach to emit it: https://learn.microsoft.com/en-us/dotnet/framework/debug-trace-profile/how-to-create-and-initialize-trace-listeners
I am working for my client using Asp.net webAPI2 and angularJS. Now my client have following requirement,but i am unable to understand what type of project i have to create like WebAPI project,window service or any other? Anyone please tell me what the client actually want and how can i do it?
QueueManager will need to be some kind of a service which would be able to run jobs on a timed basis. We envision it being a service that runs on a continuous loop, but has a Thread.Sleep at the end of each iteration with a duration of x-seconds (“x” being set in a config file.) You should create this QueueManager service as a new project within the Core.Jobs project; I would like to have the project name be “Core.Jobs.QueueManager”, along with the base namespace.
Here are the functions that the QueueManager will do for each iteration:
1) Do a worker healthcheck (JobsAPI: Queue/WorkerHealthCheck – already created)
a. This method will just return a 200 status code, and a count of workers. Not need to act on the return value.
Look at Hangfire, it is quite easy to set up and simple to use.
http://docs.hangfire.io/en/latest/background-methods/performing-recurrent-tasks.html
I have a scheduled task set up to run Scan.aspx every 3 minutes in IE7. Scan.aspx reads data from 10 files in sequence. These files are constantly being updated. The values from the file are inserted into a database.
Sporadically, the value being read is truncated or distorted. For example, if the value in the file was "Hello World", random entries such as "Hello W", "Hel", etc. will be in the database. The timestamps on these entries appear completely random. Sometimes at 1:00 am, sometimes at 3:30 am. And some nights, this doesn't occur at all.
I'm unable to reproduce this issue when I debug the code. So I know under "normal" circumstances, the code executes correctly.
UPDATE:
Here is the aspx codebehind (in Page_Load) to read a text file (this is called for each of the 10 text files):
Dim filename As String = location
If File.Exists(filename) Then
Using MyParser As New FileIO.TextFieldParser(filename)
MyParser.TextFieldType = FileIO.FieldType.Delimited
MyParser.SetDelimiters("~")
Dim currentrow As String()
Dim valueA, valueB As String
While Not MyParser.EndOfData
Try
currentrow = MyParser.ReadFields()
valueA= currentrow(0).ToUpper
valueB = currentrow(1).ToUpper
//insert values as record into DB if does not exist already
Catch ex As Exception
End Try
End While
End Using
End If
Any ideas why this might cause issues when running multiple times throughout the day (via scheduled task)?
First implement a Logger such as Log4Net in your ASP.NET solution and Log method entry and exit points in your Scan.aspx as well as your method for updating the DB. There is a chance this may provide some hint of what is going on. You should also check the System Event Log to see if any other event is associated with your failed DB entries.
ASP.NET is not the best thing for this scenario especially when paired with a Windows scheduled task; this is not a robust design. A more robust system would run on a timer inside a Windows-Service-Application. Your code for reading the files and updating to the DB could be ported across. If you have access to the server and can install a Windows Service, make sure you also add Logging to the Windows Service too!
Make sure you read the How to Debug below
Windows Service Applications intro on MSDN: has further links to:
How to: Create Windows Services
How to: Install and Uninstall Services
How to: Start Services
How to: Debug Windows Service Applications]
Walkthrough: Creating a Windows Service
Application in the Component Designer
How to: Add Installers to Your Service Application
Regarding your follow up comment about the apparent random entries that sometimes occur at 1am and 3.30am: you should:
Investigate the IIS Log for the site when these occur and find out what hit(visited) the page at that time.
Check if there is an indexing service on the server which is visiting your aspx page.
Check if Anti-Virus software is installed and ascertain if this is visiting your aspx page or impacting the Asp.Net cache; this can cause compilation issues such as file-locks on the aspnet page in the aspnet cache; (a scenario for aspnet websites as opposed to aspnet web applications) which could give weird behavior.
Find out if the truncated entries coincide with the time that the files are updated: cross reference your db entries timestamp or logger timestamp with the time the files are updated.
Update your logger to log the entire contents of the file being read to verify you've not got a 'junk-in > junk-out' scenario. Be careful with diskspace on the server by running this for one night.
Find out when the App-Pool that your web app runs under is recycled and cross reference this with the time of your truncated entries; you can do this with web.config only via ASP.NET Health Monitoring.
Your code is written with a 'try catch' that will bury errors. If you are not going to do something useful with your caught error then do not catch it. Handle your edge cases in code, not a try catch.
See this try-catch question on this site.