Azure Biztalk Services time out issue when processing a large CSV file - biztalk

I have a simple azure biztalk services project.
It has a FTP source that reads a .CSV file and writes to an on-premise sqlserver database table.
I successfully deployed and it works for small .CSV files (around 800 rows) quite well. But, when I have a large file (around 6500 rows. Actually, this is also very samll file in my opinion), it fails with the following error. Below this error, you will see my configurations for the SQLServer Adapter Service.
<?xml version="1.0" encoding="utf-16"?>
<s:Fault xmlns:s="http://www.w3.org/2003/05/soap-envelope">
<s:Code>
<s:Value>s:Receiver</s:Value>
<s:Subcode>
<s:Value>s:SendError</s:Value>
</s:Subcode>
</s:Code>
<s:Reason>
<s:Text xml:lang="en-US">The operation with action "TableOp/Insert/dbo/tblVMSData"
took longer than the specified timeout "00:01:00".</s:Text>
</s:Reason>
</s:Fault>
My on premise SQL Server adapter service has the following configuration.
<basicHttpRelayBinding>
<binding name="basicHttpRelayBinding1"
closeTimeout="00:20:00"
openTimeout="00:20:00"
receiveTimeout="00:20:00"
sendTimeout="00:20:00"
maxBufferPoolSize=" 1048576"
maxBufferSize="67108864"
maxReceivedMessageSize="67108864">
<readerQuotas maxDepth="2147483647"
maxStringContentLength="2147483647"
maxArrayLength="2147483647"
maxBytesPerRead="67108864"
maxNameTableCharCount="2147483647" />
<security mode="Transport" />
</binding>
</basicHttpRelayBinding>

You might have to make the timeout value on the server side larger too! So if you have a look in your BizTalk Services project, look for your SQL endpoint configuration file (under you itinerary in solution explorer) and edit the WCF configuration values there too.
Does that work ?

I finally found the configuration options for the timeout. They are not in any config file. They are not even in the biztalk services project.
You have to right click your SQL Target, under LOB Types under Biztalk Adapter Services, and select the properties. In the properties, Click on the Binding Configuration. It opens up Advanced Adapter Configuration. There are four timeouts "Open Timeout", "Receive Timeout", "Send Timeout" and "Close Timeout".
What is strange is that these timeouts also appear in the config file that is automatically generated when you drag and drop the SQL Target on to your MessageFlowItinerary. But, changing them in the config file seems to be not enough in my case.
Also, when you open IIS Management console, there you can change client configuration for timeouts by clicking on the Configure option under "Manage WCF and WF Services". If you do not see "Manage WCF and WF Services", you haven't installed the Windows Server App Fabric SDK. Download and install it.
Microsoft should really make it easy to find and change configuration for both server and client from a single simple page. It is really frustrating to click through different things to find the options. It should simply work out of the box. On one hand they give flexibility by providing configuration option and with the other hand they rob it by hiding the configuration options under different rocks.

Related

Failure creating BizTalk application on remote server using MSBuild extension pack

I have an msbuild script that tries to create and deploy a BizTalk application to a remote server. I use the following task:
<MSBuild.ExtensionPack.BizTalk.BizTalkApplication TaskAction="Create" MachineName="$(BizTalkServer)" Applications="$(BizTalkApplicationName)" />
When trying to deploy on a remote server, I get the following error
error : COMException: Application registration failed because the
application already exists.
But that is not true, the application does not exist.
Searching on the net results in the following link, suggesting setting the MSDTC properties: http://social.msdn.microsoft.com/Forums/sa/biztalkediandas2/thread/251250c1-3f95-4457-8fbc-0274c722c7b0
But the DTC has been setup like this on both the local and remote server:
It is also worhty of note that I run two other tasks against the remote BizTalk server to check the existence of the application, and the existence of the WCF-SQL adapter. These both succeed.
<MSBuild.ExtensionPack.BizTalk.BizTalkApplication TaskAction="CheckExists" MachineName="$(BizTalkServer)" Application="$(BizTalkApplicationName)">
<Output TaskParameter="Exists"
PropertyName="ApplicationExists" />
</MSBuild.ExtensionPack.BizTalk.BizTalkApplication>
<MSBuild.ExtensionPack.BizTalk.BizTalkAdaptor TaskAction="CheckExists" MachineName="$(BizTalkServer)" AdaptorName="WCF-SQL">
<Output TaskParameter="Exists" PropertyName="AdaptorExists" />
</MSBuild.ExtensionPack.BizTalk.BizTalkAdaptor>
Update:
I inspected the event log. Don't know why I didn't think of that to begin with. I get the following 3 events:
1st event, Level=Information, Source=Sql server
Attempting to initialize Microsoft Distributed Transaction Coordinator
(MS DTC). This is an informational message only. No user action is
required.
2nd event, Level=Information, Source=Sql server
The Microsoft Distributed Transaction Coordinator (MS DTC) service
could not be contacted. If you would like distributed transaction
functionality, please start this service.
3rd event, Level=Error, Source=BizTalk
Unable to communicate with MessageBox BizTalkMsgBoxDb on SQL Instance
.. Error Code: 0x8004d01c. Possible reasons include:
1) The MessageBox is unavailable.
2) The network link from this machine to the MessageBox is down.
3) The DTC Configuration on either this local machine or the machine hosting this MessageBox is incorrect.
I discovered that although there is no BizTalk application on the remote server, the one I'm attempting to deploy to, with that name, there was a BizTalk application on the local server, the one I'm attempting to deploy from, with the specified name.
That shouldn't prevent me from deploying the application to a remote server but it does.
This behavior is present both when I use the MSBuild extension pack BizTalk tasks, and when I try to use btstask.exe to create a remote application.

My SQL Server Express has gone nuts it seems, at times mounting db and others not

I am not sure what's going on exactly as I can't establish a strict linear correlation between
events. But below you'll find my connection string, at times it works, other times when I open the app/project I am building (no changes made) it doesn't and gives me various errors and reasons.
<add name="EFDbContext"
connectionString="Data Source=.\SQLEXPRESS;AttachDbFilename=C:\Program Files\Microsoft SQL Server\MSSQL10.SQLEXPRESS\MSSQL\DATA\EFDbContext.mdf;MultipleActiveResultSets=true;Integrated Security=True;User Instance=True"
providerName="System.Data.SqlClient" />
Errors include:
Unable to complete operation. The supplied SqlConnection does not specify an initial catalog.
or
saying Amin-PC\Admin has no access rights (I am logged in admin)
Also when I try to add a new connection at times it will tell me I have no access rights or database already exists or just adds it normally!
I tried explicitly starting visual studio as administrator and this seemed to help with the denied access to the db issue.
Basically I don't know what the heck's going on.
Have recently started with .net and it has been easy coding for asp.net mvc framework until I got to interacting with underlying asp.net infrastructure, connection strings, outdated (noon-entity framework ready) membership provider etc.
And now don't know what's going on with the SQL Server Express edition that gets installed a long with VS2010.
My recommendation would be: since you already have a server instance (.\SQLEXPRESS) installed - attach your databases to the SQL Server Express instance, and then use them under their database name (instead of using the shaky AttachDbFileName= method).
So:
launch your SQL Server Management Studio Express
find your MDF files you want
use Server > Attach Database to attach the MDF/LDF to the server - give it meaningful name
from that point on - use the database on the server and don't attach MDF's on the fly....
Your connection string would then look something like:
server=.\SQLEXPRESS;Database=YourEFDatabaseName;Integrated Security=True
I find this method is typically much more reliable and predictable - the AttachdbFileName= and User Instance=true are supposed to be easier, but tend to be more confusing that anything else....
Refer to SQL Server connection string samples here.

Unable to make the session state request to the session state server

Our site is currently having this problem. Basically it only happen when we click some particular links where it will pop-up a new window.
This is the error message we receive :
Unable to make the session state request to the session state server.
Please ensure that the ASP.NET State service is started and that
the client and server ports are the same. If the server is on a
remote machine, please ensure that it accepts remote requests by
checking the value of
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection.
If the server is on the local machine, and if the before
mentioned registry value does not exist or is set to 0, then the state
server connection string must use either 'localhost' or '127.0.0.1' as
the server name. target site : Void
MakeRequest(StateProtocolVerb, System.String, StateProtocolExclusive,
Int32, Int32, Int32, Byte[], Int32, Int32, SessionNDMakeRequestResults
ByRef)
this is the webconfig that contain sessionstate tag :
<sessionState mode="StateServer" timeout="45" />
I have check the ASP.NET State Service, and it is currently started and set to automatic
I have set the registry stated in the error message to 1, but still not working.
Start–> Administrative Tools –> Services
Right-click on the ASP.NET State Service and click “start”
Additionally you could set the service to automatic so that it will work after a reboot
If you need to change HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection to 1, remember to restart the ASP.net state service after you change the parameter.
Check that:
stateConnectionString="tcpip=server:port"
is correct. Also please check that default port (42424) is available and your system does not have a firewall that is blocking the port on your system
Type Services.msc in run panel of windows run window. It will list all the windows services in our system. Now we need to start Asp .net State service as show in the image.
Your issue will get resolved.
One of my clients was facing the same issue. Following steps are taken to fix this.
(1) Open Run.
(2) Type Services.msc
(3) Select ASP.NET State Service
(4) Right Click and Start it.
Another thing to check is whether you have Windows Firewall enabled, since that might be blocking port 42424.
I've had the same issue when some ASP.NET installation was corrupted. In that case they suggest running aspnet_regiis -i -enable
I've found that some developers will for some reason define the server's private IP outside of IIS in an unexpected location, like a nonstandard config file (i.e. not web.config) or a text file. This can cause internal operation to fail even when the service is started, ports aren't being blocked, reg keys are correct, etc.
Kaseya, in particular, places a file called serveripinternal.txt in the root IIS directory of the VSA server. I've seen the text of your error when somebody running their own Kaseya instance changed the server's internal IP. The server will be reachable, IIS will respond, and the login page will come up - but login will fail with the cited message.
Not the best answer, but it's an option anyway:
Comment the given line in the web.config.
I recently ran into this issue and none of the solutions proposed fixed it. The issue turned out to be an excessive use of datasets stored in the session. There was a flaw in the code that results in the session size to increase 10x.
There is an article on the msdn blog that also talks about this.
http://blogs.msdn.com/b/johan/archive/2006/11/20/sessionstate-performance.aspx
I used a function to write custom trace messages to measure the size of the session data on the live site.

BizTalk SSO configuration and SQL Error 18456, Severity 14, State 16

I'm having a major issue trying to configure a new install of BizTalk Server 2006 (not R2). The server had BizTalk installed on it before, and it was working fine. I've uninstalled BizTalk, removed the databases and jobs from the SQL server, which is a separate machine, and re-installed BizTalk. The install was successful, with no errors during the install, and nothing in the install logs.
I'm configuring the BizTalk server to be the SSO master secret server, along with creating a new BizTalk group and registering the BizTalk runtime. The process always errors out on creating the SSO database on the SQL server. In the ConfigLog, there are a couple of warnings that the MSSQLServerOLAPService does not exist, then it shows errors on creating the SSO database. There are 4 in a row. In order, they are:
Error ConfigHelper] [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.
Error ConfigHelper] SQL error: 08001 Native error code: 17
Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\sqlhelper.cpp(1176): FAILED hr = 80004005
Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\sqlhelper.cpp(918): FAILED hr = 80004005
It then has similar errors trying to create each of the BizTalk databases.
On the SQL server, there are corresponding errors in the SQL Server Logs - 2 for each attempt
Login failed for user '[USERNAME]'.[CLIENT: [IP ADDRESS]]
Error: 18456, Severity: 14, State: 16
The first error from the SQL logs also shows up as a failure audit in the SQL server's application event log.
The biggest issue I am having with this is that the user I am logged on to the BizTalk server is a local admin on both the BizTalk server and the SQL server, and is in the SQL sysadmin group. The user that I am configuring the BizTalk services to run under is also a local admin on both servers and in the sysadmin group on the SQL server. I've checked the MSDTC settings on both machines and made sure they are set as the BizTalk documentation recommends. SQL Browser is running on the SQL machine, and I've verified that network access is allowed using the SQL Surface Area Configuration tool.
Can anyone help me find something that I might have missed?
Re: Igal:
Yes, all of the servers and users are on the same domain. I've run across that posting on SQL protocols in researching this, but I tried to select a count from one of the tables in the default database of the logged in user while connected to another database. I had no problems at all running that query.
Re: Yossi:
I'm installing BizTalk on Windows Server 2003 R2 SP1. Yes, I have removed the SSODB (Wouldn't out it past myself to miss something like that though!). I will make sure I am providing the usernames correctly and check out the sources you linked and get back to you.
A few of pointers:
Check out the two points at the end of the Configuring Enterprise SSO Using the Configuration Manager page on MSDN:
When configuring the SSO Windows
accounts using local accounts, you
must specify the account name without
the computer name.
When using a local SQL Server named
instance as data store, you must use
LocalMachineName\InstanceName instead
of LocalMachineName\InstanceName,
PortNumber.
Check out the relevant installation guide (don't worry about the fact that it relates to R2, they seems to have hidden the 'R1' documentation, but they are the same), and specifically the section around "Windows Groups and Service Accounts"
also - just to be sure - when you have uninstalled BizTalk and removed the databases - you have removed the SSODB as well, right?! :-)
The log files are very confusing - especially when deciding which error is the acutal problem - have you tried looking up any other errors you've had? (check out this blog entry, for example)
I had everything set up properly. Unfortunately for me, the answer was the standard "Windows" answer - reboot and try again. As soon as I rebooted the SQL server, I was able to configure BizTalk just fine.
I am going to set Yossi's answer as accepted, however, since that would be the most relevant for anyone else who may be reading this question.
Just remember to reboot after all setting changes!
Make sure the BizTalkMgmtDb and BizTalkMsgBoxDb have your local admin account as DB OWNER.
Right click on the databases --> Properties --> Files --> Owner:

BizTalk 2006 Tutorial 1: EDI-to-XML Document Translation

I cannot find the translated file after running the solution in BizTalk 2006 Tutorial Lesson 3: Run the EDI-to-XML Solution.
It should be placed in the c:\Program Files\Microsoft BizTalk Server 2006 \EDI\Adapter\Getting Started with EDI\Northwind\In folder.
The Base EDI adapter picks up the file in c:\Documents and Settings\All Users\Application Data\Microsoft\BizTalk Server 2006 \EDI\Subsystem\Documents\PickupEDI folder, but I cannot find the translated file in the X-12 4010 850 document format.
I'm not immediately familiar with the tutorial you mention, but below are steps to find where any document has gone to in BizTalk.
First two places to check are in the event viewer and in the BizTalk Server Administration Console.
Check you have no errors in the event viewer.
In the admin console, click on the BizTalk Group in the left hand window and you should see two columns in the right hand pane, Work in Progress and Suspended Items. Click on Running service instances and Suspended service instances. Check that you message is not delayed for any reason (a Send Port being turned off perhaps).
Next, from Start -> All Programs -> Microsoft BizTalk Server 2006 select the Health and Activity Tracking (HAT) tool.
In HAT, select Queries -> Most recent 100 service instances. Find the pipeline that will have wrote out your file, right click the service instance and select Message Flow. In the message flow view you should see in the URL the disk location where your file was written to.
(You can also look in the admin console to check where the send port is pointing)
Thanks for your suggestion regarding how to troubleshoot an issue of BizTalk Server from generic point of view. It did help. I have resolved this problem by reading error logs.
Here is the error:
Access denied. The client user must be a member of one of the following accounts to perform this function.
SSO Administrators: SSO Administrators
SSO Affiliate Administrators: SSO Affiliate Administrators
Application Administrators: BizTalk Server Administrators
Application Users: BizTalk Application Users
It works now after adding a service account to "SSO Administrators" and restart all BizTalk related services.

Resources