I am using WebDeploy (msdeploy.exe) to deploy an ASP.NET Core app to an Azure App Service instance.
This works fine most of the time, but sometimes the deployment fails with the error message: Unable to perform the operation ("Create File") for the specified directory. (...) ERROR_INSUFFICIENT_ACCESS_TO_SITE_FOLDER.
I believe this is some file locking issue, because the only way out of this is to log into Azure Portal and stop the App Service, and then redeploy. After this, deployment is fine for a while, until the same happens again.
Having spent some time setting up automatic deployment, this hickup is very annoying.
Is there any way around this issue?
It has nothing, as i understand, with the Azure - the error is a bit common.
As for Azure Web Apps, there are two options:
1) Stop your site and deploy.
2) If the first one is not OK, then use Kudu Console, find your file (for example, d:\home...\something.blocked) and rename the file. After that, the problem should be away
If that is the helpful answer, please mark it as a helpful or mark it as the answer. Thanks!
Related
I am using VS2013 Premium to publish a site to Windows Server 2012.
All files publish ok except these:
SqlServerTypes\x64\msvcr100.dll
SqlServerTypes\x64\SqlServerSpatial110.dll
SqlServerTypes\x86\msvcr100.dll
SqlServerTypes\x86\SqlServerSpatial110.dll
I get this kind of errors for each of the above files I tried to publish:
Web deployment task failed. (The file 'msvcr100.dll' is in use. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.)
Interrestingly, these files were published the first time (when they were not on the server), then they are no longer overwritten. Tried with 2 different web servers.
I have followed the guide here:
http://blogs.msdn.com/b/webdev/archive/2013/10/30/web-publishing-updates-for-app-offline-and-usechecksum.aspx
...But it only managed to put the site offline (VS is placing the app_offline.htm) but publish still fails with the same error.
All other files publish perfectly.
Any ideas?
You can take you app offline during publishing which hopefully should free up the lock on the file and allow you to update it.
I blogged about this a while back. The support outlined was shipped inside of the Azure SDK and Visual Studio Update. I don't remember the exact releases but I can find out if needed. Any update dating around/after that blog post should be fine.
Prerequisites:
VS 2012 + VS update / VS 2013 + VS Update / VS2015
MSDeploy v3
Note: if you are publishing from a CI server the CI server will need the updates above as well
Edit the publish profile
In VS when create a Web Publish profile the settings from the dialog are stored in Properties\PublishProfiles\ as files that end with .pubxml. Note: there is also a .pubxml.user file, that file should not be modified
To take your app offline in the .pubxml file add the following property.
<EnableMSDeployAppOffline>true</EnableMSDeployAppOffline>
Notes
ASP.NET Required
The way that this has been implemented on the MSDeploy side is that an app_offline.htm file is dropped in the root of the website/app. From there the asp.net runtime will detect that and take your app offline. Because of this if your website/app doesn't have asp.net enabled this function will not work.
Cases where it may not work
The implementation of this makes it such that the app may not strictly be offline before publish starts. First the app_offline.htm file is dropped, then MSDeploy will start publishing the files. It doesn't wait for ASP.NET to detect the file and actually take it offline. Because of this you may run into cases where you still run into the file lock. By default VS enables retrys so usually the app will go offline during one of the retrys and all is good. In some cases it may take longer for ASP.NET to respond. That is a bit more tricky.
In the case that you add <EnableMSDeployAppOffline>true</EnableMSDeployAppOffline> and your app is not getting taken offline soon enough then I suggest that you take the app offline before the publish begins. There are several ways to do this remotely, but that depends on your setup. If you only have MSDeploy access you can try the following sequence:
Use msdeploy.exe to take your site offline by dropping app_offline.htm
Use msdeploy.exe to publish your app (_make sure the sync doesn't delete the app_offline.htm file_)
Wait some amount of time
Publish the site
Use msdeploy.exe to bring the app online by deleting app_offline.htm
I have blogged how you can do this at http://sedodream.com/2012/01/08/howtotakeyourwebappofflineduringpublishing.aspx. The only thing that is missing from that blog post is the delay to wait for the site to actually be taken offline. You can also create a script that just calls msdeploy.exe directly instead of integrating it into the project build/publish process.
I have found the reason why the solution at
http://blogs.msdn.com/b/webdev/archive/2013/10/30/web-publishing-updates-for-app-offline-and-usechecksum.aspx
did not work for the original poster, and I have a workaround.
The issue with the EnableMSDeployAppOffline approach is that it only recycles the app domain hosting the application. It does not recycle the app pool worker process (w3wp.exe) which the app domain lives in.
Tearing down and recreating the app domain will not affect the Sql Server Spatial dlls in question. Those dlls are unmanaged code which are manually loaded via interop LoadLibray calls. Therefore the dlls live outside the purview of the app domain.
In order to release the files locks, which the app pool process puts on them, you need to either recycle the app pool, or unload the dlls from memory manually.
The Microsoft.SqlServer.Types nuget package ships a class which is used to load the Spatial dlls called SqlServerTypes.Utilities. You can modify the LoadNativeAssemblies method to unload the unmanaged dlls when the app domain is unloaded. With this modification when msdeploy copys the app_offline.htm the app domain will unload and then unload the managed dlls as well.
[DllImport("kernel32.dll", SetLastError = true)]
internal extern static bool FreeLibrary(IntPtr hModule);
private static IntPtr _msvcrPtr = IntPtr.Zero;
private static IntPtr _spatialPtr = IntPtr.Zero;
public static void LoadNativeAssemblies(string rootApplicationPath)
{
if (_msvcrPtr != IntPtr.Zero || _spatialPtr != IntPtr.Zero)
throw new Exception("LoadNativeAssemblies already called.");
var nativeBinaryPath = IntPtr.Size > 4
? Path.Combine(rootApplicationPath, #"SqlServerTypes\x64\")
: Path.Combine(rootApplicationPath, #"SqlServerTypes\x86\");
_msvcrPtr = LoadNativeAssembly(nativeBinaryPath, "msvcr100.dll");
_spatialPtr = LoadNativeAssembly(nativeBinaryPath, "SqlServerSpatial110.dll");
AppDomain.CurrentDomain.DomainUnload += (sender, e) =>
{
if (_msvcrPtr != IntPtr.Zero)
{
FreeLibrary(_msvcrPtr);
_msvcrPtr = IntPtr.Zero;
}
if (_spatialPtr != IntPtr.Zero)
{
FreeLibrary(_spatialPtr);
_spatialPtr = IntPtr.Zero;
}
};
}
There is one caveat with this approach. It assumes your application is the only one running in the worker process that is using the Spatial dlls. Since app pools can host multiple applications the file locks will not be released if another application has also loaded them. This will prevent your deploy from working with the same file locked error.
There are known issues with IIS and file-locks (why they aren't solved yet i dont know).
The question i want to ask however is if you even need to re-deploy these files?
I recognize the file-names and recall them to be system-files which should either already be present on the server or simply not need to be re-deployed.
I am not very experienced when it comes to IIS but i have ran into this problem before and several of my more experienced co-workers have told me that this is as i said a known IIS-issue and i believe the answer to your question is:
Avoid deploying unnecessary files.
try again
Reset website
try again
iisreset
I think what would be the easiest thing to do is to make these dll's as CopyLocal as true. I am assuming these dll's are pulled out from program files folder. Try marking them as copylocal true and do a deployment.Try to stop any IIS local process running in your local machine.
Watch out you don't have one of those new-fangled cloud backup services running that is taking file locks - and also you don't have things open in explorer or a DLL inspection tool.
I think it's kind of ridiculous that MS doesn't make better provisions for this problem. I find that 9 times out of 10 my deployment works just fine, but then as our traffic increases that can become 1 in 10 times.
I am going to solve the problem with :
two applications MySite.A and MySite.B, where only one is running at a time.
I always then deploy to the dormant site.
If there's a problem during the deployment it will never cause the whole site to go down.
If there's a major problem after deployment you can revert back very easily.
Not quite sure how I'm implementing it, but I think this is what I need to do.
We're working on validating our Loupe service to run as an Azure App Service and have run into a showstopper we can't figure out. Anything that attempts to resolve a temp directory fails with the exception:
mscorlib : System.IO.IOException
The directory name is invalid.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.__Error.WinIOError()
at System.IO.Path.InternalGetTempFileName(Boolean checkHost)
The stack trace has this within the .NET method for generating a temp file name. This stack trace is common to pretty much all the areas we get the failure. For a bit it seemed that if we forced the site to restart and/or forced the underlying App Service Plan to rescale it would go away until we next updated the site but no longer.
Since the only search results we could find said this error happens when impersonation is enabled and the user the site's impersonating doesn't have access to the IIS App Pool user's temp directory we've dug into that. First, we can confirm from our logging that the thread is not impersonating at the time the failed request is made. Second, just for fun we added this to the web.config to be doubly sure:
<system.web>
<identity impersonate="false"/>
</system.web>
All to no avail. If this was a generic problem with Azure App Services then I would presume it would break many systems, so I have to conclude we've done something fascinating and wrong to cause it.
This might not be the exact answer you're looking for but it might help point you in the right direction.
I had similar issues a while back using the Azure App Services. I found that accessing the local file system was somewhat problematic. Sometimes it worked fine and other times it didn't.
Eventually, I discovered that when an Azure App Service is instantiated, it doesn't always use the same drive letters for the system behind it. In some cases, this can cause the environment variables to be blatantly incorrect. They "think" they are set properly, but that's not always the case.
Generating a temp filename will use that environment variable for the path and if it's set to C: but the machine has a D: drive instead, if will fail. The C: drive doesn't exist and therefore the path to the temp file can't exist either.
To identify if this is the problem, you need to enable RDP so you can log into it directly. https://learn.microsoft.com/en-us/azure/cloud-services/cloud-services-role-enable-remote-desktop
It's the only way I was able to eventually figure it out.
If you open up the Kudu instance for your App Service Web App you'll be able to see what the local Temp directory is on the Managed VM underneath. You can access Kudu by going to "Advanced Tools" on the App Service blade in the Azure Portal, or by navigating to the https://{web app name}.scm.azurewebsites.net domain for your Web App.
Once in Kudu, click on Environment in the top navigation. The Temp directory is usually D:\local\Temp and that path is stored in the "TEMP" environment variable made accessible to your Web App.
My Elastic Beanstalk installation won't deploy through Visual Studio due to this error:
2016-07-01 20:45:02,627 ERROR 1 AWSBeanstalkCfnDeploy.DeploymentUtils - Exception during deployment.
Microsoft.Web.Deployment.DeploymentDetailedClientServerException: Web Deploy cannot modify the file 'msvcr100.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.
The link suggests that I create a pubxml file with settings to enable AppOffline, but this file only seems to be relevant for publishing through VS using the built-in Publish feature. I haven't found any documentation suggesting that this will work for AWS.
How do I enable AppOffline for an Elastic Beanstalk deployment?
Thanks!
Sorry that this is only general advice and not the code you need, but the solution is to use hooks via .ebextensions. Please see http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref-hooks.html.
You can add the execution of a powershell script to add app_offline.htm before the update is extracted and remove it once the update is deployed.
We had a similar issue, but the DLL in question (abcPDF, v9) was only blocked because we were initializing the licensing of it during application_start(), which EB did not like. So we moved applying the license elsewhere.
However, I think this approach would work.
--
Oh, maybe this container command will work for you. It recycles the IIS app pool right before the It didn't for us because of the aforementioned licensing locking the DLL.
/.ebextensions/recycleapppool.config
container_commands:
__recycle_app_pool:
____command: c:\windows\system32\inetsrv\appcmd.exe recycle apppool DefaultAppPool
After quite a lot of experimentation, the only working solution I could find for this problem was
// in Project/.ebextensions/reset.config
container_commands:
00_nuke:
command: IISReset
waitAfterCompletion: 0
The cost was about 4 seconds of downtime (on a t2.micro), during which you get a 503, which certainly isn't great.
Note there's a Github issue for this (open at the time of writing).
If you have the option, deploy your service to Azure rather than AWS and there are configuration options to work around the issue (such as an environment variable MSDEPLOY_RENAME_LOCKED_FILES) - related Azure specific question.
I have been using meteor for quite awhile and have been deploying apps to .meteor.com. However recently after updating my app to meteor v0.8 and new collectionFS, the terminal states that the app has been deployed to whatever.meteor.com but when I go to the site, I see Meteor's Site is down.Try again later. I have narrowed it down to the new collectionFS package causing the problem, since my old app with the old collectionFS deploys fine. Any thoughts?
EDIT
The problem was due to the long startup time caused by my collectionFS path: definition.
There are several reasons why your site may not load when being deployed.
Site Inactivity
The meteor deploy service shuts down if your site hasn't been accessed in a while, and takes a while to start up again if it is requested, during which time you'll see that message.
In a few minutes after the first request, you should see the site come back up.
For more information, see this answer: https://stackoverflow.com/a/19072230/586086
Excessive Resource Use
Another reason your site can refuse to deploy is if your app takes more than 4 minutes to start or uses an excessive amount of CPU - it will get killed. Is it doing anything resource-intensive like that? For initializing really big databases, do the initialization locally and copy the contents using the url from meteor mongo -U yoursite.meteor.com.
I had to do this for the demo app for meteor-autocomplete. See the file upload-db.sh.
I had the same error.
But the following solution works deploying the app successfully.
meteor login
meteor deploy < available meteor sub domain name >
I know this is old but I was just having the same issue and removing the collectionFS package solved my problem immediately...in case that helps anyone.
I run an ASP.Net MVC3 application and have followed this Windows Azure How-To on how to store my sessions in a co-located cache in Windows Azure.
I have installed the mentioned Nuget-package, and applied the necessary information to my web.config. The web role is setup to run 2 small instances, and use a 30% co-located cache.
When I deploy to Azure, my instances just recycle. Through a remote desktop connection to one of the instances, I have found these two events in the logs, that seem related:
The CacheStatusIndicator service terminated unexpectedly. It has done
this 1 time(s). The following corrective action will be taken in 1000
milliseconds: Restart the service.
The CacheService service terminated unexpectedly. It has done this 1
time(s). The following corrective action will be taken in 60000
milliseconds: Restart the service.
However, I cannot find much information on that.
Does anyone have a clue what I might be missing here?
I am very much sure that your role is missing some of the reference and because of that your see such message from CacheStatusIndicator. Have you verified that your role does have all the modules/references set to "copy local as true" so all of them are part of deployment package. Mainly it is a MVC3 app, you may need to manually set app MVC specific references property to copy local as true to resolved this problem.
Now I spent my shared amount of time dealing with this, and finally it seems to be working. Not sure exactly what part solved the problem, but I'll wrap up the changes I made, in case someone else have the same issue at some point.
While debugging this issue, I also updated my application from MVC3 to MVC4. Not sure if that has anything to do with solving this problem, but at least it is one of the changes I made.
I noticed that another exception was thrown by the the IntelliTrace service, so I disabled that in the deployment and re-deployed the application (my guess is that this problem is what actually caused the instances to cycle.
Response from Microsoft Support on this issue:
I was troubleshooting your deployment with the issue and I got a memory dump from the CacheService.exe
This was the exception being thrown
CLR exception type: System.TypeInitializationException
"The type initializer for 'Microsoft.ApplicationServer.Caching.AzureCommon.AzureUtility' threw
an exception." ModLoad: 000007feea010000 000007feea0fa000
D:\Windows\Microsoft.NET\Framework64\v4.0.30319\diasymreader.dll
* ERROR: Symbol file could not be found. Defaulted to export symbols for D:\windows\system32\KERNEL32.dll -
ntdll!ZwWaitForMultipleObjects+0xa: 00000000`76f5739a c3
ret
The exceptions on this namespace can happen if:
You set the Microsoft.WindowsAzure.Plugins.Caching.ConfigStoreConnectionString setting on ServiceConfiguration to use the local storage emulator
On the web.config you have the identifier value of the <dataCacheClients> section not set to the correct role name
In some rare cases if you have the SDK 1.6 and 1.7 installed together this can cause some mixed up DLLs when project is being built.