MSDeploy Package has Missing Files - asp.net

Question

I'm using MSBuild to build a web application project and adding parameters to create the package file. All of that is good. I get two folders in the _PublishedWebSites output:
AppName
AppName_Package
In the ApplicationName folder, the entire site is there and I can simply copy this folder over to the website and it will run.
In the Package folder I've got the expected 5 files:
AppName.deploy.cmd
AppName.-readme.txt
AppName.SetParameters.xml
AppName.SourceManifest.xml
AppName.zip
When deploying the package however, the web application dll (AppName.dll) is missing, as are a few other important referenced dependencies.
Inspecting the package itself does reveal that the files are in fact missing from the web applications bin directory.
This is very odd, considering the files are all in the root AppName folder, but not in the package found in AppName_Package folder.
The only MSDeploy related modification I've made is that I am overriding the CopyAllFilesToSingleFolderForPackageDependsOn target to copy in some handlers from a library project, and this all works nicely.
<PropertyGroup>
<CopyAllFilesToSingleFolderForPackageDependsOn>
CustomCollectFiles;
$(CopyAllFilesToSingleFolderForPackageDependsOn);
</CopyAllFilesToSingleFolderForPackageDependsOn>
</PropertyGroup>
<Target Name="CustomCollectFiles">
<ItemGroup>
<_CustomFiles Include="..\Libraries\CodeLibrary1\**\*.ashx" />
<FilesForPackagingFromProject Include="%(_CustomFiles.Identity)">
<DestinationRelativePath>$(ProjectDir)%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>
</ItemGroup>
</Target>
So, I'm not able to determine why the web application's DLL, and only a few other (referenced)DLLs are removed from the bin directory during the package creation process (but not all - maybe 3%).
Any ideas what I should be looking for in the log file?
EDIT: (Response to Sayed):
I truly appreciate you answering my post. Unfortunately I’m not sure we are on the same page. Like I said, I might not have explained my problem well enough to actually clue others in on exactly what my issue is. Let me try once more to clarify
Getting external files copied into my web application was not the problem. I understand that scanning over my post and seeing the all too common “CopyAllFilesToSingleFolderForPackageDependsOn” element set off a few red flags, alarms, bells and whisles. . It seems to be a common thing people are struggling with, and as you point out, there have been some issues with it being executed. That worked OK for me.
My problem has more to do with project referenced DLLs, AND most importantly the application DLL itself not being copied.
More tweaking around today has revealed something I was not aware of. For instance:
Kicking off a Build on TFS outputs a couple things:
The source from TFS is output to the Source Directory (SourceDir)
This only contains the output what is checked into TFS
This does not include project or file references, only what is committed in TFS
The output of the build is into two folders
Binaries (includes project and file references)
Sources (only what is included/committed in TFS)
Inside Binaries I find the _publishedWebsites folder, as well as all project and file references, while inside Sources there are just the files that are checked into TFS.
My problem, or confusion, was thinking that when I pass in the following parameters to MSBuild Arguments, it would take the output from the build (_PublishedWebsites) and use those files to create the package. It does not do that.
The actual process builds a package based on the files in the SOURCES folder. So, here is my dilemma/confusion, because I did not commit my application dll into source, it was not being included in the package, and therefore was not being sent off to the test site on the test server.
/p:DeployOnBuild=True /p:DeployTarget=MsDeployPublish /p:CreatePackageOnPublish=True /p:MSDeployPublishMethod=RemoteAgent /p:MSDeployServiceUrl=http://<mytestsite> /p:DeployIisAppPath="<AppName>" /p:UserName=<domain\user> /p:Password=<mypassword> /p:SkipExtraFilesOnServer=false /p:AllowUntrustedCertificate=True
To wrap this up, once I “checkout for edit” my application DLL on my dev machine, and them compile the solution, and finally commit the application DLL to source control, then it is included in the package because it is included as just another source item.
So, I guess this becomes my fault for not understanding how to get the output of the TFS BUILD into the sources folder, and get that included into the package used by msdeploy.
There is probably something very fundamental that I missed or just skimmed over - and not understood, that the package creation is from the Sources folder and not the _PublishedWebsites folder. Nor do I understand how to get the MSBuild compiled application dll, and project referenced dlls, into my package – replacing the committed source items in source control.
I hope this isn’t a complete waste of your time because I missed a
source=<some-parameter>
somewhere.
I appreciate it you can direct me to any existing sources out that already explain this to where I should be able to get my head around it and get this working. Or if it’s so easy to just tell me here.

Answer 1

The issue here is that the CopyAllFilesToSingleFolderForPackage target itself is not getting called from the VS2012 targets. We made a lot of changes and this may be a regression on our side. I will look into this to see if there is anything that we can do. Fortunately it should be pretty straight forward to update this to get the behavior that you are looking for. Instead of using CopyAllFilesToSingleFolderForPackageDependsOn you should be able to use PipelineCollectFilesPhaseDependsOn as an alternative. You should be able to change what you have above to
<PropertyGroup>
<PipelineCollectFilesPhaseDependsOn>
CustomCollectFiles;
$(PipelineCollectFilesPhaseDependsOn);
</PipelineCollectFilesPhaseDependsOn>
</PropertyGroup>
<Target Name="CustomCollectFiles">
<Message Text="Inside of CustomCollectFiles" Importance="high"/>
<ItemGroup>
<_CustomFiles Include="C:\Temp\_NET\WAP-AfterPublish\MvcApplication1\additional files\**\*" />
<FilesForPackagingFromProject Include="%(_CustomFiles.Identity)">
<DestinationRelativePath>additional files\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
</FilesForPackagingFromProject>
</ItemGroup>
</Target>
I just tried this for both VS2012 as well as VS2010 so this seems like a better to solution then the CopyAllFilesToSingleFolderForPackageDependsOn approach. Can you try that out and let me know what you find out?