Silverlight authentication and ICredential - asp.net

Question

I've setup up a WCF web service to handle requests from a Silverlight application. That service has Windows authentication set up which works well with the following endpoint configuration
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior name="Test.Service.ServiceBehavior">
<serviceMetadata httpGetEnabled="true"/>
<serviceDebug includeExceptionDetailInFaults="true"/>
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<customBinding>
<binding name="customBinding0">
<binaryMessageEncoding/>
<httpTransport authenticationScheme="Negotiate"/>
</binding>
</customBinding>
</bindings>
<serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
<services>
<service behaviorConfiguration="Test.Service.ServiceBehavior" name="Test.Service.Service">
<endpoint address="" binding="customBinding" bindingConfiguration="customBinding0" contract="Test.Service.Service"/>
<endpoint address="mex" binding="customBinding" bindingConfiguration="customBinding0" contract="IMetadataExchange"/>
</service>
</services>
</system.serviceModel>
The service also has the following code in the constructor which authenticates to a TFS server
teamFoundationServer = TeamFoundationServerFactory.GetServer(tfsServer);
teamFoundationServer.EnsureAuthenticated();
workItemStore = (WorkItemStore)teamFoundationServer.GetService(typeof(WorkItemStore));
Then I have a Silverlight application that uses this web service and contains the following code to access it
proxy = new ServiceClient("Service");
Lastly there is a host web site that only contains the .xap silverlight file. That site has also Windows authentication configured.
Both the service and the host are running in IIS.
The problem I'm having is that when the service authenticates with TFS I always get an exception
Microsoft.TeamFoundation.TeamFoundationServerUnauthorizedException: TF30063: You are not authorized to access
Since the service and the host are windows authenticated then the Silverlight application is most likely causing me problems. After googling Silverlight and authentication it seems like there is some issue realated for Silverlight to forwarding the credential from the host to the service. Has someone been able to accomplish this task ?
To provide further info then I've been able to get the current users username by doing
OperationContext.Current.ServiceSecurityContext.PrimaryIdentity.AuthenticationType
But that returns a WindowsIdentity which is not compatible with ICredential which the TFS API requires :(