Forms Authentication 4.0 on Web Farm - asp.net

Question

This is quite literally driving me bananas - I'm on holiday as from tomorrow but if I can't get this working today then it's under threat - so any help much appreciated!
Firstly, my website has a manually defined <machinekey /> element defined so that both web servers in the web farm are kept in sync. I have verified this with IIS manager (hence why I'm asking, despite the similarity with so many other questions). It looks like this (keys elided - but they are the correct length):
<machineKey validationKey="[512 bit hex]"
decryptionKey="[256-bit hex]"
validation="SHA1"
decryption="AES" />
The website is running Asp.Net MVC3 and I am using Forms authentication in 'normal' mode (i.e. not 2.0 compat mode). I'm using Forms Authentication to create an authentication ticket - using code like the following:
FormsAuthentication.SetAuthCookie(userName, false);
My Forms Auth config is very simple; there are no IIS or server-wide settings in place that override the documented defaults:
<authentication mode="Forms">
<forms defaultUrl="~/Unauthorised"
loginUrl="~/Unauthorised"
ticketCompatibilityMode="Framework40" />
</authentication>
And then I've hijacked the cookie-reading functionality as per this MSDN topic so that I can create the principal and identity that I want.
Problem is - only one half of the web farm is able to decrypt the authentication cookie, the other half (i.e. whichever one didn't authenticate the user) just gives:
System.Security.Cryptography.CryptographicException: Length of the data to decrypt is invalid.
With this as the top-part of the stack trace:
[CryptographicException: Length of the data to decrypt is invalid.]
System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) [phone]
System.Security.Cryptography.CryptoStream.FlushFinalBlock() +53
System.Web.Configuration.MachineKeySection.EncryptOrDecryptData(Boolean fEncrypt, Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Boolean useValidationSymAlgo, Boolean useLegacyMode, IVType ivType) +331
System.Web.Security.FormsAuthentication.Decrypt(String encryptedTicket) +293
We are using a slightly ancient (but very reliable) Load balancer - which does not modify the HTTP traffic - and the traffic in this case is HTTPs; I'm able to verify that it's always one server and not both by having used Fiddler to decrypt and inspect the traffic.
This clearly points to out-of-sync machineKeys - but they are not - so any idea what's going on!!?
Thanks in advance!

Answer 1

When different servers have different patches, it has the potential to alter the behavior of encryption (/decryption), therefore possibly causing this runtime exception:
System.Security.Cryptography.CryptographicException: Length of the data to decrypt is invalid.
More information regarding this issue can be found here: http://blog.evonet.com.au/post/SystemSecurityCryptographyCryptographicException-Length-of-the-data-to-decrypt-is-invalid.aspx