MVC 3 client caching - asp.net

Question

I am trying to make modifications to an existing CDN. What I am trying to do is create a short cache time and use conditional GETs to see if the file has been updated.
I am tearing my hair out because even though I am setting a last modified date and seeing it in the response headers, on subsequent get requests I am not seeing an If-Modified-Since header being returned. At first I thought it was my local development environment or the fact that I was using Fiddler as a proxy for testing so I deployed to a QA server. But what I am seeing in Firebug is so different than what I am doing. I see the last modified date, for some reason it is setting my cache-control to private, and I have cleared any header Output Caching and the only header IIS 7.5 is set to write is to enable Http keep-alive, so all the caching should be driven by the code.
This seemed like such a no-brainer, yet I've been adding and removing headers all day with no luck. I checked global.asax and anywhere else (I didn't write the app so I was looking for any hidden surprises and am stumped. Below is the current code and request and response headers. I have the expiration set to 30 seconds just for testing purposes. I have looked at several samples, I don't see myself doing anything different, but it simply won't work.
Response Headersview source
Cache-Control private, max-age=30
Content-Length 597353
Content-Type image/jpg
Date Tue, 03 Sep 2013 21:33:55 GMT
Expires Tue, 03 Sep 2013 21:34:25 GMT
Last-Modified Tue, 03 Sep 2013 21:33:55 GMT
Server Microsoft-IIS/7.5
X-AspNet-Version 4.0.30319
X-AspNetMvc-Version 3.0
X-Powered-By ASP.NET
Request Headersview source
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection keep-alive
Cookie __utma=1.759556114.1354835397.1377631052.1377732484.36; __utmz=1.1354835397.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Host hqat4app1
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0
Response.Cache.SetCacheability(HttpCacheability.Public);
Response.Cache.SetLastModified(DateTime.Now);
return new FileContentResult(fileContents, contentType);
The relevant code is:
public ActionResult Resize(int id, int size, bool grayscale)
{
_logger.Debug(() => string.Format("Resize {0} {1} {2}", id, size, grayscale));
string imageFileName = null;
if (id > 0)
using (new UnitOfWorkScope())
imageFileName = RepositoryFactory.CreateReadOnly<Image>().Where(o => o.Id == id).Select(o => o.FileName).SingleOrDefault();
CacheImageSize(id, size);
if (!ImageWasModified(imageFileName))
{
Response.Cache.SetExpires(DateTime.Now.AddSeconds(30));
Response.StatusCode = (int)HttpStatusCode.NotModified;
Response.Status = "304 Not Modified";
return new HttpStatusCodeResult((int)HttpStatusCode.NotModified, "Not-Modified");
}
byte[] fileContents;
if (ShouldReturnDefaultImage(imageFileName))
fileContents = GetDefaultImageContents(size, grayscale);
else
{
bool foundImageFile;
fileContents = GetImageContents(id, size, grayscale, imageFileName, out foundImageFile);
if (!foundImageFile)
{
// No file found, clear cache, disable output cache
//ClearOutputAndRuntimeCacheForImage(id, grayscale);
//Response.DisableKernelCache();
}
}
string contentType = GetBestContentType(imageFileName);
Response.Cache.SetCacheability(HttpCacheability.Public);
Response.Cache.SetLastModified(DateTime.Now);
return new FileContentResult(fileContents, contentType);
}
private bool ImageWasModified(string fileName)
{
bool foundImageFile;
string filePath = GetFileOrDefaultPath(fileName, out foundImageFile);
if (foundImageFile)
{
string header = Request.Headers["If-Modified-Since"];
if(!string.IsNullOrEmpty(header))
{
DateTime isModifiedSince;
if (DateTime.TryParse(header, out isModifiedSince))
{
return isModifiedSince < System.IO.File.GetLastWriteTime(filePath);
}
}
}
return true;
}