Cache the dynamic image served by http handler - asp.net

Question

I have developed a http handler to serve images dynamically..
I have a respective last modified date to each image.
So I have implemented http header caching, as below :
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "image/jpeg";
DateTime lastModified = Convert.ToDateTime("2/1/2013 12:00:00 AM");
string eTag = "test.JPG" + lastModified.ToString();
// set cache info
context.Response.Cache.VaryByHeaders["If-Modified-Since"] = true;
context.Response.Cache.VaryByHeaders["If-None-Match"] = true;
context.Response.Cache.SetLastModified(lastModified);
context.Response.Cache.SetETag(eTag);
//Set cache control header
context.Response.Cache.SetCacheability(HttpCacheability.ServerAndPrivate);
Byte[] imageBytes = CreateImage();
context.Response.OutputStream.Write(imageBytes, 0, imageBytes.Length);
}
Note : Here lastModified will be fetched dynamically for each image, also the name will be different for each image.
Response Headers
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: image/jpeg
Last-Modified: Thu, 31 Jan 2013 18:30:00 GMT
Etag: test.JPG2/1/2013 12:00:00 AM
Vary: If-Modified-Since, If-None-Match
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-SourceFiles: =?UTF-8?B?RTpcTGVhcm5pbmdcQ2FjaGVcR2V0SW1hZ2UuYXNoeA==?=
X-Powered-By: ASP.NET
Date: Mon, 08 Jul 2013 11:22:26 GMT
Content-Length: 384411
Request Headers
GET /GetImage.ashx HTTP/1.1
Host: localhost:50432
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Question
This page is getting cached properly..But the next time when I am loading the page with different lastModified value, which makes the etag different too..
But the page is not hitting the handler at all..It is always showing the image from cache...
And as it is not hitting the handler, how can I check lastModified value and serve accordingly..