Timeout on WebRequest with F#-Style Asynchronous Workflows - asynchronous

Question

For a broader context, here is my code, which downloads a list of URLs.
It seems to me that there is no good way to handle timeouts in F# when using use! response = request.AsyncGetResponse() style URL fetching. I have pretty much everything working as I'd like it too (error handling and asynchronous request and response downloading) save the problem that occurs when a website takes a long time to response. My current code just hangs indefinitely. I've tried it on a PHP script I wrote that waits 300 seconds. It waited the whole time.
I have found "solutions" of two sorts, both of which are undesirable.
AwaitIAsyncResult + BeginGetResponse
Like the answer by ildjarn on this other Stack Overflow question. The problem with this is that if you have queued many asynchronous requests, some are artificially blocked on AwaitIAsyncResult. In other words, the call to make the request has been made, but something behind the scenes is blocking the call. This causes the time-out on AwaitIAsyncResult to be triggered prematurely when many concurrent requests are made. My guess is a limit on the number of requests to a single domain or just a limit on total requests.
To support my suspicion I wrote little WPF application to draw a timeline of when the requests seem to be starting and ending. In my code linked above, notice the timer start and stops on lines 49 and 54 (calling line 10). Here is the resulting timeline image.
When I move the timer start to after the initial response (so I am only timing the downloading of the contents), the timeline looks a lot more realistic. Note, these are two separate runs, but no code change aside from where the timer is started. Instead of having the startTime measured directly before use! response = request.AsyncGetResponse(), I have it directly afterwards.
To further support my claim, I made a timeline with Fiddler2. Here is the resulting timeline. Clearly the requests aren't starting exactly when I tell them to.
GetResponseStream in a new thread
In other words, synchronous requests and download calls are made in a secondary thread. This does work, since GetResponseStream respects the Timeout property on the WebRequest object. But in the process, we lose all of the waiting time as the request is on the wire and the response hasn't come back yet. We might as well write it in C#... ;)
Questions
Is this a known problem?
Is there any good solution that takes advantage of F# asynchronous workflows and still allows timeouts and error handling?
If the problem is really that I am making too many requests at once, then would the best way to limit the number of request be to use a Semaphore(5, 5) or something like that?
Side Question: if you've looked at my code, can you see any stupid things I've done and could fix?
If there is anything you are confused about, please let me know.

Answer 1

AsyncGetResponse simply ignoring any timeout value posted... here's a solution we just cooked:
open System
open System.IO
open System.Net
type Request = Request of WebRequest * AsyncReplyChannel<WebResponse>
let requestAgent =
MailboxProcessor.Start <| fun inbox -> async {
while true do
let! (Request (req, port)) = inbox.Receive ()
async {
try
let! resp = req.AsyncGetResponse ()
port.Reply resp
with
| ex -> sprintf "Exception in child %s\n%s" (ex.GetType().Name) ex.Message |> Console.WriteLine
} |> Async.Start
}
let getHTML url =
async {
try
let req = "http://" + url |> WebRequest.Create
try
use! resp = requestAgent.PostAndAsyncReply ((fun chan -> Request (req, chan)), 1000)
use str = resp.GetResponseStream ()
use rdr = new StreamReader (str)
return Some <| rdr.ReadToEnd ()
with
| :? System.TimeoutException ->
req.Abort()
Console.WriteLine "RequestAgent call timed out"
return None
with
| ex ->
sprintf "Exception in request %s\n\n%s" (ex.GetType().Name) ex.Message |> Console.WriteLine
return None
} |> Async.RunSynchronously;;
getHTML "www.grogogle.com"
i.e. We're delegating to another agent and calling it providing an async timeout... if we do not get a reply from the agent in the specified amount of time we abort the request and move on.

Answer 2

I see my other answer may fail to answer your particular question... here's another implementation for a task limiter that doesn't require the use of semaphore.
open System
type IParallelLimiter =
abstract GetToken : unit -> Async<IDisposable>
type Message=
| GetToken of AsyncReplyChannel<IDisposable>
| Release
let start count =
let agent =
MailboxProcessor.Start(fun inbox ->
let newToken () =
{ new IDisposable with
member x.Dispose () = inbox.Post Release }
let rec loop n = async {
let! msg = inbox.Scan <| function
| GetToken _ when n = 0 -> None
| msg -> async.Return msg |> Some
return!
match msg with
| Release ->
loop (n + 1)
| GetToken port ->
port.Reply <| newToken ()
loop (n - 1)
}
loop count)
{ new IParallelLimiter with
member x.GetToken () =
agent.PostAndAsyncReply GetToken}
let limiter = start 100;;
for _ in 0..1000 do
async {
use! token = limiter.GetToken ()
Console.WriteLine "Sleeping..."
do! Async.Sleep 3000
Console.WriteLine "Releasing..."
} |> Async.Start