I just spent a long time breaking my teeth on why this code was 'hanging' for some urls:
let getImage (imageUrl:string) =
async {
try
let req = WebRequest.Create(imageUrl) :?> HttpWebRequest
req.UserAgent <- "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)";
req.Method <- "GET";
req.AllowAutoRedirect <- true;
req.MaximumAutomaticRedirections <- 4;
req.Timeout <- 3000; //HAHAHA, nice try!
let! response1 = req.AsyncGetResponse()
let response = response1 :?> HttpWebResponse
use stream = response.GetResponseStream()
let ms = new MemoryStream()
let bytesRead = ref 1
let buffer = Array.create 0x1000 0uy
while !bytesRead > 0 do
bytesRead := stream.Read(buffer, 0, buffer.Length)
ms.Write(buffer, 0, !bytesRead)
return SuccessfulDownload(imageUrl, ms.ToArray())
with
ex -> return FailedDownload(imageUrl, ex.Message)
}
After managing to track down which of the 3000 urls was hanging, I learned that AsyncGetResponse doesn't take any notice of HttpWebRequest.Timeout. I've done a bit of searching which throws up suggestions of wrapping the async request in a thread with a timer. That's great for C#, but if I'm running 3000 of these through Async.Parallel |> Async.RunSynchronously, what's the best way to handle this problem?
I've only roughly tested this, but it should have the correct behavior:
type System.Net.WebRequest with
member req.AsyncGetResponseWithTimeout () =
let impl = async {
let iar = req.BeginGetResponse (null, null)
let! success = Async.AwaitIAsyncResult (iar, req.Timeout)
return if success then req.EndGetResponse iar
else req.Abort ()
raise (System.Net.WebException "The operation has timed out") }
Async.TryCancelled (impl, fun _ -> req.Abort ())
In your code, call req.AsyncGetResponseWithTimeout() instead of req.AsyncGetResponse().
Related
Lets assume we have the following C# method
public async Task DoStuffAsync(string path)
{
var fullPath = Path.Combine(
Directory.GetCurrentDirectory(),
path
);
using var stream = File.OpenText(fullPath);
while (!stream.EndOfStream)
{
var line = await stream.ReadLineAsync();
await Console.Out.WriteLineAsync(line);
}
}
and we want to implement this in F#, my first attempt was to code it like this
let doStuffAsync path =
async {
let fullPath = Path.Combine(Directory.GetCurrentDirectory(), path)
use stream = File.OpenText fullPath
while (not stream.EndOfStream) do
let! line = stream.ReadLineAsync() |> Async.AwaitTask
System.Console.Out.WriteLineAsync(line) |> Async.AwaitTask |> ignore
}
but that does not seam right to me as we have to unpack a Task<string> into a Async<string> then unpack this. Same for the Task from WriteLineAsync.
Are there no F# variantes for the methods ReadLineAsync and WriteLineAsync which return Async<_> instead of Task<_>?
Also is there somekind of async pipe operaor that would let me do
stream.ReadLineAsync()
|> Async.AwaitTask
|> System.Console.Out.WriteLineAsync
|> Async.AwaitTask
|> ignore
The use of Async.AwaitTask is correct as #NghiaBui pointed out. These are two async models and you need to convert in between them.
I'm afraid there are no Async<'T> variants of ReadlineAsync and most .net core libraries.
You could use TaskBuilder for some cleaner syntax:
task {
let fullPath = Path.Combine(Directory.GetCurrentDirectory(), path)
use stream = File.OpenText fullPath
while (not stream.EndOfStream) do
let! line = stream.ReadLineAsync()
do! System.Console.Out.WriteLineAsync line
}
Regarding the piping, I'm afraid it's not possible. There are some discussions to bring in a |>! operator but it looks like it would add more complexity than it's necessary.
I have some potentially very long running function, which may sometimes hang up. So, I thought that if I wrap it into an async workflow, then I should be able to cancel it. Here is an FSI example that does not work (but the same behavior happens with the compiled code):
open System.Threading
let mutable counter = 0
/// Emulates an external C# sync function that hung up.
/// Please, don't change it to some F# async stuff because
/// that won't fix that C# method.
let run() =
while true
do
printfn "counter = %A" counter
Thread.Sleep 1000
counter <- counter + 1
let onRunModel() =
let c = new CancellationTokenSource()
let m = async { do run() }
Async.Start (m, c.Token)
c
let tryCancel() =
printfn "Starting..."
let c = onRunModel()
printfn "Waiting..."
Thread.Sleep 5000
printfn "Cancelling..."
c.Cancel()
printfn "Waiting again..."
Thread.Sleep 5000
printfn "Completed."
#time
tryCancel()
#time
If you run it in FSI, then you will see something like that:
Starting...
Waiting...
counter = 0
counter = 1
counter = 2
counter = 3
counter = 4
Cancelling...
Waiting again...
counter = 5
counter = 6
counter = 7
counter = 8
counter = 9
Completed.
Real: 00:00:10.004, CPU: 00:00:00.062, GC gen0: 0, gen1: 0, gen2: 0
counter = 10
counter = 11
counter = 12
counter = 13
counter = 14
counter = 15
counter = 16
which means that it does not stop at all after c.Cancel() is called.
What am I doing wrong and how to make such thing work?
Here is some additional information:
When the code hangs up, it does it in some external sync C# library,
which I have no control of. So checking for cancellation token in
the code that I control is useless. That's why function run()
above was modeled that way.
I don't need any communication of completion and / or progress. It's already done via some messaging system and it is out of scope
of the question.
Basically I just need to kill background work as soon as I "decide" to do so.
You are handing off control to a code segment, which, albeit wrapped in an async block, has no means of checking for the cancellation. Were you to construct your loop directly wrapped in an async, or have it replaced by a recursive async loop, it will work as expected:
let run0 () = // does not cancel
let counter = ref 0
while true do
printfn "(0) counter = %A" !counter
Thread.Sleep 1000
incr counter
let m = async { run0 () }
let run1 () = // cancels
let counter = ref 0
async{
while true do
printfn "(1) counter = %A" !counter
Thread.Sleep 1000
incr counter }
let run2 = // cancels too
let rec aux counter = async {
printfn "(2) counter = %A" counter
Thread.Sleep 1000
return! aux (counter + 1) }
aux 0
printfn "Starting..."
let cts = new CancellationTokenSource()
Async.Start(m, cts.Token)
Async.Start(run1(), cts.Token)
Async.Start(run2, cts.Token)
printfn "Waiting..."
Thread.Sleep 5000
printfn "Cancelling..."
cts.Cancel()
printfn "Waiting again..."
Thread.Sleep 5000
printfn "Completed."
A word of caution though: Nested async calls in F# are automatically checked for cancellation, which is why do! Async.Sleep is preferable. If you are going down the recursive route, be sure to enable tail-recursion via return!. Further reading: Scott W.'s blog on Asynchronous programming, and Async in C# and F# Asynchronous gotchas in C# by Tomas Petricek.
This piece of code was developed to solve a situation where I couldn't get some calls to terminate/timeout. They would just hang. Maybe you can get some ideas that will help you solve your problem.
The interesting part for you would be only the two first functions. The rest is only to demonstrate how I'm using them.
module RobustTcp =
open System
open System.Text
open System.Net.Sockets
open Railway
let private asyncSleep (sleepTime: int) (error: 'a) = async {
do! Async.Sleep sleepTime
return Some error
}
let private asyncWithTimeout asy (timeout: int) (error: 'a) =
Async.Choice [ asy; asyncSleep timeout error ]
let private connectTcpClient (host: string) (port: int) (tcpClient: TcpClient) = async {
let asyncConnect = async {
do! tcpClient.ConnectAsync(host, port) |> Async.AwaitTask
return Some tcpClient.Connected }
match! asyncWithTimeout asyncConnect 1_000 false with
| Some isConnected -> return Ok isConnected
| None -> return Error "unexpected logic error in connectTcpClient"
}
let private writeTcpClient (outBytes: byte[]) (tcpClient: TcpClient) = async {
let asyncWrite = async {
let stream = tcpClient.GetStream()
do! stream.WriteAsync(outBytes, 0, outBytes.Length) |> Async.AwaitTask
do! stream.FlushAsync() |> Async.AwaitTask
return Some (Ok ()) }
match! asyncWithTimeout asyncWrite 10_000 (Error "timeout writing") with
| Some isWrite -> return isWrite
| None -> return Error "unexpected logic error in writeTcpClient"
}
let private readTcpClient (tcpClient: TcpClient) = async {
let asyncRead = async {
let inBytes: byte[] = Array.zeroCreate 1024
let stream = tcpClient.GetStream()
let! byteCount = stream.ReadAsync(inBytes, 0, inBytes.Length) |> Async.AwaitTask
let bytesToReturn = inBytes.[ 0 .. byteCount - 1 ]
return Some (Ok bytesToReturn) }
match! asyncWithTimeout asyncRead 2_000 (Error "timeout reading reply") with
| Some isRead ->
match isRead with
| Ok s -> return Ok s
| Error error -> return Error error
| None -> return Error "unexpected logic error in readTcpClient"
}
let sendReceiveBytes (host: string) (port: int) (bytesToSend: byte[]) = async {
try
use tcpClient = new TcpClient()
match! connectTcpClient host port tcpClient with
| Ok isConnected ->
match isConnected with
| true ->
match! writeTcpClient bytesToSend tcpClient with
| Ok () ->
let! gotData = readTcpClient tcpClient
match gotData with
| Ok result -> return Ok result
| Error error -> return Error error
| Error error -> return Error error
| false -> return Error "Not connected."
| Error error -> return Error error
with
| :? AggregateException as ex ->
(* TODO ? *)
return Error ex.Message
| ex ->
(*
printfn "Exception in getStatus : %s" ex.Message
*)
return Error ex.Message
}
let sendReceiveText (host: string) (port: int) (textToSend: string) (encoding: Encoding) =
encoding.GetBytes textToSend
|> sendReceiveBytes host port
|> Async.map (Result.map encoding.GetString)
I am confused about the
let decode buffer = Encoding.UTF8.GetString(buffer)
line, what is going on there ?
buffer is declared inside async, or is it a closure ?
https://github.com/sebfia/OffLog/blob/master/Shared/FeedStorage.fs#L22
let thaw<'T> path =
let decode buffer = Encoding.UTF8.GetString(buffer)
async {
try
match File.Exists(path) with
| false -> return None
| true ->
use fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.None)
let! buffer = fs.AsyncRead(fs.Length |> Convert.ToInt32)
let str = decode buffer
System.Diagnostics.Debug.WriteLine(str)
let bla = JsonConvert.DeserializeObject<'T>(str)
System.Diagnostics.Debug.WriteLine(typeof<'T>.GetType().FullName)
return bla |> Some
with | e -> System.Diagnostics.Debug.WriteLine(e.ToString()); return None
}
Edit: This turned out to be an F# bug which can be worked-around by using a custom option type instead of Fsharp's "Option".
In F#, I am trying to call a .net Task with Async.AwaitTask. The task is throwing an exception, and I can't seem to catch it with either try-catch or Async.Catch. And I know of no other way to catch exceptions. What is the solution? And what is the cause of the problem? Thanks for any explanation.
Here is some test code that shows my failure to catch the exception thrown by DownloadStringTaskAsync:
open System
open System.Net
[<EntryPoint>]
let main argv =
let test =
async{
let! exc = Async.Catch( async{
try
let w = new Net.WebClient();
let! str = Async.AwaitTask (w.DownloadStringTaskAsync "") // throws ArgumentException
return Some str
with
| _ ->
return None // not caught
}
)
match exc with
| Choice1Of2 r -> return r
| Choice2Of2 ext -> return None // not caught
}
let res = Async.RunSynchronously(test)
let str = Console.ReadLine();
0 // return an integer exit code
I need some way to catch the exception inside the "test" async function, preventing it to "bubble up".
I found this related question, but I can't seem to adapt the answer (which deals with Task, not with Task<'a>) to my needs.
Edit: Here is a screenshot showing the problem: https://i.gyazo.com/883f546c00255b210e53cd095b876cb0.png
"ArgumentException was unhandled by user code."
(Visual Studio 2013, .NET 4.5.1, Fsharp 3.1, Fsharp.core.dll 4.3.1.0.)
Could it be that the code is correct but some Fsharp or Visual Studio setting is preventing the exception from being caught?
TL;DR: The exception does get caught, it appears to be returning None out of async that's confusing here - the result is null, not None, which is screwing up pattern matching and generally being a bother. Use your own union type instead of Option to handle it.
Edit: Oh, and #takemyoxygen's comments about why you're seeing it straight away is correct. Debug->Exceptions, or just untick "Break when this exception type is user-unhandled" in the pop-up visible in your screenshot. It looks like VS is breaking on THROW, rather than actually on unhandled.
First, returning Some/None inside an Async.Catch makes it useless. Async.Catch returns Choice1Of2(val) unless there's an uncaught exception, so the code as is (if it worked) would return Choice1Of2(Some(string)) with no exception, or Choice1Of2(None) with exception. Either use try/with and handle the exception yourself, or only use Async.Catch.
Experiments:
Proof that we are catching: Add a debug output to the "with". The code DOES get there (add a breakpoint or look at debug output for proof). We get Choice1Of2(null), rather than Choice1Of2(None) in exc though. WEIRD. See image: http://i.imgur.com/e8Knx5a.png
open System
open System.Net
[<EntryPoint>]
let main argv =
let test =
async{
let! exc = Async.Catch( async{
try
let w = new Net.WebClient();
let! str = Async.AwaitTask (w.DownloadStringTaskAsync "") // throws ArgumentException
return Some str
with
| _ ->
System.Diagnostics.Debug.WriteLine "in with" // We get here.
return None // not caught
}
)
match exc with
| Choice1Of2 r -> return r
| Choice2Of2 ext -> return None // not caught
}
let res = Async.RunSynchronously(test)
let str = Console.ReadLine();
0 // return an integer exit code
Remove Async.Catch: Don't use Async.Catch (keep the debug output though). Again, we get to the debug output, so we're catching the exception, and as we're not wrapping in a Choice from Async.Catch, we get null instead of None. STILL WEIRD.
open System
open System.Net
[<EntryPoint>]
let main argv =
let test = async {
try
let w = new Net.WebClient();
let! str = Async.AwaitTask (w.DownloadStringTaskAsync "") // throws ArgumentException
return Some str
with
| _ ->
System.Diagnostics.Debug.WriteLine "in with"
return None }
let res = Async.RunSynchronously(test)
let str = Console.ReadLine();
0 // return an integer exit code
Only use Async.Catch: Don't use a try/with, just Async.Catch. This works a bit better. At the pattern match on exc, I have a Choice2Of2 containing the exception. However, when the None is returned out of the outermost async, it becomes null.
open System
open System.Net
[<EntryPoint>]
let main argv =
let test = async {
let! exc = Async.Catch(async {
let w = new Net.WebClient();
let! str = Async.AwaitTask (w.DownloadStringTaskAsync "") // throws ArgumentException
return str })
match exc with
| Choice1Of2 v -> return Some v
| Choice2Of2 ex -> return None
}
let res = Async.RunSynchronously(test)
let str = Console.ReadLine();
0 // return an integer exit code
Don't use Option: Interestingly, if you use a custom union type, it works perfectly:
open System
open System.Net
type UnionDemo =
| StringValue of string
| ExceptionValue of Exception
[<EntryPoint>]
let main argv =
let test = async {
let! exc = Async.Catch(async {
let w = new Net.WebClient();
let! str = Async.AwaitTask (w.DownloadStringTaskAsync "") // throws ArgumentException
return str })
match exc with
| Choice1Of2 v -> return StringValue v
| Choice2Of2 ex -> return ExceptionValue ex
}
let res = Async.RunSynchronously(test)
let str = Console.ReadLine();
0 // return an integer exit code
Using a try/with and no Async.Catch also works with a new Union:
open System
open System.Net
type UnionDemo =
| StringValue of string
| ExceptionValue of Exception
[<EntryPoint>]
let main argv =
let test = async {
try
let w = new Net.WebClient();
let! str = Async.AwaitTask (w.DownloadStringTaskAsync "") // throws ArgumentException
return StringValue str
with
| ex -> return ExceptionValue ex }
let res = Async.RunSynchronously(test)
let str = Console.ReadLine();
0 // return an integer exit code
It works even if the union is defined as the following:
type UnionDemo =
| StringValue of string
| ExceptionValue
The following gets null in moreTestRes.
[<EntryPoint>]
let main argv =
let moreTest = async {
return None
}
let moreTestRes = Async.RunSynchronously moreTest
0
Why this is the case I don't know (still happens in F# 4.0), but the exception is definitely being caught, but None->null is screwing it up.
let failing = async {
failwith "foo"
}
let test () =
try
Async.Start(failing)
with
| exn -> printf "caught"
This code doesn't catch the exception. How can I start an asynchronous workflow on a separate thread and catch the exception in the main program?
as an alternative you start the workflow as a task and use it's methods and properties instead. For example Task.Result will rethrow an exception again so this works, and is almost what you tried:
let test () =
try
Async.StartAsTask failing
|> fun t -> t.Result
with _ -> printfn "caught"
run
> test ();;
caught
val it : unit = ()
on a differnt thread
sorry - I just saw that you want it on a different thread - in this case you most likely want to use the internal approach RCH gave you - but you could use ContinueWith too (although a bit ugly):
open System.Threading.Tasks
let test () =
(Async.StartAsTask failing).ContinueWith(fun (t : Task<_>) -> try t.Result with _ -> printfn "caught")
run
> test ();;
caught
val it : Task = System.Threading.Tasks.Task {AsyncState = null;
CreationOptions = None;
Exception = null;
Id = 3;
IsCanceled = false;
IsCompleted = true;
IsFaulted = false;
Status = RanToCompletion;}
without Async.Catch
also you don't really need the Async.Catch:
let test () =
async {
try
do! failing
with _ -> printfn "caught"
} |> Async.Start
As there is no result awaited, there is no place where the exception could be caught. You need to wrap the computation. One possibility:
let failing = async {
failwith "foo"
}
let test () =
async {
let! res = failing |> Async.Catch
match res with
| Choice1Of2 _ -> printf "success"
| Choice2Of2 exn -> printfn "failed with %s" exn.Message
} |> Async.Start