I come from a C# background having used async/ await. I am trying to find a "less verbose" way of programming using a library.
( specifically the Microsoft Playwright library for browser automation )
let (~~) = Async.AwaitTask
let getLastPageNumber(page: IPage) =
let playwright = ~~Playwright.CreateAsync() |> Async.RunSynchronously
let browser = ~~playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions(Headless = false )) |> Async.RunSynchronously
~~page.GotoAsync("https://go.xero.com/BankRec/BankRec.aspx?accountID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&page1") |> Async.RunSynchronously |> ignore
let lastPageLink = ~~page.QuerySelectorAsync("#mainPagerEnd") |> Async.RunSynchronously
if lastPageLink = null then
//this is the last page
1
else
let lastPageNumber = ~~lastPageLink.GetAttributeAsync("href") |> Async.RunSynchronously
lastPageNumber |> int
I have shortened things a bit using the alias ~~ for Async.AwaitTask but it seems to be a lot of code to do something that was a lot easier in C#.
Async.RunSynchronously should only be used as a very last resort because it blocks a thread to perform the computation, which defeats the purpose of using async/tasks.
The F# equivalent of C#'s async/await is to use F#'s Async type, and the async computation expression. However, if you're using a .NET library which uses the .NET Task type then you can use the TaskBuilder.fs library which has a task computation expression.
Then you would write the function like this:
open FSharp.Control.Tasks
let getLastPageNumber(page: IPage) = task {
let! playwright = Playwright.CreateAsync()
let! browser = playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions(Headless = false ))
let! _ = page.GotoAsync("https://go.xero.com/BankRec/BankRec.aspx?accountID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&page1")
let! lastPageLink = page.QuerySelectorAsync("#mainPagerEnd")
if lastPageLink = null then
//this is the last page
return 1
else
let! lastPageNumber = lastPageLink.GetAttributeAsync("href")
return lastPageNumber |> int
}
Inside the computation expression (also called a builder) let! is used to await tasks.
Note that this function now returns Task<int> rather than int, so the caller would probably need to follow a similar pattern and propogate the task further.
You can read more about using async in F# here. Some of that knowledge can also be applied to the task computation expression.
Related
I have a function that I want to run 5 at a time from a chunked list and then wait a second so as not to hit an Api rate limit. I know the code is wrong but it is as far as I have got.
let ordersChunked = mOrders |> List.chunkBySize 5
for fiveOrders in ordersChunked do
let! tasks =
fiveOrders
|> List.map (fun (order) -> trackAndShip(order) )
|> Async.Parallel
Async.Sleep(1000) |> ignore
trackAndShip is an async function not a task and I get a little confused about the two.
Some of the answers I have read add |> Async.RunSynchronously to the end of it - and I do that at the top level but I don't feel it is good practice here ( and I would need to convert async to Task? )
This is probably a very simple answer if you are familiar with async / tasks but I am still "finding my feet" in this area.
Unlike C#'s tasks, Async computation do not start when they are created and you have to explicitly start them. Furthermore, even if you would start the Async operation, it won't do what you expect because the sleep will not execute because you just ignore it.
From the let! in your example code, I'm going to assume you're the code snippet is inside a computation expression. So the following may be what you want
let asyncComputation =
async {
let ordersChunked = [] |> List.chunkBySize 5
for fiveOrders in ordersChunked do
let! tasks =
fiveOrders
|> List.map (fun (order) -> trackAndShip(order) )
|> Async.Parallel
do! Async.Sleep(1000) // Note the do! here, without it your sleep will be ignored
}
Async.RunSynchronously asyncComputation
Note that the Async.Sleep is now chained into the CE by using do!. The Async.RunSynchronously call is one way to start the async computation: it runs the async operation and blocks the current thread while waiting for the async operation's result. There are many other ways of running it described in the documentation.
I am trying to make the code below working
type Message = {
Text: string
}
[<EntryPoint>]
let main argv =
use bus = RabbitHutch.CreateBus("host=localhost")
async {
let! result = bus.PubSub.PublishAsync({Text = "text"}) |> Async.AwaitTask
}
0
Knowing that PublishAsync returns Task<unit>. I just wanted to equivalently await this method in F# method but this triggers an error:
Program.fs(14, 5): [FS0010] Unexpected symbol '}' in expression
I already checked that answer on SO, not really sure what I am supposed to do, adding |> ignore does not seem to do anything.
[EDIT]
I added an ignore:
type Message = {
Text: string
}
[<EntryPoint>]
let main argv =
use bus = RabbitHutch.CreateBus("host=localhost")
async {
let! result = bus.PubSub.PublishAsync({Text = "text"}) |> Async.AwaitTask
result |> ignore
}
0
But now I get the following warning:
Program.fs(10, 5): [FS0020] The result of this expression has type 'Async<unit>' and is implicitly ignored. Consider using 'ignore' to discard this value explicitly, e.g. 'expr |> ignore', or 'let' to bind the result to a name, e.g. 'let result = expr'.
Can't get rid of that one though, any idea?
The result of this expression:
async {
let! result = bus.PubSub.PublishAsync({Text = "text"}) |> Async.AwaitTask
result |> ignore
}
is a value of Async<unit> type. In F# and functional programming, every expression should contribute to the result of the bigger expression, otherwise it is likely a mistake. Hence your compiler reported the warning.
To tell compiler that you are okay with that, simply transform the value to the () value of the unit type (which is similar to void type in other languages) by using the ignore function.
Now look into the body of your async, in your old code, the last statement is let! result = ... and it is invalid in F# syntax. Every let or let! is the short form of let name=... in someExpression but there is no expression here except the } that’s why compiler said Unexpected symbol '}' in expression.
You fixed that by piping result to ignore. That’s okay because that is a valid expression.
Now consider your new code, you can ignore the Async<unit> to dismiss the warning, but that makes your async nonsense. You declared the async, but did not execute it. Probably you need to execute it with the Async.RunSynchronously function:
async {
let! result = bus.PubSub.PublishAsync({Text = "text"}) |> Async.AwaitTask
result |> ignore
}
|> Async.RunSynchronously
Since result is a unit value, you can clean your code with do!:
async {
do! bus.PubSub.PublishAsync({Text = "text"}) |> Async.AwaitTask
}
|> Async.RunSynchronously
Or even cleaner:
bus.PubSub.PublishAsync({Text = "text"})
|> Async.AwaitTask
|> Async.RunSynchronously
Your async computation expression returns Async<unit>. You need to actually run the computation. You can do that with Async.RunSynchronously.
async {
let! result = bus.PubSub.PublishAsync({Text = "text"}) |> Async.AwaitTask
result |> ignore
} |> Async.RunSynchronously
Though at this point, I think the computation expression is redundant. This would do the same:
bus.PubSub.PublishAsync({Text = "text"})
|> Async.AwaitTask
|> Async.RunSynchronously
I'm trying to understand async workflows in F# but I found one part that I really don't understand.
The following code works fine:
let asynWorkflow = async{
let! result = Stream.TryOpenAsync(partition) |> Async.AwaitTask
return result
}
let stream = Async.RunSynchronously asynWorkflow
|> fun openResult -> if openResult.Found then openResult.Stream else Stream(partition)
I define a async workflow where TryOpenAsync returns a Task<StreamOpenResult> type. I convert it to Async<StreamOpenResult> with Async.AwaitTask. (Side quest: "Await"Task? It doesn't await it just convert it, does it? I think it has nothing to do with Task.Wait or the await keyword). I "await" it with let! and return it.
To start the workflow I use RunSynchronously which should start the workflow and return the result (bind it). On the result I check if the Stream is Found or not.
But now to my first question. Why do I have to wrap the TryOpenAsync call in another async computation and let! ("await") it?
E.g. the following code does not work:
let asynWorkflow = Stream.TryOpenAsync(partition) |> Async.AwaitTask
let stream = Async.RunSynchronously asynWorkflow
|> fun openResult -> if openResult.Found then openResult.Stream else Stream(partition)
I thought the AwaitTask makes it an Async<T> and RunSynchronously should start it. Then use the result. What do I miss?
My second question is why is there any "Async.Let!" function available? Maybe because it does not work or better why doesn't it work with the following code?
let ``let!`` task = async{
let! result = task |> Async.AwaitTask
return result
}
let stream = Async.RunSynchronously ( ``let!`` (Stream.TryOpenAsync(partition)) )
|> fun openResult -> if openResult.Found then openResult.Stream else Stream(partition)
I just insert the TryOpenAsync as a parameter but it does not work. By saying does not work I mean the whole FSI will hang. So it has something to do with my async/"await".
--- Update:
Result of working code in FSI:
>
Real: 00:00:00.051, CPU: 00:00:00.031, GC gen0: 0, gen1: 0, gen2: 0
val asynWorkflow : Async<StreamOpenResult>
val stream : Stream
Result of not working code in FSI:
>
And you cannot execute anything in the FSI anymore
--- Update 2
I'm using Streamstone. Here the C# example: https://github.com/yevhen/Streamstone/blob/master/Source/Example/Scenarios/S04_Write_to_stream.cs
and here the Stream.TryOpenAsync: https://github.com/yevhen/Streamstone/blob/master/Source/Streamstone/Stream.Api.cs#L192
I can't tell you why the second example doesn't work without knowing what Stream and partition are and how they work.
However, I want to take this opportunity to point out that the two examples are not strictly equivalent.
F# async is kind of like a "recipe" for what to do. When you write async { ... }, the resulting computation is just sitting there, not actually doing anything. It's more like declaring a function than like issuing a command. Only when you "start" it by calling something like Async.RunSynchronously or Async.Start does it actually run. A corollary is that you can start the same async workflow multiple times, and it's going to be a new workflow every time. Very similar to how IEnumerable works.
C# Task, on the other hand, is more like a "reference" to an async computation that is already running. The computation starts as soon as you call Stream.TryOpenAsync(partition), and it's impossible to obtain a Task instance before the task actually starts. You can await the resulting Task multiple times, but each await will not result in a fresh attempt to open a stream. Only the first await will actually wait for the task's completion, and every subsequent one will just return you the same remembered result.
In the async/reactive lingo, F# async is what you call "cold", while C# Task is referred to as "hot".
The second code block looks like it should work to me. It does run it if I provide dummy implementations for Stream and StreamOpenResult.
You should avoid using Async.RunSynchronously wherever possible because it defeats the purpose of async. Put all of this code within a larger async block and then you will have access to the StreamOpenResult:
async {
let! openResult = Stream.TryOpenAsync(partition) |> Async.AwaitTask
let stream = if openResult.Found then openResult.Stream else Stream(partition)
return () // now do something with the stream
}
You may need to put a Async.Start or Async.RunSynchronously at the very outer edge of your program to actually run it, but it's better if you have the async (or convert it to a Task) and pass it to some other code (e.g. a web framework) that can call it in a non-blocking manner.
Not that I want to answer your question with another question, but: why are you doing code like this anyway? That might help to understand it. Why not just:
let asyncWorkflow = async {
let! result = Stream.TryOpenAsync(partition) |> Async.AwaitTask
if result.Found then return openResult.Stream else return Stream(partition) }
There's little point in creating an async workflow only to immediately call RunSynchronously on it - it's similar to calling .Result on a Task - it just blocks the current thread until the workflow returns.
I have an array holding a large number of small async database queries; for example:
// I actually have a more complex function that
// accepts name/value pairs for query parameters.
let runSql connString sql = async {
use connection = new SqlConnection(connString)
use command = new SqlCommand(sql, connection)
do! connection.OpenAsync() |> Async.AwaitIAsyncResult |> Async.Ignore
return! command.ExecuteScalarAsync() |> Async.AwaitTask
}
let getName (id:Guid) = async {
// I actually use a parameterized query
let querySql = "SELECT Name FROM Entities WHERE ID = '" + id.ToString() + "'"
return! runSql connectionString querySql
}
let ids : Guid array = getSixtyThousandIds()
let asyncWorkflows = ids |> Array.map getName
//...
Now, the problem: The next expression runs all 60K workflows at once, flooding the server. This leads to many of the SqlCommands timing out; it also typically causes out of memory exceptions in the client (which is F# interactive) for reasons I do not understand and (not needing to understand them) have not investigated:
//...
let names =
asyncWorkflows
|> Async.Parallel
|> Async.RunSynchronously
I've written a rough-and-ready function to batch the requests:
let batch batchSize asyncs = async {
let batches = asyncs
|> Seq.mapi (fun i a -> i, a)
|> Seq.groupBy (fst >> fun n -> n / batchSize)
|> Seq.map (snd >> Seq.map snd)
|> Seq.map Async.Parallel
let results = ref []
for batch in batches do
let! result = batch
results := (result :: !results)
return (!results |> List.rev |> Seq.collect id |> Array.ofSeq)
}
To use this function, I replace Async.Parallel with batch 20 (or another integer value):
let names =
asyncWorkflows
|> batch 20
|> Async.RunSynchronously
This works reasonably well, but I would prefer to have a system that starts each new async as soon as one completes, so rather than successive batches of size N starting after each previous batch of size N has finished, I am always awaiting N active SqlCommands (until I get to the end, of course).
Questions:
Am I reinventing the wheel? In other words, are there library functions that do this already? (Would it be profitable to look into exploiting ParallelEnumerable.WithDegreeOfParallelism somehow?)
If not, how should I implement a continuous queue instead of a series of discrete batches?
I am not primarily seeking suggestions to improve the existing code, but such suggestions will nonetheless be received with interest and gratitude.
FSharpx.Control offers an Async.ParallelWithThrottle function. I'm not sure if it is the best implementation as it uses SemaphoreSlim. But the ease of use is great and since my application doesn't need top performance it works well enough for me. Although since it is a library if someone knows how to make it better it is always a nice thing to make libraries top performers out of the box so the rest of us can just use the code that works and just get our work done!
Async.Parallel had support for throttling added in FSharp v 4.7. You do:
let! results = Async.Parallel(workflows, maxDegreeOfParallelism = dop)
if doing more than 1200 workflows concurrently in FSharp.Core versions <= 6.0.5, see this resolved issue
Proposal for a more explicit API
Is there any primitive in the langage to compose async1 then async2, akin to what parallel does for parallel execution planning ?
to further clarify, I have 2 async computations
let toto1 = Async.Sleep(1000)
let toto2 = Async.Sleep(1000)
I would like to create a new async computation made of the sequential composition of toto1 and toto2
let toto = Async.Sequential [|toto1; toto2|]
upon start, toto would run toto1 then toto2, and would end after 2000 time units
The async.Bind operation is the basic primitive that asynchronous workflows provide for sequential composition - in the async block syntax, that corresponds to let!. You can use that to express sequential composition of two computations (as demonstrated by Daniel).
However, if you have an operation <|> that Daniel defined, than that's not expressive enough to implement async.Bind, because when you compose things sequentially using async.Bind, the second computation may depend on the result of the first one. The <e2> may use v1:
async.Bind(<e1>, fun v1 -> <e2>)
If you were writing <e1> <|> <e2> then the two operations have to be independent. This is a reason why the libraries are based on Bind - because it is more expressive form of sequential composition than the one you would get if you were following the structure of Async.Parallel.
If you want something that behaves like Async.Parallel and takes an array, then the easiest option is to implement that imperatively using let! in a loop (but you could use recursion and lists too):
let Sequential (ops:Async<'T>[]) = async {
let res = Array.zeroCreate ops.Length
for i in 0 .. ops.Length - 1 do
let! value = ops.[i]
res.[i] <- value
return res }
I'm not sure what you mean by "primitive." Async.Parallel is a function. Here are a few ways to run two asyncs:
In parallel:
Async.Parallel([|async1; async2|])
or
async {
let! child = Async.StartChild async2
let! result1 = child
let! result2 = async1
return [|result1; result2|]
}
Sequentially:
async {
let! result1 = async1
let! result2 = async2
return [|result1; result2|]
}
You could return tuples in the last two. I kept the return types the same as the first one.
I would say let! and do! in an async { } block is as close as you'll get to using a primitive for this.
EDIT
If all this nasty syntax is getting to you, you could define a combinator:
let (<|>) async1 async2 =
async {
let! r1 = async1
let! r2 = async2
return r1, r2
}
and then do:
async1 <|> async2 |> Async.RunSynchronously