I'm using the B-Pipe API to request financial securities data from Bloomberg. Large requests are divided into small groups and returned as PARTIAL_RESPONSE with the final response being RESPONSE. The recursive handleSingleEvent function recurses until a final RESPONSE is returned by reqEventQueue.NextEvent().
My problem is that only the final RESPONSE group of securities are returned. The intermediate PARTIAL_RESPONSE groups are not returned by the function. However, these PARTIAL_RESPONSE groups are received from Bloomberg.
I'm thinking I might need some accumulator parameters or should use something similar to List.collect, but I don't have a clue how to get started.
// list<Request> * EventQueue * Session -> list<list<string>>
let handleEvents (requests: Request list, reqEventQueue: EventQueue, session: Session) =
let rec handleSingleEvent (request: Request) =
try
let eventObj = reqEventQueue.NextEvent()
match eventObj.Type with
| Event.EventType.PARTIAL_RESPONSE ->
processReferenceResponseEvent eventObj
|> makeJson
|> ignore
handleSingleEvent request
| Event.EventType.RESPONSE -> processReferenceResponseEvent eventObj |> makeJson
| _ -> processMiscEvents eventObj |> makeJson
with
| ex -> failwithf "%s" ex.Message
List.map (fun request -> handleSingleEvent request) requests
Update
I made edits, but now code seems to enter infinite recursion.
let handleEvents (requests: Request list, reqEventQueue: EventQueue, session: Session) =
let rec handleSingleEvent (request: Request) : seq<list<string>> =
seq {
let eventObj = reqEventQueue.NextEvent()
match eventObj.Type with
| Event.EventType.REQUEST_STATUS -> yield processMiscEvents eventObj |> makeJson
| Event.EventType.ADMIN -> yield processAdminEvent eventObj |> makeJson
| Event.EventType.AUTHORIZATION_STATUS -> yield processAuthEvent eventObj session |> makeJson
| Event.EventType.PARTIAL_RESPONSE ->
yield processReferenceResponseEvent eventObj
|> makeJson
yield! handleSingleEvent request
| Event.EventType.RESPONSE -> yield processReferenceResponseEvent eventObj |> makeJson
| _ -> yield processMiscEvents eventObj |> makeJson
} |> ignore
handleSingleEvent request
List.map (fun request -> handleSingleEvent request) requests
I'm not familiar with the Bloomberg API, but this is a fairly common pattern. I think the easiest way to handle it is to generate a sequence of JSON strings recursively, using yield to process the current event and yield! to process the remainder of the queue recursively. That way you don't have to worry about accumulating the results manually. So something like this:
let rec loop () : seq<string> = // assumption: makeJson has type _ -> string
seq {
let eventObj = reqEventQueue.NextEvent()
match eventObj.Type with
| Event.EventType.PARTIAL_RESPONSE ->
yield processReferenceResponseEvent eventObj
|> makeJson
yield! loop ()
| Event.EventType.RESPONSE ->
yield processReferenceResponseEvent eventObj
|> makeJson
| _ ->
yield processMiscEvents eventObj
|> makeJson
}
loop ()
There are some other issues in your code that I haven't attempted to address, such as the fact that the request and session values aren't used at all. I assume you can fix that easily enough.
Related
I have a sequence of Result and I would like to accumulate all the Error values yet abort processing and return the first Ok value found. Specifically, I would like to abort processing the remainder of the list. Unfortunately, the approach I have preserves the first Ok found but does not abort processing the rest of the list.
let process : Result<'t, string list> -> Result<'t, string list> =
let st0 = Error []
let acc st e =
match st, e with
| Ok _ , _ -> st
| _ , Ok _ -> e
| Error v, Error vs -> Error (v ++ vs)
Seq.scan acc st0
|> Seq.last
Ideally, a Seq.skipToOrDefault and Seq.takeToOrDefault methods would be nice to have for this.
From your comments, it has become clear that what you'd like to do is to avoid iterating over the whole sequence, stopping once you encounter the first Ok.
Well, sequences already do that by default (they're lazy), and the scan function preserves that property. Let's check:
let mySeq = seq {
for i in 0..3 do
printfn "Returning %d" i
yield i
}
mySeq |> Seq.toList |> ignore
> Returning 0
> Returning 1
> Returning 2
> Returning 3
mySeq |> Seq.take 2 |> Seq.toList |> ignore
> Returning 0
> Returning 1
mySeq
|> Seq.scan (fun _ x -> printfn "Scanning %d" x) ()
|> Seq.take 3
|> Seq.toList |> ignore
> Returning 0
> Scanning 0
> Returning 1
> Scanning 1
Look: we never see "Returning 2" and "Returning 3" after the scan. That's because we're not iterating over the whole sequence, only the piece we need, as determined by Seq.take 3.
But the thing that does force the full iteration in your code is Seq.last. After all, in order to get the last element, you need to iterate over the whole sequence, there is no other way.
But what you can do is stop iteration when you need via Seq.takeWhile. This function takes a predicate and returns only the elements for which the predicate is true, excluding the first one that yields false:
mySeq |> Seq.takeWhile (fun x -> x < 2) |> Seq.toList |> ignore
> Returning 0
> Returning 1
> Returning 2
> val it : int list = [0; 1]
The difficulty in your case is that you also need to return the element that breaks the predicate. In order to do that, you can deploy a little hack: keep around in your folding state a special flag stop: bool, initially set it to false, and switch to true on the element immediately succeeding the one where you need to stop. To keep such state, I am going to use a record:
let st0 = {| prev = Error []; stop = false |}
let acc (s: {| prev: Result<_,string>; stop: bool |}) x =
match s.prev, x with
| Ok _, _ -> {| s with stop = true |} // Previous result was Ok => stop now
| _, Ok _ -> {| s with prev = x |} // Don't stop, but remember the previous result
| Error a, Error b -> {| s with prev = Error (a # b) |}
sourceSequence
|> Seq.scan acc st0
|> Seq.takeWhile (fun s -> not s.stop)
|> Seq.last
|> (fun s -> s.prev)
P.S. also note that in F# list concatenation is #, not ++. Are you coming from Haskell?
I think this is a better solution. However, there is some confusion as to whether Seq.tryPick is always side effect free regardless of the underlying sequence. For list it is such that Seq.tail is required here to advance through it...
let rec scanTo (pred:'u -> bool) (acc:'u -> 'a -> 'u) (st0:'u) (ss:'a seq) = seq {
let q =
ss
|> Seq.tryPick Some
|> Option.bind (acc st0 >> Some)
match q with
| None -> yield! Seq.empty
| Some v when pred v -> yield v
| Some v -> yield v; yield! (scanTo pred acc v (Seq.tail ss))
}
For instance...
let process : Result<'v, string list> seq -> Result<'v, string list> seq = fun aa ->
let mergeErrors acc e =
match acc, e with
| Error ms, Error m -> Error (m # ms)
| _, Ok v -> Ok v
| _, Error m -> Error m
let st0 = Error []
let isOk = function
| Ok _ -> true
| _ -> false
scanTo isOk mergeErrors st0 aa
For the following example, Array.mapFold produces the result ([|1; 4; 12|], 7).
let mapping s x = (s * x, s + x)
[| 1..3 |]
|> Array.mapFold mapping 1
Now suppose our mapping is asynchronous.
let asyncMapping s x = async { return (s * x, s + x) }
I am able to create Array.mapFoldAsync for the following to work.
[| 1..3 |]
|> Array.mapFoldAsync asyncMapping 1
|> Async.RunSynchronously
Is there a succinct way to achieve this without creating Array.mapFoldAsync?
I am asking as a way to learn other techniques - my attempts using Array.fold were horrible.
I don't think it would generally be of much benefit to combine mapFold with an Async function, because the expected result is a tuple ('values * 'accumulator), but using an Async function will at best give you an Async<'values * 'accumulator>. Consider the following attempt to make Array.mapFold work with Async:
let mapping s x = async {
let! s' = s
let! x' = x
return (s' * x', s' + x')
}
[| 1..3 |]
|> Array.map async.Return
|> Array.mapFold mapping (async.Return 1)
Even this doesn't work, because of the type mismatch: The type ''a * Async<'b>' does not match the type 'Async<'c * 'd>'.
You may also have noticed that while there is an Array.Parallel.map, there's no Array.Parallel.fold or Array.Parallel.mapFold. If you try to write your own mapFoldAsync, you may see why. The mapping part is pretty easy, just partially apply Array.map and compose with Async.Parallel:
let mapAsync f = Array.map f >> Async.Parallel
You can implement an async fold as well, but since each evaluation depends on the previous result, you can't leverage Async.Parallel this time:
let foldAsync f state array =
match array |> Array.length with
| 0 -> async.Return state
| length ->
async {
let mutable acc = state
for i = 0 to length - 1 do
let! value = f acc array.[i]
acc <- value
return acc
}
Now, when we try to combine these to build a mapFoldAsync, it becomes apparent that we can't leverage parallel execution on the mapping anymore, because both the values and the accumulator can be based on the result of the previous evaluation. That means our mapFoldAsync will be a modified 'foldAsync', not a composition of it with mapAsync:
let mapFoldAsync (f: 's -> 'a -> Async<'b * 's>) (state: 's) (array: 'a []) =
match array |> Array.length with
| 0 -> async.Return ([||], state)
| length ->
async {
let mutable acc = state
let results = Array.init length <| fun _ -> Unchecked.defaultof<'b>
for i = 0 to length - 1 do
let! (x,y) = f acc array.[i]
results.[i] <- x
acc <- y
return (results, acc)
}
While this will give you a way to do a mapFold with an async mapping function, the only real benefit would be if the mapping function did something with high-latency, such as a service call. You won't be able to leverage parallel execution for speed-up. If possible, I would suggest considering an alternative solution, based on your real-world scenario.
Without external libraries (I recommend to try AsyncSeq or Hopac.Streams)
you could do this:
let mapping s x = (fst s * x, snd s + x) |> async.Return
module Array =
let mapFoldAsync folderAsync (state: 'state) (array: 'elem []) = async {
let mutable finalState = state
for elem in array do
let! nextState = folderAsync finalState elem
finalState <- nextState
return finalState
}
[| 1..4 |]
|> Array.mapFoldAsync mapping (1,0)
|> Async.RunSynchronously
I'm trying to build pipeline for message handling using free monad pattern, my code looks like that:
module PipeMonad =
type PipeInstruction<'msgIn, 'msgOut, 'a> =
| HandleAsync of 'msgIn * (Async<'msgOut> -> 'a)
| SendOutAsync of 'msgOut * (Async -> 'a)
let private mapInstruction f = function
| HandleAsync (x, next) -> HandleAsync (x, next >> f)
| SendOutAsync (x, next) -> SendOutAsync (x, next >> f)
type PipeProgram<'msgIn, 'msgOut, 'a> =
| Act of PipeInstruction<'msgIn, 'msgOut, PipeProgram<'msgIn, 'msgOut, 'a>>
| Stop of 'a
let rec bind f = function
| Act x -> x |> mapInstruction (bind f) |> Act
| Stop x -> f x
type PipeBuilder() =
member __.Bind (x, f) = bind f x
member __.Return x = Stop x
member __.Zero () = Stop ()
member __.ReturnFrom x = x
let pipe = PipeBuilder()
let handleAsync msgIn = Act (HandleAsync (msgIn, Stop))
let sendOutAsync msgOut = Act (SendOutAsync (msgOut, Stop))
which I wrote according to this article
However it's important to me to have those methods asynchronous (Task preferably, but Async is acceptable), but when I created a builder for my pipeline, I can't figure out how to use it - how can I await a Task<'msgOut> or Async<'msgOut> so I can send it out and await this "send" task?
Now I have this piece of code:
let pipeline log msgIn =
pipe {
let! msgOut = handleAsync msgIn
let result = async {
let! msgOut = msgOut
log msgOut
return sendOutAsync msgOut
}
return result
}
which returns PipeProgram<'b, 'a, Async<PipeProgram<'c, 'a, Async>>>
In my understanding, the whole point of the free monad is that you don't expose effects like Async, so I don't think they should be used in the PipeInstruction type. The interpreter is where the effects get added.
Also, the Free Monad really only makes sense in Haskell, where all you need to do is define a functor, and then you get the rest of the implementation automatically. In F# you have to write the rest of the code as well, so there is not much benefit to using Free over a more traditional interpreter pattern.
That TurtleProgram code you linked to was just an experiment -- I would not recommend using Free for real code at all.
Finally, if you already know the effects you are going to use, and you are not going to have more than one interpretation, then using this approach doesn't make sense. It only makes sense when the benefits outweigh the complexity.
Anyway, if you did want to write an interpreter version (rather than Free) this is how I would do it:
First, define the instructions without any effects.
/// The abstract instruction set
module PipeProgram =
type PipeInstruction<'msgIn, 'msgOut,'state> =
| Handle of 'msgIn * ('msgOut -> PipeInstruction<'msgIn, 'msgOut,'state>)
| SendOut of 'msgOut * (unit -> PipeInstruction<'msgIn, 'msgOut,'state>)
| Stop of 'state
Then you can write a computation expression for it:
/// A computation expression for a PipeProgram
module PipeProgramCE =
open PipeProgram
let rec bind f instruction =
match instruction with
| Handle (x,next) -> Handle (x, (next >> bind f))
| SendOut (x, next) -> SendOut (x, (next >> bind f))
| Stop x -> f x
type PipeBuilder() =
member __.Bind (x, f) = bind f x
member __.Return x = Stop x
member __.Zero () = Stop ()
member __.ReturnFrom x = x
let pipe = PipeProgramCE.PipeBuilder()
And then you can start writing your computation expressions. This will help flush out the design before you start on the interpreter.
// helper functions for CE
let stop x = PipeProgram.Stop x
let handle x = PipeProgram.Handle (x,stop)
let sendOut x = PipeProgram.SendOut (x, stop)
let exampleProgram : PipeProgram.PipeInstruction<string,string,string> = pipe {
let! msgOut1 = handle "In1"
do! sendOut msgOut1
let! msgOut2 = handle "In2"
do! sendOut msgOut2
return msgOut2
}
Once you have described the the instructions, you can then write the interpreters. And as I said, if you are not writing multiple interpreters, then perhaps you don't need to do this at all.
Here's an interpreter for a non-async version (the "Id monad", as it were):
module PipeInterpreterSync =
open PipeProgram
let handle msgIn =
printfn "In: %A" msgIn
let msgOut = System.Console.ReadLine()
msgOut
let sendOut msgOut =
printfn "Out: %A" msgOut
()
let rec interpret instruction =
match instruction with
| Handle (x, next) ->
let result = handle x
result |> next |> interpret
| SendOut (x, next) ->
let result = sendOut x
result |> next |> interpret
| Stop x ->
x
and here's the async version:
module PipeInterpreterAsync =
open PipeProgram
/// Implementation of "handle" uses async/IO
let handleAsync msgIn = async {
printfn "In: %A" msgIn
let msgOut = System.Console.ReadLine()
return msgOut
}
/// Implementation of "sendOut" uses async/IO
let sendOutAsync msgOut = async {
printfn "Out: %A" msgOut
return ()
}
let rec interpret instruction =
match instruction with
| Handle (x, next) -> async {
let! result = handleAsync x
return! result |> next |> interpret
}
| SendOut (x, next) -> async {
do! sendOutAsync x
return! () |> next |> interpret
}
| Stop x -> x
First of all, I think that using free monads in F# is very close to being an anti-pattern. It is a very abstract construction that does not fit all that great with idiomatic F# style - but that is a matter of preference and if you (and your team) finds this way of writing code readable and easy to understand, then you can certainly go in this direction.
Out of curiosity, I spent a bit of time playing with your example - although I have not quite figured out how to fix your example completely, I hope the following might help to steer you in the right direction. The summary is that I think you will need to integrate Async into your PipeProgram so that the pipe program is inherently asynchronous:
type PipeInstruction<'msgIn, 'msgOut, 'a> =
| HandleAsync of 'msgIn * (Async<'msgOut> -> 'a)
| SendOutAsync of 'msgOut * (Async<unit> -> 'a)
| Continue of 'a
type PipeProgram<'msgIn, 'msgOut, 'a> =
| Act of Async<PipeInstruction<'msgIn, 'msgOut, PipeProgram<'msgIn, 'msgOut, 'a>>>
| Stop of Async<'a>
Note that I had to add Continue to make my functions type-check, but I think that's probably a wrong hack and you might need to remote that. With these definitions, you can then do:
let private mapInstruction f = function
| HandleAsync (x, next) -> HandleAsync (x, next >> f)
| SendOutAsync (x, next) -> SendOutAsync (x, next >> f)
| Continue v -> Continue v
let rec bind (f:'a -> PipeProgram<_, _, _>) = function
| Act x ->
let w = async {
let! x = x
return mapInstruction (bind f) x }
Act w
| Stop x ->
let w = async {
let! x = x
let pg = f x
return Continue pg
}
Act w
type PipeBuilder() =
member __.Bind (x, f) = bind f x
member __.Return x = Stop x
member __.Zero () = Stop (async.Return())
member __.ReturnFrom x = x
let pipe = PipeBuilder()
let handleAsync msgIn = Act (async.Return(HandleAsync (msgIn, Stop)))
let sendOutAsync msgOut = Act (async.Return(SendOutAsync (msgOut, Stop)))
let pipeline log msgIn =
pipe {
let! msgOut = handleAsync msgIn
log msgOut
return! sendOutAsync msgOut
}
pipeline ignore 0
This now gives you just plain PipeProgram<int, unit, unit> which you should be able to evaluate by having a recursive asynchronous functions that acts on the commands.
I had code that was waiting to blow up something lurking around. Using F# 4.1 Result it is similar to this:
module Result =
let unwindSeq (sourceSeq: #seq<Result<_, _>>) =
sourceSeq
|> Seq.fold (fun state res ->
match state with
| Error e -> Error e
| Ok innerResult ->
match res with
| Ok suc ->
Seq.singleton suc
|> Seq.append innerResult
|> Ok
| Error e -> Error e) (Ok Seq.empty)
The obvious bottleneck here is Seq.singleton added to Seq.append. I understand that this is slow (and badly written), but why does it have to blow up the stack? I don't think that Seq.append is inherently recursive...
// blows up stack, StackOverflowException
Seq.init 1000000 Result.Ok
|> Result.unwindSeq
|> printfn "%A"
And as an aside, to unwind a sequence of Result, I fixed this function by using a simple try-catch-reraise, but that feels sub-par too. Any ideas as to how to do this more idiomatically without force-evaluating the sequence or blowing up the stack?
Not-so-perfect unwinding (it also forces the result-fail type), but at least without pre-evaluation of the sequence:
let unwindSeqWith throwArgument (sourceSeq: #seq<Result<_, 'a -> 'b>>) =
try
sourceSeq
|> Seq.map (throwOrReturnWith throwArgument)
|> Ok
with
| e ->
(fun _ -> raise e)
|> Error
I believe the idiomatic way of folding a sequence of Results in the way you suggest would be:
let unwindSeq<'a,'b> =
Seq.fold<Result<'a,'b>, Result<'a seq, 'b>>
(fun acc cur -> acc |> Result.bind (fun a -> cur |> Result.bind (Seq.singleton >> Seq.append a >> Ok)))
(Ok Seq.empty)
Not that this will be any faster than your current implementation, it just leverages Result.bind to do most of the work. I believe the stack is overflowing because a recursive function somewhere in the F# library, likely in the Seq module. My best evidence for this is that materializing the sequence to a List first seems to make it work, as in the following example:
let results =
Seq.init 2000000 (fun i -> if i <= 1000000 then Result.Ok i else Error "too big")
|> Seq.toList
results
|> unwindSeq
|> printfn "%A"
However, this may not work in your production scenario if the sequence is too big to materialize in memory.
I'm currently generating a sequence in a similar way to:
migrators
|> Seq.map (fun m -> m())
The migrator function is ultimately returning a discriminated union like:
type MigratorResult =
| Success of string * TimeSpan
| Error of string * Exception
I want to stop the map once I encounter my first Error but I need to include the Error in the final sequence.
I have something like the following to display a final message to the user
match results |> List.rev with
| [] -> "No results equals no migrators"
| head :: _ ->
match head with
| Success (dt, t) -> "All migrators succeeded"
| Error (dt, ex) -> "Migration halted owing to error"
So I need:
A way to stop the mapping when one of the map steps produces an Error
A way to have that error be the final element added to the sequence
I appreciate there may be a different sequence method other than map that will do this, I'm new to F# and searching online hasn't yielded anything as yet!
I guess there are multiple approaches here, but one way would be to use unfold:
migrators
|> Seq.unfold (fun ms ->
match ms with
| m :: tl ->
match m () with
| Success res -> Some (Success res, tl)
| Error res -> Some (Error res, [])
| [] -> None)
|> List.ofSeq
Note the List.ofSeq at the end, that's just there for realizing the sequence. A different way to go would be to use sequence comprehensions, some might say it results in a clearer code.
The ugly things Tomaš alludes to are 1) mutable state, and 2) manipulation of the underlying enumerator. A higher-order function which returns up to and including when the predicate holds would then look like this:
module Seq =
let takeUntil pred (xs : _ seq) = seq{
use en = xs.GetEnumerator()
let flag = ref true
while !flag && en.MoveNext() do
flag := not <| pred en.Current
yield en.Current }
seq{1..10} |> Seq.takeUntil (fun x -> x % 5 = 0)
|> Seq.toList
// val it : int list = [1; 2; 3; 4; 5]
For your specific application, you'd map the cases of the DU to a boolean.
(migrators : seq<MigratorResult>)
|> Seq.takeUntil (function Success _ -> false | Error _ -> true)
I think the answer from #scrwtp is probably the nicest way to do this if your input is reasonably small (and you can turn it into an F# list to use pattern matching). I'll add one more version, which works when your input is just a sequence and you do not want to turn it into a list.
Essentially, you want to do something that's almost like Seq.takeWhile, but it gives you one additional item at the end (the one, for which the predicate fails).
To use a simpler example, the following returns all numbers from a sequence until one that is divisible by 5:
let nums = [ 2 .. 10 ]
nums
|> Seq.map (fun m -> m % 5)
|> Seq.takeWhile (fun n -> n <> 0)
So, you basically just need to look one element ahead - to do this, you could use Seq.pairwise which gives you the current and the next element in the sequence"
nums
|> Seq.map (fun m -> m % 5)
|> Seq.pairwise // Get sequence of pairs with the next value
|> Seq.takeWhile (fun (p, n) -> p <> 0) // Look at the next value for test
|> Seq.mapi (fun i (p, n) -> // For the first item, we return both
if i = 0 then [p;n] else [n]) // for all other, we return the second
|> Seq.concat
The only ugly thing here is that you then need to flatten the sequence again using mapi and concat.
This is not very nice, so a good thing to do would be to define your own higher-order function like Seq.takeUntilAfter that encapsulates the behavior you need (and hides all the ugly things). Then your code could just use the function and look nice & readable (and you can experiment with other ways of implementing this).