I'm trying to make a simple HTTP server in F#. But unfortunately I'm stuck at the response stage, I'm trying to create a simple 404 response but my browser won't get it, it will simply load and load the page forever not going anywhere. I can't figure out where my code could be wrong.
let responseString = "HTTP/1.1 404 NOT FOUND";
let buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
let processRequest (client: TcpClient) =
async { do printfn "Incoming HTTP request..."
use stream = client.GetStream()
use sr = new StreamReader(stream)
let! content = sr.ReadToEndAsync() |> Async.AwaitTask
do
printfn "Stream contains:\n%s" content
stream.Write(buffer, 0, buffer.Length)
stream.Flush() }
let ip = IPAddress.Parse("127.0.0.1")
let listener = TcpListener(ip, 8080)
let startServer =
async { do listener.Start()
do printfn "Starting server..."
while true do
let! context = listener.AcceptTcpClientAsync() |> Async.AwaitTask
do! processRequest context }
Async.Start startServer
The problem is, the TCP stream does not know when the client has sent all its data (unless you e.g. close the browser), so the StreamReader is blocked forever.
If you want to implement HTTP by yourself, you'd need (for starters) to read line by line and check whether the client is allowed to send any more data.
As #CaringDev points out, ReadToEnd doesn't terminate. That said for a simple HTTP server you'd only need to read the first line of the request. Here's an updated version of your code that reads the first line only and responds with a 200 OK:
open System.Net
let processRequest (client: Sockets.TcpClient) = async {
printfn "Incoming HTTP request..."
use stream = client.GetStream()
use sr = new System.IO.StreamReader(stream)
let line = sr.ReadLine()
printfn "Request: %s" line
let sw = new System.IO.StreamWriter(stream)
sw.WriteLine("HTTP/1.1 200 OK\r\n\r\nHello World")
sw.Flush()
}
let ip = IPAddress.Parse("127.0.0.1")
let listener = Sockets.TcpListener(ip, 8080)
let startServer = async {
do listener.Start()
do printfn "Starting server..."
while true do
let! context = listener.AcceptTcpClientAsync() |> Async.AwaitTask
processRequest context |> Async.Start
}
Async.Start startServer
I'd also be tempted to change do! processRequest context to processRequest context |> Async.Start so that you can process multiple requests simultaneously.
Related
I want to do message broadcasting: when one of the clients sends a message, the server writes it to each socket. My main problem is I can't figure out how to send the Vec to the threads. I can't use Mutex because that will lock the access of other threads to the Vec for reading. I can't clone and send because TcpStream can't be cloned and sent. Here's my attempt until now
use std::net::{TcpStream, TcpListener};
use std::io::prelude::*;
use std::sync::{Arc, Mutex};
use std::thread;
use std::sync::mpsc::{channel, Receiver};
use std::cell::RefCell;
type StreamSet = Arc<RefCell<Vec<TcpStream>>>;
type StreamReceiver = Arc<Mutex<Receiver<StreamSet>>>;
fn main() {
let listener = TcpListener::bind("0.0.0.0:8000").unwrap();
let mut connection_set: StreamSet = Arc::new(RefCell::new(vec![]));
let mut id = 0;
let (tx, rx) = channel();
let rx = Arc::new(Mutex::new(rx));
for stream in listener.incoming() {
let receiver = rx.clone();
let stream = stream.unwrap();
(*connection_set).borrow_mut().push(stream);
println!("A connection established with client {}", id);
thread::spawn(move || handle_connection(receiver, id));
id += 1;
tx.send(connection_set.clone()).unwrap();
}
}
fn handle_connection(rx: StreamReceiver, id: usize) {
let streams;
{
streams = *(rx.lock().unwrap().recv().unwrap()).borrow();
}
let mut connection = &streams[id];
loop {
let mut buffer = [0; 512];
if let Err(_) = connection.read(&mut buffer) {
break;
};
println!("Request: {}", String::from_utf8_lossy(&buffer[..]));
if let Err(_) = connection.write(&buffer[..]) {
break;
};
if let Err(_) = connection.flush() {
break;
};
}
}
Another idea is to spawn a single "controller" thread and a thread for every socket. Each thread would own the socket and have a channel to send data back to the controller. The controller would own a Vec of channels to send to each thread. When a thread receives data, you send it to the controller which duplicates it and sends it back to each worker thread. You can wrap the data in an Arc to prevent unneeded duplication, and you should provide an ID to avoid echoing the data back to the original sender (if you need to).
This moves the ownership completely within a single thread, which should avoid the issues you are experiencing.
You may also wish to look into Tokio, which should allow you to do something similar but without the need to spawn threads in a 1-1 mapping.
I can't use Mutex because that will lock the access of other threads
You can always try a different locking mechanism such as a RwLock.
because TcpStream can't be cloned
Sure it can: TcpStream::try_clone.
My objective is to
start a GUI effect,
await some async work without freezing the GUI
do a final GUI effect
I've prepared a first demo code using a viewmodel with the following
member this.RunSetStatus() =
async {
this.Status <- "!Start resetting #" + DateTime.Now.ToString "yy.MM.dd hh:mm:ss"
let! task = async {
do! Async.Sleep (10 * 1000)
return "!Reset done #" + DateTime.Now.ToString "yy.MM.dd hh:mm:ss"
}
this.Status <- task
} |> Async.StartImmediate
It behaves as expected so I'm happy with the above.
The issue is when I replace the Sleep in the demo with real blocking work, like a wcf consumer, retrieving some results.
member this.CheckReport(user : string) =
async {
let endpoint = new ServiceEndpoint(ContractDescription.GetContract(typeof<IClaimService>),
new BasicHttpBinding(),
new EndpointAddress(address))
let factory = new ChannelFactory<IClaimService>(endpoint)
let channel = factory.CreateChannel()
let resp = channel.CheckReport(user)
factory.Close()
return resp
}
called from my final delegate command
let RefreshLogic() =
this.RefreshIsActive <- true
async {
let cons = ConsumerLib.ConsumerWCF()
let! task, msg = async {
try
let! resp = cons.CheckReport(Environment.UserName.ToLower())
return resp , ""
with
|exc -> return [||], (ConsumerLib.FindInner(exc).Message + ConsumerLib.FindInner(exc).StackTrace)
}
this.Reports <- task
this.RefreshIsActive <- false
this.StatusMsg <- msg
this.ExportCommand.RaiseCanExecuteChanged()
} |> Async.StartImmediate
It unfortunately freezes the GUI while refreshing (why?)
The problem is your CheckReport function. While it's an async block, it never actually calls any asynchronous work (ie: nothing is bound via let! or do!), so the entire block runs synchronously.
Even though the work is inside of an asynchronous workflow, when you use StartImmediate, the work runs synchronously up to the first actual asynchronous function call, which would be bound by let! or do!. Since your work is completely synchronous, this propogates upwards, and ends up being synchronous, blocking the UI.
If your WCF bindings were setup to include asynchronous versions that are Task returning, the best approach here would be to use the asynchronous version of the WCF method, which would look something like:
let! resp = channel.CheckReportAsync(user) |> Async.AwaitTask
I wanna make a web crawling, currently i am reading a txt file with 12000 urls, i wanna use concurrency in this process, but the requests don't work.
typealias escHandler = ( URLResponse?, Data? ) -> Void
func getRequest(url : URL, _ handler : #escaping escHandler){
let session = URLSession(
configuration: .default,
delegate: nil,
delegateQueue: nil)
var request = URLRequest(url:url)
request.httpMethod = "GET"
let task = session.dataTask(with: request){ (data,response,error) in
handler(response,data)
}
task.resume()
}
for sUrl in textFile.components(separatedBy: "\n"){
let url = URL(string: sUrl)!
getRequest(url: url){ response,data in
print("RESPONSE REACHED")
}
}
If you have your URLSessions working correctly, all you need to go is create separate OperationQueue create a Operation for each of your async tasks you want completed, add it to your operation queue, and set your OperationQueue's maxConcurrentOperationCount to control how many of your tasks can run at one time. Puesdo code:
let operationQueue = OperationQueue()
operationQueue.qualityOfService = .utility
let exOperation = BlockOperation(block: {
//Your URLSessions go here.
})
exOperation.completionBlock = {
// A completionBlock if needed
}
operationQueue.addOperation(exOperation)
exOperation.start()
Using a OperationQueue subclass and Operation subclass will give you additional utilities for dealing with multiple threads.
As far as I understood, the use keyword disposes the bound IDisposable as soon it is out of scope, so considering this recursive function:
let rec AsyncAcceptMessages(client : WebSocket) =
async {
let! message = client.AsyncReadMessage
use reader = new StreamReader(message)
let s = reader.ReadToEnd()
printf "%s" <| s
do! AsyncAcceptMessages client
}
Let's pretend that the compiler does not find a way of using tail recursion, would that StreamReader be disposed after each recursion?
UPDATE
Tomas response show me a way of fixing it when you actually expect something back, but what if you are expecting nothing? like in this example with the StreamWriter:
let rec AsyncAcceptMessages(client : WebSocket) =
async {
let! message = client.AsyncReadMessage
if(not(isNull message)) then
let s =
use reader = new StreamReader(message)
reader.ReadToEnd()
use writer = new StreamWriter(client.CreateMessageWriter(WebSocketMessageType.Text), Encoding.UTF8)
writer.Write s
printf "%s" <| s
do! AsyncAcceptMessages client
}
As you're saying, the StreamReader would only be disposed of after the execution returns from the recursive call (i.e. never).
There is another issue, which is that do! is not treated as a tail-recursive call, so if you want to create an infinite tail-recursive loop, you need to use return! (otherwise your code will be leaking memory).
In this case, you can fix it easily using, because you're not doing any asynchronous operations with the StreamReader, so you can just create an ordinary local scope:
let rec AsyncAcceptMessages(client : WebSocket) =
async {
let! message = client.AsyncReadMessage
let s =
use reader = new StreamReader(message)
reader.ReadToEnd()
printf "%s" <| s
return! AsyncAcceptMessages client
}
If you wanted to call e.g. AsyncReadToEnd, then you could do something like:
let rec AsyncAcceptMessages(client : WebSocket) =
async {
let! message = client.AsyncReadMessage
let! s =
async { use reader = new StreamReader(message)
return! reader.ReadToEnd() }
printf "%s" <| s
return! AsyncAcceptMessages client
}
I have been trying to learn F# for the past couple of day and I keep running into something that perplexes me. My "learning project" is a screen scraper for some data I'm kind of interested in manipulating.
In F# PowerPack there is a call Stream.AsyncReadToEnd. I did not want to use the PowerPack just for that single call so I took a look at how they did it.
module Downloader =
open System
open System.IO
open System.Net
open System.Collections
type public BulkDownload(uriList : IEnumerable) =
member this.UriList with get() = uriList
member this.ParalellDownload() =
let Download (uri : Uri) = async {
let UnblockViaNewThread f = async {
do! Async.SwitchToNewThread()
let res = f()
do! Async.SwitchToThreadPool()
return res }
let request = HttpWebRequest.Create(uri)
let! response = request.AsyncGetResponse()
use responseStream = response.GetResponseStream()
use reader = new StreamReader(responseStream)
let! contents = UnblockViaNewThread (fun() -> reader.ReadToEnd())
return uri, contents.ToString().Length }
this.UriList
|> Seq.cast
|> Seq.map Download
|> Async.Parallel
|> Async.RunSynchronously
They have that function UnblockViaNewThread. Is that really the only way to asynchronously read the response stream? Isn't creating a new thread really expensive (I've seen the "~1mb of memory" thrown around all over the place). Is there a better way to do this? Is this what's really happenening in every Async* call (one that I can let!)?
EDIT: I follow Tomas' suggestions and actually came up with something independent of F# PowerTools. Here it is. This really needs error handling, but it asynchronous requests and downloads a url to a byte array.
namespace Downloader
open System
open System.IO
open System.Net
open System.Collections
type public BulkDownload(uriList : IEnumerable) =
member this.UriList with get() = uriList
member this.ParalellDownload() =
let Download (uri : Uri) = async {
let processStreamAsync (stream : Stream) = async {
let outputStream = new MemoryStream()
let buffer = Array.zeroCreate<byte> 0x1000
let completed = ref false
while not (!completed) do
let! bytesRead = stream.AsyncRead(buffer, 0, 0x1000)
if bytesRead = 0 then
completed := true
else
outputStream.Write(buffer, 0, bytesRead)
stream.Close()
return outputStream.ToArray() }
let request = HttpWebRequest.Create(uri)
let! response = request.AsyncGetResponse()
use responseStream = response.GetResponseStream()
let! contents = processStreamAsync responseStream
return uri, contents.Length }
this.UriList
|> Seq.cast
|> Seq.map Download
|> Async.Parallel
|> Async.RunSynchronously
override this.ToString() = String.Join(", ", this.UriList)
I think that AsyncReadToEnd that just synchronously calls ReadToEnd on a separate thread is wrong.
The F# PowerPack also contains a type AsyncStreamReader that contains proper asynchronous implementation of stream reading. It has a ReadLine method that (asynchronously) returns the next line and only downloads a few chunks from the source stream (using the asynchronous ReadAsync as opposed to running on a background thread).
let processStreamAsync stream = async {
use asyncReader = new AsyncStreamReader(stream)
let completed = ref false
while not (!completed) do
// Asynchrnously get the next line
let! nextLine = asyncReader.ReadLine()
if nextLine = null then completed := true
else
(* process the next line *) }
If you want to download the whole content as a string (instead of processing it line-by-line), then you can use ReadToEnd method of AsyncStreamReader. This is a proper asynchronous implementation that starts downloading block of data (asynchronously) and repeats this without blocking.
async {
use asyncReader = new AsyncStreamReader(stream)
return! asyncReader.ReadToEnd() }
Also, F# PowerPack is open-souorce and has permissive license, so the best way to use it is often to just copy the few files you need into your project.