Using gRPC bindings from https://github.com/gkkachi/firestore-grpc I was able to puzzle together something that is seemingly working but does not receive any content:
Creating the request:
let req = ListenRequest {
database: format!("projects/{}/databases/(default)", project_id),
labels: HashMap::new(),
target_change: Some(TargetChange::AddTarget(Target {
// "Rust" in hex: https://github.com/googleapis/python-firestore/issues/51
target_id: 0x52757374,
once: false,
target_type: Some(TargetType::Documents(DocumentsTarget {
documents: vec![users_collection],
})),
resume_type: None,
})),
};
Sending it:
let mut req = Request::new(stream::iter(vec![req]));
let metadata = req.metadata_mut();
metadata.insert(
"google-cloud-resource-prefix",
MetadataValue::from_str(&db).unwrap(),
);
println!("sending request");
let res = get_client(&token).await?.listen(req).await?;
let mut res = res.into_inner();
while let Some(msg) = res.next().await {
println!("getting response");
dbg!(msg);
}
(full code in this repo).
The request can be made but the stream does not contain any actual content. The only hint I get from the debug logs is
[2021-10-27T14:54:39Z DEBUG h2::codec::framed_write] send frame=GoAway { error_code: NO_ERROR, last_stream_id: StreamId(0) }
[2021-10-27T14:54:39Z DEBUG h2::proto::connection] Connection::poll; connection error error=GoAway(b"", NO_ERROR, Library)
Any idea what is missing?
The crucial thing I was missing as pointed out in the rust users forum was that the request stream was immediately ending which caused the connection to close. The send frame=GoAway was actually send by the client (facepalm).
To keep the connection open and receive responses we can keep the input stream pending: Request::new(stream::iter(vec![req]).chain(stream::pending())). There will be a better way to set things up and keep control over subsequent input requests but this is enough to fix the example.
Related
Issue
I have a client that needs to send the following custom data structure to an API:
#[derive(Serialize, Deserialize)]
pub struct FheSum {
pub server_keys: ServerKey,
pub value1: FheUint8,
pub value2: FheUint8,
}
The code for the client is the following:
let fhe_post: FheSum = FheSum {
server_keys: server_keys.to_owned(),
value1: value_api.to_owned(),
value2: value_api_2.to_owned(),
};
let client = reqwest::blocking::Client::builder()
.timeout(None)
.build().unwrap();
let response = client
.post("http://127.0.0.1:8000/computesum")
.json(&fhe_post)
.send().unwrap();
let response_json: Result<FheSumResult, reqwest::Error> = response.json();
match response_json {
Ok(j) => {
let result_api: u8 = FheUint8::decrypt(&j.result, &client_keys);
println!("Final Result: {}", result_api)
},
Err(e) => println!("{:}", e),
};
In the API, I have the following definition of an HttpServer:
HttpServer::new(|| {
let json_cfg = actix_web::web::JsonConfig::default()
.limit(std::usize::MAX);
App::new()
.app_data(json_cfg)
.service(integers::computesum)
})
.client_disconnect_timeout(std::time::Duration::from_secs(3000))
.client_request_timeout(std::time::Duration::from_secs(3000))
.max_connection_rate(std::usize::MAX)
.bind(("127.0.0.1", 8000))?
.run()
.await
And the associated endpoint the client is trying to access:
#[post("/computesum")]
pub async fn computesum(req: Json<FheSum>) -> HttpResponse {
let req: FheSum = req.into_inner();
let recovered: FheSum = FheSum::new(
req.server_keys,
req.value1,
req.value2,
);
set_server_key(recovered.server_keys);
let result_api_enc: FheSumResult = FheSumResult::new(recovered.value1 + recovered.value2);
HttpResponse::Ok()
.content_type(ContentType::json())
.json(&result_api_enc)
}
Problem
The structs are the same in both the client and the server. This code works when using common data types such as Strings. The issue is when using this data structures. The memory occupied, obtained with mem::size_of_val which returns the size in bytes, is the following:
Size 1: 2488
Size 2: 32
Size 3: 32
The result has been obtained in bytes, so, given the limit established in the HttpServer, this shouldn't be an issue. Timeouts have also been set at much higher values than commonly needed.
Even with this changes, the client always shows Killed, and doesn't display the answer from the server, not giving any clues on what the problem might be.
The client is killing the process before being able to process the server's response. I want to find a way to send these custom data types to the HTTP server without the connection closing before the operation has finished.
I have already tried different libraries for the client such as the acw crate, apart from reqwest and the result is the same. I have also tried not using reqwest in blocking mode, and the error persists.
I have set up an SSE connection using Koa like so:
const Koa = require('koa');
const Router = require('koa-router');
const app = new Koa();
const router = new Router();
// Sets up the HTTP header and sends a message via SSE
function writeSSE(ctx, message) {
ctx.res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
'Access-Control-Allow-Origin': '*',
});
ctx.res.write(`id: 01\n`);
ctx.res.write(`data: ${message}\n\n`);
}
// Router Middleware
router.get('/stream', (ctx, next) => {
writeSSE(ctx, 'Stream reached.');
});
app.use(router.routes()).use(router.allowedMethods());
app.listen(8080);
Where my React components starts the connection like so:
new EventSource("http://localhost:8080/stream")
The component then receives the answer sent by the writeSSE method on the backend.
But for some reason the /stream endpoint is reached every 3 seconds or so, as if the connection was being reestablished.
And my error listener on the front-end catches a CONNECTING event every time.
this.state.source.onerror = (e) => {
if (e.target.readyState == EventSource.CONNECTING) {
console.log("Connecting...");
}
};
And on the back-end, ctx.response equals { status: 404, message: 'Not Found', header: {} }.
Would anyone know the cause of this issue? Is it linked to the way I use Koa?
this is a bit too late, but I will write my experience with sse using Koa.
First of all using ctx.res directly is not much appreciated by Koa, if you still want to use it make sure to put ctx.respond = false to bypass koa response mecanism.
In my experience a stream is the best way to use SSE with Koa you can do something like :
const stream = require('stream');
const koa = require('koa');
const app = new koa();
app.use(async ctx => {
ctx.set({
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive'
});
ctx.status = 200;
const stream = new stream.PassThrough()
ctx.body = stream; // here koa will pipe the ctx.res to stream and end the ctx.res when ever the stream ends.
let counter = 5;
const t = setInterval(() => {
stream.write(`data: hi from koa sse ${counter}`);
counter--;
if (counter === 0) {
stream.end();
clearInterval(t);
}
}, 1000);
});
Hope this help anyone will play with SSE on koa.
PS: I wrote this on hurry if there is anything wrong with code tell me and I will correct it.
I'm in the process of implementing a Koa-based server for SSE. I've been running into the same problem, and here are my thoughts / working solution:
As far as I can tell, the reason why onmessage and onerror keep getting called is because the EventSource object on the client side is emitting an error event. This is causing the connection to be disconnected, which causes the client to send another request to initialize the stream to the server. From here, the process repeats itself indefinitely.
Based on my own testing, EventSource is emitting an error due to the data that is being sent back from the server. Per the docs, a 200 response that has as Content-Type other than 'text/event-stream' will cause a failure.
In your example, you have declared your response as 'text/event-stream' and are passing a string into the ctx.res.write method. While this looks correct, and in fact works when using comparable code and Express, it seems that it doesn't work in Koa. However, if you change the 'data' you are writing to your response to a stream, such as this example here, you'll find that the connection establishes correctly.
Maybe try the following:
//require Passthrough
const PassThrough = require('stream').PassThrough;
//then, in your writeSSE function, try this:
let stream = new PassThrough();
stream.write(`data: ${message}\n\n`);
ctx.res.write(stream);
I'm not 100% sure why this change works. My best guess is that there is something about Koa's ctx object that prevents a plain string or template literal from being viewed as valid text/event-stream data, but this is entirely supposition (this begs the question as to why it works in Express, but hopefully someone more knowledgeable can answer this for both of us). From what I've seen of other snippets published online, the stream approach is the one to take in Koa.
I'm not sure what your results will be, as it looks like you may be using a different version of Koa than I am, but I'd give it a shot. I was able to get my connection established correctly making this small change.
I am trying to spawn a server and connect to it on a different thread. I know Rust has blocking I/O, but I feel like I should be able to connect a server in a different thread. I do not have a lot of knowledge in threads. The end game is to connect to this server across a network. That is what I am simulating with the player_stream TCPStream. The player_stream will wait until there is something in its buffer. Once something has been written there, it will respond back to the server. As is, the program will not terminate.
use std::net::{TcpListener, TcpStream};
use std::io::{BufReader,BufWriter};
use std::io::Write;
use std::io::Read;
use std::thread;
fn main() {
thread::spawn(move || {
start_server();
});
let player_stream = TcpStream::connect("127.0.0.1:8000").expect("Couldn't connect");
let mut reader = BufReader::new(&player_stream);
let mut response = String::new();
reader.read_to_string(&mut response);
println!("Player received {}", response);
let mut writer = BufWriter::new(&player_stream);
writer.write_all("NAME".as_bytes());
}
fn start_server() {
let listener = TcpListener::bind("127.0.0.1:8000").unwrap();
fn handle_client(stream: TcpStream) {
println!("Client connected");
let mut writer = BufWriter::new(&stream);
writer.write_all("Red".as_bytes());
let mut reader = BufReader::new(&stream);
let mut response = String::new();
reader.read_to_string(&mut response);
println!("Server received {}", response);
}
// accept connections
for stream in listener.incoming() {
match stream {
Ok(stream) => {
handle_client(stream);
}
Err(e) => { panic!("{}",e) }
}
}
}
First off, don't ignore warnings. You have 4 errors of the type warning: unused result which must be used. Every single one of those could be cases where your code is failing and you wouldn't even know it. Use expect liberally!
Second, you have an open client read socket and you ask to "read all the data until the end into a string". What determines the end? In this case, it's when the socket is closed; so when is that?
Trick question!
The client's read socket closes when the server's write socket closes.
The server's write socket closes when the server's read socket closes.
The server's read socket closes when the the client's write socket closes.
So when does that happen? Because there's no code that does it specifically, it will close when the socket is dropped, so:
The client's write socket closes when the the client ends.
Thus the deadlock. The issue could be fixed by explicitly closing the write half of the socket:
stream.shutdown(std::net::Shutdown::Write).expect("could not shutdown");
Third, you are writing into a BufWriter. Review the documentation for it:
A BufWriter keeps an in-memory buffer of data and writes it to an underlying writer in large, infrequent batches.
The buffer will be written out when the writer is dropped.
The BufWriter is dropped at the end of the scope, after you've tried to read the response. That's another deadlock.
In the end, you need to establish a protocol for how to delimit messages sent back and forth. A simple but very limited solution is to have a line-oriented protocol: every message fits on one line ending with a newline character.
If you choose that, you can use read_to_line instead. I've also used BufWriter::flush to force the data to be sent down the wire; you could have also encapsulated writer in a block so it is dropped earlier or explicitly call drop(writer).
use std::net::{TcpListener, TcpStream};
use std::io::{BufReader, BufWriter, Write, BufRead};
use std::thread;
fn main() {
thread::spawn(start_server);
let player_stream = TcpStream::connect("127.0.0.1:8000").expect("Couldn't connect");
let mut reader = BufReader::new(&player_stream);
let mut response = String::new();
reader.read_line(&mut response).expect("Could not read");
println!("Player received >{}<", response.trim());
let mut writer = BufWriter::new(&player_stream);
writer.write_all("NAME\n".as_bytes()).expect("Could not write");
}
fn start_server() {
let listener = TcpListener::bind("127.0.0.1:8000").unwrap();
fn handle_client(stream: TcpStream) {
println!("Client connected");
let mut writer = BufWriter::new(&stream);
writer.write_all("Red\n".as_bytes()).expect("could not write");
writer.flush().expect("could not flush");
let mut reader = BufReader::new(&stream);
let mut response = String::new();
reader.read_line(&mut response).expect("could not read");
println!("Server received {}", response);
}
for stream in listener.incoming() {
let stream = stream.expect("Unable to accept");
handle_client(stream);
}
}
You'll note that the program doesn't always print out the server's response. That's because the main thread exiting exits the program.
You mentioned that your real case uses XML, which can have newlines embedded in it, making a line-oriented protocol unsuitable. Another common protocol is to send a length before sending the data itself. There are many possible implementations for this. At a previous job, we sent XML in this fashion. We started with an ASCII-encoded newline-terminated string of the length before the data itself. In that case, having the readability of the length as a string was a benefit. You could also choose to send a number of bytes that can be interpreted according to some endianness as a 2's compliment number.
See also:
Rust echo server and client using futures blocks itself forever
Executing Angular2 http call to the offline server doesn't provide much info in it's "error response" object I'm getting in the Observable's .catch(error) operator or subscription error delegate (they are both share the same info actually). But as you can see on the screen shot of the console there's actual error was displayed by zone.js somehow.
So, how can I get this specific error info (net::ERR_CONNECTION_REFUSED)?
Thanks.
Whenever server do not respond, response.status will be always equal to 0 (zero)
{
type: 3, //ResponseType.Error
status: 0, // problem connecting endpoint
}
Also note, when you are performing CORS request, but origin (url in browser) is not authorized (not in allowed list of host names configured in remote endpoint) the response would be similar to above, with exception to type attribute which will be equal to 4 = ResponseType.Opaque...
This means, connection was made, but for instance, OPTIONS request returned with headers which do not contain origin or HTTPS request was performed from HTTP origin.
You can handle the error messages so they are easier to read. This can definitely be expanded on too:
public Get() {
return this.http.get(this.URL).map(this.extractData)
.catch(this.handleError);
}
public extractData(res: Response) {
let body = res.json();
return body || {};
}
public handleError(error: any) {
let errMsg = (error.message) ? error.message :
error.status ? `${error.status} - ${error.statusText}` : 'Server error';
console.error(errMsg);
return Observable.throw(errMsg);
}
Check out this part of the docs on error handling.
Without digging in the code, my expectation is that if the server is unreachable, then no response can be returned from the server. Therefore the Response object remains its initialized state.
Currently, I use the built-in meteor http method (see http://docs.meteor.com/#http) for issuing http calls, on both my client and my server.
However, I'm experiencing two issues:
is it possible to cancel a request?
is it possible to have multiple query parameters which share the same key?
Are these just Meteor limitations, or are there ways to get both to work using Meteor?
I know I could you jquery on the clientside, and there must be a server-side solution which supports both as wel, but I'd prefer sticking with meteor code here.
"is it possible to cancel a request?"
HTTP.call() does not appear to return an object on which we could call something like a stop() method. Perhaps a solution would be to prevent execution of your callback based on a Session variable?
HTTP.call("GET", url, function(error, result) {
if (!Session.get("stopHTTP")) {
// Callback code here
}
});
Then when you reach a point where you want to cancel the request, do this:
Session.set("stopHTTP", true);
On the server, instead of Session perhaps you could use an environment variable?
Note that the HTTP.call() options object does accept a timeout key, so if you're just worried about the request never timing out, you can set this to whatever millisecond integer you want.
"is it possible to have multiple query parameters which share the same key?"
Yes, this appears to be possible. Here's a simple test I used:
Meteor code:
HTTP.call("GET", "http://localhost:1337", {
query: "id=foo&id=bar"
}, function(error, result) {
// ...
});
Separate Node.js server: (just the basic example on the Node.js homepage, with a console.log line to output the request URL with query string)
var http = require('http');
http.createServer(function(req, res) {
console.log(req.url); // Here I log the request URL, with the query string
res.writeHead(200, {
'Content-Type': 'text/plain'
});
res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');
When the Meteor server is run, the Node.js server logged:
/?id=foo&id=bar
Of course, this is only for GET URL query parameters. If you need to do this for POST params, perhaps you could store the separate values as a serialized array string with EJSON.stringify?