I am trying to connect to a socket.io socket in R using the R function socketConnection(). However, although I am able to set up the socket properly, I am not able to read data from it into R.
The javascript code I use to set up the server is:
var app = require('http').createServer(handler)
var io = require('socket.io')(app);
var fs = require('fs');
app.listen(8005);
function handler (req, res) {
fs.readFile(__dirname + '/index.html',
function (err, data) {
if (err) {
res.writeHead(500);
return res.end('Error loading index.html');
}
res.writeHead(200);
res.end(data);
});
}
io.on('connection', function (socket) {
setInterval(function() {
socket.emit('update', "test")
}, 1000);
});
The code for index.html is:
<script src="/socket.io/socket.io.js"></script>
<script>
var socket = io('http://localhost:8005');
socket.on('update', function (data) {
console.log(data);
});
</script>
I am able to verify that the socket is indeed working by going to localhost:8005 in my web browser and opening up the Developer's Console, where I can see "test" being logged. However, when I try to connect to the socket in R by:
sock <- socketConnection("localhost", 8005, blocking = FALSE, open = "r")
readLines(sock)
Every time I run readLines(sock) I get an empty character vector back. I have also confirmed I can read from other sockets in R by executing the following and indeed getting a response from readLines(sock):
sock <- socketConnection("rstudio.com", 6789, blocking = FALSE, open = "r")
readLines(sock)
Any help would be greatly appreciated. Thanks!
UPDATE (2015-09-01):
Thanks to the excellent help from Aaron Dufour, I was able to adapt this net server demo to stream data into R. The javascript code I used for the server was
var net = require('net');
var server = net.createServer(function(socket) {
setInterval(function() {
socket.write('Test\r\n');
socket.pipe(socket);
}, 1000)
});
server.listen(1337, '127.0.0.1');
And the R code was:
sock <- socketConnection("localhost", 1337, open = "r")
readLines(sock)
close(sock)
I did get the following warning warning: possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit. sometimes on the server side when I ran readLines(socket)
Also, when I ran close(socket) in R the server crashed and I get the following error: Error: This socket has been ended by the other party
With additional research, I think both the warning and error are preventable.
As I've described here, the socket.io protocol is much more than a WebSocket. Just opening a WebSocket to it won't work.
But socketConnection appears to not even be a WebSocket, but rather a raw socket. You're getting nothing back because the server is talking HTTP and you haven't finished sending an HTTP request.
You probably need a socket.io library for R, or to switch away from socket.io on the server.
Related
Using gRPC bindings from https://github.com/gkkachi/firestore-grpc I was able to puzzle together something that is seemingly working but does not receive any content:
Creating the request:
let req = ListenRequest {
database: format!("projects/{}/databases/(default)", project_id),
labels: HashMap::new(),
target_change: Some(TargetChange::AddTarget(Target {
// "Rust" in hex: https://github.com/googleapis/python-firestore/issues/51
target_id: 0x52757374,
once: false,
target_type: Some(TargetType::Documents(DocumentsTarget {
documents: vec![users_collection],
})),
resume_type: None,
})),
};
Sending it:
let mut req = Request::new(stream::iter(vec![req]));
let metadata = req.metadata_mut();
metadata.insert(
"google-cloud-resource-prefix",
MetadataValue::from_str(&db).unwrap(),
);
println!("sending request");
let res = get_client(&token).await?.listen(req).await?;
let mut res = res.into_inner();
while let Some(msg) = res.next().await {
println!("getting response");
dbg!(msg);
}
(full code in this repo).
The request can be made but the stream does not contain any actual content. The only hint I get from the debug logs is
[2021-10-27T14:54:39Z DEBUG h2::codec::framed_write] send frame=GoAway { error_code: NO_ERROR, last_stream_id: StreamId(0) }
[2021-10-27T14:54:39Z DEBUG h2::proto::connection] Connection::poll; connection error error=GoAway(b"", NO_ERROR, Library)
Any idea what is missing?
The crucial thing I was missing as pointed out in the rust users forum was that the request stream was immediately ending which caused the connection to close. The send frame=GoAway was actually send by the client (facepalm).
To keep the connection open and receive responses we can keep the input stream pending: Request::new(stream::iter(vec![req]).chain(stream::pending())). There will be a better way to set things up and keep control over subsequent input requests but this is enough to fix the example.
I have set up an SSE connection using Koa like so:
const Koa = require('koa');
const Router = require('koa-router');
const app = new Koa();
const router = new Router();
// Sets up the HTTP header and sends a message via SSE
function writeSSE(ctx, message) {
ctx.res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
'Access-Control-Allow-Origin': '*',
});
ctx.res.write(`id: 01\n`);
ctx.res.write(`data: ${message}\n\n`);
}
// Router Middleware
router.get('/stream', (ctx, next) => {
writeSSE(ctx, 'Stream reached.');
});
app.use(router.routes()).use(router.allowedMethods());
app.listen(8080);
Where my React components starts the connection like so:
new EventSource("http://localhost:8080/stream")
The component then receives the answer sent by the writeSSE method on the backend.
But for some reason the /stream endpoint is reached every 3 seconds or so, as if the connection was being reestablished.
And my error listener on the front-end catches a CONNECTING event every time.
this.state.source.onerror = (e) => {
if (e.target.readyState == EventSource.CONNECTING) {
console.log("Connecting...");
}
};
And on the back-end, ctx.response equals { status: 404, message: 'Not Found', header: {} }.
Would anyone know the cause of this issue? Is it linked to the way I use Koa?
this is a bit too late, but I will write my experience with sse using Koa.
First of all using ctx.res directly is not much appreciated by Koa, if you still want to use it make sure to put ctx.respond = false to bypass koa response mecanism.
In my experience a stream is the best way to use SSE with Koa you can do something like :
const stream = require('stream');
const koa = require('koa');
const app = new koa();
app.use(async ctx => {
ctx.set({
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive'
});
ctx.status = 200;
const stream = new stream.PassThrough()
ctx.body = stream; // here koa will pipe the ctx.res to stream and end the ctx.res when ever the stream ends.
let counter = 5;
const t = setInterval(() => {
stream.write(`data: hi from koa sse ${counter}`);
counter--;
if (counter === 0) {
stream.end();
clearInterval(t);
}
}, 1000);
});
Hope this help anyone will play with SSE on koa.
PS: I wrote this on hurry if there is anything wrong with code tell me and I will correct it.
I'm in the process of implementing a Koa-based server for SSE. I've been running into the same problem, and here are my thoughts / working solution:
As far as I can tell, the reason why onmessage and onerror keep getting called is because the EventSource object on the client side is emitting an error event. This is causing the connection to be disconnected, which causes the client to send another request to initialize the stream to the server. From here, the process repeats itself indefinitely.
Based on my own testing, EventSource is emitting an error due to the data that is being sent back from the server. Per the docs, a 200 response that has as Content-Type other than 'text/event-stream' will cause a failure.
In your example, you have declared your response as 'text/event-stream' and are passing a string into the ctx.res.write method. While this looks correct, and in fact works when using comparable code and Express, it seems that it doesn't work in Koa. However, if you change the 'data' you are writing to your response to a stream, such as this example here, you'll find that the connection establishes correctly.
Maybe try the following:
//require Passthrough
const PassThrough = require('stream').PassThrough;
//then, in your writeSSE function, try this:
let stream = new PassThrough();
stream.write(`data: ${message}\n\n`);
ctx.res.write(stream);
I'm not 100% sure why this change works. My best guess is that there is something about Koa's ctx object that prevents a plain string or template literal from being viewed as valid text/event-stream data, but this is entirely supposition (this begs the question as to why it works in Express, but hopefully someone more knowledgeable can answer this for both of us). From what I've seen of other snippets published online, the stream approach is the one to take in Koa.
I'm not sure what your results will be, as it looks like you may be using a different version of Koa than I am, but I'd give it a shot. I was able to get my connection established correctly making this small change.
I have a CasperJS Script duolingo.js and I run the script using a MeteorJS App using a Meteor Method like this:
// define server methods so that the clients will have access to server components
Meteor.methods({
runCasperJS: function() {
// This method call won't return immediately, it will wait for the
// asynchronous code to finish, so we call unblock to allow this client
// to queue other method calls (see Meteor docs)
this.unblock();
// run synchonous system command
var result = process_exec_sync('casperjs duolingo.js --engine=slimerjs --disk-cache=no --proxy=178.166.41.225:80 --proxy-type=HTTP');
// check for error
if (result.error) {
throw new Meteor.Error("exec-fail", "Error running CasperJS: " + result.error.message);
}
// success
return true;
}
});
I need to run the CasperJS Instance using a proxy IP V6. But I don't have any idea, the official documentations of slimerjs and casperjs did not say anything.
Here is the exact part of the code where I use an IP V4 Proxy.
var result = process_exec_sync('casperjs duolingo.js
--engine=slimerjs --disk-cache=no --proxy=178.166.41.225:80
--proxy-type=HTTP');
Thank you for your time and help.
I'm thinking of scenarios where I might want to perform cross-cutting/AOP or other functions at the server for my Meteor Js project when a Meteor client (or DDP client) invokes a server-side method over a DDP connection.
This link here gives a really nice example of how to perform AOP on objects, but I wanted to know if there was a way to listen for inbound client requests over the DDP connection much like express-interceptor or action filters for asp.net web api but, of course, for websocket/ddp rpc implementations.
The Meteor Js Api describes only one event "onConnection" at the server. And this SO response mentions a connection._send on the client to perform certain AOP functions...but didn't find a whole lot of official documentation beyond that.
I basically want to know if there is a way to listen at the server for all DDP method calls from all client sessions to the server as described in the DDP spec here
Thanks.
We'll there are a lot of undocumented things in Meteor you'll find. I don't really see it as a problem. Here are some ways to intercept WebSocket traffic in Meteor:
Server
Here's some stuff you can do from the server:
Server -> Client
It's a bit tedious to intercept messages from the server to the client, but this works. You'd probably want to write some code to pin logs to clients.
(function () {
var timeout = 3000
var streamServer = Meteor.server.stream_server
var standardConnect = streamServer.server._events.connection
streamServer.server._events.connection = function (socket) {
var write = socket.write
socket.write = function () {
console.log(arguments)
write.apply(this, args)
}
standardConnect.apply(this, arguments)
}
})()
Client -> Server
To intercept calls from the client on the server you can do this:
Meteor.server.stream_server.server.addListener('connection', function (socket) {
var old = socket._events.data
socket._events.data = function () {
console.log(arguments)
old.apply(this, arguments)
}
})
The above sipped can't be used with the first one. It's not hard to fix dough. If you use this snipped, Meteor.server.stream_server.server._events.connection will simply be an array of functions instead of a function.
Client
Server - > Client
To Listen to calls from server to the client on the client you can do this:
Meteor.connection._stream.on('message', console.log.bind(console))
You can also intercept them using something like this
(function () {
var cb = Meteor.connection._stream.eventCallbacks.message[0]
Meteor.connection._stream.eventCallbacks.message[0] = function () {
console.log(arguments)
cb.apply(this, arguments)
}
})()
Not entirely sure how solid that one is. But it works, so what the heck.
To test it out you can simply do
Meteor.subscribe('test')
Client -> Server
As you pointed out, you can also do similar things with outgoing messages from the client.
Meteor.connection._send = function () {
console.log(arguments)
this.__proto__._send.apply(this, arguments)
}
I implemented a graceful stop to our node.js server. Basically something like this:
var shutDown = function () {
server.on('close', function () {
console.log('Server ' + process.pid + ' closed.');
process.exit();
});
console.log('Shutting down ' + process.pid + '...');
server.close();
}
However, when I close the server like this, I get a Error: socket hang up error in my continuous requests.
I thought that server.close() would make the server stop listening and accepting new requests, but keep processing all pending/open requests. However, that should result in an Error: connect ECONNREFUSED.
What am I doing wrong?
Additional info: The server consists of a master and three forked children/workers. However, the master is not listening or binding to a port, only the children are, and they are shut down as stated above.
Looking at the docs, it sounds like server.close() only stops new connections from coming in, so I'm pretty sure the error is with the already-open connections.
Maybe your shutDown() can check server.connections and wait until there are no more?
var shutDown = function(){
if(server.connections) return setTimeout(shutDown, 1000);
// Your shutDown code here
}
Slightly uglier (and much less scalable), but without the wait: you can keep track of connections as they occur and close them yourself
server.on('connection', function(e){
// Keep track of e.connection in a list.
// You'll want to remove e.connection
// from the list if it closes on its own
}
var shutDown = function(){
// Close all remaining connections in the list
// Your shutDown code here
}