After having implemented SSE with Express I wanted to do the same with Koa like so:
const Koa = require('koa');
const Router = require('koa-router');
const app = new Koa();
const router = new Router();
router.get('/stream', (ctx, next) => {
ctx.set({
'Access-Control-Allow-Origin': '*',
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'Keep-Alive',
});
const id = new Date().toLocaleTimeString();
ctx.res.write(`id: ${id}'\n`);
ctx.res.write(`data: CONNECTION ESTABLISHED)}\n\n`);
next();
});
app.use(router.routes());
app.listen(8080, () => {
console.log('Listening on port 8080');
});
And for my client, in a React component's constructor:
constructor(props) {
super(props);
this.state = {
source: new EventSource("http://localhost:8080/stream"),
};
}
But for some reason, I received the following error message client-side:
Firefox can’t establish a connection to the server at http://localhost:8080/stream.
Even though my client's request to /stream does go through (but no answer is sent back).
What could be causing this problem with the SSE connection?
I have a Koa server listening on a given port, a route to catch the initial GET request with the correct header data and yet it fails.
A major problem here is that Koa will 'end' the HTTP response as soon as all middleware have run.
This happens immediately after the function ends, of if the function returns a promise, when the promise has resolved. To keep a connection open and circumvent Koa's response handling, you need to make sure that the function 'never ends', and the best way to do that is to simply return a promise that does not resolve.
You're effectively taking over the response handling and stopping Koa from doing so. At this point you can start doing stuff with the socket.
I'm not sure if a more appropriate way exists to handle this in Koa, but this is how I've solved this in the past.
Related
I've looked at countless other posts and cannot find the answer to this, why am I continuously getting the API resolved without sending a response for /api/git/latest-commit, this may result in stalled requests. error in next.js? As soon as I disable sentry it goes away, has anyone else struggled with this?
import type { NextApiRequest, NextApiResponse } from 'next'
import { withSentry } from "#sentry/nextjs";
const handler = async (req: NextApiRequest, res: NextApiResponse) => {
const response = await fetch(`https://api.github.com/repos/####/####/commits?per_page=1`, {
method: 'GET'
});
const data = await response.json();
const commit = data[0]
res.status(200).json({
sha: {
full: commit.sha,
short: commit.sha.substring(0,7)
},
committer: commit.commit.committer.name,
time: commit.commit.committer.date,
html_url: commit.html_url
})
};
export default withSentry(handler);
Running your code produced the following message on my end (next 12.1.4, #sentry/nextjs 6.19.7):
[sentry] If Next.js logs a warning "API resolved without sending a response", it's a false positive, which we're working to rectify.
In the meantime, to suppress this warning, set SENTRY_IGNORE_API_RESOLUTION_ERROR to 1 in your env.
To suppress the nextjs warning, use the externalResolver API route option (see https://nextjs.org/docs/api-routes/api-middlewares#custom-config for details).
To suppress the warning from Sentry, I added this environment variable to an .env.development file:
SENTRY_IGNORE_API_RESOLUTION_ERROR=1
To suppress the warning from the Next.js API route, I added this to latest-commit.ts:
// ...
export const config = {
api: {
externalResolver: true,
},
};
export default withSentry(handler);
Both warnings no longer appear and the data appears to return correctly.
After some digging, this was their explanation as to what's happening:
https://github.com/getsentry/sentry-javascript/pull/4139
In dev, nextjs checks that API route handlers return a response to the client before they resolve, and it throws a warning if this hasn't happened. Meanwhile, in withSentry(), we wrap the res.end() method to ensure that events are flushed before the request/response lifecycle finishes.
As a result, there are cases where the handler resolves before the response is finished, while flushing is still in progress.
I have been trying to implement an SSE stream with Koa for hours now but got the following error when trying to send a message to my client after initializing the connection.
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
Here's how I set up my SSE:
Client-side:
const source = new EventSource("http://localhost:8080/stream");
this.source.onmessage = (e) => {
console.log("---- RECEIVED MESSAGE: ", e.data);
};
// Catches errors
this.source.onerror = (e) => {
console.log("---- ERROR: ", e.data);
};
Server-side (Koa):
// Entry point to our SSE stream
router.get('/stream', ctx => {
// Set response status, type and headers
ctx.response.status = 200;
ctx.response.type = 'text/event-stream';
ctx.response.set({
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
});
// Called when another route is reached
// Should send to the client the following
ctx.app.on('message', data => {
ctx.res.write(`event: Test\n`);
ctx.res.write(`data: This is test data\n\n`);
});
});
The error comes when we call ctx.res.write once a message is received.
Why is my stream ended although nothing explicitly is doing it?
How may I send a message through the stream with Koa?
Koa is entirely promise based and everything is a middleware.
Every middleware returns a promise (or nothing). The middleware chain is effectively 'awaited' and once the middleware returns, Koa knows the response is done and will end the stream.
To make sure that Koa doesn't do this, you have to make sure that the chain of middlewares don't end. To do this, you need to return a promise that only resolves when you're done streaming.
A quick hack to demonstrate would be to return a promise that doesn't resolve:
return new Promise( resolve => { }});
I'm using rxjs v5.4.3, redux-observable v0.16.0.
in my application, I'd like to achieve below:
an user has auth token, and refresh token to regenerate auth token.
the user requests with auth token. (by emitting REQUEST action)
if it failed, request regenerating auth token with refresh token.
if refreshed, emit TOKEN_REFRESHED action to update auth token, and do not emit REQUEST_FAILURE.
if refreshing failed, emit REQUEST_FAILURE
after refreshing(and updating auth token reducer), retry requesting using the refreshed auth token.
if request succeeded, emit REQUEST_SUCCESS, and if failed, emit REQUEST_FAILURE.
I'd like to achieve like:
const fetchEpic = (action$: ActionsObservable<Action>, store: Store<IRootState>) => action$
.ofAction(actions.fetchPost)
.mergeMap(({ payload: { postId } })) => {
const { authToken, refreshToken } = store.getState().auth;
return api.fetchPost({ postId, authToken }) // this returns Observable<ResponseJSON>
.map(res => actions.fetchSuccess({ res })) // if success, just emit success-action with the response
.catch(err => {
if (isAuthTokenExpiredError(err) {
return api.reAuthenticate({ refreshToken })
.map(res => actions.refreshTokenSuccess({ authToken: res.authToken });
.catch(actions.fetchFailure({ err }))
// and retry fetchPost after re-authenticate!
}
return Observable.of(actions.fetchFailure({ err }))
})
}
is there any solution?
There are many ways to do it, but I would recommend splitting off the reauthentication into its own epic to make it easier to maintain/test/reuse.
Here's what that might look like:
const reAuthenticateEpic = (action$, store) =>
action$.ofType(actions.reAuthenticate)
.switchMap(() => {
const { refreshToken } = store.getState().auth;
return api.reAuthenticate({ refreshToken })
.map(res => actions.refreshTokenSuccess({ authToken: res.authToken }))
.catch(err => Observable.of(
actions.refreshTokenFailure({ err })
));
});
We'll also want to use something like Observable.defer so that each time we retry, we look up the latest version of the authToken:
Observable.defer(() => {
const { authToken } = store.getState().auth;
return api.fetchPost({ postId, authToken });
})
When we catch errors in fetchEpic and detect isAuthTokenExpiredError we return an Observable chain that:
Starts listening for a single refreshTokenSuccess, signalling we can retry
Just in case the reauthing itself fails, we listen for it with .takeUntil(action$.ofType(refreshTokenFailure)) so that we aren't waiting around forever--you might want to handle this case differently, your call.
mergeMap it to the original source, which is the second argument of the catch callback. The "source" is the Observable chain before the catch, and since Observables are lazy, when we receive the refreshTokenSuccess action it it will resubscribe to that chain again, effectively be a "retrying"
Merge the above chain with an Observable of an reAuthenticate action. This is used to kick off the actual reauth.
To summarize: the Observable chain we return from catch will first starting listening for refreshTokenSuccess, then it emits reAuthenticate, then when (and if) we receive refreshTokenSuccess we will then "retry" the source, our api.fetchPost() chain above the catch that we wrapped in an Observable.defer. If refreshTokenFailure is emitted before we receive our refreshTokenSuccess, we give up entirely.
const fetchEpic = (action$, store) =>
action$.ofType(actions.fetchPost)
.mergeMap(({ payload: { postId } })) =>
Observable.defer(() => {
const { authToken } = store.getState().auth;
return api.fetchPost({ postId, authToken });
})
.map(res => actions.fetchSuccess({ res }))
.catch((err, source) => {
if (isAuthTokenExpiredError(err)) {
// Start listening for refreshTokenSuccess, then kick off the reauth
return action$.ofType(actions.refreshTokenSuccess)
.takeUntil(action$.ofType(refreshTokenFailure))
.take(1)
.mergeMapTo(source) // same as .mergeMap(() => source)
.merge(
Observable.of(action.reAuthenticate())
);
} else {
return Observable.of(actions.fetchFailure({ err }));
}
});
);
These examples are untested, so I may have some minor issues but you hopefully get the gist. There's also probably a more elegant way to do this, but this will at least unblock you. (Others are more than welcome to edit this answer if they can decrease the complexity)
Side notes
This creates the slight potential for infinite retries, which can cause nasty issues both in the person's browser or your servers. It might be a good idea to only retry a set number of times, and/or put some sort of delay in between your retries. In practice this might not be worth worrying about, you'll know best.
You (or someone else reading this later) may be tempted to use .startWith(action.reAuthenticate()) instead of the merge, but be mindful that a startWith is just shorthand for a concat, not a merge, which means it would synchronously emit the action before we have started to listen for a success one. Usually that isn't an issue since http requests are async, but it's caused people bugs before.
I am using node-http-proxy. However, in addition to relaying HTTP requests, I also need to listen to the incoming and outgoing data.
Intercepting the response data is where I'm struggling. Node's ServerResponse object (and more generically the WritableStream interface) doesn't broadcast a 'data' event. http-proxy seems to create it's own internal request, which produces a ClientResponse object (which does broadcast the 'data' event) however this object is not exposed publically outside the proxy.
Any ideas how to solve this without monkey-patching node-http-proxy or creating a wrapper around the response object?
Related issue in issues of node-http-proxy on Github seems to imply this is not possible. For future attempts by others, here is how I hacked the issue:
you'll quickly find out that the proxy is only calling writeHead(), write() and end() methods of the res object
since res is already an EventEmitter, you can start emitting new custom events
listen for these new events to assemble the response data and then use it
var eventifyResponse = function(res) {
var methods = ['writeHead', 'write', 'end'];
methods.forEach(function(method){
var oldMethod = res[method]; // remember original method
res[method] = function() { // replace with a wrapper
oldMethod.apply(this, arguments); // call original method
arguments = Array.prototype.slice.call(arguments, 0);
arguments.unshift("method_" + method);
this.emit.apply(this, arguments); // broadcast the event
};
});
};
res = eventifyResponse(res), outputData = '';
res.on('method_writeHead', function(statusCode, headers) { saveHeaders(); });
res.on('method_write', function(data) { outputData += data; });
res.on('method_end', function(data) { use_data(outputData + data); });
proxy.proxyRequest(req, res, options)
This is a simple proxy server sniffing the traffic and writing it to console:
var http = require('http'),
httpProxy = require('http-proxy');
//
// Create a proxy server with custom application logic
//
var proxy = httpProxy.createProxyServer({});
// assign events
proxy.on('proxyRes', function (proxyRes, req, res) {
// collect response data
var proxyResData='';
proxyRes.on('data', function (chunk) {
proxyResData +=chunk;
});
proxyRes.on('end',function () {
var snifferData =
{
request:{
data:req.body,
headers:req.headers,
url:req.url,
method:req.method},
response:{
data:proxyResData,
headers:proxyRes.headers,
statusCode:proxyRes.statusCode}
};
console.log(snifferData);
});
// console.log('RAW Response from the target', JSON.stringify(proxyRes.headers, true, 2));
});
proxy.on('proxyReq', function(proxyReq, req, res, options) {
// collect request data
req.body='';
req.on('data', function (chunk) {
req.body +=chunk;
});
req.on('end', function () {
});
});
proxy.on('error',
function(err)
{
console.error(err);
});
// run the proxy server
var server = http.createServer(function(req, res) {
// every time a request comes proxy it:
proxy.web(req, res, {
target: 'http://localhost:4444'
});
});
console.log("listening on port 5556")
server.listen(5556);
I tried your hack but it didn't work for me. My use case is simple: I want to log the in- and outgoing traffic from an Android app to our staging server which is secured by basic auth.
https://github.com/greim/hoxy/
was the solution for me. My node-http-proxy always returned 500 (while the direct request to stage did not). Maybe the authorization headers would not be forwarded correctly or whatever.
Hoxy worked fine right from the start.
npm install hoxy [-g]
hoxy --port=<local-port> --stage=<your stage host>:<port>
As rules for logging I specified:
request: $aurl.log()
request: #log-headers()
request: $method.log()
request: $request-body.log()
response: $url.log()
response: $status-code.log()
response: $response-body.log()
Beware, this prints any binary content.
So I would like to do something like:
app.On_All_Incoming_Request(function(req, res){
console.log('request received from a client.');
});
the current app.all() requires a path, and if I give for example this / then it only works when I'm on the homepage, so it's not really all..
In plain node.js it is as simple as writing anything after we create the http server, and before we do the page routing.
So how to do this with express, and what is the best way to do it?
Express is based on the Connect middleware.
The routing capabilities of Express are provided by the router of your app and you are free to add your own middlewares to your application.
var app = express.createServer();
// Your own super cool function
var logger = function(req, res, next) {
console.log("GOT REQUEST !");
next(); // Passing the request to the next handler in the stack.
}
app.configure(function(){
app.use(logger); // Here you add your logger to the stack.
app.use(app.router); // The Express routes handler.
});
app.get('/', function(req, res){
res.send('Hello World');
});
app.listen(3000);
It's that simple.
(PS : If you just want some logging you might consider using the logger provided by Connect)
You should do this:
app.all("*", (req, res, next) => {
console.log(req); // do anything you want here
next();
});
You can achieve it by introducing a middleware function.
app.use(your_function) can be of help. app.use with accept a function that will get executed on every request logged to your server.
Example:
app.use((req, res, next) => {
console.log("req received from client");
next(); // this will invoke next middleware function
});
Express supports wildcards in route paths. So app.all('*', function(req, res) {}) is one way to go.
But that's just for route handlers. The difference is that an Express route handler is expected to send a response, and, if it doesn't, Express will never send a response. If you want to do something without explicitly sending a response, like check for a header, you should use Express middleware. app.use(function(req, res, next) { doStuff(); next(); }