I need to modify the topic of MQTT publish packages using NGINX proxy. I created a njs function which is called by js_filter.
function updateTopic(s) {
s.log("buf: " + s.buffer.toString('hex'));
if (!s.fromUpstream) {
if(s.buffer.indexOf("topic") != -1){
s.buffer = s.buffer.replace("topic", "mopic").toBytes();
s.log("new buffer: " + s.buffer.toString('hex'));
}
}
return s.OK;
}
returns:
buf: 300f0005746f7069636d65737361676533
new buffer: 300f00056d6f7069636d65737361676533
This function updates s.buffer correctly, but the packages are not transmitted until another type of package is received. When a subscribe, disconnect, ping message is received, all the buffered messages are transmitted at once.
If this function does not replace a package, it is transmitted instantaneously.
Should I do something special after changing s.buffer?
It turned out to be due to a bug, which will be fixed in later versions.
https://github.com/nginx/njs/issues/45
Related
I'm implementing a Thrift client in order to make connection to a built-in scribe server.
Everything is going OK if I use a standard Log method, like this:
public boolean log(List<LogEntry> messages) {
boolean ret = false;
PooledClient client = borrowClient();
try {
if ((client != null) && (client.getClient() != null)) {
ResultCode result = client.getClient().Log(messages);
ret = (result != null && result.equals(ResultCode.OK));
returnClient(client);
}
} catch (Exception ex) {
logger.error(LogUtil.stackTrace(ex));
invalidClient(client);
}
return ret;
}
However, when I use send_Log instead:
public void send_Log(List<LogEntry> messages) {
PooledClient client = borrowClient();
try {
if ((client != null) && (client.getClient() != null)) {
client.getClient().send_Log(messages);
returnClient(client);
}
} catch (Exception ex) {
logger.error(LogUtil.stackTrace(ex));
invalidClient(client);
}
}
It acctually causes some problems:
Total network connection to port 1463 (default port for a scribe server) is going to increase so much, and always in a CLOSE_WAIT state.
Cause my application got stuck without throwing any error, I think it may be an issue with network connection.
what if send without recv
As this is clearly TCP, the sender will block (in blocking mode), or incur EAGAIN/EWOULDBLOCK in non-blocking mode. EDIT It is now clear that you want to send without receiving the reply. You can do that by just sending and then closing the socket, but that may cause the peer to incur ECONNRESET, which may upset it. You should really implement the application protocol correctly.
1/ Total network connection to port 1463 (default port for a scribe server) is going to increase so much, and always in a CLOSE_WAIT state.
Lots of ports in CLOSE_WAIT state indicates a socket leak on the part of the local application.
2/ Cause my application got stuck without throwing any error. I think it may be an issues with network connection.
It is an issue with sending and not receiving.
Since you labelled this as a Thrift related question, the answer is oneway.
service foo {
oneway void FireAndForget(1: some args)
}
The oneway keyword does exactly what the name suggests. You get a client implementation that only sends and does not wait for anything to be returned from the server. This rule also includes exceptions. Hence a oneway method must always be void and can't throw any exceptions.
However, when I use send_Log instead ...
client.getClient().send_Log(messages);
Neither one of the Thrift-generated send_Xxx and recv_Xxx methods are meant to be public. That's why they are usually either private or protected methods. They should not be called directly, unless you are sure that you know what you are doing (and very obviously the latter is not the case here).
And since the real question is about performance: Why don't you just delegate the call(s) into a secondary thread? That way the I/O will not block the UI.
I'm working my way through boost's asio tutorial. I'm looking into their chat example. More specifically, I'm trying to split their chat client from a sender+receiver, to just a sender and just a receiver, but I'm seeing some behaviour that I can't explain.
The setup consists of:
boost::asio::io_service io_service;
tcp::resolver::iterator endpoint = resolver.resolve(...);
boost::thread t(boost::bind(&boost::asio::io_service::run, &io_service));
boost::asio::async_connect(socket, endpoint, bind(handle_connect, ... ));
The sending portion effectively conisists of:
while (std::cin.getline(str))
io_service.post( do_write, str );
and
void do_write (string str)
{
boost::asio::async_write(socket, str, bind( handle_write, ... ));
}
The receive section consists of
void handle_connect(...)
{
boost::asio::async_read(socket, read_msg_, bind(handle_read, ...));
}
void handle_read(...)
{
std::cout << read_msg_;
boost::asio::async_read(socket, read_msg_, bind(handle_read, ...));
}
If I comment out the content of handle_connect to isolate the send portion, my other client (compiled using the original code) does not receive anything. If I revert, then comment out the content of handle_read, my other client only receives the first message.
Why is it necessary to call async_read() in order to be able to post() an async_write()?
The full unmodified code is linked above.
The problem here is that, your io_service is running out of work and stops processing requests even before you start sending your chat messages.
If you comment out the body of handle_connect, then the only work it had to do was to dispatch the handle_connect handler and then execute it once the connection was done.
std::size_t scheduler::run(asio::error_code& ec)
{
.....
mutex::scoped_lock lock(mutex_);
std::size_t n = 0;
for (; do_run_one(lock, this_thread, ec); lock.lock())
if (n != (std::numeric_limits<std::size_t>::max)())
++n;
return n;
}
So, you have to provide it with something in it's operation queue. This was done with handle_read_header handler in the original code as this handler would always be in the need of servicing till the client gets something from the server.
You can do what you want to do by providing work to the io_service.
asio::io_context io_context;
asio::io_context::work wrk(io_context); // make `run` run forever
tcp::resolver resolver(io_context);
tcp::resolver::results_type endpoints = resolver.resolve(argv[1], argv[2]);
chat_client c(io_context, endpoints);
asio::thread t(boost::bind(&asio::io_context::run, &io_context));
I am reading a tcp-ip socket response (JSON) from the server. The issue is that sometimes if the data that is received from the server is very large it comes in batches of 2 or 3 with a random break in the JSON. Is there a way to detect how many batches are being sent from the server or is there a mechanism to tackle this at the client end? Below is the titanium code for TCP/IP:
var socket = Ti.Network.Socket.createTCP({
host: '127.0.0.1',
port: 5000,
connected: function (e) {
Ti.API.info('Socket opened!');
Ti.Stream.pump(socket, readCallback, 2048, true);
},
error: function (e) {
Ti.API.info('Error (' + e.errorCode + '): ' + JSON.stringify(e));
},
});
socket.connect();
function writeCallback(e) {
Ti.API.info('Successfully wrote to socket.'+JSON.stringify(e));
}
function readCallback(e) {
Ti.API.info('e ' + JSON.stringify(e));
if (e.bytesProcessed == -1)
{
// Error / EOF on socket. Do any cleanup here.
Ti.API.info('DONE');
}
try {
if(e.buffer) {
var received = e.buffer.toString();
Ti.API.info('Received: ' + received);
} else {
Ti.API.error('Error: read callback called with no buffer!');
socket.close();
}
} catch (ex) {
Ti.API.error('Catch ' + ex);
}
}
TCP/IP is a streaming interface and there is no guarantee from the client side how much data will be received when a read is attempted from the socket.
You would have to implement some sort of protocol between the server and the client.
TCP/IP Protocol is not designed like that, you cannot know how much data you will receive, it's in darkness lol
There are two ways to solve this problem, as I know.
The first One is EOF, you will add something as a prefix at the end of the packet, and it's a unique string like [%~Done~%]
so? you will receive everything and search in it about this string, did you find it? let's go to processing.
but I saw it as wasting of time, memory, and very primitive
the second one is Prefix Packet header, and it's the best for me, an example in .NET can be located here: Thanks to Stephen Cleary
I'm trying to wrap my head around the Redis Pub/Sub API and setup a long-polling server.
This lua script subscribes to a 'test' channel and returns new messages received:
nginx.conf:
location /poll {
lua_need_request_body on;
default_type 'text/plain';
content_by_lua_file '/usr/local/nginx/html/poll.lua';
}
poll.lua:
local redis = require "redis";
local red = redis:new();
local cjson = require "cjson";
red:set_timeout(30000) -- 30 sec
local resCon, err = red:connect("127.0.0.1", 6379)
if not resCon then
ngx.print("error")
return
end
local resSub, err = red:subscribe('r:' .. ngx.var["arg_r"]:gsub('%W',''))
if not resSub then
ngx.print("error")
return
end
if resSub == ngx.null then
ngx.print("error")
return
end
local resMsg, err = red:read_reply()
if not resMsg then
ngx.say("0")
return
end
ngx.say(cjson.encode(resMsg))
client.js:
var tmpR = 'test';
function poll() {
$.get('/poll', {'r':tmpR}, function(data){
if (data !== "error") {
console.log(data);
window.setTimeout(function(){
poll();
},1000);
} else {
console.log('poll fail');
}
})
}
Now, if I send publish r:test hello from redis-cli, I receive the message on the client and the server responds to redis-cli with 1. But, if I send two messages quickly, the second message doesn't broadcast and the server responds with 0.
Are my channels only capable of receiving a message per second, or, is this a throttle on the frequency of messages a user can broadcast to a channel?
Is this the right way to approach this polling server on nginx assuming many users may be connected at one time? Would it be more efficient to use GET requests on a timer?
Given two consecutive messages only one is going to have a subscriber listening to the result. No subscriber is listening when the second message is sent. The only subscriber is processing the previous result and returning that to the user.
Redis is not maintaining a message queue or similar to make sure that previously listening clients will receive the missing messages upon reconnect.
Are console.log/debug/warn/error in node.js asynchrounous? I mean will javascript code execution halt till the stuff is printed on screen or will it print at a later stage?
Also, I am interested in knowing if it is possible for a console.log to NOT display anything if the statement immediately after it crashes node.
Update: Starting with Node 0.6 this post is obsolete, since stdout is synchronous now.
Well let's see what console.log actually does.
First of all it's part of the console module:
exports.log = function() {
process.stdout.write(format.apply(this, arguments) + '\n');
};
So it simply does some formatting and writes to process.stdout, nothing asynchronous so far.
process.stdout is a getter defined on startup which is lazily initialized, I've added some comments to explain things:
.... code here...
process.__defineGetter__('stdout', function() {
if (stdout) return stdout; // only initialize it once
/// many requires here ...
if (binding.isatty(fd)) { // a terminal? great!
stdout = new tty.WriteStream(fd);
} else if (binding.isStdoutBlocking()) { // a file?
stdout = new fs.WriteStream(null, {fd: fd});
} else {
stdout = new net.Stream(fd); // a stream?
// For example: node foo.js > out.txt
stdout.readable = false;
}
return stdout;
});
In case of a TTY and UNIX we end up here, this thing inherits from socket. So all that node bascially does is to push the data on to the socket, then the terminal takes care of the rest.
Let's test it!
var data = '111111111111111111111111111111111111111111111111111';
for(var i = 0, l = 12; i < l; i++) {
data += data; // warning! gets very large, very quick
}
var start = Date.now();
console.log(data);
console.log('wrote %d bytes in %dms', data.length, Date.now() - start);
Result
....a lot of ones....1111111111111111
wrote 208896 bytes in 17ms
real 0m0.969s
user 0m0.068s
sys 0m0.012s
The terminal needs around 1 seconds to print out the sockets content, but node only needs 17 milliseconds to push the data to the terminal.
The same goes for the stream case, and also the file case gets handle asynchronous.
So yes Node.js holds true to its non-blocking promises.
console.warn() and console.error() are blocking. They do not return until the underlying system calls have succeeded.
Yes, it is possible for a program to exit before everything written to stdout has been flushed. process.exit() will terminate node immediately, even if there are still queued writes to stdout. You should use console.warn to avoid this behavior.
My Conclusion , after reading Node.js 10.* docs (Attached below). is that you can use console.log for logging , console.log is synchronous and implemented in low level c .
Although console.log is synchronic, it wont cause a performance issue only if you are not logging huge amount of data.
(The command line example below demonstrate, console.log async and console.error is sync)
Based on Node.js Doc's
The console functions are synchronous when the destination is a terminal or a file (to avoid lost messages in case of premature exit) and asynchronous when it's a pipe (to avoid blocking for long periods of time).
That is, in the following example, stdout is non-blocking while stderr is blocking:
$ node script.js 2> error.log | tee info.log
In daily use, the blocking/non-blocking dichotomy is not something you should worry about unless you > log huge amounts of data.
Hope it helps
Console.log is asynchronous in windows while it is synchronous in linux/mac. To make console.log synchronous in windows write this line at the start of your
code probably in index.js file. Any console.log after this statement will be considered as synchronous by interpreter.
if (process.stdout._handle) process.stdout._handle.setBlocking(true);
You can use this for synchrounous logging:
const fs = require('fs')
fs.writeSync(1, 'Sync logging\n')