I'm new to gRPC programming. I need to write a gRPC client which receives file content sent from the gRPC server and send this content to web-page. As the file content is huge, I've set it as a stream.
Below is my .proto file
service LogService {
rpc fetchLogContent(LogRequest) returns (stream LogResponse);
}
message LogRequest {
string ip = 1;
string fileName = 2;
}
message LogResponse {
string ip = 1;
string logContent = 2;
}
Now, in the client, when I use the blockingStub to access fetchLogContent, it returns an Iterator . I understand from the examples provided grpc.io - java, if there are list of response objects, (list of Feature objects in the example provided in link), an Iterator will be valid. But in my case, I need a single LogResponse which was sent as a stream. Please provide any suggestions/alternatives for this. Thanks in advance. :)
In your Method definition:
rpc fetchLogContent(LogRequest) returns (stream LogResponse);
The stream keyword means that you will get 0 or more LogResponse messages. If the content is very large, what you can do is read chunks of the file (like 4kb) and send multiple LogResponse messages each with a part of the file. The client side can read the chunks repeatedly, and piece it back together.
Since the ip field will likely not be changing each time, you can make you server only set ip on the first message. On the client, just use the very first ip received.
Related
I write a Nginx module, the client will post stream data to it. But I found the client will be blocked after send about 64KB data(http body's size). I try to change the Nginx config client_body_buffer_size, after that the client can send more data.
r->request_body_no_buffering = 1; ngx_http_read_client_request_body(r, body_handler);
Do I have to do something in the body_handler function which release the request_body->buffs?
I try to use ngx_free_chain, but it doesn't work.
i'm using syslog-ng for collecting json messages and send alarms to slack
there is parameter in json message which contains IP address of router from which I'm receiving the json message and I want to convert IP address to router hostname
i'm using two parsers 1) parse json 2) replace IP address with router hostname:
parser p_json {
json-parser(prefix(".json."));
};
parser p_acd_router {
add-contextual-data(selector("${.json.router_ip}"), database("host_map.csv"), prefix(".meta_router."));
};
until now everything works fine.
Hi,
I'm having the problem when I receive log message from router and I don't have IP_to_Hostname mapping for it in csv file. in this case i just get blank space in syslog message. there is option to return default value if mapping does not exist, but i would prefer to return original value (IP address)
if mapping exists convert IP address to hostname >> works
if mapping does not exist, return original IP address >> don't know how to set it up
is this even supported?
thanks
You can set the default-selector("UNKNOWN") option for add-contextual-data(), and add a record to your CSV file with the ID UNKNOWN, and use the following value when setting .meta_router.hostname: ${.json.router_ip}.
TLDR: templates are supported inside the CSV file as well.
Note: In case your IPs are reverse-resolvable, you can just use the $(dns-resolve-ip) template function instead of maintaining a complete CSV database:
https://github.com/syslog-ng/syslog-ng/pull/3046
I am building an UDP messaging system with Protocol Buffers. I have several messages (e.g. ChannelUpdate, MessageCreate, ACK). If I send a message to the remote, how can I also send what type of message I am sending?
Your best bet is to always send the same message type: a root message that has one of the others. For example:
message SomeRoot {
oneof content {
ChannelUpdate channelUpdate = 1;
MessageCreate msgCreate = 2;
// etc
}
}
Now you're using an inbuilt protocol feature to enforce the logic.
I receive an email with an attached file.
I retrieve the email through a receive location.
Retrieve the attachment through a pipeline.
Then I have a send Port that consumes the message from the pipeline.
I'd like the send port to publish the document with the same name the file had in the attachment.
I read that you had to use the macro %SourceFileName% when setting the file name in the send port configurations.
I tried that but the file name comes out as literally %SourceFileName%.
I stopped the send port to check what was the FileReceivedName and it is what I want.
These are the Send Port Configurations.
After I restart the send port the file that gets published.
You can promote a context property(nor especially ReceivedFileName) in your pipeline and use a macro to get this property like what you did with ReceivedFileName. have a look on this thread:
%SourceFileName% is macro for the FILE.ReceivedFileName message context property which is normally added to the message context by FILE or FTP adapter, but not SMTP. You can set this property yourself inside orchestration or your receive pipeline and then you will be able to use the %SourceFileName% macro on the send port.
Get the attachment filename from mime-properties and promote ReceivedFileName property to get %SourceFileName% during sending.
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg) {
IBaseMessagePart bodyPart = pInMsg.BodyPart;
if (bodyPart != null)
{
string fileName = (string)pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties");
fileName = fileName.Substring(fileName.LastIndexOf("\\") + 1);
pInMsg.BodyPart.PartProperties.Write("FileName", "http://schemas.microsoft.com/BizTalk/2003/mime-properties", fileName);
}
return pInMsg;}
I am writing my own http module (I know that node.js includes one)
I am using the net module
when I am getting a request for a static binary(picture) file, how do I generate HTTP response with the binary file?
When I do exactly that for text file(e.g. html file) it just works..
If you are using the net module, you are probably using a code such as :
var server = net.createServer(function (socket) {
});
What is a socket? it represents the stream of data over the network.
In objects, Socket is a WriteableStream, read more here: http://nodejs.org/docs/v0.6.5/api/streams.html
When you are reading a file you can get the content as String, Buffer or as a ReadableStream
The easiest way to read file as a stream is by using the function: http://nodejs.org/docs/v0.6.5/api/fs.html#fs.createReadStream
e.g.
var fileAsAstream = fs.createReadStream(filePath);
In order to transfer the content of a binary read-stream to a write-stream you can use the pipe function http://nodejs.org/docs/v0.6.5/api/streams.html#stream.pipe functions
e.g.
fileAsAstream.pipe(socket);