I was reading across the net, and I wasn't able to find what are the differences between those protocols when Syslog message is being sent, nor a proper example for how RFC6587 log messages look like. If someone can share some insight regarding those two questions.
The 2 RFCs are for different purposes.
RFC 5424 defines a "modern" log format with structural elements, while RFC 6587 can be considered as transport for such a log format over TCP.
RFC 6587 defines frames around syslog messages, and it also mentions/suggests RFC 5424 as payload:
https://datatracker.ietf.org/doc/html/rfc6587#section-3.4.1
SYSLOG-MSG is defined in the syslog protocol [RFC5424] and may
also be considered to be the payload in [RFC3164]
Example for RFC 5424:
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID#32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...
RFC 6587 is just about framing, so the example would be the same, but with prepending the length of the message: MSG-LEN SP SYSLOG-MSG.
Related
I am reading WebSocket specification, it's written:
The WebSocket Protocol is designed on the principle that there should be minimal framing (the only framing that exists is to make the protocol frame-based instead of stream-based and to support a distinction between Unicode text and binary frames).
I don't understand what do frame-based and stream-based mean. Could you please provide examples of them with some basic explanation of the difference?
A short overview:
Frame-Based: If I told you that I'm sending you 8 bytes and I send you 6 bytes, you would wait for the next 2 bytes and than say "this is a message".
Stream-Based: I send you 6 bytes. I send you 2 bytes. Is it one message? two messages? six messages? How do you know where one message starts and another begins?
I'm implementing a ultra simple dummy HTTPÂ server responding a message with Hello world to any requests. It is just for benchmarking the asynchronous event handling with wrk or equivalent web server benchmarking tool.
After some searching on the Web I can't find a clear EndOfMessage (EOM) marker. It seam that with HTTPÂ 1.0 we know we have received the full request when the connection is closed. Is that right ?
For HTTP 1.1, how do we know if pipelining is used ? What is the EOM in this case ?
After some searching on the Web I can't find a clear EndOfMessage (EOM) marker.
You can't find one because such a thing doesn't exist. The only marker you may find is the CRLF pair indicating the end of the header fields. In general, the enclosed entity length (that is for requests and responses!) is either communicated beforehand via the Content-Length header or through the transport coding.
with HTTP 1.0 we know we have received the full request when the connection is closed. Is that right?
That is one of two ways mandated by RFC 1945. So generally speaking: no. From RFC 1945, section 7.2.2:
When an Entity-Body is included with a message, the length of that body may be determined in one of two ways. If a Content-Length header field is present, its value in bytes represents the length of the Entity-Body. Otherwise, the body length is determined by the closing of the connection by the server.
This may read like you were generally in the right with your assertion. BUT:
Closing the connection cannot be used to indicate the end of a request body, since it leaves no possibility for the server to send back a response.
With you being on the receiving side, your assumption is simply wrong on every conceivable level: If the request contains a body, announcing the size of said body through the Content-Length header is an absolute requirement.
HTTP/1.1 is a bit relaxed in this regard, as it allows for more options. As Julian pointed out, please consult RFC 7230, section 3.3.3. That section is straightforward to read and to answer your question, I'd have to c&p it as whole.
For HTTP 1.1, how do we know if pipelining is used ?
You do if you receive multiple requests through one connection. The strongest indicator for the client non engaging into pipelining is the presence of Connection: close in the first received request. See RFC 7230, section 6.3 and section 6.3.2. If you are worried about having to support this, you are always free to just read the first request and send back a response with Connection: close in it. The client will know it has to establish a new connection.
What is the EOM in this case ?
Again, there is no marker as there is no special treatment for requests during pipelining. All pipelining is really enabling is to have multiple requests being issued in one go. See section 3.3.3 from above on how to determine the message length.
I was thinking about JSON/XML-RPC over HTTP vs TCP. In case of HTTP, the HTTP request and response provide a message framing mechanism. However, since TCP is a stream,
How are RPC messages framed?
Does the RPC spec dictate that?
Are there any other standards defining framing mechanisms?
Is there one mechanism more often used than the other?
I'm trying to guage this before inventing a framing mechanism.
There are many framing standards, HTTP being one of them. Websocket being another, on top of HTTP, more suited for bidirectional streams.
JSON-RPC 2.0 does not concern itself with transport, intentionally.
(1.0 had some transport specifics, that was removed in the new version.)
RFC 7484 provides a framing standard for "JSON text sequences": https://www.rfc-editor.org/rfc/rfc7464
Summary: Each JSON message is prefixed with a 0x1E byte (which can't appear unescaped in a JSON message) and is suffixed with 0x0A (linefeed).
Note, however, that this is not part of the JSON-RPC specification. There are libraries that support it as part of their JSON-RPC implementation.
Is there any way to support IETF / RFC 5424 syslog messages with fluentd? My application has no disk, so tailing a file is not an option. The system already outputs standard syslog messages, but it looks like fluentd's syslog plugin is not, by default, able to parse the RFC 5424 message format.
A Fluentd maintainer here.
You are correct: in the past, Fluentd only supported a "default" syslog message via in_syslog. Hopefully in the next version, we have in_tcp and in_udp which accept arbitrary input format, including RFC 5424 syslog format. Here is a pull request for it
It seems Syslog has a 1KB message limit. Is this hardcoded into the Syslog protocol, or is this a parameter that can be set for each server?
I am hoping the article I read was out of date, so if you have any info please share.
This is correct, as can be seen in the syslog protocol RFC. This, and other deficiencies in the syslog protocol, is the reason why modern syslog daemons such as rsyslog support enhanced protocols with features such as TCP transport, encryption etc. There was also some effort within the IETF to standardize an improved syslog protocol, which resulted in RFC5424, RFC5425, and RFC 5426. Here, the minimum maximum message size is relatively small (depending on the transport layer), however implementations are allowed to support larger messages as well.
From my reading of the syslog protocol spec (well, draft standard), message packets can't be more than 1KiB, but (using a fragmentation feature) messages can be. RFC 5424, however, says message size depends on transport, but is at least 480 octets.
Yes, but you can increase this limit to an arbitrary length by recompiling from source.
See instructions in this blog post I found about truncated syslog messages: http://bsdpants.blogspot.com/2010/08/truncated-syslog-messages.html