nginx returning netstring with wrong length? - nginx

I installed nginx (nginx version: nginx/1.7.9) via macports on my mac running the latest OSX.
I configured a URI to use SCGI:
location /server {
include /Users/ruipacheco/Projects/Assorted/nginx/conf/scgi_params;
scgi_pass unix:/var/tmp/rpc.sock;
#scgi_pass 127.0.0.1:9000;
}
And when I do a GET request on 127.0.0.1/server, I see the following on my SCGI server:
633:CONTENT_LENGTH0REQUEST_METHODGETREQUEST_URI/serverQUERY_STRINGCONTENT_TYPEDOCUMENT_URI/serverDOCUMENT_ROOT/opt/local/htmlSCGI1SERVER_PROTOCOLHTTP/1.1REMOTE_ADDR127.0.0.1REMOTE_PORT62088SERVER_PORT80SERVER_NAMElocalhostHTTP_HOST127.0.0.1HTTP_CONNECTIONkeep-aliveHTTP_CACHE_CONTROLmax-age=0HTTP_ACCEPTtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8HTTP_USER_AGENTMozilla/5.0
(Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/40.0.2214.115
Safari/537.36HTTP_DNT1HTTP_ACCEPT_ENCODINGgzip, deflate,
sdchHTTP_ACCEPT_LANGUAGEen-US,en;q=0.8,End of file
The problem is that the length of the netstring, 633, does not match the interpretation. If I understand the netstrings spec correctly, 633 should be the length of characters between the first : and the last ,:
Any string of 8-bit bytes may be encoded as [len]":"[string]",". Here [string] is the string and [len] is a nonempty sequence of ASCII digits giving the length of [string] in decimal. The ASCII digits are <30> for 0, <31> for 1, and so on up through <39> for 9. Extra zeros at the front of [len] are prohibited: [len] begins with <30> exactly when [string] is empty.
For example, the string hello world! is encoded as 31 32 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 2c, i.e., 12:hello world!,.
So, I'm getting the wrong length. How can this be explained?

As far as I can tell, your example response has correct length.
According to example here:
http://en.wikipedia.org/wiki/Simple_Common_Gateway_Interface
Field values are preceded and followed by <00> symbol (ASCII symbol with hex code 00), eg.:
REQUEST_METHOD <00>GET<00>
Once I added missing spaces to your response snippet – it quickly got back to 633 bytes, as advertised.
I suppose somewhere in the process of passing that response to us here, some piece of software stripped <00>'s, which is a totally normal behaviour?
Anyway, the answer seems to be – your nginx is either returning a correct length, or your response is stripping <00>'s somewhere.

Well,
The hexadecimal <31 32 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21>
in ASCII is "12:hello world!" (no quotes) and the lenght is 12 (hello world!)
And this one <31 32 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 2c> in the example is wrong (at least it didnt match the nginx norm.)(since the internal lenght is 13 and the lenght specified in hex is 12):
The ASCII "12:hello world!," should be "13:hello world!," and in hex <31 33 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 2c>
This line is the mess:
For example, the string "hello world!" is encoded as <31 32 3a 68 65
6c 6c 6f 20 77 6f 72 6c 64 21 2c>, i.e., "12:hello world!,".
OK) 12:hello world! ---> <31 *32* 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21>
KO) 12:hello world!, ---> <31 *32* 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 2c>
OK) 13:hello world!, ---> <31 *33* 3a 68 65 6c 6c 6f 20 77 6f 72 6c 64 21 2c>
The hex inside the ** is the second number of the lenght.
Then your concept about this Ok, the example is bad.

Related

Cannot get Russian Subject from Outlook using RDCOMClient

I am using Outlook under Windows 10 as my email client and am trying to use the RDCOMClient library to process some emails. Some of the emails are in Russian and I am having trouble getting the Russian part out in a usable format. Right now, I am just
focusing on the subject lines. When I extract the line and print
it out, I just get question marks except for a few Latin characters
in the subject. I have tried setting the encoding and using
iconv, but with no success. But iconv did provide a useful clue.
Based on my reproducible example below showing the raw characters
gives:
iconv(SUBJECT, toRaw=T)
[1] 53 74 61 63 6b 4f 76 65 72 66 6c 6f 77 54 65 73 74 4d 65 73 73 61 67 65 3a
[26] 20 3f 3f 3f 3f 3f 3f 3f 3f 20 3f 3f 3f 3f 3f 3f 3f 3f 3f
All of the 3f's at the end? That is the code for question mark. RDCOMClient is
actually returning the ??? from Outlook. It is not some encoding issue inside R.
I have looked at many RDCOMClient posts on SO, but do not see anything
that deals with this problem.
Is the RDCOMClient<->Outlook connection just broken? Or is there some way
around this?
Attempt at Reproducible Example
Since we are talking about accessing email, I don't see how to make a
really easy reproducible example, but here is a reproducible way to test this.
Of course, you have to have Outlook on Windows for this to make sense.
Send yourself an email with the subject line:
StackOverflowTestMessage: Тестовое сообщение
R code
We need to find the email first. Most of the code does that.
Then we inspect the subject.
## Connect to Outlook
OutApp <- COMCreate("Outlook.Application")
outlookNameSpace = OutApp$GetNameSpace("MAPI")
## Find the Inbox
INBOX = outlookNameSpace$GetDefaultFolder(6)
INBOX$Name() ## Confirm
emails <- INBOX$Items
## Find the relevant email
NumEmail = emails()$Count()
MessageNumber = 0
for(i in NumEmail:1) {
SUBJ = emails(i)$Subject()
if(grepl("StackOverflowTestMessage", SUBJ)) {
MessageNumber = i
break()
}
}
## Now try to get the subject line
SUBJECT = emails(MessageNumber)$Subject()
Encoding(SUBJECT) = 'UTF-8'
SUBJECT
[1] "StackOverflowTestMessage: ???????? ?????????"
iconv(SUBJECT, toRaw=T)
[[1]]
[1] 53 74 61 63 6b 4f 76 65 72 66 6c 6f 77 54 65 73 74 4d 65 73 73 61 67 65 3a
[26] 20 3f 3f 3f 3f 3f 3f 3f 3f 20 3f 3f 3f 3f 3f 3f 3f 3f 3f```

Why is the hex value of a period in a DNS request not 0x2E, and why does it change?

Looking at a DNS request in wireshark for www.google.com and the hex for it is 03 77 77 77 06 67 6f 6f 67 6c 65 03 63 6f 6d 00
Little confused why the first period is 03 (and why it's there), the second is 06, and the last is 03
The DNS protocol layer is defined in RFC 1035. To cite from "3.1. Name space definitions":
Domain names in messages are expressed in terms of a sequence of labels.
Each label is represented as a one octet length field followed by that
number of octets. Since every domain name ends with the null label of
the root, a domain name is terminated by a length byte of zero.
Thus www.google.com is encoded in the DNS packet as:
03 77 77 77 length 3, "www"
06 67 6f 6f 67 6c 65 length 6, "google"
03 63 6f 6d length 3, "com"
00 length 0 (end of label)

looking to understand meaning of two bytes in HTTP request made with curl --trace

tl;dr "What would the bytes 0x33 0x39 0x0d 0x0a between the end of HTTP headers and the start of HTTP response body refer to?"
I'm using the thoroughly excellent libcurl to make HTTP requests to various 3rd party endpoints. These endpoints are not under my control and are required to implement a specification. To help debug and develop these endpoints I have implemented the text output functionality you might see if you make a curl request from the command line with the -v flag using curl.setopt(pycurl.VERBOSE, 1) and curl.setopt(pycurl.DEBUGFUNCTION, debug_function)
This has been working great but recently I've come across a request which my debug function does not handle in the same way as curl's debug output. I'm sure is due to me not understanding the HTTP spec.
If making a curl request from the command line with --verbose I get the following returned.
# redacted headers
< Via: 1.1 vegur
<
{"code":"InvalidCredentials","message":"Bad credentials"}*
Connection #0 to host redacted left intact
If making the same request with --trace the following is returned
0000: 56 69 61 3a 20 31 2e 31 20 76 65 67 75 72 0d 0a Via: 1.1 vegur..
<= Recv header, 2 bytes (0x2)
0000: 0d 0a ..
<= Recv data, 1 bytes (0x1)
0000: 33 3
<= Recv data, 62 bytes (0x3e)
0000: 39 0d 0a 7b 22 63 6f 64 65 22 3a 22 49 6e 76 61 9..{"code":"Inva
0010: 6c 69 64 43 72 65 64 65 6e 74 69 61 6c 73 22 2c lidCredentials",
0020: 22 6d 65 73 73 61 67 65 22 3a 22 42 61 64 20 63 "message":"Bad c
0030: 72 65 64 65 6e 74 69 61 6c 73 22 7d 0d 0a redentials"}..
<= Recv data, 1 bytes (0x1)
0000: 30 0
<= Recv data, 4 bytes (0x4)
0000: 0d 0a 0d 0a ....
== Info: Connection #0 to host redacted left intact
All HTTP client libs I've tested don't include these parts of the bytes in the response body so I'm guessing these are part of the HTTP spec I don't know about but I can't find a reference to them and I don't know how to handle them.
If it's helpful I think curl is using this https://github.com/curl/curl/blob/master/src/tool_cb_dbg.c for building the output in the first example bit I'm not really a c/c++ programmer and I haven't been able to reverse engineer the logic.
Does anyone know what these bytes are?
0d 0a are ASCII control characters representing carriage return and line feed, respectively. CRLF is used in HTTP to mark the end of a header field (there are some historic exceptions you should not worry about at this point). A double CRLF is supposed to mark the end of the fields section of a message.
The 33 39 you observe there is "39" in ascii. This is the chunk size indicator - treated as a hexdecimal number. The presence of Transfer-Encoding: chunked in the response headers may support this.

How do i make the result of base64enc::base64decode human readable

I have some text that is base 64 encoded and want to decode it in R. The package im using is base64decode of the base64enc package. The problem I have is that its not human readable. How do i make it work
e.g. This is what i get from a text string that was endcoded from "exampleEncodedText"
base64enc::base64decode("ZXhhbXBsZUVuY29kZWRUZXh0")
[1] 65 78 61 6d 70 6c 65 45 6e 63 6f 64 65 64 54 65 78 74
For reference i encoded it on https://www.base64decode.org/
?base64decode says that this function decodes a base64-encoded string into binary data. So, using rawToChar gives a human readable character:
rawToChar(base64decode("ZXhhbXBsZUVuY29kZWRUZXh0"))
[1] "exampleEncodedText"

Errors while opening dicom files in R

I am trying to open dicom files in R using following code:
library(oro.dicom)
dcmobject <- readDICOMFile(filename)
Some files open properly and I can display them. However, some files give errors of different types:
First error: For some, I get the error:
Error in file(con, "rb") : cannot open the connection
Second error: In others, I get following error with dicom file: http://www.barre.nom.fr/medical/samples/files/OT-MONO2-8-hip.gz :
Error in readDICOMFile(filename) : DICM != DICM
Third error: This file gives following error: http://www.barre.nom.fr/medical/samples/files/CT-MONO2-16-chest.gz
Error in parsePixelData(fraw[(132 + dcm$data.seek + 1):fsize], hdr, endian, :
Number of bytes in PixelData not specified
Fourth error: One dicom file gives following error:
Error in rawToChar(fraw[129:132]) : embedded nul in string: '\0\0\b'
How can I get rid of these errors and display these images in R?
EDIT:
This sample file gives the error 'embed nul in string...':
http://www.barre.nom.fr/medical/samples/files/CT-MONO2-12-lomb-an2.gz
> jj = readDICOMFile( "CT-MONO2-12-lomb-an2.dcm" )
Error in rawToChar(fraw[129:132]) : embedded nul in string: '3\0\020'
There are four different errors highlighted in this ticket:
Error in file(con, "rb") : cannot open the connection
This is not a problem with oro.dicom, it is simply the fact that the file path and/or name has been mis-specified.
Error in readDICOMFile(filename) : DICM != DICM
The file is not a valid DICOM file. That is, section 7.1 in Part 10 of the DICOM Standard (available at http://dicom.nema.org) specifies that there should be (a) the File Preample of length 128 bytes and (b) the four-byte DICOM Prefix "DICM" at the beginning of a DICOM file. The file OT-MONO2-8-hip does not follow this standard. One can investigate this problem further using the debug=TRUE input parameter
> dcm <- readDICOMFile("OT-MONO2-8-hip.dcm", debug=TRUE)
# First 128 bytes of DICOM header =
[1] 08 00 00 00 04 00 00 00 b0 00 00 00 08 00 08 00 2e 00 00 00 4f 52 49 47 49 4e 41 4c 5c 53 45
[32] 43 4f 4e 44 41 52 59 5c 4f 54 48 45 52 5c 41 52 43 5c 44 49 43 4f 4d 5c 56 41 4c 49 44 41 54
[63] 49 4f 4e 20 08 00 16 00 1a 00 00 00 31 2e 32 2e 38 34 30 2e 31 30 30 30 38 2e 35 2e 31 2e 34
[94] 2e 31 2e 31 2e 37 00 08 00 18 00 1a 00 00 00 31 2e 33 2e 34 36 2e 36 37 30 35 38 39 2e 31 37
[125] 2e 31 2e 37
Error in readDICOMFile("OT-MONO2-8-hip.dcm", debug = TRUE) : DICM != DICM
It is apparent that the first 128 bytes contain information. One can now use the parameters skipFirst128=FALSE and DICM=FALSE to start reading information from the beginning of the file
dcm <- readDICOMFile("OT-MONO2-8-hip.dcm", skipFirst128=FALSE, DICM=FALSE)
image(t(dcm$img), col=grey(0:64/64), axes=FALSE, xlab="", ylab="")
3.
Error in parsePixelData(fraw[(132 + dcm$data.seek + 1):fsize], hdr, endian, :
Number of bytes in PixelData not specified
The file CT-MONO2-16-chest.dcm is encoded using JPEG compression. The R package oro.dicom does not support compression.
Error in rawToChar(fraw[129:132]) : embedded nul in string: '\0\0\b'
I have to speculate, since the file is not available for direct interrogation. This problem is related to the check for "DICM" characters as part of the DICOM standard. If it failed, then one can assume the file is not a valid DICM file. I will look into making this error more informative in future versions of oro.dicom.
EDIT: Thank-you for providing a link to the appropriate file. The file is in "ARC-NEMA 2" format. The R package oro.dicom has not been designed to read such a file. I have modified the code to improve the error tracking.

Resources