I'm trying to read in a complicated data file that has floating point values. Some C code has been supplied that handles this format (Met Office PP file) and it does a lot of bit twiddling and swapping. And it doesn't work. It gets a lot right, like the size of the data, but the numerical values in the returned matrix are nonsensical, have NaNs and values like 1e38 and -1e38 liberally sprinkled.
However, I have a binary exe ("convsh") that can convert these to netCDF, and the netCDFs look fine - nice swirly maps of wind speed.
What I'm thinking is that the bytes of the PP file are being read in in the wrong order. If I could compare the bytes of the floats returned correctly in the netCDF data with the bytes in the floats returned wrongly from the C code, then I might figure out the correct swappage.
So is there a plain R function to dump the four (or eight?) bytes of a floating point number? Something like:
> as.bytes(pi)
[1] 23 54 163 73 99 00 12 45 # made up values
searches for "bytes" and "float" and "binary" haven't helped.
Its trivial in C, I could probably have written it in the time it took me to write this...
rdyncall might give you what you're looking for:
library(rdyncall)
as.floatraw(pi)
# [1] db 0f 49 40
# attr(,"class")
# [1] "floatraw"
Or maybe writeBin(pi, raw(8))?
Yes, that must exist in the serialization code because R merrily sends stuff across the wire, taking care of endianness too. Did you look at eg Rserve using it, or how digest passes the char representation to chosen hash functions?
After a quick glance at digest.R:
R> serialize(pi, connection=NULL, ascii=TRUE)
[1] 41 0a 32 0a 31 33 34 39 31 34 0a 31 33 31 38 34 30 0a
[19] 31 34 0a 31 0a 33 2e 31 34 31 35 39 32 36 35 33 35 38
[37] 39 37 39 33 0a
and
R> serialize(pi, connection=NULL, ascii=FALSE)
[1] 58 0a 00 00 00 02 00 02 0f 02 00 02 03 00 00 00 00 0e
[19] 00 00 00 01 40 09 21 fb 54 44 2d 18
R>
That might get you going.
Come to think about it, this includes header meta-data.
The package mcga (machine-coded genetic algorithms) includes some functions for bytes-to-double and doubles-to-byte conversions. For handling the bytes of pi, you can use DoubleToBytes like:
> DoubleToBytes(pi)
1 24 45 68 84 251 33 9 64
For converting bytes to double again, BytesToDouble() can be used instead:
> BytesToDouble(c(24,45,68,84,251,33,9,64))
1 3.141593
Links:
CRAN page of mcga
Related
I am trying to parse the length of a message received. The length is in BCD. When I use ReadSmallInt(), I get a reading interpreted as a hex value, not as BCD.
So, if I have a message like this:
00 84 60 00 00 00 19 02 10 70 38 00 00 0E C0 00
00 16 45 93 56 00 01 79 16 62 00 00 00 00 00 00
08 00 00 00 00 02 10 43 02 04 02 35 31 35 31 35
31 35 31 35 31 35 31 53 41 4C 45 35 31 30 30 31
32 33 34 35 36 37 38 31 32 33 34 35 36 37 38 39
30 31 32 33
I am expecting ReadSmallInt() to return 84, but instead it is returning 132, which is correct if you are reading a hex value instead of a BCD one.
According to this answer, ReadSmallInt() reads BCD, as in the examples it gets 11 and 13 (BCD) as lengths instead of 17 and 19 (hex).
I have fixed this with duct tape, but is there a more elegant way?
int calculated_length;
// getting the length in Hexa
calculated_length = AContext->Connection->IOHandler->ReadSmallInt();
// converting from hex binary to hex string
UnicodeString bcdLength = UnicodeString().sprintf(L"%04x", calculated_length);
// converting from hex string to int
calculated_length = bcdLength.ToInt();
ucBuffer.Length = calculated_length -2;
AContext->Connection->IOHandler->ReadBytes(ucBuffer, calculated_length - 2);
According to this answer, ReadSmallInt reads BCD
That is incorrect. You have misinterpreted what that answer is saying. NOTHING in that answer indicates that ReadSmallInt() reads in a Binary Coded Decimal, because it doesn't, as Indy DOES NOT support reading/writing BCDs at all. ReadSmallInt() simply reads in 2 bytes and returns them as-is as a 16-bit decimal integer (swapping the byte order, if needed). So, if you need to read in a BCD instead, you will have to read in the bytes and then parse them yourself. Or find a BCD library to handle it for you.
If you re-read that other question again more carefully, in the 2 examples it gives:
24 24 00 11 12 34 56 FF FF FF FF 50 00 8B 9B 0D 0A
24 24 00 13 12 34 56 FF FF FF FF 90 02 00 0A 8F D4 0D 0A
The 3rd and 4th bytes represent the message lengths (x00 x11 and x00 x13, respectively). As 16-bit values in network byte order, they represent decimal integers 17 and 19, respectively. And if you count the bytes present, you will see those values are the correct byte lengths of those messages. So, there are no BCDs involved here.
That is different than your example. Bytes x00 x84 in network byte order represent decimal integer 132. But your message is 84 bytes in size, not 132 bytes. So clearly the bytes x00 x84 DO NOT represent a 16-bit decimal value, so ReadSmallInt() is the wrong method to use in the first place.
In your "duct tape" code, you are taking the decimal value that ReadSmallInt() returns (132), converting it to a hex string ('0084'), and then parsing that to a decimal value (84). There is no method in Indy that will do that kind of conversion for you.
That "works" in your case, but whether or not that is the correct conversion to perform, I could not say for sure as you have not provided any details about the protocol you are dealing with. But, if you think the bytes represent a BCD then you should interpret the bytes in terms of an actual BCD.
In a packed BCD, a byte can represent a 2-digit number. In this case, byte x84 (10000100b) contains two nibbles 1000b (8) and 0100b (4), thus put together they form decimal 84, which is calculated as follows:
BYTE b = 0x84;
int len = (int((b >> 4) & 0x0F) * 10) + int(b & 0x0F);
Now, how that extends to multiple bytes in a BCD, I'm not sure, as my experience with BCDs is very limited. But, you are going to have to figure that out if you need to handle message lengths greater than 99 bytes, which is the highest decimal that a single BCD byte can represent.
I having following modbus polling query but i am unable find the correct crc method,crc position and crc value please guide me to find correct crc method,crc position and crc value for the following query
01 37 37 1D 31 31 31 1D 30 1D 31 03 32 35 30 34 39 04
01 37 37 1D 31 31 31 1D 30 1D 32 03 32 35 30 34 33 04
01 37 37 1D 31 31 31 1D 30 1D 33 03 35 37 38 31 34 04
01 37 37 1D 31 31 31 1D 30 1D 34 03 32 35 30 33 31 04
01 37 37 1D 31 31 31 1D 30 1D 35 03 35 37 37 39 34 04
for the above modbus polling query i am not getting which one is crc value and what type of crc used. How it is came and 77 is id of the device. Please guide me..
I am getting following response from the device for polling
01 37 37 02 33 1D 30 39 33 31 39 30 39 34 38 32 30 30 37 31 31 42 20 30 30 30
30 30 30 30 38 34 30 30 30 30 30 30 30 30 30 30 30 30 30 30 37 38 30 30 30 30
30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 33 35 31 30 30 30 30 30
30 30 30 30 30 30 30 30 30 30 30 30 30 30 03 34 37 33 38 35 04
First of all:
The MODBUS over Serial Line standard is available here. This completely specifies low-level interface to MODBUS.
The MODBUS Application Protocol Specification is available here. Higher-level MODBUS commands are specified here.
Both of these standards are fairly short, easy to read and understand, and contain nice diagrams, so I highly recommend you look at them for answers.
Your data doesn't appear to conform to these standards, which makes it hard to answer your questions!
The serial line standard describes two communication methods, RTU and ASCII, but only the RTU method uses a CRC. So you must be using the RTU communication method since you're asking about CRC.
Assuming the RTU communication method, then the slave address should be the first byte, and the function code should be the second byte. That would mean that the slave address is 01, and the function code is 37. But that's not right, because the Application Protocol Specification page 11 lists all the function codes, and 37 is not a valid function code. Also, the slave address 01 doesn't match the slave address 77 that you claim to be using.
But let's ignore that for the moment. Your specific questions are about CRC.
The RTU communication method is the only one which uses a CRC. The ASCII communication method uses a Longitudinal Redundancy Check (LRC), which is different.
The RTU communication method uses a CRC-16, meaning the CRC has 16 bits = 2 bytes. The details are contained in the serial line standard, and there is even some sample C code for computing the CRC there. There are also multiple implementations in various languages available if you google, but do be careful because there are several different CRC-16 definitions and you need to make sure that what you use matches the MODBUS standard.
The CRC position in a MODBUS frame is at the very end of the frame. The last two bytes (16 bits) are the CRC. The low-order byte of the CRC is appended first, followed by the high-order byte.
The MODBUS queries that you provide do NOT appear to have a CRC appended to them at all. But, as I mentioned earlier, the MODBUS queries that you provide don't even look like they use the RTU communication method, which is the only method that uses a CRC.
Since your data doesn't seem to follow the RTU communication method, perhaps it uses the ASCII communication method, which is described starting on page 16 of the serial line standard. The ASCII communication method always starts with an ASCII : character, which is 3A in hex. The ASCII communication method always ends with ASCII cr and lf characters, which are 0D and 0A in hex. All other characters between these must be 0-9 or A-F characters, which are 30 through 39 and 41 through 46 in hex. Your message does not obey any of these requirements - it doesn't start with 3A, it doesn't end with 0D 0A, and it contains other characters that are not 30 through 39 or 41 through 46.
If we ignore that and assume that your data follows the ASCII communication method, then:
The LRC field would be the last two bytes before the 0D 0A at the end.
Page 18 of the serial line standard describes how LRC is computed. It is very simple:
The LRC is calculated by adding together successive 8–bit bytes of
the message, discarding any carries, and then two’s complementing the
result. It is performed on the bytes of the message, before the
encoding of each byte in the two ASCII characters corresponding to the
hexadecimal representation of each nibble. The computation does not include
the 'colon' character that begins the message, and does not include the CRLF
pair at the end of the message. The resulting LRC is ASCII encoded into two
bytes and placed at the end of the ASCII mode frame before the CRLF.
Page 38 of the serial line standard give example C code for generating the LRC. It is only 9 lines of code.
Your data does not follow either of the communication methods. We would need to know more about where your data came from to figure out what went wrong. Perhaps the data is somehow corrupted by multiple devices, or perhaps the sensor that measures the data has incorrect timing or voltage or something, causing it to mis-understand the values being sent. Perhaps there is some kind of parity mis-match, causing some bytes to be thrown away. But we just can't tell without more information.
I am trying to play a WAV file in the browser. This particular WAV file plays just fine in all browsers except IE and Edge. IE is to be expected -- it doesn't support WAV. However, it looks like it should work in Edge. I've studied the header of the file, and it looks like it is well-formed to me.
This error is printed to the console:
WEBAUDIO17014: Decoding error: The stream provided is corrupt or unsupported.
Here is the header:
52 49 46 46 : "RIFF"
24 30 0C 00 : file size (798,756 bytes)
57 41 56 45 : "WAVE"
66 6D 74 20 : "fmt "
10 00 00 00 : length of format (16 bytes)
01 00 : type of format
01 00 : Number of channels (1)
22 56 00 00 : Sample rate (22050)
88 58 01 00 : (Sample Rate * BitsPerSample * Channels) / 8.
02 00 : (BitsPerSample * Channels) / 8.1 - 8 bit mono2 - 8 bit stereo/16 bit mono4 - 16 bit stereo
10 00 : bits per sample (16)
64 61 74 61 : "data"
00 30 0C 00 : size of data (798,720 bytes)
So, the two unusual things I see are:
1) It uses a sample rate of 22050. Is that supported in Edge?
2) It is a mono file. Is that supported in Edge?
I tried to look up WEBAUDIO17014 but I didn't get any relevant answers. Is there a way to get a more specific error message about what is going wrong?
Edit:
By request, here is the HTML of the page:
<audio src="/path/to/file.wav" preload></audio>
I am trying to open dicom files in R using following code:
library(oro.dicom)
dcmobject <- readDICOMFile(filename)
Some files open properly and I can display them. However, some files give errors of different types:
First error: For some, I get the error:
Error in file(con, "rb") : cannot open the connection
Second error: In others, I get following error with dicom file: http://www.barre.nom.fr/medical/samples/files/OT-MONO2-8-hip.gz :
Error in readDICOMFile(filename) : DICM != DICM
Third error: This file gives following error: http://www.barre.nom.fr/medical/samples/files/CT-MONO2-16-chest.gz
Error in parsePixelData(fraw[(132 + dcm$data.seek + 1):fsize], hdr, endian, :
Number of bytes in PixelData not specified
Fourth error: One dicom file gives following error:
Error in rawToChar(fraw[129:132]) : embedded nul in string: '\0\0\b'
How can I get rid of these errors and display these images in R?
EDIT:
This sample file gives the error 'embed nul in string...':
http://www.barre.nom.fr/medical/samples/files/CT-MONO2-12-lomb-an2.gz
> jj = readDICOMFile( "CT-MONO2-12-lomb-an2.dcm" )
Error in rawToChar(fraw[129:132]) : embedded nul in string: '3\0\020'
There are four different errors highlighted in this ticket:
Error in file(con, "rb") : cannot open the connection
This is not a problem with oro.dicom, it is simply the fact that the file path and/or name has been mis-specified.
Error in readDICOMFile(filename) : DICM != DICM
The file is not a valid DICOM file. That is, section 7.1 in Part 10 of the DICOM Standard (available at http://dicom.nema.org) specifies that there should be (a) the File Preample of length 128 bytes and (b) the four-byte DICOM Prefix "DICM" at the beginning of a DICOM file. The file OT-MONO2-8-hip does not follow this standard. One can investigate this problem further using the debug=TRUE input parameter
> dcm <- readDICOMFile("OT-MONO2-8-hip.dcm", debug=TRUE)
# First 128 bytes of DICOM header =
[1] 08 00 00 00 04 00 00 00 b0 00 00 00 08 00 08 00 2e 00 00 00 4f 52 49 47 49 4e 41 4c 5c 53 45
[32] 43 4f 4e 44 41 52 59 5c 4f 54 48 45 52 5c 41 52 43 5c 44 49 43 4f 4d 5c 56 41 4c 49 44 41 54
[63] 49 4f 4e 20 08 00 16 00 1a 00 00 00 31 2e 32 2e 38 34 30 2e 31 30 30 30 38 2e 35 2e 31 2e 34
[94] 2e 31 2e 31 2e 37 00 08 00 18 00 1a 00 00 00 31 2e 33 2e 34 36 2e 36 37 30 35 38 39 2e 31 37
[125] 2e 31 2e 37
Error in readDICOMFile("OT-MONO2-8-hip.dcm", debug = TRUE) : DICM != DICM
It is apparent that the first 128 bytes contain information. One can now use the parameters skipFirst128=FALSE and DICM=FALSE to start reading information from the beginning of the file
dcm <- readDICOMFile("OT-MONO2-8-hip.dcm", skipFirst128=FALSE, DICM=FALSE)
image(t(dcm$img), col=grey(0:64/64), axes=FALSE, xlab="", ylab="")
3.
Error in parsePixelData(fraw[(132 + dcm$data.seek + 1):fsize], hdr, endian, :
Number of bytes in PixelData not specified
The file CT-MONO2-16-chest.dcm is encoded using JPEG compression. The R package oro.dicom does not support compression.
Error in rawToChar(fraw[129:132]) : embedded nul in string: '\0\0\b'
I have to speculate, since the file is not available for direct interrogation. This problem is related to the check for "DICM" characters as part of the DICOM standard. If it failed, then one can assume the file is not a valid DICM file. I will look into making this error more informative in future versions of oro.dicom.
EDIT: Thank-you for providing a link to the appropriate file. The file is in "ARC-NEMA 2" format. The R package oro.dicom has not been designed to read such a file. I have modified the code to improve the error tracking.
Can anyone give me information on which standard contains MAP Tags - sm-RP-UI?
04 1a - sm-RP-UI
24 - TP-RP/UDHI/SRI/MMS/MTI
0b - length
91 26 18 18 55 32 f7 - TP-Originating-Address
00 - TP-PID
00 - TP-DCS
90 40 02 91 61 42 82 - TP-Service-Centre-Time-Stamp
07 - TP-User-Data-Length: (7) depends on Data-Coding-Scheme
ca f0 3a 2c a7 87 01 - TP-User-Data
The details are needed for coding and I'd like to know which standard they are in. I have been looking in GSM 29.002, GSM 23.040, and GSM 24.011 and I haven't found them.
Any help would be greatly appreciated,
Thank you.
The SMTL PDUs are defined in 3GPP TS 23.040 - Technical realization of the Short Message Service (SMS)
More specifically:
04 1a
This is ASN.1 tag a length (OCTET STRING). Since you say this is sm-RP-UI
it would be the SignalInfo ASN.1 type defined in 3GPP TS 29.002
used with labels sm-RP-UI on different MAP operations.
24
First thing to look here are the last two bits (TP-Message-Type-Indicator: 9.2.3.1 of 23.040)
Since you have H'24 -> B'00100100. This is an SMS-DELIVER (SC to MS)
SMS-DELIVER (9.2.2.1) contains
TP-Message-Type-Indicator (TP-MTI on 9.2.3.1) (bit 0-1 --> 00)
TP-More-Messages-To-Send (TP-MMS on 9.2.3.2) (bit 2 --> 1: "No more messages are waiting for the MS in this SC
TP-Status-Report-Indication (TP-SRI on 9.2.3.4) (bit 5 --> 1: "A status report shall be returned to the SME")
TP-User-Data_Header-Indicator (TP-UDHI on 9.2.3.23) (bit 6 -> 0: "The TP-UD field contains only the short message")
TP-Reply-Path (TP-RP on 9.2.3.17) (bit 7 -> 0: "Not set")
0b 91 26 18 18 55 32 f7
TP-Originating-Address (TP-OA 9.2.3.7 - Address fields in 9.1.2.5) that works:
** Address-Length: H'B = D'11 (not this is in semi-octets)
** Type-of-Address: H'91=B'10010001 with Type-of-Nuymber (B'001: International number) and Numbering-Plan (B'0001: ISDN/E.164)
** Address-Value: BCD: 62818155327 (F is filler)
00
TP-Protocol-Identifier (TP-PID 9.2.3.9)
00
TP-Data-Coding-Scheme (TP-DCS 9.2.3.10)
90 40 02 91 61 42 82
TP-Service-Centre-Time-Stamp (TP-SCTS 9.2.3.11) 2009/04/20 19:16:24
07
TP-User-Data-Length (TP-UDL 9.2.3.16)
ca f0 3a 2c a7 87 01
* TP-User-Data (TP-UD 9.2.3.24)