Unknown CRC Method - hex

I having following modbus polling query but i am unable find the correct crc method,crc position and crc value please guide me to find correct crc method,crc position and crc value for the following query
01 37 37 1D 31 31 31 1D 30 1D 31 03 32 35 30 34 39 04
01 37 37 1D 31 31 31 1D 30 1D 32 03 32 35 30 34 33 04
01 37 37 1D 31 31 31 1D 30 1D 33 03 35 37 38 31 34 04
01 37 37 1D 31 31 31 1D 30 1D 34 03 32 35 30 33 31 04
01 37 37 1D 31 31 31 1D 30 1D 35 03 35 37 37 39 34 04
for the above modbus polling query i am not getting which one is crc value and what type of crc used. How it is came and 77 is id of the device. Please guide me..
I am getting following response from the device for polling
01 37 37 02 33 1D 30 39 33 31 39 30 39 34 38 32 30 30 37 31 31 42 20 30 30 30
30 30 30 30 38 34 30 30 30 30 30 30 30 30 30 30 30 30 30 30 37 38 30 30 30 30
30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 33 35 31 30 30 30 30 30
30 30 30 30 30 30 30 30 30 30 30 30 30 30 03 34 37 33 38 35 04

First of all:
The MODBUS over Serial Line standard is available here. This completely specifies low-level interface to MODBUS.
The MODBUS Application Protocol Specification is available here. Higher-level MODBUS commands are specified here.
Both of these standards are fairly short, easy to read and understand, and contain nice diagrams, so I highly recommend you look at them for answers.
Your data doesn't appear to conform to these standards, which makes it hard to answer your questions!
The serial line standard describes two communication methods, RTU and ASCII, but only the RTU method uses a CRC. So you must be using the RTU communication method since you're asking about CRC.
Assuming the RTU communication method, then the slave address should be the first byte, and the function code should be the second byte. That would mean that the slave address is 01, and the function code is 37. But that's not right, because the Application Protocol Specification page 11 lists all the function codes, and 37 is not a valid function code. Also, the slave address 01 doesn't match the slave address 77 that you claim to be using.
But let's ignore that for the moment. Your specific questions are about CRC.
The RTU communication method is the only one which uses a CRC. The ASCII communication method uses a Longitudinal Redundancy Check (LRC), which is different.
The RTU communication method uses a CRC-16, meaning the CRC has 16 bits = 2 bytes. The details are contained in the serial line standard, and there is even some sample C code for computing the CRC there. There are also multiple implementations in various languages available if you google, but do be careful because there are several different CRC-16 definitions and you need to make sure that what you use matches the MODBUS standard.
The CRC position in a MODBUS frame is at the very end of the frame. The last two bytes (16 bits) are the CRC. The low-order byte of the CRC is appended first, followed by the high-order byte.
The MODBUS queries that you provide do NOT appear to have a CRC appended to them at all. But, as I mentioned earlier, the MODBUS queries that you provide don't even look like they use the RTU communication method, which is the only method that uses a CRC.
Since your data doesn't seem to follow the RTU communication method, perhaps it uses the ASCII communication method, which is described starting on page 16 of the serial line standard. The ASCII communication method always starts with an ASCII : character, which is 3A in hex. The ASCII communication method always ends with ASCII cr and lf characters, which are 0D and 0A in hex. All other characters between these must be 0-9 or A-F characters, which are 30 through 39 and 41 through 46 in hex. Your message does not obey any of these requirements - it doesn't start with 3A, it doesn't end with 0D 0A, and it contains other characters that are not 30 through 39 or 41 through 46.
If we ignore that and assume that your data follows the ASCII communication method, then:
The LRC field would be the last two bytes before the 0D 0A at the end.
Page 18 of the serial line standard describes how LRC is computed. It is very simple:
The LRC is calculated by adding together successive 8–bit bytes of
the message, discarding any carries, and then two’s complementing the
result. It is performed on the bytes of the message, before the
encoding of each byte in the two ASCII characters corresponding to the
hexadecimal representation of each nibble. The computation does not include
the 'colon' character that begins the message, and does not include the CRLF
pair at the end of the message. The resulting LRC is ASCII encoded into two
bytes and placed at the end of the ASCII mode frame before the CRLF.
Page 38 of the serial line standard give example C code for generating the LRC. It is only 9 lines of code.
Your data does not follow either of the communication methods. We would need to know more about where your data came from to figure out what went wrong. Perhaps the data is somehow corrupted by multiple devices, or perhaps the sensor that measures the data has incorrect timing or voltage or something, causing it to mis-understand the values being sent. Perhaps there is some kind of parity mis-match, causing some bytes to be thrown away. But we just can't tell without more information.

Related

Parsing bytes as BCD with Indy C++ Builder

I am trying to parse the length of a message received. The length is in BCD. When I use ReadSmallInt(), I get a reading interpreted as a hex value, not as BCD.
So, if I have a message like this:
00 84 60 00 00 00 19 02 10 70 38 00 00 0E C0 00
00 16 45 93 56 00 01 79 16 62 00 00 00 00 00 00
08 00 00 00 00 02 10 43 02 04 02 35 31 35 31 35
31 35 31 35 31 35 31 53 41 4C 45 35 31 30 30 31
32 33 34 35 36 37 38 31 32 33 34 35 36 37 38 39
30 31 32 33
I am expecting ReadSmallInt() to return 84, but instead it is returning 132, which is correct if you are reading a hex value instead of a BCD one.
According to this answer, ReadSmallInt() reads BCD, as in the examples it gets 11 and 13 (BCD) as lengths instead of 17 and 19 (hex).
I have fixed this with duct tape, but is there a more elegant way?
int calculated_length;
// getting the length in Hexa
calculated_length = AContext->Connection->IOHandler->ReadSmallInt();
// converting from hex binary to hex string
UnicodeString bcdLength = UnicodeString().sprintf(L"%04x", calculated_length);
// converting from hex string to int
calculated_length = bcdLength.ToInt();
ucBuffer.Length = calculated_length -2;
AContext->Connection->IOHandler->ReadBytes(ucBuffer, calculated_length - 2);
According to this answer, ReadSmallInt reads BCD
That is incorrect. You have misinterpreted what that answer is saying. NOTHING in that answer indicates that ReadSmallInt() reads in a Binary Coded Decimal, because it doesn't, as Indy DOES NOT support reading/writing BCDs at all. ReadSmallInt() simply reads in 2 bytes and returns them as-is as a 16-bit decimal integer (swapping the byte order, if needed). So, if you need to read in a BCD instead, you will have to read in the bytes and then parse them yourself. Or find a BCD library to handle it for you.
If you re-read that other question again more carefully, in the 2 examples it gives:
24 24 00 11 12 34 56 FF FF FF FF 50 00 8B 9B 0D 0A
24 24 00 13 12 34 56 FF FF FF FF 90 02 00 0A 8F D4 0D 0A
The 3rd and 4th bytes represent the message lengths (x00 x11 and x00 x13, respectively). As 16-bit values in network byte order, they represent decimal integers 17 and 19, respectively. And if you count the bytes present, you will see those values are the correct byte lengths of those messages. So, there are no BCDs involved here.
That is different than your example. Bytes x00 x84 in network byte order represent decimal integer 132. But your message is 84 bytes in size, not 132 bytes. So clearly the bytes x00 x84 DO NOT represent a 16-bit decimal value, so ReadSmallInt() is the wrong method to use in the first place.
In your "duct tape" code, you are taking the decimal value that ReadSmallInt() returns (132), converting it to a hex string ('0084'), and then parsing that to a decimal value (84). There is no method in Indy that will do that kind of conversion for you.
That "works" in your case, but whether or not that is the correct conversion to perform, I could not say for sure as you have not provided any details about the protocol you are dealing with. But, if you think the bytes represent a BCD then you should interpret the bytes in terms of an actual BCD.
In a packed BCD, a byte can represent a 2-digit number. In this case, byte x84 (10000100b) contains two nibbles 1000b (8) and 0100b (4), thus put together they form decimal 84, which is calculated as follows:
BYTE b = 0x84;
int len = (int((b >> 4) & 0x0F) * 10) + int(b & 0x0F);
Now, how that extends to multiple bytes in a BCD, I'm not sure, as my experience with BCDs is very limited. But, you are going to have to figure that out if you need to handle message lengths greater than 99 bytes, which is the highest decimal that a single BCD byte can represent.

Decoding Thrift Object what are these extra bytes?

I'm working on writing a pure JS thrift decoder that doesn't depend on thrift definitions. I have been following this handy guide which has been my bible for the past few days: https://erikvanoosten.github.io/thrift-missing-specification/
I almost have my parser working, but there is a string type that throws a wrench into the program, and I don't quite understand what it's doing. Here is an excerpt of the hexdump, which I did my best to annotate:
Correctly parsing:
000001a0 0a 32 30 32 31 2d 31 31 2d 32 34 16 02 00 18 07 |.2021-11-24.....|
........................blah blah blah............| | |
Object End-| | |
0x18 & 0xF = 0x8 = Binary-| |
The binary sequence is 0x7 characters long-|
000001b0 53 65 61 74 74 6c 65 18 02 55 53 18 02 55 53 18 |Seattle..US..US.|
S E A T T L E |___| U S |___| U S
Another string, 2 bytes long |------------|
So far so good.
But then I get to this point:
There string I am trying to extract is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4592.0 Safari/537.36 Edg/94.0.975.1" and is 134 bytes long.
000001c0 09 54 61 68 6f 65 2c 20 43 41 12 12 00 00 08 c8 |.Tahoe, CA......|
Object ends here-| | |
0x8 & 0xF = 0x8 = Binary -| |
0xc8 bytes long (200)-|
000001d0 01 86 01 4d 6f 7a 69 6c 6c 61 2f 35 2e 30 20 28 |...Mozilla/5.0 (|
| | | M o z i l l a
???? |--|-134, encoded as var-int
000001e0 4d 61 63 69 6e 74 6f 73 68 3b 20 49 6e 74 65 6c |Macintosh; Intel|
As you can see, I have a byte sequence 0x08 0xC8 0x01 0x86 0x01 which contains the length of the string I'm looking for, is followed by the string I'm looking for but has 3 extra bytes that are unclear in purpose.
The 0x01 is especially confusing as it neither a type identifier, nor seems to have a concrete value.
What am I missing?
Thrift supports pluggable serialization schemes. In tree you have binary, compact and json. Out of tree anything goes. From the looks of it you are trying to decode compact protocol, so I'll answer accordingly.
Everything sent and everything returned in a Thrift RPC call is packaged in a struct. Every field in a struct has a 1 byte type and a 2 byte field ID prefix. In compact protocol field ids, when possible, are delta encoded into the type and all ints are compressed down to just the bits needed to store them (and some flags). Because ints can now take up varying numbers of bytes we need to know when they end. Compact protocol encodes the int bits in 7 bits of a byte and sets the high order bit to 1 if the next byte continues the int. If the high order bit is 0 the int is complete. Thus the int 5 (101) would be encoded in one byte as 0000101. Compact knows this is the end of the int because the high order bit is 0.
In your case, the int 134 (binary 10000110) will need 2 bytes to encode because it is more than 7 bits. The fist 7 bits are stored in byte 1 with the 0x80 bit set to flag "the int continues". The second and final byte encodes the last bit (00000001). What you thought was 134 was just the encoding of the first seven bits. The stray 1 was the final bit of the 134.
I'd recommend you use the in tree source to do any needed protocol encoding/decoding. It's already written and tested: https://github.com/apache/thrift/blob/master/lib/nodejs/lib/thrift/compact_protocol.js
The byte sequence reads as follows
0x08: String type, the next 2 bytes define the elementId
0xC8 0x01: ElementId, encoded in 16 bits
0x86 0x01: String length, encoded as var int
It turns out that if the type identifier does not contain bits defining the elementId, the elementId will be stored in the next 2 bytes.

tcp-check expect binary response in second packet in a row

I am trying to build a TCP checking on my backend servers using HAProxy version 1.5.8.
The behavior should be as follows:
Send binary data to server
Receive ACK as first packet
Receive confirmation data in second packet
So I need to check that after sending binary data I received ACK and after that other binary data in a second packet in a row.
Is it possible to do it with HAProxy.
I am trying to find it in documentation and also trying to create different configurations, unsuccessfully:
option tcp-check
tcp-check connect
tcp-check send-binary 303030303030
tcp-check expect binary 303030303030
Every time I received back from server ACK, connection is terminated by HAProxy with the result that the backend server is down.
EDIT:
I will receive the following:
First packet after sending data
0000 a0 66 10 09 2e 46 9c af ca bb aa 47 08 00 45 00  f...F.¯Ê»ªG..E.
0010 00 28 40 58 40 00 3e 06 d7 04 0a 1e 0b 34 0a 02 .(#X#.>.×....4..
0020 06 20 25 1c d5 80 91 0a f8 87 db 03 25 8f 50 10 . %.Õ...ø.Û.%.P.
0030 01 c9 03 d6 00 00 00 00 00 00 00 00 .É.Ö........
Second packet right after the above
0000 a0 66 10 09 2e 46 9c af ca bb aa 47 08 00 45 00  f...F.¯Ê»ªG..E.
0010 00 39 40 59 40 00 3e 06 d6 f2 0a 1e 0b 34 0a 02 .9#Y#.>.Öò...4..
0020 06 20 25 1c d5 80 91 0a f8 87 db 03 25 8f 50 18 . %.Õ...ø.Û.%.P.
0030 01 c9 2d 2e 00 00 00 0f 30 30 30 30 30 30 42 33 .É-.....000000B3
0040 30 30 43 48 45 43 4b 00CHECK
The first is without any data and I need to check that the second contains 000000.
EDIT2:
PCAP provided:
Normal behavior when communication goes directly from client to server, without HAProxy:
Normal behavior - client to server
Using HAProxy as load balancer, connecting to the same server and checking with the same command, failing to check:
failing check - HAProxy to server
backend configuration:
backend nodes
mode tcp
balance roundrobin
default-server inter 10s fall 3 rise 2
option tcp-check
tcp-check connect
tcp-check send-binary 303030303030423230303035434845434b
tcp-check expect binary 000f30303030303042333030434845434b
server server1 10.30.11.52:9500 check
server server2 10.30.11.52:9501 check
server server3 10.30.11.52:9502 check
Receive ACK as first packet
HA proxy does not work at the raw packet level but at the TCP level. At this level there is no such thing as an ACK as a single packet. There is not even the concept of a packet at this level. Instead there is only the concept of a data stream consisting of the received bytes.
Every time I received back from server ACK, connection is terminated by HAProxy with the result that the backend server is down.
Given that HA proxy does not care about packets with zero payload in the first place it is likely that your "ACK as first packet" is actually some packet which contains an ACK (as almost all TCP packets do) but also contains some payload, but not the one you expect with the "next packet". Since the payload does not match the payload you specify as expected the check fails.
Note that this is only an assumption made based on incomplete information about your "ACK as first packet". To prove the assumption one would actually need to see what is really going on on the wire, for example by having a packet capture.
EDIT#1: after the OP provided a some (undocumented) dump of the packets and some figuring out where the actual IP header in these packets starts (offset 14, i.e. prefixed with layer 2 ethernet header) it is clear that the first packet has no payload which means it gets completely ignored by the check. The second packet then has the following payload of 17 bytes:
0030 00 0f 30 30 30 30 30 30 42 33 ..000000B3
0040 30 30 43 48 45 43 4b 00CHECK
Given that the OP checks for binary 303030303030 but the actual payload is 00 0f 30 30 30 30 30 30 .... the given tcp-check expect ... does not match the actual payload and thus the check fails.
EDIT#2:
After the OP has provided the pcap of a connection without and with haproxy a difference in the behavior of both client/haproxy and server can be seen:
without haproxy:
client sends 2 bytes \x00\x11 to the server followed by 17 bytes \x30\x30....
server replies immediately with 17 bytes \x00\x0f\x30\x30....
with haproxy:
haproxy send 17 bytes \x30\x30... to the server. It does not send the initial 2 bytes \x00\x11 as done by the original server !!!
Server does not reply (except an ACK with no payload). After 6 seconds of inactivity haproxy closes the connection to the server and likely considers the check failed.
In summary: I think the haproxy check fails to send the proper request to the server, i.e. the first 2 bytes are missing. That's why the server will not respond at all and the check will fail after some timeout.

Which standards are SS7 MAP Tags defined in?

Can anyone give me information on which standard contains MAP Tags - sm-RP-UI?
04 1a - sm-RP-UI
24 - TP-RP/UDHI/SRI/MMS/MTI
0b - length
91 26 18 18 55 32 f7 - TP-Originating-Address
00 - TP-PID
00 - TP-DCS
90 40 02 91 61 42 82 - TP-Service-Centre-Time-Stamp
07 - TP-User-Data-Length: (7) depends on Data-Coding-Scheme
ca f0 3a 2c a7 87 01 - TP-User-Data
The details are needed for coding and I'd like to know which standard they are in. I have been looking in GSM 29.002, GSM 23.040, and GSM 24.011 and I haven't found them.
Any help would be greatly appreciated,
Thank you.
The SMTL PDUs are defined in 3GPP TS 23.040 - Technical realization of the Short Message Service (SMS)
More specifically:
04 1a
This is ASN.1 tag a length (OCTET STRING). Since you say this is sm-RP-UI
it would be the SignalInfo ASN.1 type defined in 3GPP TS 29.002
used with labels sm-RP-UI on different MAP operations.
24
First thing to look here are the last two bits (TP-Message-Type-Indicator: 9.2.3.1 of 23.040)
Since you have H'24 -> B'00100100. This is an SMS-DELIVER (SC to MS)
SMS-DELIVER (9.2.2.1) contains
TP-Message-Type-Indicator (TP-MTI on 9.2.3.1) (bit 0-1 --> 00)
TP-More-Messages-To-Send (TP-MMS on 9.2.3.2) (bit 2 --> 1: "No more messages are waiting for the MS in this SC
TP-Status-Report-Indication (TP-SRI on 9.2.3.4) (bit 5 --> 1: "A status report shall be returned to the SME")
TP-User-Data_Header-Indicator (TP-UDHI on 9.2.3.23) (bit 6 -> 0: "The TP-UD field contains only the short message")
TP-Reply-Path (TP-RP on 9.2.3.17) (bit 7 -> 0: "Not set")
0b 91 26 18 18 55 32 f7
TP-Originating-Address (TP-OA 9.2.3.7 - Address fields in 9.1.2.5) that works:
** Address-Length: H'B = D'11 (not this is in semi-octets)
** Type-of-Address: H'91=B'10010001 with Type-of-Nuymber (B'001: International number) and Numbering-Plan (B'0001: ISDN/E.164)
** Address-Value: BCD: 62818155327 (F is filler)
00
TP-Protocol-Identifier (TP-PID 9.2.3.9)
00
TP-Data-Coding-Scheme (TP-DCS 9.2.3.10)
90 40 02 91 61 42 82
TP-Service-Centre-Time-Stamp (TP-SCTS 9.2.3.11) 2009/04/20 19:16:24
07
TP-User-Data-Length (TP-UDL 9.2.3.16)
ca f0 3a 2c a7 87 01
* TP-User-Data (TP-UD 9.2.3.24)

Can I get the byte representation of an R float?

I'm trying to read in a complicated data file that has floating point values. Some C code has been supplied that handles this format (Met Office PP file) and it does a lot of bit twiddling and swapping. And it doesn't work. It gets a lot right, like the size of the data, but the numerical values in the returned matrix are nonsensical, have NaNs and values like 1e38 and -1e38 liberally sprinkled.
However, I have a binary exe ("convsh") that can convert these to netCDF, and the netCDFs look fine - nice swirly maps of wind speed.
What I'm thinking is that the bytes of the PP file are being read in in the wrong order. If I could compare the bytes of the floats returned correctly in the netCDF data with the bytes in the floats returned wrongly from the C code, then I might figure out the correct swappage.
So is there a plain R function to dump the four (or eight?) bytes of a floating point number? Something like:
> as.bytes(pi)
[1] 23 54 163 73 99 00 12 45 # made up values
searches for "bytes" and "float" and "binary" haven't helped.
Its trivial in C, I could probably have written it in the time it took me to write this...
rdyncall might give you what you're looking for:
library(rdyncall)
as.floatraw(pi)
# [1] db 0f 49 40
# attr(,"class")
# [1] "floatraw"
Or maybe writeBin(pi, raw(8))?
Yes, that must exist in the serialization code because R merrily sends stuff across the wire, taking care of endianness too. Did you look at eg Rserve using it, or how digest passes the char representation to chosen hash functions?
After a quick glance at digest.R:
R> serialize(pi, connection=NULL, ascii=TRUE)
[1] 41 0a 32 0a 31 33 34 39 31 34 0a 31 33 31 38 34 30 0a
[19] 31 34 0a 31 0a 33 2e 31 34 31 35 39 32 36 35 33 35 38
[37] 39 37 39 33 0a
and
R> serialize(pi, connection=NULL, ascii=FALSE)
[1] 58 0a 00 00 00 02 00 02 0f 02 00 02 03 00 00 00 00 0e
[19] 00 00 00 01 40 09 21 fb 54 44 2d 18
R>
That might get you going.
Come to think about it, this includes header meta-data.
The package mcga (machine-coded genetic algorithms) includes some functions for bytes-to-double and doubles-to-byte conversions. For handling the bytes of pi, you can use DoubleToBytes like:
> DoubleToBytes(pi)
1 24 45 68 84 251 33 9 64
For converting bytes to double again, BytesToDouble() can be used instead:
> BytesToDouble(c(24,45,68,84,251,33,9,64))
1 3.141593
Links:
CRAN page of mcga

Resources