Locating data (like pressure etc) in a raw data stream - serial-port
I am a newbie to Serial Port Analysis and I would appreciate some help on this. my specific question is...
If I have raw data from a serial port analyzer program, how will I locate measures like temperature, pressure, energy etc?
What should I look for in the raw data that will help me identify these units of measure?
What is the best way the extract relevant data from this raw data?
I would be very grateful if you can provide me any help with respect to this. I am unable to figure out how to do this.
Thanks a lot.
The best way that I know of to do this is to find the "reset" identifier, also called the "End of Stream" identifier or sequence. I am assuming that the data is a continuous flow not a one-time transmission.
If the data is continuously cycling, you need to find where the transmit begins (or ends) and then start metering your capture from there. Most devices will have an associated manual or documentation that give you the end sequence (or optionally the start sequence) and then the method by which they identifier their data.
For instance, the device may end a message by sending 4 all zero bytes in a row, then begin again by sending one byte that identifies the sensor, and another two bytes with the data, followed by the next sensor etc.
You would then watch the stream for 4 zero byte entries, and then start capturing 3 bytes at a time, one for the sensor and two for the data, until you saw 4 zero byte entries in a row again.
Related
How to reverse engineer buffer data received from a serial port
I am trying to decode some buffer data that I have received from a serial port. To me the data seems to make no sense - I am not even sure if I am splitting the messages up correctly. The data comes from a concrete crusher and while the concrete is being crushed we get an almost continuous stream of data. I get about 10 "messages" a second (but this might be multiple messages included in each message) and I am splitting them up by waiting 50 ms after each message. The data looks like this: [0,0,0,224,0,224,0,0,224,0,0,224,0,0,0,0,0,0,0,224,0,0,0,0,224,0,0,0,224,0,224,0,0,224,0,0,0,224,0,0,0,0,0,0,0,224,0,0,0,224,0,0,224,0,0,224,0,224,0,0,224,0,0,0,0,0,0,0,0,224,224,0] [0,0,0,224,0,224,0,0,224,0,0,224,0,0,0,0,0,0,0,224,0,0,0,0,0,0,0,224,0,224,0,0,0,0,0,0,0,0,0,0,0,0,0,224,0,0,0,224,0,0,224,0,0,224,0,0,0,0,224,0,0,0,0,0,0,0,0,224,224,0] [0,0,0,224,0,224,0,0,224,0,0,224,0,0,0,0,0,0,0,224,0,224,0,0,224,0,0,0,224,0,224,224,224,0,0,0,224,0,0,0,0,0,0,0,0,224,0,0,0,224,0,224,224,224,0,0,224,0,0,0,224,0,0,0,224,0,0,0,0,0,224,224,0] as you can see there are no values at all other than 0 and 224... The last message is: [0,0,0,0,0,0,224,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,224,0,0,224,0,0,0,0,0,0,0,0,0,0,0,0,0,0,224,0,0,0,0,224,224,0,0,224,0,0,224,0,0,0,224,0,0,0,0,0,0,0,0,0,0,0,224,0,0,0,0,0,224,0,224,224,0,0,224,0,0,0,224,224,0] the value displayed on the machine was 427.681kN but I can't see any way that this data can produce that. Each message ends with 224,224,0 so I am wondering if that is the split sequence? I am getting this data with node-red and this is the format that I can copy it from the debug panel in. I am very lost so any guidance or directions that I can look in would be much appreciated.
DNP3 protocol - How is the data stored?
I am new to DNP3 protocol and I have a question. I understand that the data is stored in arrays But, I did not understand if the array could be noncontinuous? In addition, Is there any beginners source information for DNP3 protocol? (I have tried to read the DNP3 specification but it was unclear to me) I would appreciate your answer!
Yes, data indexes may be non-contiguous.
To achieve "more efficient" transmission of data, section 5.1.2 of the IEEE Standard for Electric Power Systems Communications— Distributed Network Protocol (DNP3) states that "gaps in the point index range are permissible but should be avoided wherever possible."
The DNP3 standard does not specify how data is stored, but rather how it is transmitted. The indexes are part of an addressing scheme used to identify individual pieces of data in a device. A given piece of data, or point, is identified by its Group number and an Index. For example, "Group 30 : Index 9" is the 10th readable, analog value ("10th" because the lists are zero-based).
Another way to state the answer is that point addresses (meaning indexes within a Group) are not required to be contiguous.
Note that even if points in a device are indexed contiguously, the device could return data with non-contiguous indexes in a single transmission packet. For example, a packet of data from a device might contain the 2nd, 5th, and 12th readable, analog input.
I don't have any specific recommendation for beginners information.
Can't get temperature reading out of ble beacon .. at my wits end now. This needs a super-hero I guess
I have a task where I need to read 2 parameters from a BLE Beacon. The documentation was seriously lacking and after a fair amount of effort, I managed to get some basic information about reading the data from the BLE Beacon. The parameters to read are 1) Battery Voltage of the sensor 2) Temperature the beacon has a built in temperature sensor. I think I have tried almost every popular Python BLE library out there but I just can't seem to get the temperature reading out of the beacon. "I think" I am able to read the voltage. The reason why I said "I think" is because the value seems to match what was provided in the minimal document. And also when I put the beacon into the charger, I can see the value go up - an indication that it is the voltage reading. As I could not read the temperature ( because the UUIDs that are mentioned in the document, the value doesn't seem to change ). I have tried enabling the sensor in every possible way and method described - by writing 01:00 etc. I spent a fair amount of time to reverse engineer the thing. I ran a packet sniffer and managed to capture the data that was being transferred between the beacon and the mobile app ( They have a mobile app ). But then again I am not able to figure out how the temperature readings are being communicated between the beacon and the app. Let me break the whole stuff in smaller blocks. Hardware: BLE beacon from which voltage and temperature can be read. The temperature sensor is built into the beacon. And the beacon itself is from Texas Instruments but the temperature, voltage sensing part is done by a third party. They provided us with some minimal information and it was difficult to make sense of some of the sentences as they have trouble communicating in English. The sequence to get the data goes like this Scan for beacons When the beacon is found then connect to it Enable notification Set notification interval Get the voltage and temperature reading. I have been able to do the first 4 real fast, and "half" of No. 5, i.e getting the voltage part. When I say real fast I mean I got that stuff with nearly no documentation available at that time. As per the info that I have the data resides in these characteristics/UUIDs. Also please note that the UUID are not standard 128 bit and this caused me issues when using certain libraries. But after some tries I got to read/write to them using handles etc. The handles and other stuff I printed are ones that I read using PYGATT (A Python wrapper for gatttool). The UUIDs are marked as 1st, 2nd, 3rd and 4th parameters and it has the following to say about the parameters - A: 1 byte (2nd Param) - B: Maj + Min values, 4 bytes (4th Param) - C: 4 bytes (3rd Param) - D: Enable/disable notification ( I have been able to turn this on ) - E: Set notification interval ( I have been able to set this and can notice the change in notification interval ) This is minimal so as to not have a large file. All it does is this - the mobile app connects to the beacon, then the notifications start and the temperate readings are retrieved by the mobile app. Like I had mentioned, I don't seem to have problem reading the voltage, it's only the temperature that I am getting stuck at. I have been at it for a week now. I think I have tried nearly everything that I could think of. I even enumerated all the writable characteristics and tried writing numbers like 1 ( enables the sensor? ). I could have offered a bounty for this straight away if it were possible. I rarely get stuck for so long with a problem. This is driving me a little crazy. I am getting close to my wits end - I guess it's time for a super hero - anyone out there? :) I can provide for every bit of information needed if someone could indicate what is wrong. I even wrote a cordova app ... and tried a bunch of stuff from my Android phone. I can connect ... write to characteristics, read stuff etc but temperature ready, nah!!! It just won't budge. All I get is the same set of values ( I used a JSON.stringify to display A, B and C). I can bother about the byte order later. I guess that is a smaller problem. The communication between the beacon and a third party mobile app is fine, it is able to read the temperature info just fine. I have been looking at wireshark data and I am fairly sure that the temperature data is being communicated at this stage. But then when I decode the "value", it looks like it's the voltage. It mentions l2cap but I am not sure how that is being used here to send the temperature readings ( if it is using that in the first place ). Update: Wrote to every writable characteristics. Wrote values like 1, 0100, 2, 7 on every writable characteristics. At the same time I was reading every readable characteristic ( in a loop ) and doing a comparison (just true/false) with the previous set of values. This seemed like a quick and easier way to know if something changed. Didn't want to take chances with converting the hex to a float. I can figure out the byte order later. From the sniffed data (wireshark) I can only see 3 writes happening on the beacon.
I am not fully sure, even after a long discussion, but it seems that the four bytes of the notification are used for the voltage as well as the temperature, since the temperature can most probably be derived from the voltage. From the values it seems that those four bytes represent the voltage in float (if you ignore the absurd factor of 10^-38 that comes in because only 4 bytes instead of 8 bytes are used). Since typically the temperature T is derived from a resistivity measurement, where the resistivity R is proportional to the voltage U (if the current is constant), you can in principle calculate the temperature T from the voltage U. The problem is that T(R) is relatively linear, but not perfectly (in contrast to U(R) which is assumed to be U=RI). So you may need to plot the values for T(U) to find out the curve that they are using. To add to the confusion, I got the best results when only using the first five bits of the third byte and the eight bits of the fourth byte. I am not aware why this is the case, and it might point to some trouble still. The best option is to ask for their function T(U) that they are using. If they can and will provide it for you...
Best approach for transfering large data chunks over BLE
I'm new to BLE and hope you will be able to point me towards the right implementation approach. I'm working on an application in which the peripheral (battery operated) device continuously aggregate sensor readings. On the mobile side application there will be a "sync" button, upon button press, I would like to transfer all the sensor readings that were accumulated in the peripheral to the mobile application. The maximal duration between sync's can be several days, hence, the accumulated data can reach a size of 20Kbytes. Now, I'm wondering what will be the best approach to perform the data transfer from the peripheral to the central application. I thought about creating an array of characteristics where each characteristic will contain a fixed amount of samples (e.g. representing 1hour of readings). Then, upon sync, I will: Read the characteristics count (how many 1hours cells). Then read the characteristics (1hour cells) one by one. However, I have no idea if this is a valid approach ? I'm not sure if this is the most "power efficient" way that I can use. I'm not sure if Characteristic READ is the way to go, or maybe I need to use indication instead. Any help here will be highly appreciated :) Thanks in advance, Moti.
I would simply use notifications. Use one characteristic which you write something to in order to trigger the transfer start. Then have another characteristic which you simply stream data over by sending 20 bytes at a time. Most SDKs for BLE system-on-a-chips have some way to control the flow of data so you don't send too fast. Normally by having a callback triggered when it is ready to take the next notification. In order to know the size of the data being sent, you can for example let the first notification contain the size, and rest of them the data. This is the most time and power efficient way since there can be sent many notifications per connection interval, compared if you do a lot of reads instead which normally requires two round trips each. Don't use indications since they also require basically two round trips per indication. They're also quite useless anyway. You could possibly increase the speed also by some % by exchanging a larger MTU (which leads to lower L2CAP/ATT headers overhead).
How to determine the length of an Ethernet II frame?
The Ethernet II frame format does not contain a length field, and I'd like to understand how the end of a frame can be detected without it. Unfortunately, I have no idea of physics, but the following sounds reasonable to me: we assume that Layer 1 (Physical Layer) provides us with a way of transmitting raw bits in such a way that it is possible to distinguish between the situation where bits are being sent and the situation where nothing is sent (if digital data was coded into analog signals via phase modulation, this would be true, for example - but I don't know if this is really what's done). In this case, an ethernet card could simply wait until a certain time intervall occurs where no more bits are being transmitted, and then decide that the frame transmission has to be finished. Is this really what's happening? If yes: where can I find these things, and what are common values for the length of "certain time intervall"? Why does IEEE 802.3 have a length field? If not: how is it done instead? Thank you for your help! Hanno
Your assumption is right. The length field inside the frame is not needed for layer1. Layer1 uses other means to detect the end of a frame which vary depending on the type of physical layer. with 10Base-T a frame is followed by a TP_IDL waveform. The lack of further Manchester coded data bits can be detected. with 100Base-T a frame is ended with an End of Stream Delimiter bit pattern that may not occur in payload data (because of its 4B/5B encoding). A rough description you can find e.g. here: http://ww1.microchip.com/downloads/en/AppNotes/01120a.pdf "Ethernet Theory of Operation"