BLE: Where does the "handle" come from? - bluetooth-lowenergy

Question

I'm trying to learn how to use BLE on a Pi4.
I have installed hcitool and gatttool and have figured out how to use them. I have also set up a virtual device using LightBlue on a Mac.
pi#raspsky:~ $ sudo hcitool -i hci0 lescan
LE Scan ...
98:9E:63:39:8B:ED Blank
98:9E:63:39:8B:ED (unknown)
That "Blank" is the virtual device.
pi#raspsky:~ $ gatttool -b 98:9E:63:39:8B:ED -I
[98:9E:63:39:8B:ED][LE]> connect
Attempting to connect to 98:9E:63:39:8B:ED
Connection successful
[98:9E:63:39:8B:ED][LE]> primary
attr handle: 0x0001, end grp handle: 0x0005 uuid: 00001800-0000-1000-8000-00805f9b34fb
<snip>
attr handle: 0x0028, end grp handle: 0x002b uuid: 00001111-0000-1000-8000-00805f9b34fb
[98:9E:63:39:8B:ED][LE]> char-desc
handle: 0x0001, uuid: 00002800-0000-1000-8000-00805f9b34fb
<snip>
handle: 0x002b, uuid: 00002901-0000-1000-8000-00805f9b34fb
Now, I'd like to be able to read from the virtual device, but I have no idea which "handle" to use. I figured out by trial and error that I need to use 002b:
[98:9E:63:39:8B:ED][LE]> char-read-hnd 002b
Characteristic value/descriptor: 53 6f 6d 65 74 68 69 6e 67
That hex turns out to be the ascii string "Something" which is what I put in the virtual device.
But, where does that handle come from? It seems like I ought to be able to figure it out without having to use trial and error:
UPDATE:
In this example Using bash and gatttool to get readings from Xiaomi Mijia LYWSD03MMC Temperature Humidity sensor, the author shows how to use hcitool to get the mac address of a BLE Device. The article then goes on with an example:
bt=$(timeout 15 gatttool -b A4:C1:38:8C:77:CA --char-write-req --handle='0x0038' --value="0100" --listen)
But the author never explains where the handle (handle='0x0038') came from. I see how you can get a list of handles using gatttool, but the only way I can figure out what data goes with each handle is to try each one until I find the one I want. There must be an easier way.

Answer 1

Since you updated your question, I have another answer for you. But since your topic is not facing any programming issue, you better ask it again at another StackExchange.
the only way I can figure out what data goes with each handle is to try each one until I find the one I want. There must be an easier way.
I guess I have to disappoint you here.
While there are some predefined GATT profiles by Bluetooth SIG, most of the proprietary implementations do not expose any meaningful information about their characteristics but the mandatory UUIDs.
If you are lucky, you might be able to find a Characteristic User Description (Bluetooth Core Spec Vol. 3 Part G 3.3.3.2) or deduce some information from a Characteristic Presentation Format (Bluetooth Core Spec Vol. 3 Part G 3.3.3.5) characteristic descriptor; but since these are optional, I doubt that a proprietary implementation will provide these.
So for a proprietary implementation, you have to reverse engineer that information or try to find it online (often other people already reversed engineered common devices).

Answer 2

hcitool, and gatttool were deprecated by the BlueZ project in 2017. If you are following a tutorial that uses them, there is a chance that it might be out of date. The current BlueZ tool for generic scanning and exploration is bluetoothctl.
With BLE the UUID is the key to identifying the service/characteristic/descriptor that you are interested in.
The 16-bit UUID Numbers Document lists the adopted UUIDs. SIG-adopted attribute types (UUIDs) share all but 16 bits of a special 128-bit base UUID:0000xxxx-0000-1000-8000-00805F9B34FB. The document lists the 16-bit value that goes into that base.
As you have with the virtual device using LightBlue, custom UUIDs can be created. These need to be outside of the 128-bit base reserved for SIG-adopted values.
As SO is about software development, I'll point you at the BlueZ API documentation should you want to do any of this with code. There are also examples in the BlueZ source tree.

Answer 3

The thing you are looking for is generally speaking GATT/ATT itself.
Basically you query a device implementing a GATT server
The best (res)source and reference for anything related to Bluetooth® is IMHO the Bluetooth Core Specification in the version your devices support and you are developing for.
Attribute Protocol (ATT)
Bluetooth Core Specification 5.2 Vol. 3 Part F describes the ATTRIBUTE PROTOCOL (ATT) as follows:
This Part defines the Attribute protocol; a protocol for discovering, reading, and writing attributes on a peer device
The ATT is some sort of "low level" since it describe the actual protocol itself and its messages / Protocol Data Units (PDUs), but it's definitely worth a (periphere) read, especial chapters 3.1 Introduction and 3.2 Basic Concepts.
Generic Attribute Profile (GATT)
Bluetooth Core Specification 5.2 Vol. 3 Part G describes the GENERIC ATTRIBUTE PROFILE (GATT) as follows:
This Part defines the Generic Attribute Profile that describes a service framework using the Attribute protocol for discovering services, and for reading and writing characteristic values on a peer device.
You can think of GATT like a high-level API for issuing ATT queries and commands.
You should definitely give the complete chapter 2.6 GATT Profile Hierarchy a read and then head over to 4 GATT Feature Requirements, especially chapters 4.4 Primary Service Discovery, 4.6 Charateristic discovery and *4.7 Characteristic Descriptor Discovery - regarding your question
But, where does that handle come from?