Is there any languages for querying CBOR? - jsonpath

I'm looking for a languages for querying CBOR, like JsonPath or jq but for CBOR binary format. I don't want to convert from CBOR to JSON because some CBOR type is not existed in JSON, and performance issue.

The C++ library jsoncons allows you to query CBOR with JSONPath, for example,
#include <jsoncons/json.hpp>
#include <jsoncons_ext/cbor/cbor.hpp>
#include <jsoncons_ext/jsonpath/json_query.hpp>
#include <iomanip>
using namespace jsoncons; // For convenience
int main()
{
std::vector<uint8_t> v = {0x85,0xfa,0x40,0x0,0x0,0x0,0xfb,0x3f,0x12,0x9c,0xba,0xb6,0x49,0xd3,0x89,0xc3,0x49,0x1,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0xc4,0x82,0x38,0x1c,0xc2,0x4d,0x1,0x8e,0xe9,0xf,0xf6,0xc3,0x73,0xe0,0xee,0x4e,0x3f,0xa,0xd2,0xc5,0x82,0x20,0x3};
/*
85 -- Array of length 5
fa -- float
40a00000 -- 5.0
fb -- double
3f129cbab649d389 -- 0.000071
c3 -- Tag 3 (negative bignum)
49 -- Byte string value of length 9
010000000000000000
c4 -- Tag 4 (decimal fraction)
82 -- Array of length 2
38 -- Negative integer of length 1
1c -- -29
c2 -- Tag 2 (positive bignum)
4d -- Byte string value of length 13
018ee90ff6c373e0ee4e3f0ad2
c5 -- Tag 5 (bigfloat)
82 -- Array of length 2
20 -- -1
03 -- 3
*/
// Decode to a json value (despite its name, it is not JSON specific.)
json j = cbor::decode_cbor<json>(v);
// Serialize to JSON
std::cout << "(1)\n";
std::cout << pretty_print(j);
std::cout << "\n\n";
// as<std::string>() and as<double>()
std::cout << "(2)\n";
std::cout << std::dec << std::setprecision(15);
for (const auto& item : j.array_range())
{
std::cout << item.as<std::string>() << ", " << item.as<double>() << "\n";
}
std::cout << "\n";
// Query with JSONPath
std::cout << "(3)\n";
json result = jsonpath::json_query(j,"$.[?(# < 1.5)]");
std::cout << pretty_print(result) << "\n\n";
// Encode result as CBOR
std::vector<uint8_t> val;
cbor::encode_cbor(result,val);
std::cout << "(4)\n";
for (auto c : val)
{
std::cout << std::hex << std::setprecision(2) << std::setw(2)
<< std::setfill('0') << static_cast<int>(c);
}
std::cout << "\n\n";
/*
83 -- Array of length 3
fb -- double
3f129cbab649d389 -- 0.000071
c3 -- Tag 3 (negative bignum)
49 -- Byte string value of length 9
010000000000000000
c4 -- Tag 4 (decimal fraction)
82 -- Array of length 2
38 -- Negative integer of length 1
1c -- -29
c2 -- Tag 2 (positive bignum)
4d -- Byte string value of length 13
018ee90ff6c373e0ee4e3f0ad2
*/
}
Output:
(1)
[
2.0,
7.1e-05,
"-18446744073709551617",
"1.23456789012345678901234567890",
[-1, 3]
]
(2)
2.0, 2
7.1e-05, 7.1e-05
-18446744073709551617, -1.84467440737096e+19
1.23456789012345678901234567890, 1.23456789012346
1.5, 1.5
(3)
[
7.1e-05,
"-18446744073709551617",
"1.23456789012345678901234567890"
]
(4)
83fb3f129cbab649d389c349010000000000000000c482381cc24d018ee90ff6c373e0ee4e3f0ad2

Sure, you can use any general purpose programming language for querying CBOR, for example JavaScript might be a good choice. But if you are looking for a "query language" like JsonPath, I'm not aware of any specifically developed for CBOR.

Related

vector iterator not behaving properly

I want to iteratively use insert to modify the first element in a vector<int>(I know that with vector it's better to insert element in the back, I was just playing).
int main() {
vector<int> v1 = {1,2,2,2,2};
auto itr = v1.begin();
print_vector(v1);
cout<<*itr<<endl; // ok, itr is pointing to first element
v1.insert(itr,3);
cout<<*itr<<endl; // after inserting 3 itr is still pointing to 1
print_vector(v1);
cout<<*itr<<endl; // but now itr is pointing to 3
v1.insert(itr,7);
print_vector(v1);
cout<<*itr<<endl;
return 0;
}
v[]: 1 2 2 2 2
1
1
v[]: 3 1 2 2 2 2
3
v[]: 131072 3 1 2 2 2 2
Process finished with exit code 0
So my problem here are mainly 2:
After v1.insert(itr,3), itr is still pointing to 1. After the call of print_vector() now itr is pointing to 3. Why?
Ok now itr its pointing to 3 (the first element of v1). I call v1.insert(itr,7) but instead of placing 7 as the first element, it place 131072. Again, why?
The print_vector function I have implemented is the following:
void print_vector(vector<int> v){
cout<<"v[]: ";
for(int i:v){
cout<<i<<" ";
}
cout<<endl;
}
After inserting an element to a vector, all of its iterators are invalidated, meaning any behavior involving them falls under undefined behavior. You can find a list of iterator invalidation conditions in the answers on Iterator invalidation rules for C++ containers.
Anything you're experiencing after the first v1.insert() call falls under undefined behavior, as you can clearly see with the placement of 131072 (an arbitrary value).
If you refresh the iterator after every insertion call, you should get normal behavior:
int main()
{
vector<int> v1 = { 1,2,2,2,2 };
auto itr = v1.begin();
print_vector(v1);
cout << *itr << endl;
v1.insert(itr, 3);
itr = v1.begin(); // Iterator refreshed
cout << *itr << endl;
print_vector(v1);
cout << *itr << endl;
v1.insert(itr, 7);
itr = v1.begin(); // Iterator refreshed
print_vector(v1);
cout << *itr << endl;
return 0;
}
And the output:
v[]: 1 2 2 2 2
1
3
v[]: 3 1 2 2 2 2
3
v[]: 7 3 1 2 2 2 2
7

why is a pointer twice as big on a 64 bit computer?

A pointer on a 32 bit machine has size 4 bytes, whereas on a 64 bit machine it has 8 bytes.
For example,
int i = 1;
ptr2int* int = &i
then
std::cout << "Size of pointer to i is " << sizeof(ptr2int) << std::endl;
gives 4 on a 32 bit machine, but 8 if calculated by a 64 bit processor.
Why is that?

How to bruteforce a lossy AND routine?

Im wondering whether there are any standard approaches to reversing AND routines by brute force.
For example I have the following transformation:
MOV(eax, 0x5b3e0be0) <- Here we move 0x5b3e0be0 to EDX.
MOV(edx, eax) # Here we copy 0x5b3e0be0 to EAX as well.
SHL(edx, 0x7) # Bitshift 0x5b3e0be0 with 0x7 which results in 0x9f05f000
AND(edx, 0x9d2c5680) # AND 0x9f05f000 with 0x9d2c5680 which results in 0x9d045000
XOR(edx, eax) # XOR 0x9d045000 with original value 0x5b3e0be0 which results in 0xc63a5be0
My question is how to brute force and reverse this routine (i.e. transform 0xc63a5be0 back into 0x5b3e0be0)
One idea i had (which didn't work) was this using PeachPy implementation:
#Input values
MOV(esi, 0xffffffff) < Initial value to AND with, which will be decreased by 1 in a loop.
MOV(cl, 0x1) < Initial value to SHR with which will be increased by 1 until 0x1f.
MOV(eax, 0xc63a5be0) < Target result which I'm looking to get using the below loop.
MOV(edx, 0x5b3e0be0) < Input value which will be transformed.
sub_esi = peachpy.x86_64.Label()
with loop:
#End the loop if ESI = 0x0
TEST(esi, esi)
JZ(loop.end)
#Test the routine and check if it matches end result.
MOV(ebx, eax)
SHR(ebx, cl)
TEST(ebx, ebx)
JZ(sub_esi)
AND(ebx, esi)
XOR(ebx, eax)
CMP(ebx, edx)
JZ(loop.end)
#Add to the CL register which is used for SHR.
#Also check if we've reached the last potential value of CL which is 0x1f
ADD(cl, 0x1)
CMP(cl, 0x1f)
JNZ(loop.begin)
#Decrement ESI by 1, reset CL and restart routine.
peachpy.x86_64.LABEL(sub_esi)
SUB(esi, 0x1)
MOV(cl, 0x1)
JMP(loop.begin)
#The ESI result here will either be 0x0 or a valid value to AND with and get the necessary result.
RETURN(esi)
Maybe an article or a book you can recommend specific to this?
It's not lossy, the final operation is an XOR.
The whole routine can be modeled in C as
#define K 0x9d2c5680
uint32_t hash(uint32_t num)
{
return num ^ ( (num << 7) & K);
}
Now, if we have two bits x and y and the operation x XOR y, when y is zero the result is x.
So given two numbers n1 and n2 and considering their XOR, the bits or n1 that pairs with a zero in n2 would make it to the result unchanged (the others will be flipped).
So in considering num ^ ( (num << 7) & K) we can identify num with n1 and (num << 7) & K with n2.
Since n2 is an AND, we can tell that it must have at least the same zero bits that K has.
This means that each bit of num that corresponds to a zero bit in the constant K will make it unchanged into the result.
Thus, by extracting those bits from the result we already have a partial inverse function:
/*hash & ~K extracts the bits of hash that pair with a zero bit in K*/
partial_num = hash & ~K
Technically, the factor num << 7 would also introduce other zeros in the result of the AND. We know for sure that the lowest 7 bits must be zero.
However K already has the lowest 7 bits zero, so we cannot exploit this information.
So we will just use K here, but if its value were different you'd need to consider the AND (which, in practice, means to zero the lower 7 bits of K).
This leaves us with 13 bits unknown (the ones corresponding to the bits that are set in K).
If we forget about the AND for a moment, we would have x ^ (x << 7) meaning that
hi = numi for i from 0 to 6 inclusive
hi = numi ^ numi-7 for i from 7 to 31 inclusive
(The first line is due to the fact that the lower 7 bits of the right-hand are zero)
From this, starting from h7 and going up, we can retrive num7 as h7 ^ num0 = h7 ^ h0.
From bit 7 onward, the equality doesn't work and we need to use numk (for the suitable k) but luckily we already have computed its value in a previous step (that's why we start from lower to higher).
What the AND does to this is just restricting the values the index i runs in, specifically only to the bits that are set in K.
So to fill in the thirteen remaining bits one have to do:
part_num7 = h7 ^ part_num0
part_num9 = h9 ^ part_num2
part_num12 = h12 ^ part_num5
...
part_num31 = h31 ^ part_num24
Note that we exploited that fact that part_num0..6 = h0..6.
Here's a C program that inverts the function:
#include <stdio.h>
#include <stdint.h>
#define BIT(i, hash, result) ( (((result >> i) ^ (hash >> (i+7))) & 0x1) << (i+7) )
#define K 0x9d2c5680
uint32_t base_candidate(uint32_t hash)
{
uint32_t result = hash & ~K;
result |= BIT(0, hash, result);
result |= BIT(2, hash, result);
result |= BIT(3, hash, result);
result |= BIT(5, hash, result);
result |= BIT(7, hash, result);
result |= BIT(11, hash, result);
result |= BIT(12, hash, result);
result |= BIT(14, hash, result);
result |= BIT(17, hash, result);
result |= BIT(19, hash, result);
result |= BIT(20, hash, result);
result |= BIT(21, hash, result);
result |= BIT(24, hash, result);
return result;
}
uint32_t hash(uint32_t num)
{
return num ^ ( (num << 7) & K);
}
int main()
{
uint32_t tester = 0x5b3e0be0;
uint32_t candidate = base_candidate(hash(tester));
printf("candidate: %x, tester %x\n", candidate, tester);
return 0;
}
Since the original question was how to "bruteforce" instead of solve here's something that I eventually came up with which works just as well. Obviously its prone to errors depending on input (might be more than 1 result).
from peachpy import *
from peachpy.x86_64 import *
input = 0xc63a5be0
x = Argument(uint32_t)
with Function("DotProduct", (x,), uint32_t) as asm_function:
LOAD.ARGUMENT(edx, x) # EDX = 1b6fb67c
MOV(esi, 0xffffffff)
with Loop() as loop:
TEST(esi,esi)
JZ(loop.end)
MOV(eax, esi)
SHL(eax, 0x7)
AND(eax, 0x9d2c5680)
XOR(eax, esi)
CMP(eax, edx)
JZ(loop.end)
SUB(esi, 0x1)
JMP(loop.begin)
RETURN(esi)
#Read Assembler Return
abi = peachpy.x86_64.abi.detect()
encoded_function = asm_function.finalize(abi).encode()
python_function = encoded_function.load()
print(hex(python_function(input)))

What is `uvw` property of QPickTriangleEvent class

QPickTriangleEvent class has a uvw property. What is it? Is uvw the normal vector of the triangle?
I log the QPickTriangleEvent properties:
Qt3DRender::QPickTriangleEvent *eventTri = static_cast<Qt3DRender::QPickTriangleEvent *>(event);
qDebug() << "Triangle Index: " << eventTri->triangleIndex();
qDebug() << "Triangle Vertex 1: " << eventTri->vertex1Index();
qDebug() << "Triangle Vertex 2: " << eventTri->vertex2Index();
qDebug() << "Triangle Vertex 3: " << eventTri->vertex3Index();
qDebug() << "Triangle UVW: " << eventTri->uvw();
The output:
Triangle Index: 79540
Triangle Vertex 1: 238620
Triangle Vertex 2: 238621
Triangle Vertex 3: 238622
Triangle UVW: QVector3D(0.0390438, 0.151772, 0.809184)
Even with examining the output I cannot figure uvw out. Is it the normal vector of the triangle?
I'm examining these source codes to figure out what uvw is:
triangleboundingvolume_p.h
triangleboundingvolume.cpp
In source code triangleboundingvolume.cpp it is commented:
// RealTime Collision Detection page 192
bool intersectsSegmentTriangle(const RayCasting::QRay3D &ray, ...
Page 193 of the book comments
... also returns the barycentric coordinates (u,v,w) of the intersection point s, ...
and page 194
... Segment intersects tri at distance t in position s (s = uA + vB + w*C) ...

On the usage of clCreateImage2D

Usually the second parameter in clCreateImage2D is a flag CL_MEM_READ etc. But I found it 0 in one of the sample codes (P. no: 80, Heterogeneous Computing using openCL ):
//Create space for the source image on the device
cl_mem bufferSourceImage = clCreateImage2D(
context,0,&format, width,height,0,NULL,NULL);
Why it is so?
cl_mem_flags are bitfields:
cl.h
/* cl_mem_flags - bitfield */
#define CL_MEM_READ_WRITE (1 << 0)
#define CL_MEM_WRITE_ONLY (1 << 1)
#define CL_MEM_READ_ONLY (1 << 2)
#define CL_MEM_USE_HOST_PTR (1 << 3)
#define CL_MEM_ALLOC_HOST_PTR (1 << 4)
#define CL_MEM_COPY_HOST_PTR (1 << 5)
// reserved (1 << 6)
#define CL_MEM_HOST_WRITE_ONLY (1 << 7)
#define CL_MEM_HOST_READ_ONLY (1 << 8)
#define CL_MEM_HOST_NO_ACCESS (1 << 9)
Here, 0 is a default value for CL_MEM_READ_WRITE :
A bit-field that is used to specify allocation and usage information
such as the memory arena that should be used to allocate the buffer
object and how it will be used. The following table describes the
possible values for flags. If value specified for flags is 0, the
default is used which is CL_MEM_READ_WRITE.
From: clCreateBuffer

Resources