Is there a way to see, via hex editor or otherwise, if data in a binary file is aligned or packed, specifically for an HPUX system?
If you know what you are looking for and can recognize it in a hex dump, then you can make informed estimates about whether the data is aligned or not, or packed. But in many ways, your question is unanswerable. Where did the data come from? Why can't you ask the person (driving a program, presumably) how it was created?
If you are asking 'what tools could I use to view the data', then you can consider:
od (octal dump - probably with the -c option too)
hd (hex dump - not always available, and seems to be absent on HP-UX)
sed l (that's a lower-case ell - it means list the data; not a good option unless the majority of the data is plain text)
Or you could do it in Perl. Once upon a long time ago (1987 or so), I wrote a program odx (octal dump in hex - weird) that I continue to use - it gives me a hex dump, 16 bytes per line, plus an image of the printable characters. This example isn't very exciting (odx run on itself - on a Sun Sparc):
Black JL: odx odx | sed 10q
0x0000: 7F 45 4C 46 01 02 01 00 00 00 00 00 00 00 00 00 .ELF............
0x0010: 00 02 00 12 00 00 00 01 00 01 0D 84 00 00 00 34 ...............4
0x0020: 00 00 77 9C 00 00 01 00 00 34 00 20 00 05 00 28 ..w......4. ...(
0x0030: 00 24 00 23 00 00 00 06 00 00 00 34 00 01 00 34 .$.#.......4...4
0x0040: 00 00 00 00 00 00 00 A0 00 00 00 A0 00 00 00 05 ................
0x0050: 00 00 00 00 00 00 00 03 00 00 00 D4 00 00 00 00 ................
0x0060: 00 00 00 00 00 00 00 11 00 00 00 00 00 00 00 04 ................
0x0070: 00 00 00 00 00 00 00 01 00 00 00 00 00 01 00 00 ................
0x0080: 00 00 00 00 00 00 22 86 00 00 22 86 00 00 00 05 ......"...".....
0x0090: 00 01 00 00 00 00 00 01 00 00 22 88 00 02 22 88 .........."...".
Black JL:
Related
RSA Encryption/Decryption Error
I am having a problem with an RSA encryption algorithm. I have an app (APP1) on an embedded device. The device has
a proprietary (old) embedded OS. I have 1 RSA crypto function ("rsa_calc") that I am using. In APP1
it has the public key of an RSA Key Pair. My issue is periodically, some encryptions can't be decrypted
in the receiving application APP2 (modern application/os). I am using BP-TOOLS Crypto Calculator to verify
results. I have included the trace of 2 encryptions below. The first works, i.e. it can be decrypted using
BP-TOOLS Crypto Calculator. The Second cannot. The vast majority of encryptions verify, but every now and again
some do not.
The issue appears to be based on the input data to be encrypted. Taking my traces, I found a working value, and a
non-working value. And performed the encryptions. I get the same results. So basically encrypting some data works,
other not so much. Please see traces below.
This leads me to believe I am having some padding error. The RSA encrypt function does not document padding schemes.
I am not implementing any padding manually other than using a 128 byte buffer. On BP-TOOLS Crypto Calculator,
I am using "No Padding" on the decryption, it only seems to work when I select that.
If I select "PKCS1", I get padding errors when decrypting. There is no error decrypting the data on BP-TOOLS, I just
don't get clear text/original text.
I am not concerned about the strength of the encryption and not using padding etc. I am just trying to get this to
work consistently. Do I have to implement one of the padding schemes for RSA manually? Does padding make certain
RSA buffers impossible to decrypt (I wouldn't think so, I know it weakens the cipher, don't care)?
Can anyone see some other issue that may be causing my problem?
Thanks for the help
Function Documentation
rsa_calc()
Performs a public key RSA computation. It supports keys up to 2048 bits and exponent values of 2, 3, and 65537.
Prototype:
int rsa_calc(unsigned short * msg, unsigned short * mod, int wds, int exp, unsigned short * result);
Parameters:
msg - Array of unsigned 16-bit integers holding the input.
mod - Array of unsigned 16-bit integers holding the modulus. High bit must be set.
Typically an RSA modulus always has the most-significant bit set. This
implementation verifies and enforces that, because if the bit is not set, it
might cause unexpected behaviors. The high bit of the modulus is the MSB
of the first 16-bit integer of the array.
wds - Number of 16-bit unsigned shorts in msg and mod.
exp - Code for exponent: actual exponent is 2^exp+1. Acceptable values are 0, 1,
16 that correspond to exponents of 2,3, and 65537, respectively.
Return Values:
result Array of 16-bit integers holding the result on exit.
RSA Key Pair
MODULUS:
BEDB6B21E12D2B6EB590EC129FCC847EA4C00BE41CA530A5FA2CCDDE3B7DB3A0F50E6D3E348CD9258D7076973DD01FDC5B7E00F1F714F4E55C650DF88AAA293ED9376D2B0905F589108FB5C2835EA025D036F369891E5F5F3BA4F5E96CF25D164C1B26215B8D9627CDB95C22F00EBF50DF821A984E01309C1677B5D013E2BEEF
PUBLIC EXP(E):
010001
PRIVATE EXP(D):
A6898BC7FA5691C97EC1405D57F6FBBE0E404D9FF4A6E7F64C807FFAE4EA60AD9867C847394F95C340D1DB894934AC3879D54F39D3A203B78791DE48FBA65369B077BD6541AA8200392CA0BF56EEE2AA8478598852BFB537498095C087910176E1E90F92F8564F3012D7DC52B8D7145C40143F51229FE4416CEF50657511E2F1
TRACES
!!!! This Works !!!!
-------------------------
|*** RSAEncryptKey: rsa_calc :: ENTER
|MSG (256):
|0000 01 2A 9E 34 EA 19 63 07 F9 AD A8 9B 02 5B D6 6D - .*.4..c......[.m
|0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|rsa_calc: 0 ::
|rsa_calc result (256):
|0000 2D 0A F1 93 29 6A 47 D7 94 A6 16 32 E8 AD 2D 3B - -...)jG....2..-;
|0010 72 64 7B 23 55 79 1B 71 A1 34 AD E4 E5 E7 E6 FA - rd{#Uy.q.4......
|0020 6D 78 AA 6E 7F 32 D0 BF 26 87 3A 5F 8F B9 23 42 - mx.n2..&.:_..#B
|0030 C7 02 D6 8B 70 24 76 90 59 40 A3 42 D5 B5 2B 00 - ....p$v.Y#.B..+.
|0040 CB 08 A4 C3 07 E7 12 4D 02 C0 2D 3A 43 15 27 41 - .......M..-:C.'A
|0050 17 42 F7 B0 F8 B7 06 B1 25 D9 90 12 88 C1 05 16 - .B..............
|0060 75 64 8E 87 AA 68 82 96 1E 3E 23 F7 32 10 E5 8A - ud...h...>#.2...
|0070 17 D4 39 18 21 FC 5A C3 38 BB 7F CB DA B0 D2 85 - ..9.!.Z.8......
|0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|*** RSAEncryptKey: :: EXIT
!!!! DOES NOT DECRYPT !!!!
-------------------------------------
|*** RSAEncryptKey: rsa_calc :: ENTER
|MSG (256):
|0000 FA 68 E9 19 F5 D2 64 E0 F6 A8 FB 88 DC 67 35 C0 - .h....d......g5.
|0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|rsa_calc: 0 ::
|rsa_calc result (256):
|0000 36 B8 4D F2 76 DA 21 31 A3 8E C8 26 75 84 52 48 - 6.M.v.!1...&u.RH
|0010 3B CE 87 15 58 D9 F0 21 04 CE A0 83 9E 5E 58 31 - ;...X..!.....^X1
|0020 19 CF 82 70 00 BD A2 6D 2C E8 34 F4 27 D1 B2 4C - ...p...m,.4.'..L
|0030 1A F4 19 1D DA C4 B6 CD 10 C2 29 32 B6 85 E7 CB - ..........)2....
|0040 BD 4D 53 70 B3 27 47 B8 87 48 0C FD 86 B8 C0 4C - .MSp.'G..H.....L
|0050 B4 B8 23 4C 11 8C A7 93 41 2E AF E4 FF F1 EC 9A - ..#L....A.......
|0060 44 13 FF DB 9B ED B6 9E E2 2A 11 41 64 C5 2C E0 - D........*.Ad.,.
|0070 FE 4D 05 30 EE 49 9A 7A C5 46 BD 17 2F 87 72 D1 - .M.0.I.z.F../.r.
|0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|00F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
|*** RSAEncryptKey: :: EXIT
Thank you President James K. Polk. Your comments resolved the issue. I wouldn't mind understanding better, so please comment if you can. It appears you are indicating that if I treat the first 16 bytes as a 16 byte integer, then the encryption block cannot be a larger integer than the Modulus, I don't really understand why. I don't really see the relationship between what I want to encrypt versus the key used to encrypt it.
However, shifting the 16 bytes in the encryption block (I shifted it 16 bytes). Every encryption/decryption appears to work (so far anyway).
Thank you for your help.
So I am learning mach-o file.
Now I was able to tell how to map a pointer to a class struct, but I found that, if I use otool, sometimes it will print out something has meta class:
00000001000990d8 0x1000b75c8
isa 0x1000b75f0
superclass 0x0
cache 0x0
vtable 0x0
data 0x1000a5a00 (struct class_ro_t *)
flags 0x184 RO_HAS_CXX_STRUCTORS
instanceStart 8
instanceSize 152
reserved 0x0
ivarLayout 0x10007bb99
layout map: 0x1f 0x02
name 0x10007bb93 AASDK
and after this class_ro_t *, there is one section right after it:
Meta Class
isa 0x0
superclass 0x0
cache 0x0
vtable 0x0
data 0x1000a5730 (struct class_ro_t *)
flags 0x185 RO_META RO_HAS_CXX_STRUCTORS
instanceStart 40
instanceSize 40
reserved 0x0
ivarLayout 0x0
name 0x10007bb93 AASDK
baseMethods 0x1000a5698 (struct method_list_t *)
And this seems like meta class has the methods declared as class methods (marked with + in ObjC code)
The memory layout looks like:
1000B75C8: F0 75 0B 00 01 00 00 00 00 00 00 00 00 00 00 00
1000B75D8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1000B75E8: 00 5A 0A 00 01 00 00 00 00 00 00 00 00 00 00 00
1000B75F8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1000B7608: 00 00 00 00 00 00 00 00 30 57 0A 00 01 00 00 00
1000B7618: 40 76 0B 00 01 00 00 00 00 00 00 00 00 00 00 00
We can obviously see that 30 57 0A 00 01 00 00 00 is the meta class data pointer.
I wonder, how to tell if the class has meta class info?
I checked the otool source code, but I can't read it, it has lots of calculations and offsets. Can someone help explain how to check it?
In my application I'm using an amalgamated embeded version of SQLite v3.8.4. In the past I had compatibility issues with v3.7 regarding page_size > 32768. Now I'd like to keep backwards compatibility on database files with v3.6.20 (default on RHEL6), but when I try to open my DB file on my RHEL6 machine I see the error:
Error: file is encrypted or is not a database
I can dump the database to an SQL file and then again import it to v3.7. This seems to do the trick to be readable on v3.6.20 but my DB is 3 GB in size and is not feasible for me to wait 2 hours for each dump.
What options can I set on my v3.8 to generate v3.6 compatible databases?
Databases dump
$ hexdump -C works-on-3.6.20.sqlite | head -n 8
00000000 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 |SQLite format 3.|
00000010 04 00 01 01 00 40 20 20 00 00 00 01 00 2d 8c 51 |.....# .....-.Q|
00000020 00 00 00 00 00 00 00 00 00 00 00 12 00 00 00 01 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 |................|
00000060 00 2d e2 21 05 00 00 00 02 03 f6 00 00 22 fd 07 |.-.!........."..|
00000070 03 fb 03 f6 01 8f 02 38 00 7c 01 61 81 62 07 07 |.......8.|.a.b..|
$ hexdump -C fails-on-3.6.20.sqlite | head -n 8
00000000 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 |SQLite format 3.|
00000010 80 00 02 02 00 40 20 20 00 00 00 04 00 00 00 2b |.....# .......+|
00000020 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 01 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 |................|
00000060 00 2d e6 04 0d 7f fc 00 18 75 a4 00 7f 36 7f c9 |.-.......u...6..|
00000070 7e 6b 7f 0b 7d 8f 7e 38 7c 7e 7d 62 7b ab 7c 47 |~k..}.~8|~}b{.|G|
The non-working database has enabled WAL mode, which was introduced in version 3.7.0. Disable it.
Edit: It turns out that this is a custom checksum algorithm, not a CRC-32. For the curious here is a snippet of C code calculating the 21 FF 1D E4 checksum in the below example.
I am working with some hex files that seem to be protected by a CRC-32 of sorts, but recalculating using the standard CRC-32 and other known parameters failed to produce a match.
All I know is, the data spans 116 bytes and four additional bytes for the checksum. I can produce tons of examples of message-key pairs, I just cannot find any relation between them.
I don't want to fill this post with hex dumps so I pasted a couple more here: http://mathb.in/12246.
11 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 43
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 21 FF 1D E4
Could this be a CRC-32 with some strange settings, or is it completely different? What would be a way to determine how the checksum is produced?
Update: I was able to make minute changes of 1 bit in the messages:
00000000000000000000000000000000000000000000000000000000, 32C9A1E6
00000000000000000000000000000000000000000000000000000001, BD25904E
00000000000000000000000000000000000000000000000000000002, B437A286
00000000000000000000000000000000000000000000000000000003, 8AB790EE
00000000000000000000000000000000000000000000000000000004, 2DDC3AEB
00000000000000000000000000000000000000000000000000000005, 208B3859
00000000000000000000000000000000000000000000000000000006, 87E0925C
00000000000000000000000000000000000000000000000000000007, E59391AE
00000000000000000000000000000000000000000000000000000008, B07292FC
00000000000000000000000000000000000000000000000000000009, 830EA655
Changing anything without updating the checksum appropriately creates a message that is rejected.
If this cannot be a CRC-32 due to the superposition principle (exclusive-or check), would there be any way to perform an analysis to find any properties or patterns in pairs of message and checksums?
Assuming that you got the messages and check values correct in your linked examples, then it is not a CRC. For a CRC or any linear function over GF(2) of the message, it should be that if message 1 exclusive-or'ed with message 2 is equal to message 3 exclusive-or'ed with message 4, then the same should be true of the check values.
There is only one byte that differs in your messages. Since 45 ^ 4A == 43 ^ 4C, then the associated check values when exclusive-or'ed should also be equal. They are not.
If you did have a CRC, then you could use reveng to try to tease out the CRC parameters.
R use 32-bit integers, 4 bytes (4*8=32, 1 byte=8 bits),max integer is 2^31-1= 2147483647.
intToBits(125)
[1] 01 00 01 01 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
There are 64 bits ,not 32 bits,32 pairs of 00 or 01(32*2=64) ,why?
in the manual ,a raw vector with entries 0 or 1,what is the meaning of entries 0 or 1?
In my opinion, intToBits(125) will generate
10111110 00000000 00000000 00000000 # it is 32 bits
not
01 00 01 01 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00. # it is 64 bits
Function intToBits returns 32 numbers, each 0 or 1.
> length(intToBits(125))
[1] 32
The confusion is caused by the hex representation of the raw type of the output of intToBits.
> intToBits(125)[3]
[1] 01
The raw numbers 0 or 1 are shown as 00 or 01.