Is this home-grown algorithm safe for encryption? [closed] - encryption

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 5 years ago.
Improve this question
Jasoos (Cryptography Algorithm)
I am working on encrypting and decrypt web application. I have built an algorithm that uses 24-byte key to encrypt/decrypt the message.
Review this algorithm and please suggest anything important and fault in this algorithm that can make it perform better. Your contribution can help us to improve our algorithm.
Code is provided on my GitHub
Algorithm:-
1] 24 digit entered/generated key will be converted into ASCII code of 24 digit code.
public void setKey(char[] arr){
for(int i=0;i<24;i++){
key[i] = (int)arr[i];
}
}
2] Entered String will be changed into a character array.
Every character will be then incremented first with the key’s value and changed into 10-bit binary code.
public void Encryption(String text){
char[] msg = text.toCharArray();
int flag = 0;
int l = msg.length;
for(int i=0;i<l;i++){
int a = (int)msg[i];
// System.out.print(msg[i]+" "+a+"-> ");
if(flag>23)
flag=0;
int b=a+key[flag];
flag++;
//System.out.print(b+" | ");
String z = binary(b);
sb.append(lookUpTool(z));
//Character.toString((char)b);
}
//sb.append(sumBinary);
sb = comp1(sb);
}
3] lookUp(): - It will take a 10-bit string as input and a matrix, and divide that string into two 5 bit binary code.
We will then calculate decimal value of each 5-bit binary code.
Example: 0011101101 -> 00111 = 7 and 01101 = 13
We have a matrix of 32 X 32 dimensions which has unique random values from 0 to 1023 and will not be shared publicly.
For 0011101101 we will look for 7th row and 13th column value.
That value will be changed into 10 bits binary code.
public String lookUp(String bits, int[][] mat){
int mid = Math.round((float) bits.length() / 2);
String part1 = bits.substring(0, mid);
String part2 = bits.substring(mid, bits.length());
int row=binaryValue(part1);
int col=binaryValue(part2);;
//System.out.print("row: "+row);
// System.out.println("|| col: "+col);
int a = mat[row][col];
return binary(a);
}
4] We will perform this steps ten times with ten different private matrices by lookUpTool method.
public String lookUpTool(String s){
String s1 = lookUp(s,matrix1);
String s2 = lookUp(s1,matrix2);
String s3 = lookUp(s2,matrix3);
String s4 = lookUp(s3,matrix4);
String s5 = lookUp(s4,matrix5);
String s6 = lookUp(s5,matrix6);
String s7 = lookUp(s6,matrix7);
String s8 = lookUp(s7,matrix8);
String s9 = lookUp(s8,matrix9);
String s10 = lookUp(s9,matrix10);
return s10;
}
Similarly, we will do this for each character in the text/string and encrypt it.
Example:-
Key: c|H#yLzd3PkRte0H,u16zt8N
Message: abcd ef$
After Encryption: 11001111000001101010000010000101101000001110100000101010111001110000011000001000

Your algorithm is completely worthless by any reasonable standard. The most obvious problem is this:
You just gave us a key, plaintext, and corresponding encoded message. This leaks out numerous entries from your super-secret matrix that you weren't supposed to share publicly. (Each ten-bit chunk of the encrypted message is an entry from that array, and with the key and plaintext, I can figure out which one it is.)
Imagine if an adversary had a collection of messages that were already encrypted by your algorithm and then you posted this challenge. He can now decrypt a significant fraction of those messages, just from what you leaked in this challenge. And if there are obvious missing bits, say he has "trans_ormer", he can work out another entry in your formerly super-secret array.
But please read the links in the comments. Trying to design your own encryption algorithm for actual use and reliance in this way is absolutely foolish. A new algorithm cannot even be considered for actual use before it has been reviewed thoroughly by experts in each type of known cryptanalysis.
Another algorithmic flaw is immediately obvious. An attacker will know that the key repeats every 24 characters. With a long enough message, say in English, the attacker can do a frequency analysis for each set of every 24th character. It's even worse if the attacker knows the message format and that format has an even more unequal frequency distribution.

Related

How can a 1 byte int conversion of a QByteArray fail?

So here is the thing, I'm receiving 1 byte from Bluetooth transmission. When using QDebug I get this message:
The array with error has "\x06"
The line that fails is this:
bool ok = true;
int v = value.toInt(&ok,0);
Because ok has false. But I'm trying to wrap my head around the fact that, How can the conversion fail in the first place if the data represented in that byte (as a sequence of zeros and ones) will always have a valid integer representation. (one byte can always be represented as a int between -127 and 128). So I'm left with the question, how can the conversion fail?
Reading the documentation does not provide many clues as it does not say how the byte array will be interpreted.
QByteArray::toInt converts a string representation in the default C locale to an integer. That means to successfully convert the value in your example, your byte array must contain the string "0x06", which consists of 4 bytes.
To convert a single byte to an int, just extract it:
int i = value[0];
Type promotion will widen the char to an int

Is RSACryptoServiceProvider working correctly?

I'm using .NET's implementation of RSA, and two things looked odd to me. I'd like to confirm that it's operating properly.
Background
Using System.Security.Cryptography.RSACryptoServiceProvider with 2048-bit keyword size to perform asymmetric encryption/decrpytion, initially following the example in this question, "AES 256 Encryption: public and private key how can I generate and use it .net".
As a first implementation, this seems to work:
public const int CSPPARAMETERS_FLAG = 1; // Specifies RSA: https://msdn.microsoft.com/en-us/library/ms148034(v=vs.110).aspx
public const bool USE_OAEP_PADDING = false;
public const int KEYWORD_SIZE = 2048;
public static byte[] Encrypt(byte[] publicKey, byte[] dataToEncrypt)
{
var cspParameters = new System.Security.Cryptography.CspParameters(CSPPARAMETERS_FLAG);
byte[] encryptedData = null;
using (var rsaProvider = new System.Security.Cryptography.RSACryptoServiceProvider(cspParameters))
{
try
{
rsaProvider.PersistKeyInCsp = false;
rsaProvider.ImportCspBlob(publicKey);
encryptedData = rsaProvider.Encrypt(dataToEncrypt, USE_OAEP_PADDING);
}
finally
{
rsaProvider.PersistKeyInCsp = false;
rsaProvider.Clear();
}
}
return encryptedData;
}
public static byte[] Decrypt(byte[] privateKey, byte[] dataToDecrypt)
{
var cspParameters = new System.Security.Cryptography.CspParameters(CSPPARAMETERS_FLAG);
byte[] encryptedData = null;
using (var rsaProvider = new System.Security.Cryptography.RSACryptoServiceProvider(cspParameters))
{
try
{
rsaProvider.PersistKeyInCsp = false;
rsaProvider.ImportCspBlob(privateKey);
encryptedData = rsaProvider.Decrypt(dataToDecrypt, USE_OAEP_PADDING);
}
finally
{
rsaProvider.PersistKeyInCsp = false;
rsaProvider.Clear();
}
}
return encryptedData;
}
After looking into these methods a bit more, it seems that the public key that I've been generating as from the example seemed to have a lot of very predictable data at its start, and it was 276-bytes long.
Apparently rsaProvider.ExportCspBlob(bool includePrivateParameters) is a functional alternative to rsaProvider.ExportParameters(bool includePrivateParameters); the main difference is that the blob is already serialized as a byte[] while the other emits the object version, RSAParameters.
Two observations about the methods:
The .Exponent is always 0x010001$=65537$.
The exported blobs contain 17 extra bytes versus the serialized typed versions.
rsaProvider.ExportCspBlob():
Public key is 276 bytes.
Private key is 1172 bytes.
RSAParameters:
Public key is 259 bytes.
.Exponent.Length = 3
.Modulus .Length = 256
Private key is 1155 bytes.
.D .Length = 256
.DP .Length = 128
.DQ .Length = 128
.Exponent.Length = 3
.InverseQ.Length = 128
.Modulus .Length = 256
.P .Length = 128
.Q .Length = 128
The extra 17 bytes appear to be at the header of the binary blob.
Concerns
From this, two concerns:
Is it okay for the exponent to not be random?
If the exponent is defined as a constant, then it'd seem like that's another 3 bytes I could shave off the serialization?
Another question, Should RSA public exponent be only in {3, 5, 17, 257 or 65537} due to security considerations?, seems to suggest that $\left{3, 5, 17, 257, 65537\right}$ are all common values for the exponent, so 0x101$=65537$ seems reasonable if it's true that there's no harm in always using the same constant exponent.
Are the 17 extra bytes an information leak?
Do they represent the option parameters like key length and method?
Is it a good idea to be transmitting option parameter information when I already know that both the sender and receiver are using the same, hard-coded method?
Question
Is RSACryptoServiceProvider's behavior a cause for concern, or are these things normal?
Update 1
In Should RSA public exponent be only in {3, 5, 17, 257 or 65537} due to security considerations?, the accepted answer starts off by noting:
There is no known weakness for any short or long public exponent for RSA, as long as the public exponent is "correct" (i.e. relatively prime to p-1 for all primes p which divide the modulus).
If this is so, then I'd guess that the apparently-constant exponent of 0x010001$=65537$ is sufficient as long as it's relatively prime to $p-1$. So, presumably the .NET implementation of RSA checks for this condition.
But then what does RSACryptoServiceProvider do if that condition isn't satisfied? If it selects a different exponent, then that'd seem to leak information about $p$ whenever the exponent isn't 0x010001. Or, if a different key is selected, then it'd seem like we can just assume that the exponent is always 0x010001 and omit it from the serialization.
Everything reported is normal, and non-alarming.
It is perfectly OK for the public exponent e to be short and non-random. e = 216+1 = 65537 = 0x010001 is common and safe. Some authorities mandate it (or some range including it). Using it (or/and something significantly larger than the bit size of the public modulus) gives some protection against some of the worst RSA paddings.
No, the 17 extra bytes in the public key are unlikely to be an information leak; they more likely are a header part of the data format chosen for an RSA public key by the software you use. My guess is that you are encountering the MS-specific format detailed in this answer (perhaps, within endianness), which also uses precisely 276 bytes for an RSA public key with a 2048-bit public modulus. In that case, you should find that the extra bytes are always the same (thus they demonstrably leak nothing). And there are countless more subtle ways to leak information about the private key, like in the public modulus itself.
Many RSA key generators used in practice, including I guess RSACryptoServiceProvider, first choose e, then somewhat avoid generating primes p such that gcd(e, p-1) ≠ 1. Since e = 65537 is prime, it is enough that ( p % e ) ≠ 1, and this is easily checked, or otherwise insured by the process generating p.

Storing binary data in QR codes

I'm trying to store binary data in a QR code. Apparently QR codes do support storing raw binary data (or ISO-8859-1 / Latin1). Here is what I want to encode (hex):
d1 50 01 00 00 00 f6 5f 05 2d 8f 0b 40 e2 01
I've tried the following encoders:
qr.js
Google Charts
qrcode.js
Decoding with zxing.org produces various incorrect results. The two javascript ones produce this (it's wrong; the first text character should be Ñ.
Whereas Google Charts produces this...
What is going on? Are any of these correct? What's really weird is that if I encode this sequence (with the JS ones at least) then it works fine - I would have thought the issue was non-ASCII characters but Ñ (0xd1) is non-ASCII.
d1 50 01 00 00 00 01 02 03 04 05 06 40 e2 01
Does anyone know what is going on?
Update
It occurred to me to try scanning them with a ZBar-based scanner app I found. It scans both JS versions ok (at least they start with ÑP). The Google Charts one is just wrong. So it seems like the issue is with ZXing (which is surprisingly shit - I wouldn't recommend it to anyone).
Update 2
ZBar can't handle null bytes. :-(
"What is going on? Are any of these correct?"
Except for the google chart (which is just empty), your QR codes are correct.
You can see the binary data from zxing is what you would expect:
4: Byte mode indicator
0f: length of 15 byte
d15001...: your 15 bytes of data
ec11 is just padding
The problem comes from the decoding. Because most decoders will try to interpret it as text. But since it's binary data, you should not try to handle it as text. Even if you think you can convert it from text to binary, as you saw this may cause issues with values which are not valid text.
So the solution is to use a decoder that will output you the binary data, and not text data.
Now about interpreting the QR code binary data as text, you said the first character should be 'Ñ' which is true if interpreted it as "ISO-8859-1",
which according to the QR code standard, is what should be done when there is no ECI mode defined.
But in practice, most smartphone QR code reader will interpret it as UTF-8 in this case (or at least try to auto-detect the encoding).
Even though this is not the standard, this had become common practice:
binary mode with no ECI, UTF-8 encoded text.
Maybe the reason behind it is that no one wants to waste these precious bytes adding an ECI mode specifying UTF-8. And actually, not all decoders support ECI.
There are two issues that you have to overcome to store binary data in QR codes.
ISO-8859-1 does not allow bytes in ranges of 00-1F and 7F-9F. If you
need to encode these bytes anyway, quote or encode them, i.e. use
quoted-printable or Base-64 encoding to avoid these ranges.
Since you are trying to store binary data in QR codes, you have to
rely only on your own scanner that will handle this binary data. You
don’t have to display text from your QR codes by other software,
like web application at zxing.org, because most QR decoders,
including that of zxing.org use heuristics to detect the character
set used. These heuristics may detect a character set other than
ISO-8859-1 and thus fail to properly display your binary data. Some
scanners use heuristics to detect a character set even if the
character set is explicitly given by ECI. This is why providing ECI
may not help much – scanners still use heuristics even with ECI.
So, using US-ASCII printable characters only (e.g., binary data encoded in Base64 before passing it to a QR Code generator) is the safest choice for QR code against the heuristics. This will also overcome another complication: that ISO-8859-1 was not the default encoding in earlier QR code standard published in 2000 (ISO/IEC 18004:2000). That standard did specify 8-bit Latin/Kana character set in accordance with JIS X 0201 (JIS8 also known as ISO-2022-JP) as default encoding for 8-bit mode, while the updated standard published in 2005 did change the default to ISO-8859-1.
As an alternative to Base-64, you can encode each byte with two hexadecimal characters (0-9, A-F), so, in the QR code your data will be encoded in the alphanumeric mode, not in 8-bit mode. This will disable all heuristics for sure and should not produce larger QR Code than with Base-64, because each character in the alphanumeric mode takes only 6 bits in the QR code stream.
Update:
I recently went back and published the referenced code as a project on GitHub for anyone who wants to use it.
https://github.com/yurelle/Base45Encoder
This is a bit necro, but I just hit this problem, and figured out a solution.
The problem with reading QR Codes with ZXING is that it assumes all QR Payloads are Strings. If you're willing to generate the QR Code in java with ZXING, I developed a solution which enables storing a binary payload in ZXING QR Codes with a storage efficiently loss of only -8%; better than the 33% inflation from Base64.
It exploits an internal compression optimization of the ZXING library based around pure Alphanum Strings. If you want a full explanation, with math and Unit Tests, check out my other answer.
But the short answer is this:
Solution
I implemented it as a self-contained static utility class, so all you have to do is call:
//Encode
final byte[] myBinaryData = ...;
final String encodedStr = BinaryToBase45Encoder.encodeToBase45QrPayload(myBinaryData);
//Decode
final byte[] decodedBytes = BinaryToBase45Encoder.decodeBase45QrPayload(encodedStr);
Alternatively, you can also do it via InputStreams:
//Encode
final InputStream in_1 = ... ;
final String encodedStr = BinaryToBase45Encoder.encodeToBase45QrPayload(in_1);
//Decode
final InputStream in_2 = ... ;
final byte[] decodedBytes = BinaryToBase45Encoder.decodeBase45QrPayload(in_2);
Here's the implementation
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Map;
/**
* For some reason none of the Java QR Code libraries support binary payloads. At least, none that
* I could find anyway. The commonly suggested workaround for this is to use Base64 encoding.
* However, this results in a 33% payload size inflation. If your payload is already near the size
* limit of QR codes, this is a lot.
*
* This class implements an encoder which takes advantage of a built-in compression optimization
* of the ZXING QR Code library, to enable the storage of Binary data into a QR Code, with a
* storage efficiency loss of only -8%.
*
* The built-in optimization is this: ZXING will automatically detect if your String payload is
* purely AlphaNumeric (by their own definition), and if so, it will automatically compress 2
* AlphaNumeric characters into 11 bits.
*
*
* ----------------------
*
*
* The included ALPHANUMERIC_TABLE is the conversion table used by the ZXING library as a reverse
* index for determining if a given input data should be classified as alphanumeric.
*
* See:
*
* com.google.zxing.qrcode.encoder.Encoder.chooseMode(String content, String encoding)
*
* which scans through the input string one character at a time and passes them to:
*
* getAlphanumericCode(int code)
*
* in the same class, which uses that character as a numeric index into the the
* ALPHANUMERIC_TABLE.
*
* If you examine the values, you'll notice that it ignores / disqualifies certain values, and
* effectively converts the input into base 45 (0 -> 44; -1 is interpreted by the calling code
* to mean a failure). This is confirmed in the function:
*
* appendAlphanumericBytes(CharSequence content, BitArray bits)
*
* where they pack 2 of these base 45 digits into 11 bits. This presents us with an opportunity.
* If we can take our data, and convert it into a compatible base 45 alphanumeric representation,
* then the QR Encoder will automatically pack that data into sub-byte chunks.
*
* 2 digits in base 45 is 2,025 possible values. 11 bits has a maximum storage capacity of 2,048
* possible states. This is only a loss of 1.1% in storage efficiency behind raw binary.
*
* 45 ^ 2 = 2,025
* 2 ^ 11 = 2,048
* 2,048 - 2,025 = 23
* 23 / 2,048 = 0.01123046875 = 1.123%
*
* However, this is the ideal / theoretical efficiency. This implementation processes data in
* chunks, using a Long as a computational buffer. However, since Java Long's are singed, we
* can only use the lower 7 bytes. The conversion code requires continuously positive values;
* using the highest 8th byte would contaminate the sign bit and randomly produce negative
* values.
*
*
* Real-World Test:
*
* Using a 7 byte Long to encode a 2KB buffer of random bytes, we get the following results.
*
* Raw Binary Size: 2,048
* Encoded String Size: 3,218
* QR Code Alphanum Size: 2,213 (after the QR Code compresses 2 base45 digits to 11 bits)
*
* This is a real-world storage efficiency loss of only 8%.
*
* 2,213 - 2,048 = 165
* 165 / 2,048 = 0.08056640625 = 8.0566%
*/
public class BinaryToBase45Encoder {
public final static int[] ALPHANUMERIC_TABLE;
/*
* You could probably just copy & paste the array literal from the ZXING source code; it's only
* an array definition. But I was unsure of the licensing issues with posting it on the internet,
* so I did it this way.
*/
static {
final Field SOURCE_ALPHANUMERIC_TABLE;
int[] tmp;
//Copy lookup table from ZXING Encoder class
try {
SOURCE_ALPHANUMERIC_TABLE = com.google.zxing.qrcode.encoder.Encoder.class.getDeclaredField("ALPHANUMERIC_TABLE");
SOURCE_ALPHANUMERIC_TABLE.setAccessible(true);
tmp = (int[]) SOURCE_ALPHANUMERIC_TABLE.get(null);
} catch (NoSuchFieldException e) {
e.printStackTrace();//Shouldn't happen
tmp = null;
} catch (IllegalAccessException e) {
e.printStackTrace();//Shouldn't happen
tmp = null;
}
//Store
ALPHANUMERIC_TABLE = tmp;
}
public static final int NUM_DISTINCT_ALPHANUM_VALUES = 45;
public static final char[] alphaNumReverseIndex = new char[NUM_DISTINCT_ALPHANUM_VALUES];
static {
//Build AlphaNum Index
final int len = ALPHANUMERIC_TABLE.length;
for (int x = 0; x < len; x++) {
// The base45 result which the alphanum lookup table produces.
// i.e. the base45 digit value which String characters are
// converted into.
//
// We use this value to build a reverse lookup table to find
// the String character we have to send to the encoder, to
// make it produce the given base45 digit value.
final int base45DigitValue = ALPHANUMERIC_TABLE[x];
//Ignore the -1 records
if (base45DigitValue > -1) {
//The index into the lookup table which produces the given base45 digit value.
//
//i.e. to produce a base45 digit with the numeric value in base45DigitValue, we need
//to send the Encoder a String character with the numeric value in x.
alphaNumReverseIndex[base45DigitValue] = (char) x;
}
}
}
/*
* The storage capacity of one digit in the number system; i.e. the maximum
* possible number of distinct values which can be stored in 1 logical digit
*/
public static final int QR_PAYLOAD_NUMERIC_BASE = NUM_DISTINCT_ALPHANUM_VALUES;
/*
* We can't use all 8 bytes, because the Long is signed, and the conversion math
* requires consistently positive values. If we populated all 8 bytes, then the
* last byte has the potential to contaminate the sign bit, and break the
* conversion math. So, we only use the lower 7 bytes, and avoid this problem.
*/
public static final int LONG_USABLE_BYTES = Long.BYTES - 1;
//The following mapping was determined by brute-forcing -1 Long (all bits 1), and compressing to base45 until it hit zero.
public static final int[] BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION = new int[] {0,2,3,5,6,8,9,11,12};
public static final int NUM_BASE45_DIGITS_PER_LONG = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION[LONG_USABLE_BYTES];
public static final Map<Integer, Integer> BASE45_TO_BINARY_DIGIT_COUNT_CONVERSION = new HashMap<>();
static {
//Build Reverse Lookup
int len = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION.length;
for (int x=0; x<len; x++) {
int numB45Digits = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION[x];
BASE45_TO_BINARY_DIGIT_COUNT_CONVERSION.put(numB45Digits, x);
}
}
public static String encodeToBase45QrPayload(final byte[] inputData) throws IOException {
return encodeToBase45QrPayload(new ByteArrayInputStream(inputData));
}
public static String encodeToBase45QrPayload(final InputStream in) throws IOException {
//Init conversion state vars
final StringBuilder strOut = new StringBuilder();
int data;
long buf = 0;
// Process all input data in chunks of size LONG.BYTES, this allows for economies of scale
// so we can process more digits of arbitrary size before we hit the wall of the binary
// chunk size in a power of 2, and have to transmit a sub-optimal chunk of the "crumbs"
// left over; i.e. the slack space between where the multiples of QR_PAYLOAD_NUMERIC_BASE
// and the powers of 2 don't quite line up.
while(in.available() > 0) {
//Fill buffer
int numBytesStored = 0;
while (numBytesStored < LONG_USABLE_BYTES && in.available() > 0) {
//Read next byte
data = in.read();
//Push byte into buffer
buf = (buf << 8) | data; //8 bits per byte
//Increment
numBytesStored++;
}
//Write out in lower base
final StringBuilder outputChunkBuffer = new StringBuilder();
final int numBase45Digits = BINARY_TO_BASE45_DIGIT_COUNT_CONVERSION[numBytesStored];
int numB45DigitsProcessed = 0;
while(numB45DigitsProcessed < numBase45Digits) {
//Chunk out a digit
final byte digit = (byte) (buf % QR_PAYLOAD_NUMERIC_BASE);
//Drop digit data from buffer
buf = buf / QR_PAYLOAD_NUMERIC_BASE;
//Write Digit
outputChunkBuffer.append(alphaNumReverseIndex[(int) digit]);
//Track output digits
numB45DigitsProcessed++;
}
/*
* The way this code works, the processing output results in a First-In-Last-Out digit
* reversal. So, we need to buffer the chunk output, and feed it to the OutputStream
* backwards to correct this.
*
* We could probably get away with writing the bytes out in inverted order, and then
* flipping them back on the decode side, but just to be safe, I'm always keeping
* them in the proper order.
*/
strOut.append(outputChunkBuffer.reverse().toString());
}
//Return
return strOut.toString();
}
public static byte[] decodeBase45QrPayload(final String inputStr) throws IOException {
//Prep for InputStream
final byte[] buf = inputStr.getBytes();//Use the default encoding (the same encoding that the 'char' primitive uses)
return decodeBase45QrPayload(new ByteArrayInputStream(buf));
}
public static byte[] decodeBase45QrPayload(final InputStream in) throws IOException {
//Init conversion state vars
final ByteArrayOutputStream out = new ByteArrayOutputStream();
int data;
long buf = 0;
int x=0;
// Process all input data in chunks of size LONG.BYTES, this allows for economies of scale
// so we can process more digits of arbitrary size before we hit the wall of the binary
// chunk size in a power of 2, and have to transmit a sub-optimal chunk of the "crumbs"
// left over; i.e. the slack space between where the multiples of QR_PAYLOAD_NUMERIC_BASE
// and the powers of 2 don't quite line up.
while(in.available() > 0) {
//Convert & Fill Buffer
int numB45Digits = 0;
while (numB45Digits < NUM_BASE45_DIGITS_PER_LONG && in.available() > 0) {
//Read in next char
char c = (char) in.read();
//Translate back through lookup table
int digit = ALPHANUMERIC_TABLE[(int) c];
//Shift buffer up one digit to make room
buf *= QR_PAYLOAD_NUMERIC_BASE;
//Append next digit
buf += digit;
//Increment
numB45Digits++;
}
//Write out in higher base
final LinkedList<Byte> outputChunkBuffer = new LinkedList<>();
final int numBytes = BASE45_TO_BINARY_DIGIT_COUNT_CONVERSION.get(numB45Digits);
int numBytesProcessed = 0;
while(numBytesProcessed < numBytes) {
//Chunk out 1 byte
final byte chunk = (byte) buf;
//Shift buffer to next byte
buf = buf >> 8; //8 bits per byte
//Write byte to output
//
//Again, we need to invert the order of the bytes, so as we chunk them off, push
//them onto a FILO stack; inverting their order.
outputChunkBuffer.push(chunk);
//Increment
numBytesProcessed++;
}
//Write chunk buffer to output stream (in reverse order)
while (outputChunkBuffer.size() > 0) {
out.write(outputChunkBuffer.pop());
}
}
//Return
out.flush();
out.close();
return out.toByteArray();
}
}
Just at a glance, the qr formats are different. I'd compare the qr formats to see if it's a problem of error correction or encoding or something else.
It turned out that ZXing is just crap, and ZBar does some weird stuff with the data (converting it to UTF-8 for example). I managed to get it to output the raw data including null bytes though. Here is a patch for the best Android ZBar library I found, that has now been merged.
I used System.Convert.ToBase64String to convert the supplied sample byte array into a Base64-encoded string, then I used ZXing to create a QRCode image.
Next I called ZXing to read the string back from the generated QRCode, and then called System.Convert.FromBase64String to convert the string back into a byte array.
I confirm that the data completed the round trip successfully.
The informational RFC 9285 - The Base45 Data Encoding document describing the optimal scheme for storing binary data within the constraints of QR Alphanumeric Mode was recently published by the IETF.
(one positive side-effect of ongoing standardization work surrounding Health Certificate QR-codes)

Does it make sense to store byte values in Map or it will still use 4 bytes?

In Java in-memory there is no difference between byte or int - both will be represented as 4 bytes.
Does for Chronicle Map the difference exist, i.e. does Chronicle Map store byte values as 8 bits or still use 32?
Same question if byte is an object property.
In primitive map implementations (fastutil, koloboke, gs, hppc) byte values are implemented as a separate byte[] array, so they actually take only 1 byte. If a byte is a field of another on-heap Java object (which is a Map value), indeed, the object size is rounded up to 8-byte boundary, so a single byte field could "take" 8 bytes. But more often, it "takes" 0 bytes, because the field is placed in the already existing alignment holes.
For Chronicle Map, a value could freely be 1 byte in size. (And even 0 bytes, this is how ChronicleSet is currently implmeneted -- a ChronicleMap with 0-byte dummy values.) This is true for all Chronicle Map versions (2, 3).
Edit -- answer to the comment.
If you have a constantly sized structure e. g. 6 byte fields, easiest and efficient way - to use data value generation mechanishm:
interface MyValue {
byte getA(); void setA(byte a);
byte getB(); void setB(byte b);
byte getC(); void setC(byte c);
byte getD(); void setD(byte d);
byte getE(); void setE(byte e);
byte getF(); void setF(byte f);
}
map = ChronicleMapBuilder.of(Key.class, MyValue.class).entries(1000).create();
// Chronicle Map 2 syntax
MyValue value = DataValueClasses.newDirectReference(MyValue.class);
try (Closeable handle = map.getUsingLocked(key, value)) {
// access the value here
System.out.println(value);
}
// Chronicle Map 3 syntax
try (ExternalMapQueryContext<Key, MyValue, ?> q = map.queryContext(key)) {
// if not sure the key is present in the map, check q.entry() != null
MyValue value = q.entry().value().get();
// access the value here
System.out.println(value);
}
It will take exactly 6 bytes per value.
I think I know the response. At least at the version 2.3.8 offheap value will be 1 byte for a byte (work done in SerializationBuilder class).

Get four 16bit numbers from a 64bit hex value

I have been through these related questions:
How to convert numbers between hexadecimal and decimal in C#?
How to Convert 64bit Long Data Type to 16bit Data Type
Way to get value of this hex number
But I did not get an answer probably because I do not understand 64bit or 16bit values.
I had posted a question on Picasa and face detection, to use the face detection that Picasa does to get individual pics from a photo containing many pictures. Automatic Face detection using API
In an answer #Joel Martinez linked to an answer on picasa help which said:
The number encased in rect64() is a 64-bit hexadecimal number.
Break that up into four 16-bit numbers.
Divide each by the maximum unsigned 16-bit number (65535) and you'll have four
numbers between 0 and 1.
the full text
#oedious wrote:- This is going to be
somewhat technical, so hang on. * The
number encased in rect64() is a 64-bit
hexadecimal number. * Break that up
into four 16-bit numbers. * Divide
each by the maximum unsigned 16-bit
number (65535) and you'll have four
numbers between 0 and 1. * The four
numbers remaining give you relative
coordinates for the face rectangle:
(left, top, right, bottom). * If you
want to end up with absolute
coordinates, multiple the left and
right by the image width and the top
and bottom by the image height.
A sample picasa.ini file:
[1.jpg]
backuphash=65527
faces=rect64(5520c092dfb2f8d),615eec1bb18bdec5;rect64(dcc2ccf1fd63e93e),bc209d92a3388dc3;rect64(52524b7c785e6cf6),242908faa5044cb3
crop=rect64(0)
How do I get the 4 numbers from the 64 bit hex?
I am sorry people, currently I do not understand the answers. I guess I will have to learn some C++ (I am a PHP & Java Web Developer with weakness in Math) before I can jump in and write a something which will cut up an image into multiple images with the help of some co-ordinates. I am looking into CodeLab and creating plugins for Paint.net too
If you want basics, say you have this hexadecimal number:
4444333322221111
We split it into your 4 parts on paper, so all that's left is to extract them. This involves using a ffff mask to block out everything else besides our number (f masks nothing, 0 masks everything) and sliding it over each part. So we have:
part 1: 4444333322221111 & ffff = 1111
part 2: 4444333322221111 & ffff0000 = 22220000
part 3: 4444333322221111 & ffff00000000 = 333300000000
part 4: 4444333322221111 & ffff000000000000 = 4444000000000000
All that's left is to remove the 0's at the end. All in all, in C, you'd write this as:
int GetPart(int64 pack, int n) // where you define int64 as whatever your platform uses
{ // __int64 in msvc
return (pack & (0xffff << (16*n)) >> (16*n);
}
So basically, you calculate the mask as 0xffff (2 bytes) moved to the right 16*n bits (0 for the first, 16 for the 2nd, 32 for the 3rd and 48 for the 4th), apply it over the number to mask out everything but the part we're interested in, then shift the result back 16*n bits to clear out those 0's at the end.
Some additional reading: Bitwise operators in C.
Hope that helps!
Here is the algorithm:
The remainder of the division by 0x10000 (65536) will give you the first number.
Take the result then divide by 0x10000 (65536) again, the remainder will give you the second number.
Take the result the divide by 0x10000 (65536) again, the remainder will give you the third number.
The result is the fourth number.
It depends on your programming language - in C# i.e. you can use the BitConverter class, which allows you to extract a number based on the byte position within a byte array.
UInt64 largeHexNumber = 420404334;
byte[] hexData = BitConverter.GetBytes(largeHexNumber);
UInt16 firstValue = BitConverter.ToUInt16(hexData, 0);
UInt16 secondValue = BitConverter.ToUInt16(hexData, 2);
UInt16 thirdValue = BitConverter.ToUInt16(hexData, 4);
UInt16 forthValue = BitConverter.ToUInt16(hexData, 6);
It depends on the language. For the C-family of languages, it can be done like this (in C#):
UInt64 number = 0x4444333322221111;
//to get the ones, use a mask
// 0x4444333322221111
const UInt64 mask1 = 0xFFFF;
UInt16 part1 = (UInt16)(number & mask1);
//to get the twos, use a mask then shift
// 0x4444333322221111
const UInt64 mask2 = 0xFFFF0000;
UInt16 part2 = (UInt16)((number & mask2) >> 16);
//etc.
// 0x4444333322221111
const UInt64 mask3 = 0xFFFF00000000;
UInt16 part3 = (UInt16)((number & mask3) >> 32);
// 0x4444333322221111
const UInt64 mask4 = 0xFFFF000000000000;
UInt16 part4 = (UInt16)((number & mask4) >> 48);
What I think you are being asked to do is take the 64 bits of data you have and treat it like 4 16-bit integers. From there you are taking the 16-bit values and converting them to percentages. Those percentages, when multiplied to the image height/width, give you 4 coordinates.
How you do this depends on the language you're programming in.
I needed to convert the crop=rect64() values from picasa.ini file.
I created the following ruby method with the above information.
def coordinates(hex_num)
[
hex_num.divmod(65536)[1],
hex_num.divmod(65536)[0].divmod(65536)[1],
hex_num.divmod(65536)[0].divmod(65536)[0].divmod(65536)[1],
hex_num.divmod(65536)[0].divmod(65536)[0].divmod(65536)[0].divmod(65536)[1]
].reverse
end
It works, but I needed to add the .reverse method on the array to achieve the desired result.

Resources