How to encrypt large data in C++ using RSA encryption - encryption

Question

I have used following code to encrypt string in C++ with OpenSSL and it works fine only if text is up to 256 bytes, it doesn't work for larger size:
void encrypt(char* message, int sourceSzie, char* encryptedMessage, int &destSize)
{
char err[130];
RSA *rsa_pubkey = NULL;
FILE *rsa_pub_file = fopen("pubkey_file.bin", "rb");
if(PEM_read_RSAPublicKey(rsa_pub_file, &rsa_pubkey, NULL, NULL) == NULL)
{
printf("Error reading public key \n");
}
if((destSize = RSA_public_encrypt(sourceSzie, (unsigned char*)message, (unsigned char*)encryptedMessage, rsa_pubkey, RSA_PKCS1_OAEP_PADDING)) == -1)
{
ERR_load_crypto_strings();
ERR_error_string(ERR_get_error(), err);
fprintf(stderr, "Error encrypting message: %s\n", err);
}
fclose(rsa_pub_file);
}
char msg[1024];
stcpy(msg, "A test message");
int size;
char encrypted[1024];
encrypt(msg, strlen(msg), encrypted, size);
But if size of string crosses 256, it doesn't work and generates Data too large error.
What should I do to make it work for string of any size?

Answer 1

But if size of string crosses 256, it doesn't work and generates Data too large error.
Correct. RSA encryption is defined as c = m ^ e mod n. The message cannot be larger than the modulus size n. You can get the modulus size with RSA_size().
More correctly, the limit is modulus size - padding size because messages are padded. OAEP padding size is approximately 41 bytes, so the limit is somewhere around RSA_size(rsa) - 41.
I omitted PKCS padding because its insecure. In the sources, it is #define RSA_PKCS1_PADDING_SIZE 11. See A bad couple of years for the cryptographic token industry for an approachable discussion.
What should I do to make it work for string of any size?
You would need an arbitrarily large key ;) But OpenSSL limits your key size to 16K-bits. So you can't make the keys arbitrarily large.
Plus, its hard to generate those large keys. Generation time grows with key size, and you could spend a couple of days generating large keys.
In your particular case:
char msg[1024];
...
encrypt(rsa, msg, sizeof(msg), ...);
Try generating a key that is (1024 + 64) * 8 or 8704-bits. That should handle the 1024-byte buffer with padding.
To use public key encryption in this case, you should encrypt the string with a symmetric cipher like AES. Then, encrypt the AES key with the public RSA key. That's basically how SSL/TLS and others operate.
If you use the hybrid encryption, you should choose a mode like EAX or GCM; and not CBC mode. EAX and GCM are authenticated encryption modes, and they provide both confidentiality and authenticity. With authenticated encryption, you will provide privacy and be able to detect tampering.
If you chose to use OpenSSL with AES/GCM, then see the examples of how to use it on the OpenSSL wiki at EVP Authenticated Encryption and Decryption.
There are a couple of cryptosystems that provide the service as a package. See Shoupe's Elliptic Curve Integrated Encryption Scheme (ECIES); or Abdalla, Bellare, and Rogaway's Diffie-Hellman Authenticates Encryption Schemes (DHAES). Unfortunately, OpenSSL does not provide either.
But if you are interested in ECIES or DHAES, then take a look at Crpyto++. The library offers both of them. For an example of its usage in Crypto++, see Elliptic Curve Integrated Encryption Scheme.

Answer 2

I have found the solution at link https://shanetully.com/2012/06/openssl-rsa-aes-and-c/.
The functions Crypto::aesEncrypt() and Crypto::aesDecrypt() are able to encrypt/decrypt strings of any size.