FFI Encryption/Decryption with LuaJit - encryption

Question

I'm trying to encrypt and decrypt using OpenSSL through FFI in LuaJIT - I've tried a lot of different variations but I'm not having a lot of luck. My code seems to return empty strings all the time.
I'm attempting to following the pattern described as part of the OpenSSL docs: https://www.openssl.org/docs/manmaster/crypto/EVP_PKEY_decrypt.html
local ffi = require "ffi"
ffi.cdef[[
EVP_PKEY_CTX *EVP_PKEY_CTX_new(EVP_PKEY *pkey, ENGINE *e);
void *malloc(size_t size);
void free(void *ptr);
int EVP_PKEY_encrypt_init(EVP_PKEY_CTX *ctx);
int EVP_PKEY_encrypt(EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen, const unsigned char *in, size_t inlen);
int EVP_PKEY_decrypt_init(EVP_PKEY_CTX *ctx);
int EVP_PKEY_decrypt(EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen, const unsigned char *in, size_t inlen);
]]
local s = "hello world"
local s_len = #s
local out_len1 = ffi.new("size_t[1]")
local ctx = ffi.C.EVP_PKEY_CTX_new(gen_key, nil)
if not ctx then
return nil
end
if ffi.C.EVP_PKEY_encrypt_init(ctx) == 0 then
return nil
end
if ffi.C.EVP_PKEY_encrypt(ctx, nil, out_len1, s, s_len) == 0 then
return nil
end
local buf = ffi.new("unsigned char[?]", out_len1[0])
if ffi.C.EVP_PKEY_encrypt(ctx, buf, out_len1, s, s_len) == 0 then
return nil
end
local s = ffi.string(buf, out_len1[0])
local s_len = #s
local out_len2 = ffi.new("size_t[1]")
if ffi.C.EVP_PKEY_decrypt_init(ctx) == 0 then
return nil
end
if ffi.C.EVP_PKEY_decrypt(ctx, nil, out_len2, s, s_len) == 0 then
return nil
end
local buf = ffi.new("unsigned char[?]", out_len2[0])
if ffi.C.EVP_PKEY_decrypt(ctx, buf, out_len2, s, s_len) == 0 then
return nil
end
return ffi.string(buf, out_len2[0])

Answer 1

I think your code has missing declarations for certain types such as EVP_PKEY_CTX, EVP_PKEY, etc. You would need to add the definition of those data types as well. Basically, ffi.cdef defines all function names that are going to be used through the FFI, so LuaJIT can parse them.
On the other hand, the call to those functions should be made through a library that actually implement those functions, such as OpenSSL, not ffi.C. The C namespace is used to access libc plus other C libraries such as libm and libdl. Example:
local ffi = require("ffi")
local ssl = ffi.load("ssl")
ffi.cdef[[
struct evp_pkey_ctx_st {
};
typedef struct evp_pkey_ctx_st EVP_PKEY_CTX;
struct evp_pkey_st {
};
typedef struct evp_pkey_st EVP_PKEY;
EVP_PKEY_CTX *EVP_PKEY_CTX_new(EVP_PKEY *key, void *b);
]]
local ctx = ssl.EVP_PKEY_CTX_new(gen_key, nil)
if ctx then
print("ctx created")
else
return nil
end

Answer 2

Answering my own question.
My original code did not strictly follow the C implementation example from OpenSSL since it only ever got the length of the encrypted data. It also never used any padding, it reused variables and wasn't split into methods and it was intentionally missing some dependancies to hide some implementation details.
The following code now works (standalone) and is better structured but it intentionally does no error checking and also will struggle with content longer than (KeyLength - 42).
To give some context this code is expecting Certificates and Keys in PEM format:
local ffi = require "ffi"
local ssl = ffi.load "ssl"
ffi.cdef[[
typedef struct bio_st BIO;
typedef struct bio_method_st BIO_METHOD;
BIO *BIO_new(BIO_METHOD *type);
BIO *BIO_new_mem_buf(void *buf, int len);
typedef struct evp_pkey_ctx_st EVP_PKEY_CTX;
typedef struct evp_pkey_st EVP_PKEY;
typedef struct engine_st ENGINE;
EVP_PKEY *EVP_PKEY_new(void);
void EVP_PKEY_free(EVP_PKEY *key);
typedef struct rsa_st RSA;
typedef int pem_password_cb(char *buf, int size, int rwflag, void *userdata);
RSA * PEM_read_bio_RSAPrivateKey(BIO *bp, RSA **rsa, pem_password_cb *cb, void *u);
int EVP_PKEY_set1_RSA(EVP_PKEY *pkey,RSA *key);
EVP_PKEY_CTX *EVP_PKEY_CTX_new(EVP_PKEY *pkey, ENGINE *e);
int EVP_PKEY_CTX_ctrl(EVP_PKEY_CTX *ctx, int keytype, int optype, int cmd, int p1, void *p2);
typedef struct x509_st X509;
X509 *PEM_read_bio_X509(BIO *bp, X509 **x, pem_password_cb *cb, void *u);
EVP_PKEY * X509_get_pubkey(X509 *x);
void X509_free(X509 *a);
int EVP_PKEY_encrypt_init(EVP_PKEY_CTX *ctx);
int EVP_PKEY_encrypt(EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen, const unsigned char *in, size_t inlen);
int EVP_PKEY_decrypt_init(EVP_PKEY_CTX *ctx);
int EVP_PKEY_decrypt(EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen, const unsigned char *in, size_t inlen);
]]
function encrypt(publicPEM, body)
local bioIn = ffi.new("unsigned char[?]", #publicPEM)
ffi.copy(bioIn, publicPEM, #publicPEM)
local bio = ffi.C.BIO_new_mem_buf(bioIn, -1)
local x509 = ffi.C.PEM_read_bio_X509(bio, nil, nil, nil)
ffi.gc(x509, ffi.C.X509_free)
local pKey = ffi.C.X509_get_pubkey(x509)
local ctx = ffi.C.EVP_PKEY_CTX_new(pKey, nil)
ffi.C.EVP_PKEY_encrypt_init(ctx)
-- Adds OEAP padding
ffi.C.EVP_PKEY_CTX_ctrl(ctx, 6, -1, 4097, 4, null)
-- Get the length
local outputLength = ffi.new("size_t[1]")
ffi.C.EVP_PKEY_encrypt(ctx, nil, outputLength, body, #body)
-- Encrypt into outputBuffer
local outputBuffer = ffi.new("unsigned char[?]", outputLength[0])
ffi.C.EVP_PKEY_encrypt(ctx, outputBuffer, outputLength, body, #body)
-- Turn it into a string
return ffi.string(outputBuffer, outputLength[0])
end
function decrypt(privatePEM, body)
local bioIn = ffi.new("unsigned char[?]", #privatePEM)
ffi.copy(bioIn, privatePEM, #privatePEM)
local bio = ffi.C.BIO_new_mem_buf(bioIn, -1)
if not bio then
return nil
end
local rsa = ffi.C.PEM_read_bio_RSAPrivateKey(bio, nil, nil, nil)
local pKey = ffi.C.EVP_PKEY_new()
ffi.C.EVP_PKEY_set1_RSA(pKey, rsa)
ctx = ffi.C.EVP_PKEY_CTX_new(pKey, nil)
ffi.C.EVP_PKEY_decrypt_init(ctx)
-- Adds OEAP padding
ffi.C.EVP_PKEY_CTX_ctrl(ctx, 6, -1, 4097, 4, null)
-- Get the length
local outputLength = ffi.new("size_t[1]")
ffi.C.EVP_PKEY_decrypt(ctx, nil, outputLength, body, #body)
-- Decrypt into outputBuffer
local outputBuffer = ffi.new("unsigned char[?]", outputLength[0])
ffi.C.EVP_PKEY_decrypt(ctx, outputBuffer, outputLength, body, #body)
-- Turn it into a string
return ffi.string(outputBuffer, outputLength[0])
end
io.write("Result: "..tostring(decrypt([[-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQCoOeFfldK3bcsun1klFb+d3egSKkfq3oFAf6n6hQ2R3TrzY3Bb
+4hYSr5LhrP/HYOvc7bxk+T3GQe6C8B/7aOYJQ+DOweKoNK90uVEQRtFO8EZ4Z7r
JfaS2rhPuX71AnfwuvNG/TZ5UFruvwUqvs2hzw57gl+IzFgAtG8rtVX/zwIDAQAB
AoGAEeGFGRndue2LqTr6yLxVD7ykjDm+RzK7XlWzhZNa6+Qt/ezV5pEH3wqiy3hX
7Yf/lUiha3Ai6Dja32WcYnyp5Lf9vvxMcdyMlv3r78N7KUccXo6qvh0dE5VdrNxH
4U5oDs5U4OXaTC3/pCgBCV9w7IxbrLvsj1yKYQ7QBOLnJTECQQDQuHo6e1C2miyI
VZv5YzTdXucpshAhpDNf65Z214e/Ww1OOFNOw9sBaGGyOVv+F9EfHC8kO4pA32S7
HOx+knRlAkEAzlUpafetWL+Ht6yguyc9yBbfeoFL6v576GpMjkIV7w1oqmiDa5ep
P71U1evgYAwH6X0ZcnPXZwZU7eOkBZdeIwJBAKG2nPUcwC+KioBjHAMAa2As/Jug
m8EE8M0bwit32HRZfpihKWK4esG/dxpYOL9JArzA4IGJJBgZPXl/8ngqzsUCQQCy
Z1xJrcgKxoC4xeCsMf/vdCeDKyzTYXsNuGu9TVLdwcBQJ9IKQ7Yp0LD7ztnQ8lYd
AvfvyE3lXMoubvgxhXH1AkBDTjC3cHtlXygZcrqviIq/lOblm2voR2YW520079Yd
h0u2xcG80J53NkBk/q0IaTOamESi1IrD2ds3xp5mZulI
-----END RSA PRIVATE KEY-----
]], encrypt([[-----BEGIN CERTIFICATE-----
MIICYTCCAcoCCQCT+Ubn23B63TANBgkqhkiG9w0BAQUFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNjbzEN
MAsGA1UEChMES29uZzEWMBQGA1UECxMNSVQgRGVwYXJ0bWVudDESMBAGA1UEAxMJ
bG9jYWxob3N0MB4XDTE2MDExODEwNDUyOVoXDTE3MDExNzEwNDUyOVowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFu
Y2lzY28xDTALBgNVBAoTBEtvbmcxFjAUBgNVBAsTDUlUIERlcGFydG1lbnQxEjAQ
BgNVBAMTCWxvY2FsaG9zdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqDnh
X5XSt23LLp9ZJRW/nd3oEipH6t6BQH+p+oUNkd0682NwW/uIWEq+S4az/x2Dr3O2
8ZPk9xkHugvAf+2jmCUPgzsHiqDSvdLlREEbRTvBGeGe6yX2ktq4T7l+9QJ38Lrz
Rv02eVBa7r8FKr7Noc8Oe4JfiMxYALRvK7VV/88CAwEAATANBgkqhkiG9w0BAQUF
AAOBgQAEC5ugqY6rOd3BbIam172OVQQwxcVx8BVfuiqX0zsFdBwTm/AvvdyJXRwo
64AqEIanvJF8Htq3Q9As6PNgHJ4eWEAZYTKlsf0PRM+d1T/uce3HY2SePuwr1Kqx
gFQjYTabjv361j8X3zB3HwrGsuED2UXxerXczszyiQNv/6BQlg==
-----END CERTIFICATE-----
]], "hello world"))).."\n")