I've writing a client-side app in Go that needs to interact with a C program on the server-side. The client does an AES CFB encrypt and the server decrypts. Unfortunately the server-side has a bug with reusing an initialization vector. It tries to do 3 decrypt operations based on:-
key1, iv
key2, iv
key3, iv
Due to this issue the iv is actually modified between decrypt operations. My problem now is how to reproduce this behaviour on the client side using Go.
By inserting a Println into the encrypt function below, I can see the cfb struct which, I think, contains the modified IV for the next block but because it's a stream interface, I'm not sure how to extract it into a byte slice. Any suggestions?
Thanks
package main
import (
"fmt"
"encoding/hex"
"crypto/cipher"
"crypto/aes"
)
func encrypt_aes_cfb(plain, key, iv []byte) (encrypted []byte) {
block, err := aes.NewCipher(key)
if err != nil {
panic(err)
}
encrypted = make([]byte, len(plain))
stream := cipher.NewCFBEncrypter(block, iv)
stream.XORKeyStream(encrypted, plain)
fmt.Println(stream)
return
}
func main() {
plain := []byte("Hello world...16Hello world...32")
key := make([]byte, 32)
iv := make([]byte, 16)
enc := encrypt_aes_cfb(plain, key, iv)
fmt.Println("Key: ", hex.EncodeToString(key))
fmt.Println("IV: ", hex.EncodeToString(iv))
fmt.Println("Enc: ", hex.EncodeToString(enc))
}
Going down the path you're hinting at is a bit ugly, and prone to break when the implementation changes.
You can get the IV from the stream by:
s := reflect.Indirect(reflect.ValueOf(stream))
lastIV := s.FieldByName("next").Bytes()
But, there's an easier way! Concatenate the plain text inputs, so that the stream for the second starts with the IV from the end of the first (and so on).
Playground Example
combined := append(plain, plain2...)
encCombined := encrypt_aes_cfb(combined, key, iv)
enc := encCombined[:len(plain)]
enc2 := encCombined[len(plain):]
Related
For a program I'm making this function is ran as a goroutine in a for loop depending on how many urls are passed in (no set amount).
func makeRequest(url string, ch chan<- string, errors map[string]error){
res, err := http.Get(url)
if err != nil {
errors[url] = err
close(ch)
return
}
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
ch <- string(body)
}
The entire body of the response has to be used so ioutil.ReadAll seemed like the perfect fit but with no restriction on the amount of urls that can be passed in and the nature of ReadAll being that it's all stored in memory it's starting to feel less like the golden ticket. I'm fairly new to Go so if you do decide to answer, if you could give some explanation behind your solution it would be greatly appreciated!
One insight that I got as I learned how to use Go is that ReadAll is often inefficient for large readers, and like in your case, is subject to arbitrary input being very big and possibly leaking out memory. When I started out, I used to do JSON parsing like this:
data, err := ioutil.ReadAll(r)
if err != nil {
return err
}
json.Unmarshal(data, &v)
Then, I learned of a much more efficient way of parsing JSON, which is to simply use the Decoder type.
err := json.NewDecoder(r).Decode(&v)
if err != nil {
return err
}
Not only is this more concise, it is much more efficient, both memory-wise and time-wise:
The decoder doesn't have to allocate a huge byte slice to accommodate for the data read - it can simply reuse a tiny buffer which will be used against the Read method to get all the data and parse it. This saves a lot of time in allocations and removes stress from the GC
The JSON Decoder can start parsing data as soon as the first chunk of data comes in - it doesn't have to wait for everything to finish downloading.
Now, of course your question has nothing to do with JSON, but this example is useful to illustrate that if you can use Read directly and parse data chunks at a time, do it. Especially with HTTP requests, parsing is faster than reading/downloading, so this can lead to parsed data being almost immediately ready the moment the request body finishes arriving.
In your case, you seem not to be actually doing any handling of the data for now, so there's not much to suggest to aid you specifically. But the io.Reader and the io.Writer interfaces are the Go equivalent of UNIX pipes, and so you can use them in many different places:
Writing data to a file:
f, err := os.Create("file")
if err != nil {
return err
}
defer f.Close()
// Copy will put all the data from Body into f, without creating a huge buffer in memory
// (moves chunks at a time)
io.Copy(f, resp.Body)
Printing everything to stdout:
io.Copy(os.Stdout, resp.Body)
Pipe a response's body to a request's body:
resp, err := http.NewRequest("POST", "https://example.com", resp.Body)
In order to bound the amount of memory that you're application is using, the common approach is to read into a buffer, which should directly address your ioutil.ReadAll problem.
go's bufio package offers utilities (Scanner) which supports reading until a delimiter, or reading a line from the input, which is highly related to #Howl's question
While that is pretty much simple in go
Here is the client program:
package main
import (
"fmt"
"net/http"
)
var data []byte
func main() {
data = make([]byte, 128)
ch := make(chan string)
go makeRequest("http://localhost:8080", ch)
for v := range ch {
fmt.Println(v)
}
}
func makeRequest(url string, ch chan<- string) {
res, err := http.Get(url)
if err != nil {
close(ch)
return
}
defer res.Body.Close()
defer close(ch) //don't forget to close the channel as well
for n, err := res.Body.Read(data); err == nil; n, err = res.Body.Read(data) {
ch <- string(data[:n])
}
}
Here is the serve program:
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/", hello)
http.ListenAndServe("localhost:8080", nil)
}
func hello(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, "movie.mkv")
}
I have a program that combines multiple http responses and writes to the respective seek positions on a file. I am currently doing this by
client := new(http.Client)
req, _ := http.NewRequest("GET", os.Args[1], nil)
resp, _ := client.Do(req)
defer resp.Close()
reader, _ := ioutil.ReadAll(resp.Body) //Reads the entire response to memory
//Some func that gets the seek value someval
fs.Seek(int64(someval), 0)
fs.Write(reader)
This sometimes results in a large memory usage because of the ioutil.ReadAll.
I tried bytes.Buffer as
buf := new(bytes.Buffer)
offset, _ := buf.ReadFrom(resp.Body) //Still reads the entire response to memory.
fs.Write(buf.Bytes())
but it was still the same.
My intention was to use a buffered write to the file, then seek to the offset again, and to continue write again till the end of stream is received (and hence capturing the offset value from buf.ReadFrom). But it was also keeping everything in the memory and writing at once.
What is the best way to write a similar stream directly to the disk, without keeping the entire content in buffer?
An example to understand would be much appreciated.
Thank you.
Use io.Copy to copy the response body to the file:
resp, _ := client.Do(req)
defer resp.Close()
//Some func that gets the seek value someval
fs.Seek(int64(someval), 0)
n, err := io.Copy(fs, resp.Body)
// n is number of bytes copied
I have a problem with decryption when I try to decrypt the same byte slice again.
Example of code for clarification:
package main
import (
"fmt"
"crypto/cipher"
"crypto/des"
)
const (
// tripleKey is TripleDES key string (3x8 bytes)
tripleKey = "12345678asdfghjkzxcvbnmq"
)
var (
encrypter cipher.BlockMode
decrypter cipher.BlockMode
)
func init() {
// tripleDESChiper is chiper block based on tripleKey used for encryption/decryption
tripleDESChiper, err := des.NewTripleDESCipher([]byte(tripleKey))
if err != nil {
panic(err)
}
// iv is Initialization Vector used for encrypter/decrypter creation
ciphertext := []byte("0123456789qwerty")
iv := ciphertext[:des.BlockSize]
// create encrypter and decrypter
encrypter = cipher.NewCBCEncrypter(tripleDESChiper, iv)
decrypter = cipher.NewCBCDecrypter(tripleDESChiper, iv)
}
func main() {
message := "12345678qwertyuia12345678zxcvbnm,12345678poiuytr"
data := []byte(message)
hash := encrypt(data)
decoded1 := decrypt(hash)
decoded2 := decrypt(hash)
decoded3 := decrypt(hash)
decoded4 := decrypt(hash)
fmt.Printf("encrypted data : %x\n", data)
fmt.Printf("1 try of decryption result : %x\n", decoded1)
fmt.Printf("2 try of decryption result : %x\n", decoded2)
fmt.Printf("3 try of decryption result : %x\n", decoded3)
fmt.Printf("4 try of decryption result : %x\n", decoded4)
}
func encrypt(msg []byte) []byte {
encrypted := make([]byte, len(msg))
encrypter.CryptBlocks(encrypted, msg)
return encrypted
}
func decrypt(hash []byte) []byte {
decrypted := make([]byte, len(hash))
decrypter.CryptBlocks(decrypted, hash)
return decrypted
}
This code is also available and runnable
on the playground.
It gives the following result:
encrypted data : 313233343536373871776572747975696131323334353637387a786376626e6d2c3132333435363738706f6975797472
1 try of decryption result : 313233343536373871776572747975696131323334353637387a786376626e6d2c3132333435363738706f6975797472
2 try of decryption result : 5e66fa74456402c271776572747975696131323334353637387a786376626e6d2c3132333435363738706f6975797472
3 try of decryption result : 5e66fa74456402c271776572747975696131323334353637387a786376626e6d2c3132333435363738706f6975797472
4 try of decryption result : 5e66fa74456402c271776572747975696131323334353637387a786376626e6d2c3132333435363738706f6975797472
As you can see the first decryption works well and returns valid result,
but all other tries returns the wrong result.
The first 16 bytes of result is not as in source byte slice.
Can somebody describe what I am doing wrong?
Short version: don't reuse the decrypter object.
Longer version: You're using a cipher in CBC mode: when encrypting the data, the plaintext for block N is XOR-ed with the ciphertext for block N-1 (or the IV, on the first block). On decryption this is done in reverse.
This means that when you try and reuse your decrypter object you don't get the correct results because the state isn't correct - it is decrypting the blocks as if they were subsequent blocks in your message. A peculiarity of CBC is that an incorrect IV will only affect the first decrypted block.
I'm writing a Go script that will decrypt some legacy data that is encrypted with EVP_aes_256_cbc and an RSA public key.
In C this would be something like:
key_size = EVP_OpenInit(&ctx, EVP_aes_256_cbc(), evp_key, eklen, iv, pkey);
//...
EVP_OpenUpdate(&ctx, destination, &len_out, buffer_in, buffer_size)
//...
EVP_OpenFinal(&ctx, destination+len_out, &len_out);
I have the evp_key and iv byte array equivalents in Go, but I must confess the order of how EVP works in OpenSSL eludes me (I'm fairly competent in C, but I can't get a grasp on the process by which this decryption happens from looking at the OpenSSL source.)
In Go, I can get this far:
pKey := //rsa.PrivateKey
eklen := 32
evpKey := "// hidden 32 byte array"
iv := "// hidden 16 byte array"
c, err := aes.NewCipher(iv)
cbc := cipher.NewCBCDecrypter(c, iv)
And here's where I get lost. I have an evpKey and the pKey, but I'm not sure how to decrypt the data from here. OpenSSL uses RSA_decrypt_old or something like that, but I'm unable to track down what that actually means.
Is there a Go equivalent or do I need to bust out the much-too-expensive cgo package and roll up my sleeves?
Update (Resolution):
For anyone looking to replicate the EVP behavior in Go or just wondering how EVP works exactly, the following is breakdown.
If you know the C (or Java or whatever OpenSSL implementation) was encrypting with something like:
// pseudo-code: don't copypasta and expect amazing
EVP_PKEY_assign_RSA(pkey, public_key);
EVP_CIPHER_CTX_init(&ctx);
EVP_SealInit(&ctx, EVP_aes_256_cbc(), &evp_key, &evp_key_len, iv, &pkey, 1);
EVP_SealUpdate(&ctx, buffer_out, &encrypt_len, (unsigned char*)buffer_in, len);
EVP_SealFinal(&ctx, buffer_out+encrypt_len, &encrypt_len);
The "Seal" actually just encrypts the key with the RSA public key.
In Go to decrypt something like that:
evpKeyBytes := "// the rsa.PublicKey encoded evpKey"
evpKey, err := rsa.DecryptPKCS1v15(rand.Reader, PrivateKeyRSA, evpKeyBytes)
c, err := aes.NewCipher(evpKey)
cbc := cipher.NewCBCDecrypter(c, iv)
decryptedDataBytes := make([]bytes, 2048) // some message size
cbc.CryptBlocks(decryptedDataBytes, encryptedDataBytes)
data = string(decryptedDataBytes)
// data should have the expected decrypted result.
NewCipher expects the key not the iv, and since you're passing it a 128bit iv it works as aes128cbc.
Trying to emulate an algorithm in Go that is basically AES ECB Mode encryption.
Here's what I have so far
func Decrypt(data []byte) []byte {
cipher, err := aes.NewCipher([]byte(KEY))
if err == nil {
cipher.Decrypt(data, PKCS5Pad(data))
return data
}
return nil
}
I also have a PKCS5Padding algorithm, which is tested and working, which pads the data first. I cant find any information on how to switch the encryption mode in the Go AES package (it's definitely not in the docs).
I have this code in another language, which is how I know this algorithm isn't working quite correctly.
EDIT: Here is the method as I have interpreted from on the issue page
func AESECB(ciphertext []byte) []byte {
cipher, _ := aes.NewCipher([]byte(KEY))
fmt.Println("AESing the data")
bs := 16
if len(ciphertext)%bs != 0 {
panic("Need a multiple of the blocksize")
}
plaintext := make([]byte, len(ciphertext))
for len(plaintext) > 0 {
cipher.Decrypt(plaintext, ciphertext)
plaintext = plaintext[bs:]
ciphertext = ciphertext[bs:]
}
return plaintext
}
This is actually not returning any data, maybe I screwed something up when changing it from encripting to decripting
Electronic codebook ("ECB") is a very straightforward mode of operation. The data to be encrypted is divided into byte blocks, all having the same size. For each block, a cipher is applied, in this case AES, generating the encrypted block.
The code snippet below decrypts AES-128 data in ECB (note that the block size is 16 bytes):
package main
import (
"crypto/aes"
)
func DecryptAes128Ecb(data, key []byte) []byte {
cipher, _ := aes.NewCipher([]byte(key))
decrypted := make([]byte, len(data))
size := 16
for bs, be := 0, size; bs < len(data); bs, be = bs+size, be+size {
cipher.Decrypt(decrypted[bs:be], data[bs:be])
}
return decrypted
}
As mentioned by #OneOfOne, ECB is insecure and very easy to detect, as repeated blocks will always encrypt to the same encrypted blocks. This Crypto SE answer gives a very good explanation why.
Why? We left ECB out intentionally: it's insecure, and if needed it's
trivial to implement.
https://github.com/golang/go/issues/5597
I used your code so I feel the need to show you how I fixed it.
I am doing the cryptopals challenges for this problem in Go.
I'll walk you through the mistake since the code is mostly correct.
for len(plaintext) > 0 {
cipher.Decrypt(plaintext, ciphertext)
plaintext = plaintext[bs:]
ciphertext = ciphertext[bs:]
}
The loop does decrypt the data but does not put it anywhere. It simply shifts the two arrays along producing no output.
i := 0
plaintext := make([]byte, len(ciphertext))
finalplaintext := make([]byte, len(ciphertext))
for len(ciphertext) > 0 {
cipher.Decrypt(plaintext, ciphertext)
ciphertext = ciphertext[bs:]
decryptedBlock := plaintext[:bs]
for index, element := range decryptedBlock {
finalplaintext[(i*bs)+index] = element
}
i++
plaintext = plaintext[bs:]
}
return finalplaintext[:len(finalplaintext)-5]
What this new improvement does is store the decrypted data into a new []byte called finalplaintext. If you return that you get the data.
It's important to do it this way since the Decrypt function only works one block size at a time.
I return a slice because I suspect it's padded. I am new to cryptography and Go so anyone feel free to correct/revise this.
Ideally you want to implement the crypto/cipher#BlockMode interface. Since an official one doesn't exist, I used crypto/cipher#NewCBCEncrypter as a starting point:
package ecb
import "crypto/cipher"
type ecbEncrypter struct { cipher.Block }
func newECBEncrypter(b cipher.Block) cipher.BlockMode {
return ecbEncrypter{b}
}
func (x ecbEncrypter) BlockSize() int {
return x.Block.BlockSize()
}
func (x ecbEncrypter) CryptBlocks(dst, src []byte) {
size := x.BlockSize()
if len(src) % size != 0 {
panic("crypto/cipher: input not full blocks")
}
if len(dst) < len(src) {
panic("crypto/cipher: output smaller than input")
}
for len(src) > 0 {
x.Encrypt(dst, src)
src, dst = src[size:], dst[size:]
}
}
I was confused by a couple of things.
First i needed a aes-256 version of the above algorithm, but apparently the aes.Blocksize (which is 16) won't change when the given key has length 32. So it is enough to give a key of length 32 to make the algorithm aes-256
Second, the decrypted value still contains padding and the padding value changes depending on the length of the encrypted string. E.g. when there are 5 padding characters the padding character itself will be 5.
Here is my function which returns a string:
func DecryptAes256Ecb(hexString string, key string) string {
data, _ := hex.DecodeString(hexString)
cipher, _ := aes.NewCipher([]byte(key))
decrypted := make([]byte, len(data))
size := 16
for bs, be := 0, size; bs < len(data); bs, be = bs+size, be+size {
cipher.Decrypt(decrypted[bs:be], data[bs:be])
}
// remove the padding. The last character in the byte array is the number of padding chars
paddingSize := int(decrypted[len(decrypted)-1])
return string(decrypted[0 : len(decrypted)-paddingSize])
}