Reverse engineering a file encryption (most likely XOR) - encryption

Question

Im trying to reverse engineer a file format which is encrypted. Most likely it uses an XOR encryption. I can create encrypted files with known plaintext, which I analyzed:
enc 71 8d 7e 84 29 20 b8 cb 6c ed bb 8a 62 a1
dec 74 68 69 73 20 69 73 20 61 20 74 65 73 74
xor 05 e5 17 f7 09 49 cb eb 0d cd cf ef 11 d5
txt t h i s i s a t e s t
enc 61 ad be 84 29 20 b8 cb 6c ed bb 8a 62 a1
dec 64 68 69 73 20 69 73 20 61 20 74 65 73 74
xor 05 c5 d7 f7 09 49 cb eb 0d cd cf ef 11 d5
txt d h i s i s a t e s t
enc 62 a5 ae a4 e9 a0 b8 cb 6c ed bb 8a 62 a1
dec 67 68 69 73 20 69 73 20 61 20 74 65 73 74
xor 05 cd c7 d7 c9 c9 cb eb 0d cd cf ef 11 d5
txt g h i s i s a t e s t
It is obvious that the original text is part of the encryption. The first byte of the key is always 05. The second byte of the key can be calculated like this:
(enc1 + dec1) OR xor1
The rather low entropy of the key implies a similar rule for the other key-bytes.
Any ideas?

Answer 1

You almost got it!
The key's byte at the m position is given by :
km = [(en + dn) ^ kn] | secret
where :
en is the previous encrypted byte
dn is the previous plain text byte
kn is the previous key byte (k0 = 5)
secret is an arbitrary number starting at 5 and incremented by 2 every two turns
^ is the xor operator
| is the or operator
A simple C# key generator :
namespace Sample.CustomEncrypt {
using System.Collections.Generic;
using System.Text;
class Program {
static void Main() {
var key1 = GenerateKey("this is a test");
var key2 = GenerateKey("dhis is a test");
var key3 = GenerateKey("ghis is a test");
}
public static byte[] GenerateKey(string input) {
var plain = Encoding.UTF8.GetBytes(input);
var secret = 5;
var key = new List<byte> {
0x05
};
for (var i = 0; i < plain.Length - 1; i++) {
var dn = plain[i];
var kn = key[i];
var en = (byte)(dn ^ kn);
var km = (byte)(((dn + en) ^ kn) | secret);
key.Add(km);
if (i % 2 == 0) {
secret += 2;
}
}
return key.ToArray();
}
}
}
PS: As pointed out by Eugene you should post on Reverse Engineering or Cryptography next time.