I tried to use aes-256-ebc encryption with Openssl, but it seems my output is not what I expected. The key and test vectors I used are taken from http://www.inconteam.com/software-development/41-encryption/55-aes-test-vectors#aes-ecb-256
My input is the following:
openssl enc -aes-256-ecb -in in.bin -nosalt -out out.bin -kfile k.bin -nopad
For example, I'm using 6bc1bee22e409f96e93d7e117393172a as test vector.
My out.bin is:
BD E8 25 14 C9 30 E8 86 CA B7 55 93 D7 B3 AB F1
instead of:
f3eed1bdb5d2a03c064b5a7e3db181f8
My question is if I'm using Openssl the wrong way or what could have caused this mistake?
Oh well, time to test my command line fu. Didn't know I still had any :)
echo "6bc1bee22e409f96e93d7e117393172a" | perl -pe 's/([0-9a-f]{2})/chr hex $1/gie' | openssl enc -e -aes-256-ecb -K `cat key.hex` | xxd -p
Related
I read that AES 128 CTR mode should work on blocks of 16 bytes (128bit), like CBC mode.
indeed, if I try with openssl to encode 18 bytes plaintext with :
max#jarvis:~$ printf 0123456789abcdefgh | openssl enc -e -nopad -nosalt -aes-128-cbc -K 00000000000000000000000000000000 -iv 00000000000000000000000000000000 | hd
bad decrypt
140670739715200:error:0607F08A:digital envelope routines:EVP_EncryptFinal_ex:data not multiple of block length:../crypto/evp/evp_enc.c:425:
00000000 14 f5 fe 74 69 66 f2 92 65 1c 22 88 bb ff 46 09 |...tif..e."...F.|
00000010
max#jarvis:~$ printf 0123456789abcdefgh | openssl enc -e -nosalt -aes-128-cbc -K 00000000000000000000000000000000 -iv 00000000000000000000000000000000 | hd
00000000 14 f5 fe 74 69 66 f2 92 65 1c 22 88 bb ff 46 09 |...tif..e."...F.|
00000010 c2 ae b2 99 18 cd 6e ee 55 92 77 d9 e8 f3 1f bf |......n.U.w.....|
00000020
the ciphertext is 16 or 32 bytes, depending on the presence of -nopad parameter.
but if I try CTR:
max#jarvis:~$ printf 0123456789abcdefgh | openssl enc -e -nopad -nosalt -aes-128-ctr -K 00000000000000000000000000000000 -iv 00000000000000000000000000000000 | hd
00000000 56 d8 79 e7 db bf 1a 0c b0 75 9b 3b a9 50 4e 48 |V.y......u.;.PNH|
00000010 3f 8a |?.|
00000012
max#jarvis:~$ printf 0123456789abcdefgh | openssl enc -e -nosalt -aes-128-ctr -K 00000000000000000000000000000000 -iv 00000000000000000000000000000000 | hd
00000000 56 d8 79 e7 db bf 1a 0c b0 75 9b 3b a9 50 4e 48 |V.y......u.;.PNH|
00000010 3f 8a |?.|
00000012
the ciphertext is 18 bytes, like plaintext, in each case.
I cannot figure out why
CTR mode makes a block cipher (like AES) behave as a stream cipher. The results you are getting are expected - the size of the input data will equal the size of the output data.
CBC operates block by block, which is what mandates a padding scheme. CTR passes blocks of a counter value to AES, still honoring the block size, but then takes the output block and XORs it with the plaintext.
In your case above, two full 128-bit blocks will have been passed through AES to produce the key stream, but only 128 + 16 bits (18 bytes) of that key stream are used and XORed with your plaintext, producing a ciphertext of equal length.
AES 128 CTR is very much NOT like AES 128 CBC, and the wikipedia page you linked to makes it very clear. Note how the cleartext is injected into the encryption block in the CBC mode, but never passes through the encryption block in the CTR mode.
CTR encryption is done by taking a nonce, a block counter (CTR!), and the key, and encrypting them to make a very-hard-to-guess block of pseudorandom data, called the keystream. The keystream is XOR-ed with the cleartext, to obtain ciphertext. The XOR is done byte-wise, so you can truncate the keystream to match the length of the cleartext. Notice that the cleartext is never subject itself to AES block cipher encryption. The block cipher is used as a one-way function to generate pseudorandom keystream. CTR can be used with a variety of keystream sources, not only with the AES encryption block.
CBC is started, for the first block, by taking the cleartext xor-ed with a random initialization vector (IV), and then actually passing it through the AES block cipher encryption. The ciphertext from the preceding block is used as the new IV for the successor block, i.e. IV is never reused.
I am using a tail function to read close to 460 log files which keep appending all at the same time. The data I am trying to read is byte separated fixed width.Please find below the command I use:
find ###Directory### -mmin -2 -type f -name FileNameString*.log | xargs tail -qf -n -1
The expected format of log files is given below:
KS0A2018020723594007G58P5CNSSHAGPRGGWS G NH 0962201803061535PEK HND C 999 9 9CC91 990C 900 99
KS0A2018020723594007G58P5CNSSHAGPRGGWS G NH 5702201803060910PEK NRT C 444 0 4 0 40 00 44
but the format I see in the output is as below:
KS0A2018020723594912V1KY7USSCNTNPRAAPI P AA 3735201802111632IAH OR3903G7YI0HKSQUNAPRAAPI P AA 1583201812241935DEN DFW P 7 7 777777777 7 7 7 7
KS0A2018020723593952G56SCKRSGKORPRGFLCNG AZ 0758201809301515FCO ICN P07100007017070010 00 7007
The tail function is distorting the way files are being read.
Any guidance in reading the format right using tail or any equivalent command will greatly help.
You need the -z option for tail.
$ find /path/to/ -mmin -2 -type f -name FileNameString*.log | xargs tail -qf -z -n -1
-z, --zero-terminated
line delimiter is NUL, not newline
Better to use, -exec for find
$ find /path/to/ -mmin -2 -type f -name "FileNameString*.log" -exec tail -qf -z -n -1 {} \+
If we do an Xargs without assigning arguments to the Xargs, there will always be distortion in the output (as it tends to distort the line formation).
So,
find ###Directory### -mmin -2 -type f -name FileNameString*.log | xargs tail -qf -n -1
will always lead to distorted output as there is no controlled way to read in or write the output.
However, if we can pass the input variables to Xargs in a controlled manner by using -I, it was functioning well. In my case,
find ###Directory### -mmin -2 -type f -name FileNameString*.log | xargs -I% tail % -qf -n -1
produced the output format I expected. However, this can be a on the slower side of execution if the variable list to be passed to Xargs is lengthy.
I'm running this on an AIX 6.1.
The intended purpose of this command is to display the following information in the following format:
GetUsedRAM:GetUsedSwap:CPU_0_System:CPU_0_User:…CPU_N_System:CPU_N_User
The command is composed of several sub commands:
echo `vmstat 1 2 | tr -s ' ' ':' | cut -d':' -f4,5,14-15 | tail -1 | sed 's/\([0-9]*:[0-9]*:\)\([0-9]*:[0-9]*\)/\1/'``mpstat -a 1 1 | tr -s ' ' '|' | head -8 | tail -4 | cut -d'|' -f 25,27 | awk -F "|" '{printf "%.0f:%.0f:",$2,$1}' | sed '$s/.$//'| sed -e "s/ \{1,\}$//"| awk '{int a[10];split($1, a,":");printf("%d:%d:%d:%d:%d:%d:%d:%d",a[0],a[1],a[2],a[3],a[4],a[5],a[6],a[7])}'`
Which I'll re format for clarity:
echo \
`vmstat 1 2 |
tr -s ' ' ':' |
cut -d':' -f4,5,14-15 |
tail -1 |
sed 's/\([0-9]*:[0-9]*:\)\([0-9]*:[0-9]*\)/\1/' \
` \
`mpstat -a 1 1 |
tr -s ' ' '|' |
head -8 |
tail -4 |
cut -d'|' -f 25,27 |
awk -F "|" '{printf "%.0f:%.0f:",$2,$1}' |
sed '$s/.$//' |
sed -e "s/ \{1,\}$//" |
awk '{int a[10];split($1, a,":");printf("%d:%d:%d:%d:%d:%d:%d:%d",a[0],a[1],a[2],a[3],a[4],a[5],a[6],a[7])}' \
`
I understand all of the tr, cut, head tail, and (roughly) vmstat/mpstat commands. The first sed is where I get lost, I've tried running the command in smaller segments and not quite sure why it seems to work as a whole but not when I truncate the command before the next tr.
I'm also not so sure on the awk command although I understand the premise vaguely, as a function allowing formatted output.
Similarly, I have a vague understanding of sed being a command allowing certain strings/characters being replaced in some file.
I'm not able to make out what this specific implementation in the above case is.
Could anyone provide some clarity or direction as to exactly what is happening at each sed and awk step within the context of the entire command?
Thanks for your help.
Simplification
This two simpler commands will get the exact same output:
# GetUsedRAM:GetUsedSwap:CPU_0_System:CPU_0_User:…CPU_N_System:CPU_N_User
# Select fields 4,5 of last line, and format with :
comm1=`vmstat 1 2 |
awk '$4~/[0-9]/{avm=$4;fre=$5} END{printf "%s:%s",avm,fre}'
`
# Select fields 27 (sy) and 25 (us) for four cpu, print as decimal.
comm2=`mpstat -A 1 1 |
awk -v firstline=6 -v cpus=4 '
BEGIN{start=firstline-1; end=firstline+cpus;}
NR>start && NR<end {printf( ":%d:%d", $27,$25)}'
`
echo "${comm1}${comm2}"
Description.
Description of original commands
The whole command is the concatenation of two commands.
The first command:
The output of the vmstat is shown in this link.
The columns 4 and 5 are 'avm' and 'fre'. The output in columns 14 and 15,
seem to be 'us' (user) and 'sy' (system). And I say seem as no output
from the user is available to confirm.
The first command
`vmstat 1 2 | # Execute the command vmstat.
tr -s ' ' ':' | # convert all spaces to colon (:).
cut -d':' -f4,5,14-15 | # select fields 4,5,14,and 15
tail -1 | # select last line.
sed 's/\([0-9]*:[0-9]*:\)\([0-9]*:[0-9]*\)/\1/' \ # See below.
`
The sed command selects inside braces all digits [0-9]* before a colon
repeated twice. And then again (without the last colon). That's the whole
string in two parts: « (dd:dd:)(dd:dd) » (d means digit).
And finally, it replaces such whole string by what was selected inside
the first braces /\1/.
All this complexity just removes fields 14 and 15 as selected by cut.
A simpler command with exactly the same output is:
Select fields 4,5 of last line, and format with (:).
`vmstat 1 2 | awk '
$4~/[0-9]/{avm=$4;fre=$5} END{printf "%s:%s:",avm,fre}'
`
The second command:
The output of mpstat -A is similar to this one from Linux.
And also similar to this AIX mpstat -d output.
However, the exact output of AIX 6.1 for mpstat -a (ALL) on the computer
used could have several variations. Anyway, guided by the intended final
output desired: CPU_0_System:CPU_0_User:…CPU_N_System:CPU_N_User.
It seems that the columns to be selected should be us (user) and sy
(sys) percent of time that used the cpu for all cpu in use,
which seem to be four on the computer measured.
The manual for AIX 6.1 mpstat is here.
It has a list of all the 40 columns that are presented when the option
-a ALL is used:
CPU min maj mpcs mpcr dev soft dec ph cs ics bound rq push
S3pull S3grd S0rd S1rd S2rd S3rd S4rd S5rd S3hrd S4hrd S5hrd
sysc us sy wa id pc %ec ilcs vlcs lcs %idon %bdon %istol %bstol %nsp
us and sy are listed as the fields 27 and 28, however the command presented
by the user selects fields number 25 and 27. Close but not the same. The
only way to confirm would be to receive the output of the command from the user.
For testing I will be using the output of mpstat 5 1 from here.
# mpstat 5 1
System configuration: lcpu=4 ent=1.0 mode=Uncapped
cpu min maj mpc int cs ics rq mig lpa sysc us sy wt id pc %ec lcs
0 4940 0 1 632 685 268 0 320 100 263924 42 55 0 4 0.57 35.1 277
1 990 0 3 1387 2234 805 0 684 100 130290 28 47 0 25 0.27 16.6 649
2 3943 0 2 531 663 223 0 389 100 276520 44 54 0 3 0.57 34.9 270
3 1298 0 2 1856 2742 846 0 752 100 82141 31 40 0 29 0.22 13.4 650
ALL 11171 0 8 4406 6324 2142 0 2145 100 752875 39 51 0 10 1.63 163.1 1846
The second command
`mpstat -A 1 1 | # execute command
tr -s ' ' '|' | # replace all spaces with (|).
head -8 | # select 8 first lines.
tail -4 | # select last four lines.
cut -d'|' -f 25,27 | # select fields 25 and 27
awk -F "|" '{printf "%.0f:%.0f:",$2,$1}' | # print the fields as integers.
sed '$s/.$//' | # on the last line ($), substitute the last character (.$) by nothing.
sed -e "s/ \{1,\}$//" | # remove trailing space(s).
awk '{
int a[10];
split($1, a,":");
printf("%d:%d:%d:%d:%d:%d:%d:%d",a[0],a[1],a[2],a[3],a[4],a[5],a[6],a[7])
}' \
`
About the int: For older versions of awk, calling a function without the parentheses is equivalent to call the function on $0. int is equivalent to int($0), which is not printed, nor used. The same happens to the value of a[10].
The split sets each value of the command in a[i]. Then, all values of a[i] are printed as decimals.
The equivalent, and way simpler is:
Command #2
`mpstat -A 1 1 |
awk -v firstline=6 -v cpus=4 '
BEGIN{start=firstline-1; end=firstline+cpus;}
NR>start && NR<end {printf( ":%d:%d", $27,$25)}'
`
How Do i Find Count of a particular word in Different Files in Unix:
I have: 50 file in a Directory (abc.txt, abc.txt.1,abc.txt.2, etc)
What I want: To Find number of instances of word 'Hello' in each file.
What I have used is grep -c Hello abc* | grep -v :0
It gave me result in Form of,
<<File name>> : <<count>>
I want Output to be in a form
<<Date>> <<File_Name>> <<Number of Instances of word Hello in the file>>
1-1-2001 abc.txt 23
1-1-2014 abc.txt.19 57
2-5-2015 abc.txt.49 16
You can use gnu awk >=4.0 (due to ENDFILE) to get the number.
If we know where the data comes from, I will add it.
awk '{for (i=1;i<=NF;i++) if ($i~/Hello/) a++} ENDFILE {print FILENAME,a;a=0}' abc.txt*
### Sample code for you to tweak for your needs:
touch test.txt
echo "ravi chandran marappan 30" > test.txt
echo "ramesh kumar marappan 24" >> test.txt
echo "ram lakshman marappan 22" >> test.txt
sed -e 's/ /\n/g' test.txt | sort | uniq | awk '{print "echo """,$1,
"""`grep -wc ",$1," test.txt`"}' | sh
Results:
22 -1
24 -1
30 -1
chandran -1
kumar -1
lakshman -1
marappan -3
ram -1
ramesh -1
ravi -1`
Basically I want to print the 10
most used commands that are stored in the
bash history but they still have to be proceeded
by the number that indicates when it was used;
I got this far:
history | cut -f 2 | cut -d ' ' -f 3,5 | sort -k 2 -n
Which should sort the second column of the number of occurrences from the command in that row... But it doesn't do that. I know I can head -10 the pipe at the end to take the highest ten of them, but I'm kinda stuck with the sorting part.
The 10 most used commands stored in your history:
history | sed -e 's/ *[0-9][0-9]* *//' | sort | uniq -c | sort -rn | head -10
This gives you the most used command line entries by removing the history number (sed), counting (sort | uniq -c), sorting by frequency (sort -rn) and showing only the top ten entries.
If you just want the commands alone:
history | awk '{print $2;}' | sort | uniq -c | sort -rn | head -10
Both of these strip the history number. Currently, I have no idea, how to achieve that in one line.
If you want to find the top used commands in your history file, you will have to count the instances in your history. awk can be used to do this. In the following code, the awk segment will create a hashtable with commands as the key and the number of times they appear as the value. This is printed out with the last history number for that command and sorted:
history | cut -f 2 | cut -d ' ' -f 3,5 | awk '{a[$2]++;b[$2]=$1} END{for (i in a) {print b[i], i, a[i]}}' | sort -k3 -rn | head -n 10
Output looks like:
975 cd 142
972 vim 122
990 ls 118
686 hg 90
974 mvn 51
939 bash 39
978 tac 32
958 cat 28
765 echo 27
981 exit 17
If you don't want the last column you could pipe the output through cut -d' ' -f1,2.