In the structured document that I'm parsing (with pyparsing), there are fields that consist of a piece of text (possibly including newlines) where the length of this text is specified beforehand. E.g.
len: 141
txt: Some text, possibly including newlines,
with length (counting each newline as 1 character) 141,
as specified in the preceding
len: 141 field.
len: 83
txt: There are multiple pieces of text, each preceded by their length (83 in this case).
I'm looking for help on how to do this.
Related
I have a 1.2 GB .csv file on my disk. I use R's filename = read.csv(path)-function and then I check the object size via object.size(filename) and it turns out, that it's 3721MB large. Why is this difference?
A CSV file is a plain text file and might look like this:
1,2,3,4
3,2,3,2
3,4,2,1
each character (ie digit and comma) is a byte. This file is 24 bytes big (there's an invisible "new line" character at the end of each row).
When read into R each number is stored as a floating point decimal number, which is 8 bytes. The file above would then be 8*24 (values) = 96 bytes big.
It can go the other way. If the above file was instead written:
1.0000000000, 2.0000000000, 3.00000000000, 4.000000000
[etc]
then in the CSV each number is taking about 12 bytes - each digit, decimal point, command and zero takes a byte - and when read in to R would still only take 8 bytes as floating point decimal values.
How do I know exactly what characters are used for the encrypted output using jasypt? Can I force that my output does not contain certain characters or are always all ASCII characters used?
Reason I am asking is that the encrypted text is part of a file with delimiters and I would like to avoid that this delimiter is part of the encrypted text. The delimiter should also not be a hidden character, like SOH, because the file can be edited manually.
"Base64 only uses 6 bits (corresponding to 2^6 = 64 characters) to ensure encoded data is printable and humanly readable. None of the special characters available in ASCII are used. The 64 characters (hence the name Base64) are 10 digits, 26 lowercase characters, 26 uppercase characters as well as '+' and '/'."
So, looks like I can use ASCII special characters as a delimiter.
I noticed that the symbol . doesn't represent the same hexadecimal number when I tried to tune my YARA rules that I run on VirusTotal. When I tried to exclude the false positive-generating text string .sample., it would not get excluded because . converted from text representation was 2E in this case, meanwhile in the string, that was actually contained in the false positives, . represented 00.
I assume that when the files are matched, text is converted to hex, the hex string is then matched in a hexdump of a file and the whole hexdump is converted to text in the VT preview.
Then I noticed that there were actually more hexadecimal numbers that were represented as . in VirusTotal's text preview. For example, 0A, 99, 09 (screenshot).
I tried seeing the text representation of these hex numbers using an online converter (http://www.unit-conversion.info/texttools/hexadecimal/) and some of them were represented as � or a blank symbol (not a space symbol, as the number 20, but just a blank space).
So my questions are - why do different numbers seem to represent the same symbol? In addition, what do the "blank spaces" represent in a file's hexdump?
The 0A characters are line feed characters, as can be seen from the table in this doc, while the 2E characters are actual periods.
As per this answer on the same issue:
These are whitespace characters, and if included literally would mess up the ASCII table. That's why they (as well as the unprintable control characters below 32, and any binary values above 127, which aren't defined by ASCII and would need another character set to be interpreted correctly) are represented by .
Essentially, the '.' character is a catch-all for things which can't be shown properly in the table.
As for the online converter, it appears to generate characters until 7F, after which ASCII's 128 bit implementation is no longer defined and the translator provides a � symbol. Even from 00 to 7F we find the translator has issues with a few hex values including the line feed character 0A.
The ASCII table linked earlier hints at a few characters which the translator might have trouble with, such the DEL character (7F), the bell (07), and ENQ (05).
I would expect that blank spaces are whitespace characters, this should be possible to verify in the ASCII table.
I am trying to create a simple encryptation scheme for strings. Each character of the string is given another ascii value.
It entails writing ascii characters upto 246 to a simple file on disk.
I want to find out if it is safe to write these special characters to the disk or can it cause untoward results. Thanks for your help.
Edit: I am considering algorithm similar to following:
* Convert each character of string to its integer number (hence 110 for 'n' and 122 for 'z')
* Double that number (get 220 and 244)
* Convert this to character (will get extended ascii codes)
* Save these characters to file.
Is it safe to save these extended ascii characters to disk files using usual text file writing functions?
There is only a limited set of ASCII characters. There are 95 printable characters such as 'A' but also the space character. There are 33 printable characters such as Line Feed, Carriage Return, NUL but also DELETE. So you cannot use 246 characters of ASCII as there are only 128 total available. ASCII is strictly 7 bits giving you 2^7 = 128 possible values.
Even if you would use the ISO 8859 Latin character set or the Windows-1252 character set you would still have the unprintable control characters to deal with, leaving you with 256 - 33 - 5 characters or 218 characters. Windows-1252 still has 5 undefined characters.
What you can do is of course save your data as bytes. Each byte has 256 possible values (usually 0 to 255 or -128 to 127). As long as you open files as binary this pose no problem.
You can of course store as many characters in a file as you want, up to the file system or operating system limit. So I presume you didn't ask that.
I need to be able to delimit a stream of binary data. I was thinking of using something like the ASCII EOT (End of Transmission) character to do this.
However I'm a bit concerned -- how can I know for sure that the particular binary sequence used for this (0b00000100) won't appear in my own binary sequences, thus giving a false positive on delimitation?
In other words, how is binary delimiting best handled?
EDIT: ...Without using a length header. Sorry guys, should have mentioned this before.
You've got five options:
Use a delimiter character that is unlikely to occur. This runs the risk of you guessing incorrectly. I don't recommend this approach.
Use a delimiter character and an escape sequence to include the delimiter. You may need to double the escape character, depending upon what makes for easier parsing. (Think of the C \0 to include an ASCII NUL in some content.)
Use a delimiter phrase that you can determine does not occur. (Think of the mime message boundaries.)
Prepend a length field of some sort, so you know to read the following N bytes as data. This has the downside of requiring you to know this length before writing the data, which is sometimes difficult or impossible.
Use something far more complicated, like ASN.1, to completely describe all your content for you. (I don't know if I'd actually recommend this unless you can make good use of it -- ASN.1 is awkward to use in the best of circumstances, but it does allow completely unambiguous binary data interpretation.)
Usually, you wrap your binary data in a well known format, for example with a fixed header that describes the subsequent data. If you are trying to find delimeters in an unknown stream of data, usually you need an escape sequence. For example, something like HDLC, where 0x7E is the frame delimeter. Data must be encoded such that if there is 0x7E inside the data, it is replaced with 0x7D followed by an XOR of the original data. 0x7D in the data stream is similarly escaped.
If the binary records can really contain any data, try adding a length before the data instead of a marker after the data. This is sometimes called a prefix length because the length comes before the data.
Otherwise, you'd have to escape the delimiter in the byte stream (and escape the escape sequence).
You can prepend the size of the binary data before it. If you are dealing with streamed data and don't know its size beforehand, you can divide it into chunks and have each chunk begin with size field.
If you set a maximum size for a chunk, you will end up with all but the last chunk the same length which will simplify random access should you require it.
As a space-efficient and fixed-overhead alternative to prepending your data with size fields and escaping the delimiter character, the escapeless encoding can be used to trim off that delimiter character, probably together with other characters that should have special meaning, from your data.
#sarnold's answer is excellent, and here I want to share some code to illustrate it.
First here is a wrong way to do it: using a \n delimiter. Don't do it! the binary data could contain \n, and it would be mixed up with the delimiters:
import os, random
with open('test', 'wb') as f:
for i in range(100): # create 100 binary sequences of random
length = random.randint(2, 100) # length (between 2 and 100)
f.write(os.urandom(length) + b'\n') # separated with the character b"\n"
with open('test', 'rb') as f:
for i, l in enumerate(f):
print(i, l) # oops we get 123 sequences! wrong!
...
121 b"L\xb1\xa6\xf3\x05b\xc9\x1f\x17\x94'\n"
122 b'\xa4\xf6\x9f\xa5\xbc\x91\xbf\x15\xdc}\xca\x90\x8a\xb3\x8c\xe2\x07\x96<\xeft\n'
Now the right way to do it (option #4 in sarnold's answer):
import os, random
with open('test', 'wb') as f:
for i in range(100):
length = random.randint(2, 100)
f.write(length.to_bytes(2, byteorder='little')) # prepend the data with the length of the next data chunk, packed in 2 bytes
f.write(os.urandom(length))
with open('test', 'rb') as f:
i = 0
while True:
l = f.read(2) # read the length of the next chunk
if l == b'': # end of file
break
length = int.from_bytes(l, byteorder='little')
s = f.read(length)
print(i, s)
i += 1
...
98 b"\xfa6\x15CU\x99\xc4\x9f\xbe\x9b\xe6\x1e\x13\x88X\x9a\xb2\xe8\xb7(K'\xf9+X\xc4"
99 b'\xaf\xb4\x98\xe2*HInHp\xd3OxUv\xf7\xa7\x93Qf^\xe1C\x94J)'