I'm exploring a database from a third-party application and I was wondering if it is possible to infer how to decode a BLOB in a SQLite database if you don't know what is stored inside the BLOB?
Is there any way or are there tools to solve this?
Is there any way or are there tools to solve this?
A BLOB is binary data. There are ways to reconstruct the data format (these reverse engineering methods are related to those you use for deciphering unknown file formats), but without further information what is stored in the binary BLOB it is rather difficult, so I can only give some vague hints:
think about: if you were the programmer to encode the data that is stored in the BLOB - how would you do it? Often the way that is used is similar
look at the first bytes of the data - often it tells what file format it could be/is (there are documentations of those "magic numbers" for many file formats available); also don't forget to look whether the data could be compressed (i. e. look for zlib header, since zlib is often used for compression)
if legal (depends on your country), it is often helpful to apply reverse engineering tools like IDA Pro or if not available a good debugger to have a look what the program does with the BLOB data after reading
If you save the BLOB to a file, you can use the Unix file command to determine what kind of data is stored in it.
use
sqlite3 db.sqlite 'select writefile('data.bin', value) from Record limit 1;'
(assuming value volumn contains type BLOB, like in IndexedDB)
then you can print contents of this file with cat data.bin
Related
The files I have been given are sample CDR files(Call Detail Records)
SGSN / GGSN data format: ASN.1 Basic Encoding Rules (BER).
The files have no extensions and I do not have a schema to work with. How can I approach this?
Vasil is correct that, to a degree, BER can be decoded without a schema. However, if the schema uses implicit tags, you won't get very far until you have blocks of data that you have no idea how to interpret. You will very likely need to either get the schema files or use a tool that has the appropriate schema definitions built-in.
If the files follow 3GPP 32.297 and 32.298, those specifications are freely available and you may be interested in https://www.3gpp.org/ftp/Specs/archive/32_series/32.298/ASN.1/
My company has a visual editor that can handle 32.297 CDR files. You can get a trial at: https://www.obj-sys.com/products/asn1ve/index.php. It comes with some CDR specs built in, so you might not need to get the schemas yourself.
To a certain extent it is possible to decode any valid BER encoded data without having the ASN.1 schema.
Try decoding the file using the unber tool from the asn1c project or this online decoder.
I have inherited a uniVerse database (link to Rocketsoftware site) and would like to know if it's possible to read/parse the underlying data files (which I believe are hash tables?) into 'R'?
I'm aware there are ODBC drivers as well as .NET libraries, but I'm interested in parsing the files in R (if possible) without these drivers?
(I've searched and seen a few topics on parsing hash tables in Java and C#, but nothing in R yet)
It's a propriety format, so unless you want to reverse engineer it and re-implement in R that isn't the path forward. Also note that it isn't a single hash-table format either, aside from the standard modulo and bucket sizes, there are several different formats you'll encounter.
If you don't want work with any of the native APIs of the database to read the data, you can issue database commands that will dump it to CSV or XML flat files. Take a look into the RetrieVe query language manuals to learn more.
I'm trying to get to the original data stored on a Micros POS.
Under the DB folder, I found over a 100 files, arranged in pairs: x.bin, x.key; y.bin, y.key etc. The file names look like table names, and each has a .key and a .bin.
After searching a lot, I got hints and rumors that the DB used by the POS is SQLite and that the files are encrypted, each with it's own key.
My question: is there a programmatic way to get at the data in those .bin files?
Bonus: is there a way to create one unencrypted SQLite file containing all tables and all data?
Thanks for your time!
Just staring at the encrypted files will not likely to do much good (unless you have experience with crypto analysis). However, if you have the whole firmware from the device, there's a simpler (IMO) way:
Find the code which works with those files (e.g. by searching for .key and .bin in files).
Reverse-engineer (disassemble/decompile) it and figure out what it does.
Reproduce the decryption step either manually or write a small program to do it.
Check if the decrypted data is SQLite format or not.
I will be creating a structure more or less of the form:
type FileState struct {
LastModified int64
Hash string
Path string
}
I want to write these values to a file and read them in on subsequent calls. My initial plan is to read them into a map and lookup values (Hash and LastModified) using the key (Path). Is there a slick way of doing this in Go?
If not, what file format can you recommend? I have read about and experimented with with some key/value file stores in previous projects, but not using Go. Right now, my requirements are probably fairly simple so a big database server system would be overkill. I just want something I can write to and read from quickly, easily, and portably (Windows, Mac, Linux). Because I have to deploy on multiple platforms I am trying to keep my non-go dependencies to a minimum.
I've considered XML, CSV, JSON. I've briefly looked at the gob package in Go and noticed a BSON package on the Go package dashboard, but I'm not sure if those apply.
My primary goal here is to get up and running quickly, which means the least amount of code I need to write along with ease of deployment.
As long as your entiere data fits in memory, you should't have a problem. Using an in-memory map and writing snapshots to disk regularly (e.g. by using the gob package) is a good idea. The Practical Go Programming talk by Andrew Gerrand uses this technique.
If you need to access those files with different programs, using a popular encoding like json or csv is probably a good idea. If you just have to access those file from within Go, I would use the excellent gob package, which has a lot of nice features.
As soon as your data becomes bigger, it's not a good idea to always write the whole database to disk on every change. Also, your data might not fit into the RAM anymore. In that case, you might want to take a look at the leveldb key-value database package by Nigel Tao, another Go developer. It's currently under active development (but not yet usable), but it will also offer some advanced features like transactions and automatic compression. Also, the read/write throughput should be quite good because of the leveldb design.
There's an ordered, key-value persistence library for the go that I wrote called gkvlite -
https://github.com/steveyen/gkvlite
JSON is very simple but makes bigger files because of the repeated variable names. XML has no advantage. You should go with CSV, which is really simple too. Your program will make less than one page.
But it depends, in fact, upon your modifications. If you make a lot of modifications and must have them stored synchronously on disk, you may need something a little more complex that a single file. If your map is mainly read-only or if you can afford to dump it on file rarely (not every second) a single csv file along an in-memory map will keep things simple and efficient.
BTW, use the csv package of go to do this.
Once I think about new software projects and current-age data uses, I cannot stand raw files anymore. they seem unnatural now.
Basically a file should contain one or more data "streams", metadata/attributes, etc.
The file should be optimized for sequential, parallel read (like mkv I think) but have reasonable performance for direct ("random") read access, possiby write access, and direct insertions.
Maybe even explicit logical data structure (inside each stream) might be useful.
mkv seems a pretty generic container format, but it seems to not completely fit the need and iirc has some multimedia-specific features in the container format itself
I see sqlite suggested as an alternative, but has an all-file locking mechanism and I just don't know how stream data gets organized (if it stays multimedia-friendly enough or if it's more single-access optimized)
Do you have any hint for me? Are there any other open, cross-platform, generic container, generic-access optimized, insertable data formats? What could I study?
How about HDF5?