When I use images database for my asp.net web application , i found two methods !
First method is using "Image" data type , second is convert image to base64 string and store in "byte" data type .
I want to know which method is better in process and why ?
Image type is deprecated so is not an option. There is no byte type in SQL Server, there is a type called tinyint which is equivalent to a byte type but I'm pretty sure that's not what you're asking.
So here is your actual question, corrected of wrong terminology:
When I use images database for my asp.net web application , i found two methods ! First method is using VARBINARY(max) data type , second is convert image to base64 string and store in VARCHAR(max) data type.
Base64 encoding adds significant size to binary data, it converts 3 octets into 4 octets, so it adds 33% overhead. That is real estate consumed in your storage (be it database or file).
However a web application will serve media files overwhelmingly as base64 encoded responses. Having the file already encoded in storage saves CPU on the web server as it can stream the storage content directly, w/o going through base64 encoding. Not that base64 encoding is expensive (is very cheap and very much cache friendly) when done properly, using streaming semantics.
So choose your poison: save binary content for smaller disk space used, or take the 33% size penalty for faster response write and less CPU on your web tier.
For an example of properly storing and serving media files from the database see Download and Upload images from aSP.Net
As for the neverending discussion about the storage be database or file system, I point you toward To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem? and its conclusion:
The study indicates that if objects are larger than
one megabyte on average, NTFS has a clear advantage
over SQL Server. If the objects are under 256
kilobytes, the database has a clear advantage. Inside
this range, it depends on how write intensive the
workload is.
This study doe snot consider the administrative problems of an out-of database storage (consistent backup-restore, failover) but those kind of problems are only discussed in v2 of products.
Related
I am trying to write firmware code for RFID device which will have config data storage as well as the temporary storage that maybe can be read and then if convenient be removed.
I am using Arduino IDE to program this on an ESP32 Wroom32. I have tried to understand how the storage actually works, finding various resources. One being datasheet of the same, that says that there could be 4 MB of program code storage possible, and that sounds fantastic, my question is if for example I take EEPROM library and save about 214 bytes to config which will rarely be touched, where is it exactly being stored? Is it simply in NVS? I can see that the default settings show me about 1310720 Bytes of storage and I know that I can utilise other partitions as well to store more in case I ever try to have more sketch storage than 1310720 Bytes.
My question is if I am trying to store data such as config and real time data, how much would I possibly be able to store? Is there a limit? Would it cause any kind of problems if I try to use the other such partitions to write the code? Will it be only NVS that is storing that data or can I utilise the other app0, app1, spiffs etc to store extra Bytes? A lot of the resources are confusing me, here are the data that I am referring to from online 1 and 2. Any idea would help me proceed very further.
P.S. I am aware that the EEPROM library has been deprecated and I shall use either Preferences or littlefs for better management but if I am aware correctly I can still utilise them, and without much issue that will work since there is still compatibility for that. I am also curious about using inbuilt SRAM of RTC with the RTC attribute RTC_DATA_ATTR, since I hope to also utilise deep sleep mode incorporated.
My question is if I am trying to store data such as config and real time data, how much would I possibly be able to store? Is there a limit?
It depends. First on the module; there is ESP32-WROOM with 4MB flash but you could also order different flash sizes.
Then the question is: how big is your application (code)? Obviously this needs to be saved on the flash as well, reducing the total usable amount for data storage (by the size of the application). Also there is a bootloader which needs some small space as well.
Next, ESP32 is using a partition scheme. One partition is reserved for the bootloader. The rest can be divided between one or more application partitions, NVS partitions, and possibly other utility partitions (i.e. OTAData).
If you are using the OTA functions, there will be at least 3 application partitions of equal size, further reducing the total usable amount for data storage.
So the absolute upper limit of what you can store using NVS functions is the size of your NVS partition. However since it's a key-value storage, you must take into account the size of the key, which can be considerably larger than the data you store (up to 12 times for a 12 character key and a uint8 value).
So there is no way to say exactly how much data you can put into the system without knowing exactly how you're going to use it. For example, you could store one very large "blob" value that could take "up to 97.6%" of the partition size. But you could not store 10 "blob" values of 1/10 (9.76%) the size since you must take into account the keys and some flash metadata used internally.
Would it cause any kind of problems if I try to use the other such partitions to write the code?
That depends on what these partitions are used for. If you override the partition table, or bootloader, or your application code, yes there will be problems. If there is "free space" then it won't be a problem, but then you should redefine this free space as NVS space. It's nice of Espressif to provide this NVS library, dont work around it, work with it.
Using Espressif's esptool you can create custom partition tables where you could minimize the size of the application partition to just barely fit your application, and maximize the NVS partition size. This way you will get the most storage out of your device without manually implementing a filesystem. If you are using OTA, you should leave some empty room in your application partition, in case your application code grows, as it usually does.
Will it be only NVS that is storing that data or can I utilise the other app0, app1, spiffs etc to store extra Bytes?
You absolutely can, but you will destroy whatever data is on that partition. And you will have lots of work to do, because you'll have to implement all of this yourself (basically roll your own flash driver).
If you don't need OTA, you dont need app0/app1 partitions at all.
Note that SPIFFS is also a way to store data, except it's not key-value but file-based. If you dont need it, remove that partition, and fill the space with your NVS partition.
On the other hand, SPIFFS is probably a better alternative if you are really tight on flash space, since you can omit the key and do your own referencing.
I'm trying to build simple app that would stream video from camera using browser to the remote server.
For the camera access from browser I've found a wonderful WebRTC API: getUserMedia.
Now for the streaming it to the server IIUC the best way would be to use some of the WebRTC_API for transporting and then use some server side library to deal with it.
However, at first I went with a bit different approach:
I've user MediaRecorder based on the stream from camera. And then I was setting the timeslice for the MediaRecorder.start() to be few hundred Ms, e.g. 200. And I had some assumptions in wrt MediaRecorder which are not in sync with what I was observing:
I've observed weird behaviour(wrt to my assumptions about MediaRecorder):
If there was only 1 chunk uploaded to server -> it opens just fine.
If there are multiple chunks -> none of them loads correctly, they give errors: Could not determine type of stream. But then if I use ffmpeg to concat all the chunks - resulting file is fine. Same happens if I'm concatenating the blobs from MediaRecorder.ondataavailable on the client.
Thus the question:
Can the chunks in theory be independent video files? Or it is not what MediaRecorder was designed for? If it is not - then why do we even have the option to give timeslice parameter in its start() method?
Bonus question
If we're setting timeslice comparatively small, e.g. 10ms -> lots of data blobs that are sent to MediaRecorder.ondataavailable are of size 0. Where we can find some sort of guarantees/specs on the minimal timeslice that we can use, so that the data blobs are meaningful?
In the documentation there are the following:
If timeslice is not undefined, then once a minimum of timeslice milliseconds of data have been collected, or some minimum time slice imposed by the UA, whichever is greater, start gathering data into a new Blob blob, and queue a task, using the DOM manipulation task source, that fires a blob event named dataavailable at recorder with blob.
So, my guess is that it is somehow related to some data blobs being of 0 size. What does it "some minimum time slice imposed by the UA" mean?
PS
Happy to provide code if needed. But the question is not about some specific code. It is to get understanding of the assumptions behind the MediaRecorder API and why they are there.
The timeslice parameter does not allow to create independent media chunks; instead, it gives an opportunity to save data (e.g. on the filesystem, or uploaded to a server) on a regular basis, rather than holding potentially large media content in memory.
We have an application which will need to store thousands of fairly small CSV files. 100,000+ and growing annually by the same amount. Each file contains around 20-80KB of vehicle tracking data. Each data set (or file) represents a single vehicle journey.
We are currently storing this information in SQL Server, but the size of the database is getting a little unwieldy and we only ever need to access the journey data one file at time (so the need to query it in bulk or otherwise store in a relational database is not needed). The performance of the database is degrading as we add more tracks, due to the time taken to rebuild or update indexes when inserting or deleting data.
There are 3 options we are considering:
We could use the FILESTREAM feature of SQL to externalise the data into files, but I've not used this feature before. Would Filestream still result in one physical file per database object (blob)?
Alternatively, we could store the files individually on disk. There
could end being half a million of them after 3+ years. Will the
NTFS file system cope OK with this amount?
If lots of files is a problem, should we consider grouping the datasets/files into a small database (one peruser) so that each user? Is there a very lightweight database like SQLite that can store files?
One further point: the data is highly compressible. Zipping the files reduces them to only 10% of their original size. I would like to utilise compression if possible to minimise disk space used and backup size.
I have a few thoughts, and this is very subjective, so your mileage ond other readers' mileage may vary, but hopefully it will still get the ball rolling for you even if other folks want to put differing points of view...
Firstly, I have seen performance issues with folders containing too many files. One project got around this by creating 256 directories, called 00, 01, 02... fd, fe, ff and inside each one of those a further 256 directories with the same naming convention. That potentially divides your 500,000 files across 65,536 directories giving you only a few in each - if you use a good hash/random generator to spread them out. Also, the filenames are pretty short to store in your database - e.g. 32/af/file-xyz.csv. Doubtless someone will bite my head off, but I feel 10,000 files in one directory is plenty to be going on with.
Secondly, 100,000 files of 80kB amounts to 8GB of data which is really not very big these days - a small USB flash drive in fact - so I think any arguments about compression are not that valid - storage is cheap. What could be important though, is backup. If you have 500,000 files you have lots of 'inodes' to traverse and I think the statistic used to be that many backup products can only traverse 50-100 'inodes' per second - so you are going to be waiting a very long time. Depending on the downtime you can tolerate, it may be better to take the system offline and back up from the raw, block device - at say 100MB/s you can back up 8GB in 80 seconds and I can't imagine a traditional, file-based backup can get close to that. Alternatives may be a filesysten that permits snapshots and then you can backup from a snapshot. Or a mirrored filesystem which permits you to split the mirror, backup from one copy and then rejoin the mirror.
As I said, pretty subjective and I am sure others will have other ideas.
I work on an application that uses a hybrid approach, primarily because we wanted our application to be able to work (in small installations) in freebie versions of SQL Server...and the file load would have thrown us over the top quickly. We have gobs of files - tens of millions in large installations.
We considered the same scenarios you've enumerated, but what we eventually decided to do was to have a series of moderately large (2gb) memory mapped files that contain the would-be files as opaque blobs. Then, in the database, the blobs are keyed by blob-id (a sha1 hash of the uncompressed blob), and have fields for the container-file-id, offset, length, and uncompressed-length. There's also a "published" flag in the blob-referencing table. Because the hash faithfully represents the content, a blob is only ever written once. Modified files produce new hashes, and they're written to new locations in the blob store.
In our case, the blobs weren't consistently text files - in fact, they're chunks of files of all types. Big files are broken up with a rolling-hash function into roughly 64k chunks. We attempt to compress each blob with lz4 compression (which is way fast compression - and aborts quickly on effectively-incompressible data).
This approach works really well, but isn't lightly recommended. It can get complicated. For example, grooming the container files in the face of deleted content. For this, we chose to use sparse files and just tell NTFS the extents of deleted blobs. Transactional demands are more complicated.
All of the goop for db-to-blob-store is c# with a little interop for the memory-mapped files. Your scenario sounds similar, but somewhat less demanding. I suspect you could get away without the memory-mapped I/O complications.
I'm new on Java Card applications. At this moment I would like to store a hash table (dictionary) that contains the configuration of a terminal that reads this type of cards. If the hash table has values, those must be retrieved to the terminal (I think using APDU's right?) but also if there are no values, the terminal must create a "default" initial configuration.
Is it possible to do this? If it is, how? Maybe there is an applet ready for that (like Musclecard for key generation and signing) but I haven't found any.
Any advice? Thanks!
Java Card is pretty limited regarding support for data structures. It has a few basic types such as byte and short and optionally int, which is not used anywhere in the classic API. For those types you can generate two types of transient (RAM) arrays using JCSystem.makeTransientByteArray() and friends. Furthermore, the default byte[], short[] and Object[] created using new are stored in EEPROM.
The Object class in Java Card has been stripped down as well. This means that there is no such thing as hashCode(). If it was present then you would run into problems as the Java SE version of hashCode() returns an integer, which is probably not present. All defined data containers are either smart card or security related (e.g. the APDU and Key classes).
So basically, if you want to create a HashMap - the common type of dictionary on Java SE - then you will have to create it yourself. It is in that case a good idea to define a Hashable interface that classes can implement to act as a key. The structures should be generated in the right type of memory. For the kind of application you specify you probably need persistent memory, which is kind of the default for object instances created using the new key word.
Personally, I would make very sure you need a hashCode() method for your solution. It is probably easier to create an Object array and simply iterate over the elements.
Since there is no hash table in the smart card, you can store the terminal configurations in byte arrays. The smart card only stores the configuration (and optionally protect the data), and the instruction to get stored configuration or to update it shall be sent by terminal via APDU command.
Suggestion 1
Put your configuration in a Linear Fixed EF, if the card supports file system. No applet needs to be created/installed. It's the terminal job to read all records of the file to determine whether a configuration exist or not, and to write configuration into the file using standard APDU (UPDATE RECORD, READ RECORD).
NOTE:
set record length as number of terminal configuration bytes
number of records denotes the number of configurations that can be stored
you can put initial condition to indicate that the record is unused, e.g. 00...00
Suggestion 2
Create your own javacard applet. The applet must handle at least three proprietary APDUs:
Get list of terminal configurations
Update a record of terminal configuration
Delete a record of terminal configuration
NOTE:
You need to handle how to store and return the bytes between APDU format and your storage
I have got an ActiveX Control that gets an image from a fingerprint device as base64 string. The Active works great and I could transfer the returned base64 string to the server to be converted to a binary data and then to be saved to a database. I use ASP.NET as server side technology and JavaScript as client side technology. The problem is that the base64 string is tool large and it would take from 30 to 40 seconds for the string to be transferred to the server. My question is: Is there any way to compress this base64 string on client (Browser) and deflate it back on server.
If the base64 image is really a jpeg or some other compressed image format, I wouldn't expect you to be able to get much extra compression out of it in the first place. You'd also need to work out a way of posting the binary compressed data afterwards in a portable way, which may not be terribly easy. (You may be able to pretend it's a file or something like that.)
Just how large is this image? 30 seconds is a very long time...
on my linux system, using the bzip2 utility (which uses burrows-wheeler transform and then compresses), I reduce a jpg encoded in Base64 from 259.6KB to 194.5KB.
Using gzip (which uses an LZ algorithm of some variety), I reduce it to 194.4KB.
So, yes you can compress it. the question is why do you want to? It sounds as though your problems are really lying elsewhere.
Base64 is a format that is usually only used to get around technical limitaions of sending binary data. For example, a base64 string could be used in a GET request, like "website.com/page?q=BASE64ENCODED".
You have two options:
Figure out how to send/recieve binary data in a POST request, then send the data in a different form and convert it appropriately on the server.
-OR-
Since this is a fingerprint device, I assume you're using it as a password, so you actually don't have to send the raw fingerprint data every time. Just make an SHA-1 hash of it, and compare it to a stored hash on the server. This is just as secure and will take a fraction of a second to upload.
Hope I helped!