Is there a way to connect to a DBF and work on it without loading the file completely into memory?
I understand that the foreign package can be used to read DBFs, but this method loads the file into memory. This is problematic if the DBF in question is heavy in terms of file size. I'm aware of solutions that enable loading of heavy files into memory, but any solution that connects to the file and makes changes to it without loading it into memory is welcome.
Related
The environment is:
R: 3.6.1
readxl version: ‘1.3.1’
When I close the Excel program, read_excel takes a second or 2, but when I have the file opened in Excel, then read_excel in R can take a few minutes.
I wonder why was that?
Some programs, like Excel, put access restrictions on files while the files are open. This prevents accidental conflicts from external changes to the same file while it is open.
I don't know why specifically it would affect other tools from reading the file and why the effect would manifest as slower speed instead of complete inability. Maybe Excel is trying to monitor the access to the file and compare it to the content it has loaded.
I'm loading an R rds file into Julia with
using RData
objs = load(rds, convert=true)
The original rds file is ~3GB. When I run the load function about, the memory spikes to ~40GB.
Any ideas what's going on?
The rds files are actually compressed using gzip. Try unzipping your file and see how big it actually is (on Windows you could use 7-zip for that). The compression level for a dataframe easily could be around 80-90% so your numbers look fine.
I would like to set my temporary directory using the scratch space of the cluster. I have tried various methods and this one: How to change directory for temporary files - problems with huge temporary raster files but nothing works.
I have to read a large file (12 GB) in R and run some code using it.
I would like to read the file in this way:
library(data.table)
mydata<-fread("path/file")
But first I believe it is necessary to set the temporary directory as scratch/ otherwise the job has been killed.
Feel free to suggest any other approach.
I'm using Qt with phonon to play some mp3 files. The problem is that I need multiple mp3 files running together and they are not playing in a synchronized fashion, especially when I order to seek or something.
I've noticed that from the hard drive synchronization is better than from an USB drive. It seems that the program doesn't load the whole file into memory. Since I need to put this program on a USB drive, is there any way to allocate a file into memory and then play from that?
If your concern is reading from the filesystem, maybe you can just cache your sound files into QBuffer objects ahead of time, and then use them in the Phonon::MediaSource(QIODevice * ioDevice)
That way you are no longer depending on the filesystem to maintain stable IO. Its in memory like you wanted.
I'm using PHPExcel 1.6.7 on wamp.
I'm trying to load a big xlsm file of ~2000kb (~2.0mb)
At first, php complained of the time the script takes to load,
then I changed that time in php.ini, then it complained of the memory size it consumes, again I increased that parameter in php.ini, finally I'm standing with Maxinum Execution Time ~ 5minuts and Memory Limit ~ 400mb and it's steel cannot be loaded.
Is there any way to optimise the loading process meaningfully? something like telling it not to load styles or pictures or only load text?
(Do you know how ASP.NET loading excel files? would it be the same?)
Version 1.6.7 is a pretty old version of PHPExcel: the latest is 1.7.6 which allows options for caching cell data outside of PHP memory (either in an external cache like memcache, wincache, apc; or to disk) or in a compressed form in PHP memory (which reduces the overall memory usage). There are also options to load only the cell data rather than the formatting. All of this is fully described in the PHPExcel manual.
Some additional techniques are also descibed in this thread
Note that xlsm (Excel Macro) files aren't officially supported by PHPExcel