I trying to optimize my model with 30000 variables and 1700 contraints, but i got this error´s when i put some more contraints.
n<-lp ("max", f.obj, f.con, f.dir, f.rhs)$solution
Error: cannot allocate vector of size 129.9 Mb
I´m working in win 32 bit, 2gb ram.
What can i do to work and optimize my model using a large dataset?
That's a tiny machine by modern standards, and a non-tiny problem. Short answer is that you should run on a machine with a lot more RAM. Note that the problem isn't that R can't allocate 130 MB vectors in general -- it can -- it's that it's run out of memory on your specific machine.
I'd suggest running on a 64-bit instance of R 3.0 on a machine with 16 GB of RAM, and see if that helps.
You may want to look into spinning up a machine on the cloud, and using RStudio remotely, which will be a lot cheaper than buying a new computer.
Related
I'm relatively new to R and seem to be having a memory limit issue with a new laptop. When I run a large dataset of about 50K survey respondents and over 500 variables I receive the error message: Error: cannot allocate vector of size 413 Kb
I got around this issue fine on my old laptop by increasing the memory limit size via the code: memory.limit(size = 10000). Everything worked fine but on my new laptop which is faster and more powerful, the memory limit fills up very fast and will crash at size 27000 after I run about 7 models.
I have tried closing all unnecessary programs, removing all the unneeded objects in R, and clearing the garbage can: gc(). I was using latest version of R 4.14 and have now gone back to 4.04 where it worked fine on my old PC - but none of these help really.
I am running the 64bit version of R on a 64bit PC that has 8GB capacity.
Does anyone know why this might be occurring on a brand new laptop that runs faster while running slower on my 4-year old PC but atleast worked with it.
Also, how high can you set the memory limit as the manual says R can handle 8TB? And how do you permanently set a memory limit?
Thanks
Whenever I run R's base-function solve(A), with A being a large matrix, my R instance uses all 8 cores of my Linux machine (Ubuntu 18.04) to 100 % so that my whole system is slowed down.
Is there a way to tell solve() how many cores it should use?
Alternative, is it possible to tell my R instance (from within R) to never use more than say 90 % of a core?
Thanks for your help!
I have been using 64 bit Windows machine. The RStudio as well as R is also 64 bit. I am using these on an EC2 instance which is r5.4xlarge. It has 16 core and about 128G memory. If I run command memory.limit() I see 100GB. Since , in the .RProfile file I have set this memory limit. Still when I use the RScript I see only 10GB memory being in use in Task Manager.
How should I make sure R uses optimum memory so that the script is run much more faster? If I run same script on my local with 64 GB RAM the script finishes in 5 minutes with 100% CPU usage but on EC2 it finishes in 15 minutes with only 25% CPU usage. Please let me know if additional information is required.
I'm not sure that memory is the issue on this.
Since you note that the server only runs with 25% of CPU usage and 100% locally, it could that your code is parallelized locally and an not on the VM.
Another thing to look at is are you running Microsoft R Open locally? and not on the VM?
R Open uses the Intel MKL (Math Kernel library) by default. which is a much faster implementation of the BLAS libraries.
sessionInfo()
for the standard R library
other attached packages:
[1] Matrix_1.2-12
and for R Open (something like )
other attached packages:
[1] RevoUtilsMath_10.0.0
I selected the first 50,000 rows of the Homesite dataset - a current Kaggle competition. (30MB out of the full 200MB in the dataset.) I imported the .csv file and connected it to a Random Forest model. I changed one default - the RF will build 50 trees rather than 10.
If I click on any other task icon after I start the process the laptop hangs, requiring a power down - which I've never had to do before. If I don't select any other icon and just stay in RapidMiner, it hangs at 1:15 (the displayed timer).
This was my very first attempt at using RapidMiner Studio 6.5. I have the much older RapidMiner 5, which doesn't hang, but it is painfully slow as compared to r, and it doesn't even have Random Forests or many other models found in 6.5. 6.5 is also supposed to be much faster, and it is supposed to be able to use r scripts.
I performed a complete checkup on my Dell laptop. Everything passes.
I ran a complete scan with MalwareBytes.
I don't have any issues with any other software.
R builds neural networks (etc) at the expected speeds without any problems.
RapidMiner Studio 6.5 (64 Bit; Basic Edition)
Windows 8.0 (64 Bit); Intel Core I3 #2 Ghz; 6 GB Ram.
Is there any way of telling how much RAM is used per mb stored in the workspace in R?
I've got ~700 mb of stuff in the workspace and it brings my pc to a complete freeze, even though it has 4GB of ram, on Ubuntu which is a lightweight OS.
This is just data, I am just doing basic exploratory stats on it like plotting, averaging e.t.c