I am running a programme called 'stdeconvolve', on my Spatial single-cell data. I have ~ 100,000 column (cells) and ~26K rows (genes). I am getting the "Cholmod error 'problem too large' at file". I am unable to debug it, how can I resolve this error?
> dim(pdach1.mat)
[1] 26273 100974
My codes are as follows
library(Seurat)
library(STdeconvolve)
pdac.int <- readRDS("pd_integ.rds")
## extract the counts matrix
pdac.mat <- Matrix((pdac.int#assays$Spatial#counts), sparse = TRUE)
## remove poor genes and pixels
pdac.mat <- cleanCounts(pdac.mat, min.lib.size = 100)
## filter for features in less than 100% of pixels but more than 5% of pixels
pdac.mat <- restrictCorpus(pdac.mat, removeAbove = 1.0, removeBelow = 0.05)
pdf("pdac.int.perplexity.pdf")
pdac.ldas = fitLDA(pdac.mat), Ks = 6:20, plot=T, verbose=TRUE)
dev.off()
pdac.optLDA <- optimalModel(models = pdac.ldas, opt = "min")
pdac.results <- getBetaTheta(pdac.optLDA, perc.filt = 0.05, betaScale = 1000)
#you can obtain the spatial coordinates of the pixels (if available) by doing something like:
now the scary part - The ERROR
Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
Calls: restrictCorpus ... is.data.frame -> as.matrix -> as.matrix.Matrix -> as -> asMethod
Execution halted
Related
Hello
I am practicing ShortRead package for analyzing fastq file format. A text book I am reading, write a function for writing large fastq files as below:
trim.file <- function (f1,
destination=sprintf("%s_filtered.fastq", fq)){
stream <- open(FastqStreamer(f1))
on.exit(close(stream))
repeat {
fq <- yield(stream)
if (length(fq) == 0){break}
fq <- fq[nFilter()(fq)]
fq <- trimTails(fq, 5, "A", successive = T)
fq <- fq[width(fq) > 80]
writeFastq(fq, destination, "a", compress = F)
}
}
trim.file(FastqFiles [1])
The FastqFiles [1] is the path/filename of the original fastq file that is created with this code FastqFiles <- list.files(path=fasrQDir, pattern = "*.fastq", full.names = T) and fasrQDir is the directory of my original fastq file.
When I run this code I encounter this error:
Error in h(simpleError(msg, call)) :
error in evaluating the argument 'file' in selecting a method for function 'writeFastq': no method for coercing this S4 class to a vector
In addition: Warning message:
In open.connection(con$con) :
Please let me know what is the solution.
Best wishes
I'm new at using GIS with R and I'm trying to open an ENVI file containing hyperspectral data following the suggestions from this post R how to read ENVI .hdr-file?, but I don't seem to be able to do so. I tried three different approaches but all of them failed. I also can't seem to find any other posts where my problem is described.
# install.packages("rgdal")
# install.packages("raster")
# install.packages("caTools")
library("rgdal")
library("raster")
library("caTools")
dirname <- "S:/LAB-cavender/4_Project_Folders/oakWilt/oak_wilt_image_analyses/R_input/6.15.2021 - Revisions/ENVI export/AISA/Resampled_flights"
filename <- file.path(dirname, "AISA_Flight_4_resampled")
file.exists(filename)
The first option that I tried was using file name only
x <- read.ENVI(filename)
But I got the following error message:
#Error in read.ENVI(filename) :
# read.ENVI: Could not open input file: S:/LAB-cavender/4_Project_Folders/oakWilt/oak_wilt_image_analyses/R_input/6.15.2021 - Revisions/ENVI export/AISA/Resampled_flights/AISA_Flight_4_resampled
#In addition: Warning message:
# In nRow * nCol * nBand : NAs produced by integer overflow
I tried then the second option which is using file name + header file name read using file.path
headerfile <- file.path(dirname, "AISA_Flight_4_resampled")
x <- read.ENVI(filename = filename,headerfile = headerfile)
Again, I got an error message that says:
#Error in read.ENVI(filename = filename, headerfile = headerfile) :
# read.ENVI: Could not open input header file: S:/LAB-cavender/4_Project_Folders/oakWilt/oak_wilt_image_analyses/R_input/6.15.2021 - Revisions/ENVI export/AISA/Resampled_flights/AISA_Flight_4_resampled
Finally, I tried the third option by using file name + header file name read using readLines
hdr_file <- readLines(con = "S:/LAB-cavender/4_Project_Folders/oakWilt/oak_wilt_image_analyses/R_input/6.15.2021 - Revisions/ENVI export/AISA/Resampled_flights/AISA_Flight_4_resampled.hdr")
x <- read.ENVI(filename = filename,headerfile = hdr_file)
But I got the error message:
#Error in read.ENVI(filename = filename, headerfile = hdr_file) :
# read.ENVI: Could not open input header file: ENVIdescription = { Spectrally Resampled File. Input number of bands: 63, output number of bands: 115. [Fri Jun 25 16:57:21 2021]}samples = 5187lines = 6111bands = 115header offset = 0file type = ENVI Standarddata type = 4interleave = bilsensor type = Unknownbyte order = 0map info = {UTM, 1.000, 1.000, 482828.358, 5029367.353, 7.5000000000e-001, 7.5000000000e-001, 15, North, WGS-84, units=Meters}coordinate system string = {PROJCS["UTM_Zone_15N",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-93.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]}default bands = {46,31,16}wavelength units = Nanometersdata ignore value = -9999.00000000e+000band names = { Resampled
# In addition: Warning message:
# In if (!file.exists(headerfile)) stop("read.ENVI: Could not open input header file: ", :
# the condition has length > 1 and only the first element will be used
Any help would be really appreciated!
I am trying to compare some recommender algorithms against each other but am running into some memory issues. The dataset that i am using is https://drive.google.com/open?id=0By5yrncwiz_VZUpiak5Hc2l3dkE
Following is my code:
library(recommenderlab)
library(Matrix)
Amazon <- read.csv(Path to Reviews.csv, header = TRUE,
col.names = c("ID","ProductId","UserId","HelpfulnessNumerator","HelpfulnessDenominator","Score",
"Time","Summary","Text"),
colClasses = c("NULL","character","character","NULL","NULL","integer","NULL","NULL","NULL"))
Amazon <- Amazon[,c("UserId","ProductId","Score")]
Amazon <- Amazon[!duplicated(Amazon[1:2]),] ## To get unique values
scheme <- evaluationScheme(r, method = "split", train = .7,
k = 1, given = 1 ,goodRating = 4)
algorithms <- list(
"user-based CF" = list(name="UBCF", param=list(normalize = "Z-score",
method="Cosine",
nn=50, minRating=3)),
"item-based CF" = list(name="IBCF", param=list(normalize = "Z-score"
))
)
results <- evaluate(scheme, algorithms, n=c(1, 3, 5))
I get the following errors :
UBCF run fold/sample [model time/prediction time]
1 Timing stopped at: 1.88 0 1.87
Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105
IBCF run fold/sample [model time/prediction time]
1 Timing stopped at: 4.93 0.02 4.95
Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105
Warning message:
In .local(x, method, ...) :
Recommender 'user-based CF' has failed and has been removed from the results!
Recommender 'item-based CF' has failed and has been removed from the results!
I tried to use recommenderlabrats package which i thought would solve this problem but could not install it. https://github.com/sanealytics/recommenderlabrats
It gave me some errors which i am not bale to make sense of:
c:/rbuildtools/3.3/gcc-4.6.3/bin/../lib/gcc/i686-w64- mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -llapack
collect2: ld returned 1 exit status
Then i came to this link for solving the recommenderlabrats problem but it did not work for me
Error while installing package from github in R. Error in dyn.load
Any help on how to get around the memory issue is appreciated
I am the author of recommenderlabrats. Try to install now, it should be fixed. Then use RSVD/ALS to solve. Your matrix is too big even when it's sparse for your computer.
Also, it might be a good idea to experiment with a smaller sample before spending on an AWS memory instance.
Here's a sample of my single column data set:
Lines
141,523
146,785
143,667
65,560
88,524
148,422
I read this file as a .csv file, convert it into a ts object and then plot it:
##Read the actual number of lines CSV file
Aclines <- read.csv(file.choose(), header=T, stringsAsFactors = F)
Aclinests <- ts(Aclines[,1], start = c(2013), end = c(2015), frequency = 52)
plot(Aclinests, ylab = "Actual_Lines", xlab = "Time", col = "red")
I get the following error message:
Error in plot.window(xlim, ylim, log, ...) : need finite 'ylim' values
In addition: Warning messages:
1: In xy.coords(x, NULL, log = log) : NAs introduced by coercion
2: In min(x) : no non-missing arguments to min; returning Inf
3: In max(x) : no non-missing arguments to max; returning -Inf
I thought this might be because of the "," in the columns and tried to use sapply to take care of that as advised here:
need finite 'ylim' values-error
plot(sapply(Aclinests, function(x)gsub(",",".",x)))
But I got the following error:
Error in plot(sapply(Aclinests, function(x) gsub(",", ".", x))) :
error in evaluating the argument 'x' in selecting a method for function 'plot': Error in sapply(Aclinests, function(x) gsub(",", ".", x)) :
'names' attribute [105] must be the same length as the vector [1]
Here is the head of my original and ts data set if it might help:
> head(Aclines)
Lines
1 141,523
2 146,785
3 143,667
4 65,560
5 88,524
6 148,422
> head(Aclinests)
[1] "141,523" "146,785" "143,667" "65,560" "88,524" "148,422"
Also, if I read the .csv file as:
Aclines <- read.csv(file.choose(), header=T, **stringsAsFactors = T**)
Then, I am able to plot the ts object, but head(Aclinests)gives the below output which is not consistent with my original data:
> head(Aclinests)
[1] 14 27 17 84 88 36
Please advice on how I can plot this ts object.
The simplest way to avoid this, in my case, is to remove the commas in the excel file containing the data. This can be done using simple excel commands and it worked for me.
I'm working the npreg example in the R np package documentation (by T. Hayfield, J. Racine), section 3.1 Univariate Regression.
library("np")
data("cps71")
model.par = lm(logwage~age + I(age^2),data=cps71)
summary(model.par)
#
attach(cps71)
bw = npregbw(logwage~age) # thislne not in example 3.1
model.np = npreg(logwage~age,regtype="ll", bwmethod="cv.aic",gradients="TRUE",
+ data=cps71)
This copied directly from the example, but the npreg call results in error message
*Rerun with Debug
Error in npreg.rbandwidth(txdat = txdat, tydat = tydat, bws = bws, ...) :
NAs in foreign function call (arg 15)
In addition: Warning message:
In npreg.rbandwidth(txdat = txdat, tydat = tydat, bws = bws, ...) :
NAs introduced by coercion*
The npreg R documentation indicates the first argument should be BW specificaion. I tried setting bws=1
model.np = npreg(bws=1,logwage~age,regtype="ll",
+ bwmethod="cv.aic",gradients="TRUE", data=cps71)
which gives the following error
*Error in toFrame(xdat) :
xdat must be a data frame, matrix, vector, or factor*
First time working with density estimation in R. Please suggest how to resolve these errors.