Difficulty in downloading TCGA data - r
I am trying to download the TCGA data but I am getting this error:
Error in summarizeMaf(maf = maf, anno = clinicalData, chatty =
verbose): Tumor_Sample_Barcode column not found in provided clinical
data. Rename column containing sample names to Tumor_Sample_Barcode if
necessary.
This is my code:
library("TCGAbiolinks")
library("tidyverse")
library(maftools)
query <- GDCquery( project = "TCGA-LIHC",
data.category = "Clinical",
file.type = "xml",
legacy = FALSE)
GDCdownload(query,directory = ".")
clinical <- GDCprepare_clinic(query, clinical.info = "patient",directory = ".")
#getting the survival time of event data
survival_data <- as_tibble(clinical[,c("days_to_last_followup","days_to_death","vital_status","bcr_patient_barcode","patient_id")])
survival_data <- filter(survival_data,!is.na(days_to_last_followup)|!is.na(days_to_death)) #not both NA
survival_data <- filter(survival_data,!is.na(days_to_last_followup)|days_to_last_followup>0 &is.na(days_to_death)|days_to_death > 0 ) #ensuring positive values
survival_data <- survival_data[!duplicated(survival_data$patient_id),] #ensuring no duplicates
dim(survival_data) #should be 371
maf <- GDCquery_Maf("LIHC", pipelines = "muse")
#maf <- GDCquery_Maf("LIHC", pipelines = "somaticsniper")
#clin <- GDCquery_clinic("TCGA-LIHC","clinical")
#print(clin )
laml = read.maf(
maf,
clinicalData = clinical,
removeDuplicatedVariants = TRUE,
useAll = TRUE,
gisticAllLesionsFile = NULL,
gisticAmpGenesFile = NULL,
gisticDelGenesFile = NULL,
gisticScoresFile = NULL,
cnLevel = "all",
cnTable = NULL,
isTCGA = TRUE,
vc_nonSyn = NULL,
verbose = TRUE
)
You should have: a) loaded with library(maftools) and b) included what was printed out before that error message:
-Validating
-Silent variants: 18306
-Summarizing
--Possible FLAGS among top ten genes:
TTN
MUC16
OBSCN
FLG
-Processing clinical data
Available fields in provided annotations..
[1] "bcr_patient_barcode" "additional_studies"
[3] "tissue_source_site" "patient_id"
# snipped remaining 78 column names
Notice that the first column is not named "Tumor_Sample_Barcode", so you need to follow the helpful error message directions and rename the appropriate column which appears to be the first one:
ns. After doing so I get:
-Validating
-Silent variants: 18306
-Summarizing
--Possible FLAGS among top ten genes:
TTN
MUC16
OBSCN
FLG
-Processing clinical data
-Finished in 1.911s elapsed (2.470s cpu)
Related
Exporting Seurat Object Data by Cluster
I'm using Seurat to perform a single cell analysis and am interested in exporting the data for all cells within each of my clusters. I tried to use the below code but have had no success. My Seurat object is called Patients. I also attached a screenshot of my Seurat object. I am looking to extract all the clusters (i.e. Ductal1, Macrophage1, Macrophage2, etc...) meta.data.cluster <- unique(x = Patients#meta.data$active.ident) for(group in meta.data.cluster) { group.cells <- WhichCells(object = Patients, subset.name = "active.ident" , accept.value = group) data_to_write_out <- as.data.frame(x = as.matrix(x = Patients#raw.data[, group.cells])) write.csv(x = data_to_write_out, row.names = TRUE, file = paste0(save_dir,"/",group, "_cluster_outfile.csv")) } I am new to R and coding so any help is greatly appreciated! :)
It doesn't work because there is no active.ident column under your metadata. For example if we use an example dataset like yours and set the ident: library(Seurat) M = matrix(rnbinom(5000,mu=20,size=1),ncol=50) colnames(M) = paste0("P",1:50) rownames(M) = paste0("gene",1:100) Patients = CreateSeuratObject(M) Patients$grp = sample(c("Ductal1","Macrophage1","Macrophage2"),50,replace=TRUE) Idents(Patients) = Patients$grp You can see this line of code gives you no value: meta.data.cluster <- unique(x = Patients#meta.data$active.ident) meta.data.cluster NULL You can do: meta.data.cluster <- unique(Idents(Patients)) for(group in meta.data.cluster) { group.cells <- WhichCells(object = Patients, idents = group) data_to_write_out <- as.data.frame(GetAssayData(Patients,slot = 'counts')[,group.cells]) write.csv(data_to_write_out, row.names = TRUE, file = paste0(save_dir,"/",group, "_cluster_outfile.csv")) } Note also you can get the counts out using GetAssayData . You can subset one group and write out like this: wh <- which(Idents(Patients) =="Macrophage1" ) da = as.data.frame(GetAssayData(Patients,slot = 'counts')[,wh]) write.csv(da,...)
Parallelizing a loop with updating during each iteration
I have some R code that puts together demographic data from the Census for all of states in the US into a list object. The block-level code can take a week to run as a sequential loop since there are ~11M blocks, so I am trying to parallelize the loop over states to make it faster. I have accomplished this goal with this: states <- c("AL","AK","AZ","AR","CA","CO","CT","DE","FL","GA","HI", "ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI", "MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC", "ND","OH","OK","OR","PA","RI","SC","SD","TN","TX","UT", "VT","VA","WA","WV","WI","WY","DC","PR") library(future.apply) plan(multiprocess) ptm <- proc.time() CensusObj_block_age_sex = list() CensusObj_block_age_sex[states] <- future_lapply(states, function(s){ county <- census_geo_api(key = "XXX", state = s, geo = "county", age = TRUE, sex = TRUE) tract <- census_geo_api(key = "XXX", state = s, geo = "tract", age = TRUE, sex = TRUE) block <- census_geo_api(key = "XXX", state = s, geo = "block", age = TRUE, sex = TRUE) censusObj[[s]] <- list(state = s, age = TRUE, sex = TRUE, block = block, tract = tract, county = county) } ) However, I need to make it more robust. Sometimes there are problem with the Census API, so I would like the CensusObj to be updated at each state iteration so that I don't loose my completed data if something wrong. That way I can restart the loop over the remaining state if something does goes wrong (like if I spell "WY" as "WU") Would it be possible to accomplish this somehow? I am open to other methods of parallelization. The code above runs, but it seems to run into memory issues: Error: Failed to retrieve the value of MultisessionFuture (future_lapply-3) from cluster RichSOCKnode #3 (PID 80363 on localhost ‘localhost’). The reason reported was ‘vector memory exhausted (limit reached?)’. Post-mortem diagnostic: A process with this PID exists, which suggests that the localhost worker is still alive. I have R_MAX_VSIZE = 8Gb in my .Renviron, but I am not sure how that would get divided between the 8 cores on my machine. This all suggests that I need to store the results of each iteration rather than try to keep it all in memory, and then append the objects together at the end.
Here is a solution that uses doParallel (with the options for UNIX systems, but you can also use it on Windows, see here) and foreach that stores the results for every state separately and afterwards reads in the single files and combines them to a list. library(doParallel) library(foreach) path_results <- "my_path" ncpus = 8L registerDoParallel(cores = ncpus) states <- c("AL","AK","AZ","AR","CA","CO","CT","DE","FL","GA","HI", "ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI", "MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC", "ND","OH","OK","OR","PA","RI","SC","SD","TN","TX","UT", "VT","VA","WA","WV","WI","WY","DC","PR") results <- foreach(state = states) %dopar% { county <- census_geo_api(key = "XXX", state = state, geo = "county", age = TRUE, sex = TRUE) tract <- census_geo_api(key = "XXX", state = state, geo = "tract", age = TRUE, sex = TRUE) block <- census_geo_api(key = "XXX", state = state, geo = "block", age = TRUE, sex = TRUE) results <- list(state = state, age = TRUE, sex = TRUE, block = block, tract = tract, county = county) # store the results as rds saveRDS(results, file = paste0(path_results, "/", state, ".Rds")) # remove the results rm(county) rm(tract) rm(block) rm(results) gc() # just return a string paste0("done with ", state) } library(purrr) # combine the results to a list result_files <- list.files(path = path_results) CensusObj_block_age_sex <- set_names(result_files, states) %>% map(~ readRDS(file = paste0(path_results, "/", .x)))
You could use a tryCatch inside future_lapply to try to relaunch the calculation in case of API error, for a maximum of maxtrials. In the resulting list, you get for each calculation the number of trials and the final status, OK or Error: states <- c("AL","AK","AZ","AR","CA","CO","CT","DE","FL","GA","HI", "ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI", "MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC", "ND","OH","OK","OR","PA","RI","SC","SD","TN","TX","UT", "VT","VA","WA","WV","WI","WY","DC","PR") library(future.apply) #> Le chargement a nécessité le package : future plan(multiprocess) ptm <- proc.time() maxtrials <- 3 census_geo_api <- function(key = "XXX", state = s, geo = "county", age = TRUE, sex = TRUE) { paste(state,'-', geo) } CensusObj_block_age_sex <- future_lapply(states, function(s) { ntrials <- 1 while (ntrials <= maxtrials) { hasError <- tryCatch({ #simulate random error if (runif(1)>0.3) {error("API failed")} county <- census_geo_api(key = "XXX", state = s, geo = "county", age = TRUE, sex = TRUE) tract <- census_geo_api(key = "XXX", state = s, geo = "tract", age = TRUE, sex = TRUE) block <- census_geo_api(key = "XXX", state = s, geo = "block", age = TRUE, sex = TRUE) }, error = function(e) e) if (inherits(hasError, "error")) { ntrials <- ntrials + 1 } else { break} } if (ntrials > maxtrials) { res <- list(state = s, status = 'Error', ntrials = ntrials-1, age = NA, sex = NA, block = NA, tract = NA, county = NA) } else { res <- list(state = s, status = 'OK' , ntrials = ntrials, age = TRUE, sex = TRUE, block = block, tract = tract, county = county) } res } ) CensusObj_block_age_sex[[1]] #> $state #> [1] "AL" #> #> $status #> [1] "OK" #> #> $ntrials #> [1] 3 #> #> $age #> [1] TRUE #> #> $sex #> [1] TRUE #> #> $block #> [1] "AL - block" #> #> $tract #> [1] "AL - tract" #> #> $county #> [1] "AL - county" <sup>Created on 2020-08-19 by the [reprex package](https://reprex.tidyverse.org) (v0.3.0)</sup>
One possible solution that I have is to log the value of CensusObj to a text file i.e print the CensusObj in each iteration. The doSNOW package can be used for logging for example library(doSNOW) cl <- makeCluster(1, outfile="abc.out") registerDoSNOW(cl) states <- c("AL","AK","AZ","AR","CA","CO","CT","DE","FL","GA","HI", "ID","IL","IN","IA","KS","KY","LA","ME","MD","MA","MI", "MN","MS","MO","MT","NE","NV","NH","NJ","NM","NY","NC", "ND","OH","OK","OR","PA","RI","SC","SD","TN","TX","UT", "VT","VA","WA","WV","WI","WY","DC","PR") foreach(i=1:length(states), .combine=rbind, .inorder = TRUE) %dopar% { county <- "A" tract <- "B" block <- "C" censusObj <- data.frame(state = states[i], age = TRUE, sex = TRUE, block = block, tract = tract, county = county) # edit: print json objects to easily extract from the file cat(sprintf("%s\n",rjson::toJSON(censusObj))) } stopCluster(cl) This would log the value of censusObj in abc.out and also logs the error if program crashes but you will get the latest value of censusObj logged in abc.out. Here is the output of the last iteration from the log file: Type: EXEC {"state":"PR","age":true,"sex":true,"block":"C","tract":"B","county":"A"} Type: DONE Type:EXEC means that the iteration has started and Type:DONE means execution is completed. The result of cat will be present between these two statements of each iteration. Now, the value of CensusObj can be extracted from the log file as shown below: Lines = readLines("abc.out") results = list() for(i in Lines){ # skip processing logs created by doSNOW if(!startsWith(i, "starting") && !startsWith(i, "Type:")){ results = rlist::list.append(results, jsonlite::fromJSON(i)) } } results will contain the elements all the values printed in abc.out. > head(results, 1) [[1]] [[1]]$state [1] "AL" [[1]]$age [1] TRUE [[1]]$sex [1] TRUE [[1]]$block [1] "C" [[1]]$tract [1] "B" [[1]]$county [1] "A" It is not a very clean solution but works.
R: M3C library - Duplicate row.names error message
I am trying to run consensus clustering using M3C library in R. My dataset contains 451 samples and ~2500 genes. The row names are the ENTREZ IDs (numeric values) of the genes. I have crosschecked the dataset using "any(duplicated(colnames(MyData)))" command to make sure that there is no duplicate entries in the row names. I ran the following command to perform the consensus clustering using M3C library: res <- M3C(MyData, cores=8, seed = 123, des = annotation, removeplots = TRUE, analysistype = 'chi', doanalysis = TRUE, variable = 'class') I am getting the following error: Warning message: "non-unique values when setting 'row.names': " Error in `.rowNamesDF<-`(x, value = value): duplicate 'row.names' are not allowed Traceback: 1. M3C(MyData, cores = 8, seed = 123, des = meta, removeplots = TRUE, . analysistype = "chi", doanalysis = TRUE, variable = "class") 2. M3Creal(as.matrix(mydata), maxK = maxK, reps = repsreal, pItem = 0.8, . pFeature = 1, clusterAlg = clusteralg, distance = distance, . title = "/home/christopher/Desktop/", printres = printres, . showheatmaps = showheatmaps, printheatmaps = printheatmaps, . des = des, x1 = pacx1, x2 = pacx2, seed = seed, removeplots = removeplots, . silent = silent, doanalysis = doanalysis, analysistype = analysistype, . variable = variable, fsize = fsize, method = method) 3. `row.names<-`(`*tmp*`, value = newerdes$ID) 4. `row.names<-.data.frame`(`*tmp*`, value = newerdes$ID) 5. `.rowNamesDF<-`(x, value = value) 6. stop("duplicate 'row.names' are not allowed") Can anyone please help me to resolve the issue? Thanks
I ran the equivalent of the following using M3C: df_wide_matrix # my expression matrix any(duplicated(colnames(df_wide_matrix))) # result = FALSE M3C::M3C(df_wide_matrix, iters=2, repsref=2, repsreal=2, clusteralg="hc", objective="PAC") I ran into the exact same error message with M3C, in addition to: In addition: Warning message: non-unique values when setting 'row.names': ‘ABCDEF’, ‘ABCDGH’ I assumed the issue is caused by the fact the first four characters of each of these features are equal. I therefore temporarily changed their respective names prior to running M3C: dup_ids <- which(colnames(dissADJ) %in% c("ABCDEF", "ABCDGH")) colnames(dissADJ)[dup_ids] <- c("A", "B") M3C::M3C(df_wide_matrix, iters=2, repsref=2, repsreal=2, clusteralg="hc", objective="PAC") M3C then runs correctly. Not an ideal solution but worked for me - I've posted it as an issue: https://github.com/crj32/M3C/issues/6.
R - XGBoost: Error building DMatrix
I am having trouble using the XGBoost in R. I am reading a CSV file with my data: get_data = function() { #Loading Data path = "dados_eye.csv" data = read.csv(path) #Dividing into two groups train_porcentage = 0.05 train_lines = nrow(data)*train_porcentage train = data[1:train_lines,] test = data[train_lines:nrow(data),] rownames(train) = c(1:nrow(train)) rownames(test) = c(1:nrow(test)) return (list("test" = test, "train" = train)) } This function is Called my the main.R lista_dados = get_data() #machine = train_svm(lista_dados$train) #machine = train_rf(lista_dados$train) machine = train_xgt(lista_dados$train) The problem is here in the train_xgt train_xgt = function(train_data) { data_train = data.frame(train_data[,1:14]) label_train = data.frame(factor(train_data[,15])) print(is.data.frame(data_train)) print(is.data.frame(label_train)) dtrain = xgb.DMatrix(data_train, label=label_train) machine = xgboost(dtrain, num_class = 4 ,max.depth = 2, eta = 1, nround = 2,nthread = 2, objective = "binary:logistic") return (machine) } This is the Error: becchi#ubuntu:~/Documents/EEG_DATA/Dados_Eye$ Rscript main.R [1] TRUE [1] TRUE Error in xgb.DMatrix(data_train, label = label_train) : xgb.DMatrix: does not support to construct from list Calls: train_xgt -> xgb.DMatrix Execution halted becchi#ubuntu:~/Documents/EEG_DATA/Dados_Eye$ As you can see, they are both DataFrames. I dont know what I am doing wrong, please help!
Just convert data frame to matrix first using as.matrix() and then pass to xgb.Dmatrix().
Check if all columns have numeric data in them- I think this could be because you have some column that has data stored as factors/ characters which it won't be able to convert to a matrix. if you have factor variables, you can use one-hot encoding to convert them into dummy variables.
Try: dtrain = xgb.DMatrix(as.matrix(sapply(data_train, as.numeric)), label=label_train) instead of just: dtrain = xgb.DMatrix(data_train, label=label_train)
Mapdist: Error is.character(from)
My dataset includes a column "pickup" corresponding to the starting coordinates and a "dropoff" for the ending coordinates, of a trip. Like: pickup dropoff 40.77419,-73.872608 40.78055,-73.955042 40.7737,-73.870721 40.757007,-73.971953 I want to calculate the shortest route suggested by Google Maps, and saved the calculations in a new column. This is what I'm doing: X$GoogleDist <- mapdist(from= list(X$pickup), to = list(X$dropoff), mode = "driving" , output = "simple", messaging = FALSE, sensor = FALSE, language = "en-EN", override_limit = FALSE) Which gives me the following error: Error: is.character(from) is not TRUE
You could do library(ggmap) X <- read.table(header=TRUE, text="pickup dropoff 40.77419,-73.872608 40.78055,-73.955042 40.7737,-73.870721 40.757007,-73.971953") X <- as.data.frame(lapply(X, function(x) sapply(as.character(x), function(y) URLencode(y, T) ) ), stringsAsFactors = F) rownames(X) <- NULL res <- mapdist(from= X$pickup, to = X$dropoff, mode = "driving" , output = "simple", messaging = FALSE, sensor = FALSE, language = "en-EN", override_limit = FALSE) cbind(X, res) # pickup dropoff from to m km miles seconds minutes hours # 1 40.77419%2C-73.872608 40.78055%2C-73.955042 40.77419%2C-73.872608 40.78055%2C-73.955042 12805 12.805 7.957027 1212 20.20 0.3366667 # 2 40.7737%2C-73.870721 40.757007%2C-73.971953 40.7737%2C-73.870721 40.757007%2C-73.971953 14038 14.038 8.723213 1437 23.95 0.3991667 Your columns are probably of type factor (check with str(X)). mapdist needs character vectors (check ?mapdist). So you have to convert the columns using as.character beforehand. Also, when using geo coordinates, I think you got to URL encode them. I.e. the comma , becomes %2C. Otherwise it didn`t work for me...