How to create sub-matrices in R? - r

I have a matrix "Mat.return" with 390 rows and 2749 columns and I want to create 2499 sub-matrices from it, each with 250 columns and 80 rows.
The first sub-matrix would be:
B1=(Mat.return)[sample(nrow((Mat.return)),size=80,replace=TRUE),][,c(1:250)]
The second one, would start from the second column of "Mat.return" and would select 250 following columns. It would thus be:
B2=(Mat.return)[sample(nrow((Mat.return)),size=80,replace=TRUE),][,c(2:251)]
The third one would start from the third column and would select the 250 following column, and so on [until matrix n°2499]
Is there a function or a code that could do this, instead of computing it manually?
Thank you!

Just make a loop from 1 to 2499 around your function. this code will give you a list of 2499 matrix on 80 rows and 251 columns
Mat.return <- matrix(rnorm(390*2749), nrow = 390, ncol = 2749)
lmat <- lapply(1:2499, function(i){
(Mat.return)[sample(nrow((Mat.return)),size=80,replace=TRUE),][,c(i:(250 + i))]
})
str(lmat, list.len = 10)
#> List of 2499
#> $ : num [1:80, 1:251] 0.493 -0.295 2.299 -1.427 -0.174 ...
#> $ : num [1:80, 1:251] -0.4 -1.632 1.21 0.529 -1.045 ...
#> $ : num [1:80, 1:251] -1.71 -1.458 0.186 0.808 -1.179 ...
#> $ : num [1:80, 1:251] 0.237 -0.952 -0.632 -0.204 -1.702 ...
#> $ : num [1:80, 1:251] -1.828 -0.895 -1.31 1.009 -0.451 ...
#> $ : num [1:80, 1:251] 0.128 0.461 -0.393 0.358 1.549 ...
#> $ : num [1:80, 1:251] -0.44814 0.52248 0.28651 0.39365 -0.00774 ...
#> $ : num [1:80, 1:251] 0.136 0.615 -0.435 -0.846 0.788 ...
#> $ : num [1:80, 1:251] 0.761 0.11 -1.486 -0.488 0.118 ...
#> $ : num [1:80, 1:251] -0.9064 -1.3382 -0.9678 0.0654 -0.5952 ...
#> [list output truncated]

Related

Turn raster files (4-dimensional) into structure that allows to conduct a random forest classification

My goal is to conduct a random forest classification for agricultural landuse forms (crop classification). I have several ground truth points for all classes. Furthermore, I have 37 raster files (.tif) each having the same 12 bands and same extent, with one file representing one date in the time series. The time series is NOT constant.
The following shows the files, the dates and band names plus and first file read with terra:
> files <- list.files("C:/temp/final2",full.names = T,pattern = ".tif$",recursive = T)
> files[1:3]
[1] "C:/temp/final2/20190322T100031_20190322T100524_T33UXP.tif" "C:/temp/final2/20190324T095029_20190324T095522_T33UXP.tif"
[3] "C:/temp/final2/20190329T095031_20190329T095315_T33UXP.tif"
> dates <- as.Date(substr(basename(files),1,8),"%Y%m%d")
> band_names <- c("B02","B03","B04","B05","B08","B11","B12","NDVI","NDWI","SAVI")
> rast(files[1])
class : SpatRaster
dimensions : 386, 695, 12 (nrow, ncol, nlyr)
resolution : 10, 10 (x, y)
extent : 634500, 641450, 5342460, 5346320 (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / UTM zone 33N (EPSG:32633)
source : 20190322T100031_20190322T100524_T33UXP.tif
names : B2, B3, B4, B5, B6, B7, ...
I want to extract the value for every date and band. This should result in a dataframe with oberserved variables and the respective class for each point (see below). With this dataframe I want to train a random forest model in order to predict the crop classes for each raster (resulting in a single raster layer with classes as values).
The following structure (copied from https://gdalcubes.github.io/source/tutorials/vignettes/gc03_ML_training_data.html) is what I need as observed values, which serve as the training data for the rf model.
## FID time B2 ... more bands ... and class of respective FID
## 1 16 2018-01-01 13.33
## 2 17 2018-01-01 13.63
## 3 18 2018-01-01 13.33
## 4 19 2018-01-01 12.15
## 5 20 2018-01-01 14.73
## 6 21 2018-01-01 15.91
## 7 16 2018-01-09 12.23
## 8 17 2018-01-09 12.15
## 9 18 2018-01-09 12.07
## 10 19 2018-01-09 10.19
## 11 20 2018-01-09 9.83
I (1) read all the rasters into list called 'cube' and
(2) combined all the spatRasters in the list into one spatRaster.
> cube <- c()
> for (file in files){
+ ras <- rast(file)
+ cube<-c(cube,ras)
+ }
> names(cube) <- dates
> cubef <- rast(cube)
> cubef
class : SpatRaster
dimensions : 386, 695, 444 (nrow, ncol, nlyr)
resolution : 10, 10 (x, y)
extent : 634500, 641450, 5342460, 5346320 (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / UTM zone 33N (EPSG:32633)
sources : 20190322T100031_20190322T100524_T33UXP.tif (12 layers)
20190324T095029_20190324T095522_T33UXP.tif (12 layers)
20190329T095031_20190329T095315_T33UXP.tif (12 layers)
... and 34 more source(s)
names : 2019-03-22_1, 2019-03-22_2, 2019-03-22_3, 2019-03-22_4, 2019-03-22_5, 2019-03-22_6, ...
When I extract the values of all the layers for sample points, I get the follwing result.
> s_points <- st_read(connex,query="SELECT * FROM s_points WHERE NOT ST_IsEmpty(geom);")
> str(s_points)
Classes ‘sf’ and 'data.frame': 286 obs. of 3 variables:
$ s_point_id: int 1 1 2 2 4 4 6 6 7 7 ...
$ kf_klasse : chr "ERBSEN - GETREIDE GEMENGE" "ERBSEN - GETREIDE GEMENGE" "ERBSEN - GETREIDE GEMENGE" "ERBSEN - GETREIDE GEMENGE" ...
$ geom :sfc_POINT of length 286; first list element: 'XY' num 637052 5345218
- attr(*, "sf_column")= chr "geom"
- attr(*, "agr")= Factor w/ 3 levels "constant","aggregate",..: NA NA
..- attr(*, "names")= chr [1:2] "s_point_id" "kf_klasse"
> s_points_coords <- st_coordinates(s_points)
> e <- terra::extract(cubef, s_points)
> str(e)
'data.frame': 286 obs. of 445 variables:
$ ID : num 1 1 2 2 3 3 4 4 5 5 ...
$ 2019-03-22_1 : num 0.0789 0.0901 0.0587 0.063 0.0937 0.0901 0.0517 0.0528 0.0819 0.0882 ...
$ 2019-03-22_2 : num 0.096 0.1056 0.0728 0.0771 0.1072 ...
$ 2019-03-22_3 : num 0.108 0.1226 0.0734 0.0788 0.125 ...
$ 2019-03-22_4 : num 0.1301 0.1437 0.0998 0.1017 0.1395 ...
$ 2019-03-22_5 : num 0.166 0.174 0.157 0.151 0.156 ...
$ 2019-03-22_6 : num 0.183 0.188 0.174 0.163 0.169 ...
$ 2019-03-22_7 : num 0.196 0.196 0.183 0.169 0.186 ...
$ 2019-03-22_8 : num 0.27 0.293 0.171 0.172 0.282 ...
$ 2019-03-22_9 : num 0.236 0.269 0.138 0.142 0.252 ...
$ 2019-03-22_10: num 0.29 0.229 0.427 0.365 0.196 ...
$ 2019-03-22_11: num -0.343 -0.299 -0.43 -0.374 -0.268 ...
$ 2019-03-22_12: num 0.1353 0.1108 0.1739 0.1452 0.0928 ...
$ 2019-03-24_1 : num 0.099 0.1088 0.0919 NA 0.1058 ...
$ 2019-03-24_2 : num 0.111 0.115 0.11 NA 0.114 ...
$ 2019-03-24_3 : num 0.116 0.127 0.104 NA 0.131 ...
$ 2019-03-24_4 : num 0.145 0.154 0.147 NA 0.152 ...
$ 2019-03-24_5 : num 0.19 0.19 0.258 NA 0.171 ...
$ 2019-03-24_6 : num 0.208 0.21 0.294 NA 0.186 ...
$ 2019-03-24_7 : num 0.231 0.222 0.31 NA 0.197 ...
$ 2019-03-24_8 : num 0.318 0.341 0.281 NA 0.331 ...
$ 2019-03-24_9 : num 0.283 0.314 0.217 NA 0.305 ...
$ 2019-03-24_10: num 0.329 0.271 0.497 NA 0.202 ...
$ 2019-03-24_11: num -0.35 -0.317 -0.477 NA -0.268 ...
$ 2019-03-24_12: num 0.1698 0.1405 0.291 NA 0.0997 ...
$ 2019-03-29_1 : num NA NA 0.0476 NA 0.0891 0.0847 0.0664 0.0719 NA NA ...
$ 2019-03-29_2 : num NA NA 0.0642 NA 0.0965 ...
$ 2019-03-29_3 : num NA NA 0.0607 NA 0.1196 ...
$ 2019-03-29_4 : num NA NA 0.0904 NA 0.1351 ...
$ 2019-03-29_5 : num NA NA 0.162 NA 0.149 ...
$ 2019-03-29_6 : num NA NA 0.18 NA 0.167 ...
$ 2019-03-29_7 : num NA NA 0.182 NA 0.183 ...
$ 2019-03-29_8 : num NA NA 0.167 NA 0.337 ...
$ 2019-03-29_9 : num NA NA 0.125 NA 0.311 ...
$ 2019-03-29_10: num NA NA 0.5 NA 0.209 ...
$ 2019-03-29_11: num NA NA -0.479 NA -0.309 ...
$ 2019-03-29_12: num NA NA 0.1955 NA 0.0971 ...
$ 2019-04-01_1 : num 0.0616 0.0703 0.0543 0.0573 0.0733 0.0783 0.0675 0.0693 0.0557 0.0584 ...
$ 2019-04-01_2 : num 0.0742 0.0838 0.073 0.076 0.0849 0.0872 0.0783 0.0821 0.0733 0.073 ...
$ 2019-04-01_3 : num 0.0798 0.0945 0.066 0.0758 0.0987 ...
$ 2019-04-01_4 : num 0.101 0.114 0.104 0.106 0.116 ...
$ 2019-04-01_5 : num 0.144 0.143 0.205 0.188 0.129 ...
$ 2019-04-01_6 : num 0.157 0.157 0.231 0.209 0.143 ...
$ 2019-04-01_7 : num 0.17 0.165 0.249 0.214 0.153 ...
$ 2019-04-01_8 : num 0.24 0.259 0.208 0.212 0.275 ...
$ 2019-04-01_9 : num 0.207 0.232 0.152 0.168 0.256 ...
$ 2019-04-01_10: num 0.362 0.272 0.581 0.476 0.216 ...
$ 2019-04-01_11: num -0.393 -0.326 -0.547 -0.475 -0.287 ...
$ 2019-04-01_12: num 0.1449 0.1119 0.2783 0.2137 0.0871 ...
$ 2019-04-16_1 : num 0.0639 0.0695 0.0539 0.0541 0.0767 0.081 0.0754 0.0739 0.0606 0.0621 ...
$ 2019-04-16_2 : num 0.0733 0.0797 0.0717 0.07 0.0834 0.0862 0.0835 0.0854 0.0748 0.0785 ...
$ 2019-04-16_3 : num 0.0832 0.0923 0.0658 0.0626 0.1042 ...
$ 2019-04-16_4 : num 0.108 0.115 0.111 0.107 0.118 ...
$ 2019-04-16_5 : num 0.164 0.159 0.229 0.223 0.136 ...
$ 2019-04-16_6 : num 0.183 0.179 0.26 0.26 0.149 ...
$ 2019-04-16_7 : num 0.202 0.198 0.284 0.275 0.166 ...
$ 2019-04-16_8 : num 0.255 0.27 0.205 0.202 0.288 ...
$ 2019-04-16_9 : num 0.219 0.244 0.141 0.144 0.278 ...
$ 2019-04-16_10: num 0.416 0.364 0.623 0.63 0.23 ...
$ 2019-04-16_11: num -0.467 -0.426 -0.596 -0.595 -0.332 ...
$ 2019-04-16_12: num 0.1846 0.1638 0.3228 0.3181 0.0979 ...
$ 2019-04-18_1 : num 0.0702 0.0792 0.0636 0.063 0.0875 0.094 0.0858 0.0868 0.0662 0.0709 ...
$ 2019-04-18_2 : num 0.0838 0.0946 0.0898 0.0872 0.101 ...
$ 2019-04-18_3 : num 0.0908 0.1038 0.0785 0.0765 0.1206 ...
$ 2019-04-18_4 : num 0.121 0.13 0.13 0.125 0.138 ...
$ 2019-04-18_5 : num 0.186 0.183 0.266 0.253 0.154 ...
$ 2019-04-18_6 : num 0.213 0.205 0.299 0.289 0.167 ...
$ 2019-04-18_7 : num 0.221 0.214 0.312 0.297 0.186 ...
$ 2019-04-18_8 : num 0.275 0.294 0.228 0.228 0.314 ...
$ 2019-04-18_9 : num 0.227 0.255 0.154 0.157 0.296 ...
$ 2019-04-18_10: num 0.418 0.346 0.598 0.59 0.214 ...
$ 2019-04-18_11: num -0.45 -0.387 -0.553 -0.546 -0.297 ...
$ 2019-04-18_12: num 0.199 0.167 0.335 0.321 0.101 ...
$ 2019-04-21_1 : num 0.0404 0.0619 0.0373 0.0351 0.0814 0.0844 0.0764 0.0801 0.0563 0.0626 ...
$ 2019-04-21_2 : num 0.0592 0.0823 0.0614 0.0579 0.0927 0.0966 0.0933 0.0952 0.0776 0.0869 ...
$ 2019-04-21_3 : num 0.0542 0.0873 0.048 0.0433 0.1118 ...
$ 2019-04-21_4 : num 0.082 0.105 0.0933 0.0841 0.1279 ...
$ 2019-04-21_5 : num 0.15 0.163 0.225 0.207 0.144 ...
$ 2019-04-21_6 : num 0.173 0.184 0.259 0.247 0.155 ...
$ 2019-04-21_7 : num 0.174 0.199 0.274 0.251 0.172 ...
$ 2019-04-21_8 : num 0.192 0.237 0.168 0.156 0.291 ...
$ 2019-04-21_9 : num 0.1352 0.1804 0.0994 0.0903 0.2674 ...
$ 2019-04-21_10: num 0.525 0.391 0.702 0.706 0.213 ...
$ 2019-04-21_11: num -0.493 -0.415 -0.634 -0.625 -0.3 ...
$ 2019-04-21_12: num 0.1954 0.174 0.3422 0.3212 0.0941 ...
$ 2019-05-01_1 : num 0.0342 0.0435 0.0282 0.0292 0.07 0.0684 0.0722 0.0757 0.0458 0.061 ...
$ 2019-05-01_2 : num 0.0516 0.055 0.0517 0.048 0.0781 0.0793 0.0861 0.0919 0.0613 0.0839 ...
$ 2019-05-01_3 : num 0.0422 0.0538 0.0299 0.0325 0.0991 ...
$ 2019-05-01_4 : num 0.0753 0.0836 0.0761 0.0755 0.1112 ...
$ 2019-05-01_5 : num 0.182 0.177 0.247 0.235 0.124 ...
$ 2019-05-01_6 : num 0.21 0.203 0.3 0.287 0.138 ...
$ 2019-05-01_7 : num 0.214 0.19 0.314 0.293 0.157 ...
$ 2019-05-01_8 : num 0.164 0.182 0.148 0.146 0.264 ...
$ 2019-05-01_9 : num 0.0988 0.1156 0.0777 0.0763 0.235 ...
$ 2019-05-01_10: num 0.67 0.559 0.826 0.801 0.225 ...
$ 2019-05-01_11: num -0.611 -0.552 -0.717 -0.719 -0.334 ...
$ 2019-05-01_12: num 0.273 0.2196 0.4226 0.3935 0.0916 ...
$ 2019-05-26_1 : num 0.0537 0.0633 0.0431 0.0444 0.118 ...
$ 2019-05-26_2 : num 0.0675 0.0835 0.0611 0.0564 0.1284 ...
[list output truncated]
What I have now is a dataframe, that has a column for every band of each image (12 columns for each image), which results in 37x12 columns. From here on, I don't know how to add the extracted values to the s_points dataframe, in order to have the ID and classname of the extracted values. This isn't possible, because I have 444 values for every point.
My questions are:
How can I combine the extracted values and the sample_points?
How can I train a rf-model with this extracted data?
Does it make more sense to use a datacube here (gdalcubes in R)? I forget this idea, mainly because of the unconstant character of the time series, which would result in problem with the temporal aggregation. This isn't expedient in the research question.
Thanks
You mention that you want a dataset with four dimensions. But how you are going to train your model and make predictions (you can only use two dimensions for that)? So it would seem to me that what you need is a three-dimensional SpatRaster that you can make with
cube <- rast(files)
Unless you want to run a separate model for each file --- but then you should loop over the files.
Here is an example (taken from ?terra::predict showing how you might then run a RandomForest, or any other regression or classification model.
library(terra)
logo1 <- rast(system.file("ex/logo.tif", package="terra"))
logo2 <- sqrt(logo1)
cube <- c(logo1, logo2)
names(cube) <- c("red1", "green1", "blue1", "red2", "green2", "blue2")
p <- matrix(c(48, 48, 48, 53, 50, 46, 54, 70, 84, 85, 74, 84, 95, 85,
66, 42, 26, 4, 19, 17, 7, 14, 26, 29, 39, 45, 51, 56, 46, 38, 31,
22, 34, 60, 70, 73, 63, 46, 43, 28), ncol=2)
a <- matrix(c(22, 33, 64, 85, 92, 94, 59, 27, 30, 64, 60, 33, 31, 9,
99, 67, 15, 5, 4, 30, 8, 37, 42, 27, 19, 69, 60, 73, 3, 5, 21,
37, 52, 70, 74, 9, 13, 4, 17, 47), ncol=2)
xy <- rbind(cbind(1, p), cbind(0, a))
e <- extract(cube, xy[,2:3])
v <- data.frame(cbind(pa=xy[,1], e))
library(randomForest)
rfm <- randomForest(formula=pa~., data=v)
p <- predict(cube, rfm)
Perhaps you can edit your question and explain why this would not work for you. And include a toy example of how you intend to fit your model. I suppose the rasters are your predictors, but what are you predicting (your y variable)? Is it constant or is it different for each time step (raster file)?
If the issue is that you want to distinguish between variables with the same names at different dates you can concatenate them. Something like this with SpatRaster x
names(x) <- paste0(names(x), "_", time(x))
If you want to write a single netCDF file you could do
sds <- rast(files)
writeCDF(sds, "test.nc")

How to normalize all variables in an R dataframe (except for the one variable that's a factor)

I'm having difficulty applying the max-min normalize function to the predictor variables (30 of them) in my data frame without excluding the diagnosis variable (as it is a factor and not subject to the function) from the data frame.
```{r}
cancer_data <- as.data.frame(lapply(cancer_data, normalize))
```
This won't run bc it will prompt an error message referencing the factor column, but I don't want the new data frame to be created without that column. I would just like to apply the normalize function I created to the 30 predictor variables.
Here is the structure of my data frame if it provides helpful context at all:
str(cancer_data)
## 'data.frame': 569 obs. of 31 variables:
## $ diagnosis : Factor w/ 2 levels "Benign","Malignant": 1 1 1 1 1 1 1 2 1 1 ...
## $ radius_mean : num 12.3 10.6 11 11.3 15.2 ...
## $ texture_mean : num 12.4 18.9 16.8 13.4 13.2 ...
## $ perimeter_mean : num 78.8 69.3 70.9 73 97.7 ...
## $ area_mean : num 464 346 373 385 712 ...
## $ smoothness_mean : num 0.1028 0.0969 0.1077 0.1164 0.0796 ...
## $ compactness_mean : num 0.0698 0.1147 0.078 0.1136 0.0693 ...
## $ concavity_mean : num 0.0399 0.0639 0.0305 0.0464 0.0339 ...
## $ points_mean : num 0.037 0.0264 0.0248 0.048 0.0266 ...
## $ symmetry_mean : num 0.196 0.192 0.171 0.177 0.172 ...
## $ dimension_mean : num 0.0595 0.0649 0.0634 0.0607 0.0554 ...
## $ radius_se : num 0.236 0.451 0.197 0.338 0.178 ...
## $ texture_se : num 0.666 1.197 1.387 1.343 0.412 ...
## $ perimeter_se : num 1.67 3.43 1.34 1.85 1.34 ...
## $ area_se : num 17.4 27.1 13.5 26.3 17.7 ...
## $ smoothness_se : num 0.00805 0.00747 0.00516 0.01127 0.00501 ...
## $ compactness_se : num 0.0118 0.03581 0.00936 0.03498 0.01485 ...
## $ concavity_se : num 0.0168 0.0335 0.0106 0.0219 0.0155 ...
## $ points_se : num 0.01241 0.01365 0.00748 0.01965 0.00915 ...
## $ symmetry_se : num 0.0192 0.035 0.0172 0.0158 0.0165 ...
## $ dimension_se : num 0.00225 0.00332 0.0022 0.00344 0.00177 ...
## $ radius_worst : num 13.5 11.9 12.4 11.9 16.2 ...
## $ texture_worst : num 15.6 22.9 26.4 15.8 15.7 ...
## $ perimeter_worst : num 87 78.3 79.9 76.5 104.5 ...
## $ area_worst : num 549 425 471 434 819 ...
## $ smoothness_worst : num 0.139 0.121 0.137 0.137 0.113 ...
## $ compactness_worst: num 0.127 0.252 0.148 0.182 0.174 ...
## $ concavity_worst : num 0.1242 0.1916 0.1067 0.0867 0.1362 ...
## $ points_worst : num 0.0939 0.0793 0.0743 0.0861 0.0818 ...
## $ symmetry_worst : num 0.283 0.294 0.3 0.21 0.249 ...
## $ dimension_worst : num 0.0677 0.0759 0.0788 0.0678 0.0677 ...
Assuming you already have normalize function in your environment. You can get the numeric variables in your data and apply the function to selected columns using lapply.
cols <- sapply(cancer_data, is.numeric)
cancer_data[cols] <- lapply(cancer_data[cols], normalize)
Or without creating cols.
cancer_data[] <- lapply(cancer_data, function(x)
if(is.numeric(x)) normalize(x) else x)
If you want to exclude only 1st column, you can also use :
cancer_data[-1] <- lapply(cancer_data[-1], normalize)
This should work, but do look into tidymodels
Thanks to akrun for the new shorter answer.
library(tidyverse)
cancer_data <-cancer_data %>% mutate_if(negate(is.factor), normalize)

Merge 2 list of lists in R [duplicate]

This question already has answers here:
Combining elements of list of lists by index
(3 answers)
in r combine a list of lists into one list
(3 answers)
Closed 4 years ago.
I have 2 list of lists in R with the same list name as follow :
str(total_delta_final[[1]])
List of 4
$ sector1_T02 :'data.frame': 24 obs. of 3 variables:
..$ DeltaF_1: num [1:24] 0.737 0.737 0.693 0.738 0.738 ...
..$ DeltaF_2: num [1:24] 0.24 0.24 0.279 0.239 0.239 ...
..$ DeltaF_3: num [1:24] 0.0233 0.0233 0.0275 0.0232 0.0232 ...
$ sector2_T03 :'data.frame': 24 obs. of 3 variables:
..$ DeltaF_1: num [1:24] 0.582 0.582 0.568 0.69 0.69 ...
..$ DeltaF_2: num [1:24] 0.377 0.377 0.39 0.282 0.282 ...
..$ DeltaF_3: num [1:24] 0.0406 0.0406 0.0426 0.0278 0.0278 ...
$ sector3_T03 :'data.frame': 24 obs. of 3 variables:
..$ DeltaF_1: num [1:24] 0.607 0.607 0.495 0.409 0.375 ...
..$ DeltaF_2: num [1:24] 0.356 0.356 0.451 0.519 0.544 ...
..$ DeltaF_3: num [1:24] 0.0373 0.0373 0.0541 0.072 0.0809 ...
$ sector12_T02:'data.frame': 24 obs. of 3 variables:
..$ DeltaF_1: num [1:24] 0.743 0.743 0.758 0.689 0.705 ...
..$ DeltaF_2: num [1:24] 0.234 0.234 0.22 0.283 0.269 ...
..$ DeltaF_3: num [1:24] 0.0226 0.0226 0.0213 0.028 0.0263 ...
> str(total_TI_final[[1]])
List of 4
$ sector1_T02 :'data.frame': 24 obs. of 3 variables:
..$ I_1: num [1:24] NA 0.0756 0.083 0.0799 0.0799 ...
..$ I_2: num [1:24] 0.122 NA 0.163 0.172 0.172 ...
..$ I_3: num [1:24] 0.212 0.211 NA 0.266 0.273 ...
$ sector2_T03 :'data.frame': 24 obs. of 3 variables:
..$ I_1: num [1:24] NA 0.0986 0.1013 0.1011 0.101 ...
..$ I_2: num [1:24] 0.15 NA 0.184 0.211 0.211 ...
..$ I_3: num [1:24] 0.249 0.249 NA 0.331 0.337 ...
$ sector3_T03 :'data.frame': 24 obs. of 3 variables:
..$ I_1: num [1:24] NA 0.119 0.115 0.113 0.105 ...
..$ I_2: num [1:24] 0.193 NA 0.2 0.193 0.177 ...
..$ I_3: num [1:24] 0.323 0.323 NA 0.277 0.256 ...
$ sector12_T02:'data.frame': 24 obs. of 3 variables:
..$ I_1: num [1:24] NA 0.0825 0.0681 0.0723 0.0706 ...
..$ I_2: num [1:24] 0.138 NA 0.146 0.145 0.144 ...
..$ I_3: num [1:24] 0.24 0.24 NA 0.22 0.226 ...
How could I merge these 2 list of lists in a way that my final output looks like total_TI_final[[1]][1] and the second list total_delta_final[[1]][1] then total_TI_final[[1]][2] and total_delta_final[[1]][2] and so on ...
We can use Map
Map(c, total_delta_final, total_TI_final)

Why does using c(data frame1 * data frame2) return a list when data frame1*data frame2 return a data frame

In R when I am multiplying two data frames within a c( ) function I get a list as an output but would like to have a data frame returned how can I do this.
Specifically to test it
str (c(Diesel_Col_Monthly_Prop_df[-1] * Product_Prop_Mch_df [ , 2]))
List of 12
$ Jan: num [1:14] 0.0335 0.0335 0.0335 0.0335 0.0335 ...
$ Feb: num [1:14] 0.0365 0.0365 0.0365 0.0365 0.0365 ...
$ Mar: num [1:14] 0.0581 0.0581 0.0581 0.0581 0.0581 ...
$ Apr: num [1:14] 0.0936 0.0936 0.0936 0.0936 0.0936 ...
$ May: num [1:14] 0.0783 0.0783 0.0783 0.0783 0.0783 ...
$ Jun: num [1:14] 0.0616 0.0616 0.0616 0.0616 0.0616 ...
$ Jul: num [1:14] 0.0713 0.0713 0.0713 0.0713 0.0713 ...
$ Aug: num [1:14] 0.107 0.107 0.107 0.107 0.107 ...
$ Sep: num [1:14] 0.0987 0.0987 0.0987 0.0987 0.0987 ...
$ Oct: num [1:14] 0.0654 0.0654 0.0654 0.0654 0.0654 ...
$ Nov: num [1:14] 0.034 0.034 0.034 0.034 0.034 ...
$ Dec: num [1:14] 0.0326 0.0326 0.0326 0.0326 0.0326 ...
> str (Diesel_Col_Monthly_Prop_df [-1] * Product_Prop_Mch_df [ ,2])
'data.frame': 14 obs. of 12 variables:
$ Jan: num 0.0335 0.0335 0.0335 0.0335 0.0335 ...
$ Feb: num 0.0365 0.0365 0.0365 0.0365 0.0365 ...
$ Mar: num 0.0581 0.0581 0.0581 0.0581 0.0581 ...
$ Apr: num 0.0936 0.0936 0.0936 0.0936 0.0936 ...
$ May: num 0.0783 0.0783 0.0783 0.0783 0.0783 ...
$ Jun: num 0.0616 0.0616 0.0616 0.0616 0.0616 ...
$ Jul: num 0.0713 0.0713 0.0713 0.0713 0.0713 ...
$ Aug: num 0.107 0.107 0.107 0.107 0.107 ...
$ Sep: num 0.0987 0.0987 0.0987 0.0987 0.0987 ...
$ Oct: num 0.0654 0.0654 0.0654 0.0654 0.0654 ...
$ Nov: num 0.034 0.034 0.034 0.034 0.034 ...
$ Dec: num 0.0326 0.0326 0.0326 0.0326 0.0326 ...
c(DF) where DF is a data frame always returns a list and DF * DF is a data frame so c(DF * DF) returns a list. If you want a data frame don't use c.
Using the builtin data frame BOD:
> str(BOD)
'data.frame': 6 obs. of 2 variables:
$ Time : num 1 2 3 4 5 7
$ demand: num 8.3 10.3 19 16 15.6 19.8
- attr(*, "reference")= chr "A1.4, p. 270"
> str(BOD * BOD)
'data.frame': 6 obs. of 2 variables:
$ Time : num 1 4 9 16 25 49
$ demand: num 68.9 106.1 361 256 243.4 ...
> str(c(BOD))
List of 2
$ Time : num [1:6] 1 2 3 4 5 7
$ demand: num [1:6] 8.3 10.3 19 16 15.6 19.8
> str(c(BOD*BOD))
List of 2
$ Time : num [1:6] 1 4 9 16 25 49
$ demand: num [1:6] 68.9 106.1 361 256 243.4 ...

transform all columns that are NAs from numeric to factors

I have a data.table called td.br.2, in which some columns are completely NAs. These columns are of type numeric. What I would like to do, is only for these columns to transform them to factors.
I have tried the following, but it does not work ( I do not get an error but it does not do the job either)
td.br.2[] <- td.br.2[,lapply(.SD, function(x) {ifelse(sum(is.na(x)==nrow(td.br.2)),as.factor(x),x)})]
n=10#nr of rows
m=10#nr of cols
N<-n*m
m1<-matrix(runif(N),nrow=n,ncol = m)
dt<-data.table(m1)
names(dt)<-letters[1:m]
dt<-cbind(dt,xxx=rep(NA,nrow(dt)))#adding NA column
At this point
str(dt)
Classes ‘data.table’ and 'data.frame': 10 obs. of 11 variables:
$ a : num 0.661 0.864 0.152 0.342 0.989 ...
$ b : num 0.06036 0.67587 0.00847 0.37674 0.30417 ...
$ c : num 0.3938 0.6274 0.0514 0.882 0.1568 ...
$ d : num 0.777 0.233 0.619 0.117 0.132 ...
$ e : num 0.655 0.926 0.277 0.598 0.237 ...
$ f : num 0.649 0.197 0.547 0.585 0.685 ...
$ g : num 0.6877 0.3676 0.009 0.6975 0.0327 ...
$ h : num 0.519 0.705 0.457 0.465 0.966 ...
$ i : num 0.43777 0.00961 0.30224 0.58172 0.37621 ...
$ j : num 0.44 0.481 0.485 0.125 0.263 ...
$ xxx: logi NA NA NA NA NA NA ...
So by executing:
dt<-dt[, lapply(.SD, function(x){ if(all(is.na(x)))as.factor(as.character(x)) else x}),]
yields:
str(dt)
Classes ‘data.table’ and 'data.frame': 10 obs. of 11 variables:
$ a : num 0.0903 0.0448 0.5956 0.418 0.1316 ...
$ b : num 0.672 0.582 0.687 0.113 0.371 ...
$ c : num 0.404 0.16 0.848 0.863 0.737 ...
$ d : num 0.073 0.129 0.243 0.334 0.285 ...
$ e : num 0.485 0.186 0.539 0.486 0.784 ...
$ f : num 0.4685 0.4815 0.585 0.3596 0.0764 ...
$ g : num 0.958 0.194 0.549 0.71 0.737 ...
$ h : num 0.168 0.355 0.552 0.765 0.605 ...
$ i : num 0.665 0.88 0.23 0.575 0.413 ...
$ j : num 0.1113 0.8797 0.1244 0.0741 0.8724 ...
$ xxx: Factor w/ 0 levels: NA NA NA NA NA NA NA NA NA NA
I am not sure why you would want to do that, but here you are:
naColumns <- sapply(td.br.2, function(x) { all(is.na(x)) })
for (col in which(naColumns))
set(td.br.2, j=col, value=as.factor(x[[col]]))
The factors will have no levels, but you can deal with that as necessary.
(The for loop is partly based on this.)

Resources