Trying to use R cv.glmnet() for cross validation on loans data.
I have a data set on loan data (Kaggle) and have already split into train, test.
Separated the y response from the predictive variables in select(1) and select(-1).
Created matrix so as to avoid the "Error in storage.mode(y) <- "double" : 'list' object cannot be coerced to type 'double' " problem earlier.
Now seeking to run cv.glmnet() for cross validation, but this error stops me now.
"Error in y - predmat : non-numeric argument to binary operator"
Error in non-numeric argument, yet all my data is numeric, save for one factor for response y.
As a side question, what is the predmat in "y - predmat" refer to?
x_vars <- as.matrix(data.sample.train.split %>% select(-1))
y_resp <- as.matrix(data.sample.train.split %>% select(1))
cv_output <- cv.glmnet(x_vars, y_resp, type.measure = "deviance", nfolds = 5)
cv_output <- cv.glmnet(x_vars, y_resp,
type.measure = "deviance",
lambda = NULL,
nfolds = 5)
I am also considering to try this function:
ddd.lasso <- cv.glmnet(x_vars, y_resp, alpha = 1, family = "binomial")
ddd.model <- glmnet(x_vars, y_resp, alpha = 1, family = "binomial", lambda = ddd.lasso$lambda.min)
Data sample is as follows, just some of the columns:
c("loan_amnt", "funded_amnt",
"funded_amnt_inv", "grade", "emp_length", "annual_inc", "dti",
"mths_since_last_delinq", "mths_since_last_record", "open_acc",
"pub_rec", "revol_bal", "revol_util", "total_acc", "out_prncp",
"out_prncp_inv", "total_pymnt", "total_pymnt_inv", "total_rec_prncp",
"total_rec_int", "total_rec_late_fee", "recoveries", "collection_recovery_fee",
"last_pymnt_amnt", "collections_12_mths_ex_med", "acc_now_delinq"
)))
loan_amnt funded_amnt funded_amnt_inv grade emp_length annual_inc dti
3 10000 10000 10000.000 60 10 49200.00 20.00
10 10000 10000 10000.000 60 4 42000.00 18.60
14 20250 20250 19142.161 60 3 43370.00 26.53
17 15000 15000 15000.000 80 2 92000.00 29.44
18 4000 4000 4000.000 80 10 106000.00 5.63
31 4400 4400 4400.000 40 10 55000.00 20.01
35 10000 10000 10000.000 100 10 60000.00 12.74
37 25600 25600 25350.000 80 9 110000.00 15.71
41 10000 10000 10000.000 80 1 39000.00 18.58
64 9200 9200 9200.000 80 2 60000.00 19.96
72 7000 7000 7000.000 80 4 39120.00 21.01
74 3500 3500 3500.000 100 10 83000.00 2.31
77 9500 9500 9500.000 100 7 50000.00 8.18
89 10000 10000 10000.000 100 1 43000.00 25.26
98 7000 7000 7000.000 80 1 30000.00 15.80
112 21600 21600 20498.266 20 8 60000.00 16.74
117 7200 7200 7200.000 80 5 48000.00 17.43
118 12000 12000 11975.000 60 1 57000.00 10.86
125 10000 10000 10000.000 100 5 70000.00 16.78
126 8000 8000 8000.000 60 3 28000.00 12.60
128 6000 6000 6000.000 60 10 94800.00 24.53
138 35000 35000 35000.000 80 2 168000.00 3.17
144 14000 14000 14000.000 100 10 66000.00 11.15
149 3000 3000 3000.000 60 5 71000.00 21.84
152 12000 12000 11975.000 80 2 60000.00 15.50
153 6000 6000 6000.000 100 3 34000.00 14.51
155 7000 7000 7000.000 80 7 82000.00 12.00
166 24250 18100 18075.000 -1 7 120000.00 12.96
170 2500 2500 2500.000 80 7 29000.00 18.70
172 4225 4225 4225.000 80 5 55200.00 17.61
180 6000 6000 6000.000 60 5 50000.00 15.58
192 5000 5000 5000.000 80 5 38004.00 23.78
193 8000 8000 8000.000 80 3 31000.00 16.22
199 12000 12000 12000.000 80 4 40000.00 22.20
203 3200 3200 3200.000 80 9 61200.00 2.16
209 5000 5000 5000.000 80 2 70000.00 20.06
220 13250 13250 13250.000 40 10 52000.00 23.70
224 12000 12000 12000.000 100 10 68000.00 7.08
mths_since_last_delinq mths_since_last_record open_acc pub_rec revol_bal revol_util
3 35 59 10 0 5598 21.0
10 61 114 14 0 24043 70.2
14 18 107 8 0 17813 85.6
17 54 79 8 0 13707 93.9
18 18 97 12 0 6110 37.7
31 68 119 7 0 25237 99.0
35 37 93 11 0 14019 19.5
37 11 118 11 0 26088 62.0
41 58 17 5 0 12874 72.7
64 39 95 8 0 23299 78.7
72 26 33 8 1 9414 52.4
74 35 59 6 0 3092 13.4
77 46 118 8 0 13422 60.5
89 59 105 8 0 8215 37.2
98 68 101 7 0 15455 47.6
112 23 26 6 0 13354 78.1
117 24 19 7 0 16450 80.2
118 47 87 7 0 9273 81.5
125 32 92 9 0 10770 69.0
126 66 112 8 0 6187 54.3
128 10 101 13 0 71890 95.9
138 22 97 16 0 1099 1.4
144 26 102 7 0 12095 35.4
149 59 103 4 0 15072 88.7
152 46 94 7 0 12168 85.7
153 70 81 9 0 13683 64.8
155 79 83 6 0 25334 71.6
166 66 118 7 0 31992 99.0
170 63 99 5 0 2668 66.7
172 69 104 6 0 4055 73.7
180 49 94 8 0 7361 83.6
192 5 85 12 0 10023 57.3
193 28 77 13 0 2751 34.4
199 78 109 9 0 16273 55.5
203 79 113 5 1 2795 33.3
209 27 62 14 0 13543 54.2
220 70 86 8 0 15002 91.5
224 21 70 7 0 15433 55.6
total_acc out_prncp out_prncp_inv total_pymnt total_pymnt_inv total_rec_prncp
3 37 0 0 12226.302 12226.30 10000.00
10 28 0 0 12519.260 12519.26 10000.00
14 22 0 0 27663.043 25417.68 20250.00
17 31 0 0 15823.480 15823.48 15000.00
18 44 0 0 4484.790 4484.79 4000.00
31 11 0 0 5626.893 5626.89 4400.00
35 18 0 0 10282.670 10282.67 10000.00
37 27 0 0 29695.623 29405.63 25600.00
41 10 0 0 11474.760 11474.76 10000.00
64 19 0 0 10480.840 10480.84 9200.00
72 26 0 0 7932.300 7932.30 7000.00
74 28 0 0 3834.661 3834.66 3500.00
77 13 0 0 10493.710 10493.71 9500.00
89 16 0 0 11264.010 11264.01 10000.00
98 11 0 0 8452.257 8452.26 7000.00
112 21 0 0 27580.750 24853.63 21600.00
117 10 0 0 8677.156 8677.16 7200.00
118 11 0 0 14396.580 14366.62 12000.00
125 18 0 0 10902.910 10902.91 10000.00
126 11 0 0 8636.820 8636.82 8000.00
128 30 0 0 7215.050 7215.05 6000.00
138 22 0 0 38059.760 38059.76 35000.00
144 46 0 0 15450.084 15450.08 14000.00
149 14 0 0 3723.936 3723.94 3000.00
152 21 0 0 13919.414 13890.44 12000.00
153 16 0 0 6857.261 6857.26 6000.00
155 31 0 0 8290.730 8290.73 7000.00
166 20 0 0 22188.250 22157.63 18100.00
170 13 0 0 2894.740 2894.74 2500.00
172 12 0 0 5081.023 5081.02 4225.00
180 14 0 0 7325.299 7325.30 6000.00
192 17 0 0 6534.430 6534.43 5000.00
193 29 0 0 8306.470 8306.47 8000.00
199 23 0 0 14006.680 14006.68 12000.00
203 17 0 0 3709.193 3709.19 3200.00
209 26 0 0 5501.160 5501.16 5000.00
220 18 0 0 15650.390 15650.39 13250.00
224 34 0 0 12554.010 12554.01 12000.00
total_rec_int total_rec_late_fee recoveries collection_recovery_fee last_pymnt_amnt
3 2209.33 16.97000 0 0 357.48
10 2519.26 0.00000 0 0 370.46
14 7413.04 0.00000 0 0 6024.09
17 823.48 0.00000 0 0 2447.05
18 484.79 0.00000 0 0 2638.77
31 1226.89 0.00000 0 0 162.44
35 282.67 0.00000 0 0 8762.05
37 4095.62 0.00000 0 0 838.27
41 1474.76 0.00000 0 0 5803.94
64 1280.84 0.00000 0 0 365.48
72 932.30 0.00000 0 0 4235.03
74 334.66 0.00000 0 0 107.86
77 993.71 0.00000 0 0 5378.43
89 1264.01 0.00000 0 0 4.84
98 1452.26 0.00000 0 0 238.06
112 5980.75 0.00000 0 0 17416.49
117 1462.16 15.00000 0 0 19.26
118 2396.58 0.00000 0 0 5359.38
125 902.91 0.00000 0 0 4152.52
126 636.82 0.00000 0 0 6983.56
128 1215.05 0.00000 0 0 1960.88
138 3059.76 0.00000 0 0 272.59
144 1450.08 0.00000 0 0 2133.17
149 723.94 0.00000 0 0 107.29
152 1919.41 0.00000 0 0 395.05
153 857.26 0.00000 0 0 198.16
155 1290.73 0.00000 0 0 2454.29
166 4088.25 0.00000 0 0 16499.75
170 394.74 0.00000 0 0 1168.50
172 856.02 0.00000 0 0 146.48
180 1325.30 0.00000 0 0 215.51
192 1534.43 0.00000 0 0 1561.93
193 306.47 0.00000 0 0 7778.22
199 2006.68 0.00000 0 0 5971.51
203 509.19 0.00000 0 0 317.41
209 501.16 0.00000 0 0 3833.62
220 2400.39 0.00000 0 0 9026.78
224 554.01 0.00000 0 0 473.95
collections_12_mths_ex_med acc_now_delinq
3 0 0
10 0 0
14 0 0
17 0 0
18 0 0
31 0 0
35 0 0
37 0 0
41 0 0
64 0 0
72 0 0
74 0 0
77 0 0
89 0 0
98 0 0
112 0 0
117 0 0
118 0 0
125 0 0
126 0 0
128 0 0
138 0 0
144 0 0
149 0 0
152 0 0
153 0 0
155 0 0
166 0 0
170 0 0
172 0 0
180 0 0
192 0 0
193 0 0
199 0 0
203 0 0
209 0 0
220 0 0
224 0 0
Looks like a incorrect glmnet family, I accidently chose the default 'deviance' for cv.glmnet, when in fact my data was binomial. My next solution is to figure out "Convergence for 1th lambda value not reached after maxit=100000 iterations; solutions for larger lambdas returned"
Code that improved the solution:
cv.lasso <- cv.glmnet(x_vars, y_resp, alpha = 1, family = "binomial", nfolds = 5)
cv.model <- glmnet(x_vars, y_resp, alpha = 1, relax=TRUE, family="binomial", lambda=cv.lasso$lambda.min)
Related
I have the following data frame in R (heavily truncated, obviously):
X PLAYER_ID PLAYER_NAME LOC_X LOC_Y SHOT_MADE_FLAG
1 0 201935 James Harden 14 55 0
2 1 201935 James Harden 0 24 0
3 2 201935 James Harden 50 74 0
4 3 201935 James Harden 160 215 0
5 4 201935 James Harden 22 21 1
6 5 201935 James Harden -43 278 1
7 6 201935 James Harden 221 6 0
8 7 201935 James Harden -27 21 0
9 8 201935 James Harden -119 235 0
10 9 201935 James Harden -223 101 0
I named this data frame shots in R, and then tried the following:
league_model_19 <- gam(SHOT_MADE_FLAG ~ ti(LOC_X) +
ti(LOC_Y) +
ti(LOC_X, LOC_Y),
data = shots)
But this gives the error shown in the title:
Error in model.frame.default(formula = SHOT_MADE_FLAG ~ ti(LOC_X) + ti(LOC_Y) + : invalid type (list) for variable 'ti(LOC_X)'
I checked the type of each value in LOC_X, and all of them are integers. I understand the column itself is a list/series, but shouldn't this work anyways?
Any way to avoid getting an error here would be great. Thanks in advance.
Edit:
str(shots) outputs:
> str(shots)
'data.frame': 1456 obs. of 6 variables:
$ X : int 0 1 2 3 4 5 6 7 8 9 ...
$ PLAYER_ID : int 201935 201935 201935 201935 201935 201935 201935 201935 201935 201935 ...
$ PLAYER_NAME : chr "James Harden" "James Harden" "James Harden" "James Harden" ...
$ LOC_X : int 14 0 50 160 22 -43 221 -27 -119 -223 ...
$ LOC_Y : int 55 24 74 215 21 278 6 21 235 101 ...
$ SHOT_MADE_FLAG: int 0 0 0 0 1 1 0 0 0 0 ...
and dput(shots) outputs (truncated):
> dput(shots)
structure(list(X = 0:1455, PLAYER_ID = c(201935L, 201935L, 201935L,
201935L, 201935L, 201935L, 201935L, 201935L,....)
Edit 2:
> shots[c('LOC_X','LOC_Y','SHOT_MADE_FLAG')]
LOC_X LOC_Y SHOT_MADE_FLAG
1 14 55 0
2 0 24 0
3 50 74 0
4 160 215 0
5 22 21 1
6 -43 278 1
7 221 6 0
8 -27 21 0
9 -119 235 0
10 -223 101 0
11 60 238 0
12 232 74 0
13 -136 239 0
14 -14 7 1
15 192 168 0
16 157 206 0
17 172 189 0
18 -168 194 0
19 10 4 1
20 -5 10 0
21 -127 228 0
22 -63 259 0
23 -95 241 1
24 227 27 0
25 -12 12 1
26 192 160 0
27 -144 236 0
28 6 281 0
29 40 250 1
30 29 52 0
31 -120 224 0
32 3 10 1
33 -131 221 0
34 32 276 0
35 -18 10 1
36 30 258 0
37 -31 251 0
38 141 210 0
39 13 77 0
40 -6 19 1
41 -18 62 0
42 47 140 0
43 210 132 0
44 139 217 0
45 163 196 1
46 -30 16 1
47 27 256 1
48 -195 173 0
49 0 251 0
50 2 22 1
51 157 203 0
52 54 249 0
53 -132 233 0
54 20 1 1
55 197 210 0
56 -147 224 1
57 1 29 1
58 -31 22 0
59 -128 236 0
60 -35 13 0
61 -29 14 1
62 234 89 0
63 196 172 0
64 -149 220 0
65 94 252 1
66 -92 269 0
67 14 20 1
68 -6 17 1
69 -171 190 1
70 163 205 0
71 2 4 1
72 11 14 1
73 117 256 0
74 0 35 0
75 -3 0 1
76 -232 120 0
77 -8 7 1
78 -2 298 1
79 -6 16 1
80 135 243 1
81 4 17 0
82 149 222 0
83 -6 31 1
84 122 97 1
85 -27 15 1
86 -2 34 0
87 -179 227 1
88 0 17 1
89 -55 310 0
90 223 159 0
91 3 -1 1
92 -11 17 0
93 -78 247 1
94 19 20 0
95 -9 16 0
96 -44 248 0
97 122 220 0
98 -15 -2 0
99 0 5 1
100 101 237 0
101 237 17 0
102 -32 250 1
103 -5 11 1
104 109 216 0
105 -228 45 0
106 18 29 1
107 -25 11 1
108 128 117 1
109 -38 246 0
110 39 57 0
111 -111 284 0
112 -44 83 0
113 111 223 0
114 -14 101 1
115 2 86 0
116 90 232 0
117 98 228 0
118 0 34 1
119 8 57 0
120 130 209 0
121 81 244 1
122 5 15 1
123 -18 27 0
124 -6 17 0
125 210 122 0
126 199 149 0
127 120 221 0
128 -142 221 0
129 -11 12 0
130 226 60 1
131 -152 212 0
132 -188 181 0
133 -1 4 1
134 -22 253 0
135 -16 253 0
136 -6 23 1
137 -120 275 1
138 8 52 1
139 -15 8 0
140 -237 8 1
141 190 159 0
142 9 8 0
143 -50 74 0
144 -17 17 0
145 -143 231 0
146 -136 222 1
147 200 147 1
148 -176 186 1
149 -229 48 0
150 -16 17 1
151 -65 11 0
152 -54 89 0
153 6 16 1
154 -9 8 0
155 -11 2 1
156 -94 251 0
157 -141 246 1
158 45 239 0
159 230 82 1
160 7 4 0
161 -16 13 1
162 131 227 1
163 125 235 0
164 20 260 0
165 -46 252 0
166 144 211 0
167 -2 4 1
168 104 253 1
169 8 16 0
170 109 226 0
171 -1 71 1
172 -201 147 0
173 76 241 0
174 0 2 1
175 21 52 0
176 -79 16 0
177 -196 157 0
178 168 194 1
179 -131 236 1
180 2 6 1
181 42 254 1
182 -50 262 1
183 -6 0 1
184 57 278 0
185 229 14 1
186 127 226 1
187 139 230 0
188 -234 111 0
189 157 211 0
190 82 255 0
191 -216 138 0
192 230 101 0
193 14 2 0
194 -47 252 0
195 -12 10 1
196 113 231 0
197 -28 264 1
198 2 248 0
199 -17 4 1
200 -235 13 1
201 48 253 1
202 20 256 0
203 -1 18 0
204 -109 233 0
205 -107 238 0
206 -116 14 0
207 14 57 0
208 63 240 1
209 96 246 0
210 13 47 0
211 -188 172 1
212 24 252 1
213 123 257 1
214 -144 206 0
215 11 61 1
216 77 247 1
217 158 201 1
218 107 315 0
219 11 283 0
220 161 209 0
221 1 45 0
222 -105 225 0
223 9 30 1
224 27 -6 1
225 3 58 1
226 -19 3 0
227 -165 208 0
228 3 55 1
229 -176 201 1
230 -18 11 0
231 -13 78 1
232 -16 7 1
233 -27 254 0
234 -117 263 0
235 95 233 0
236 211 146 0
237 -6 46 1
238 3 20 1
239 -22 79 1
240 -8 270 1
241 161 190 1
242 20 70 0
243 -14 10 1
244 144 249 1
245 14 0 1
246 -22 29 1
247 8 53 0
248 4 104 0
249 236 15 0
250 113 222 1
251 -40 260 1
252 51 115 0
253 -181 178 0
254 -144 218 0
255 62 49 0
256 116 257 1
257 131 237 1
258 38 81 1
259 -6 114 1
260 -21 21 0
261 111 266 0
262 -36 14 0
263 103 272 0
264 -27 41 0
265 3 80 0
266 -20 17 1
267 -44 316 1
268 152 253 0
269 -65 255 1
270 76 243 1
271 -35 284 0
272 5 69 1
273 -114 249 0
274 32 127 0
275 192 172 1
276 -159 205 0
277 0 13 0
278 200 155 0
279 11 10 0
280 -11 38 1
281 -98 256 1
282 87 239 0
283 -88 259 1
284 8 71 0
285 1 8 1
286 -6 82 1
287 -27 55 1
288 -14 267 0
289 0 262 0
290 -36 69 1
291 -11 59 1
292 -177 271 1
293 -62 277 1
294 -4 21 0
295 84 243 0
296 -47 48 1
297 -36 48 0
298 -94 236 0
299 -6 20 0
300 -1 34 0
301 -88 251 1
302 2 11 0
303 -225 29 0
304 -25 31 0
305 3 76 1
306 -16 31 1
307 -36 262 1
308 -164 221 0
309 5 21 0
310 -1 11 1
311 -1 264 1
312 6 260 0
313 0 28 0
314 -98 239 0
315 -6 91 1
316 -98 251 0
317 85 244 1
318 -51 273 0
319 14 24 0
320 -9 21 0
321 57 253 1
322 50 251 1
323 -231 1 0
324 -5 15 1
325 -143 214 0
326 -51 248 0
327 147 219 0
328 39 258 0
329 92 150 0
330 72 282 0
331 0 -1 0
332 13 32 1
333 -22 258 1
So far I still can't replicate this. This should be a comment but is a bit too long.
Read in the data you've provided:
shots <- read.csv(text=
"X,PLAYER_ID,PLAYER_NAME,LOC_X,LOC_Y,SHOT_MADE_FLAG
0,201935,James Harden,14,55,0
1,201935,James Harden,0,24,0
2,201935,James Harden,50,74,0
3,201935,James Harden,160,215,0
4,201935,James Harden,22,21,1
5,201935,James Harden,-43,278,1
6,201935,James Harden,221,6,0
7,201935,James Harden,-27,21,0
8,201935,James Harden,-119,235,0
9,201935,James Harden,-223,101,0
")
The numeric variables here come out as numeric, but we'll convert to integer (to match your data as closely as possible) below.
Extend the data (randomly) to the size of your full data set:
sfun <- function(x) sample(x,replace=TRUE,size=1456)
set.seed(101)
shots2 <- with(shots,
data.frame(SHOT_MADE_FLAG=sfun(SHOT_MADE_FLAG),
LOC_X=as.integer(sfun(LOC_X)),
LOC_Y=as.integer(sfun(LOC_Y))))
Fit:
library(mgcv)
league_model_19 <- gam(SHOT_MADE_FLAG ~ ti(LOC_X) +
ti(LOC_Y) +
ti(LOC_X, LOC_Y),
data = shots2)
This works fine. You should try this code on your system and see if it works or not.
I also tried this with the data you provided above (333 rows) and it works fine.
So either (1) there's something weird about the remainder of your data that you haven't shown us, or (2) there's something weird about your R environment/package versions.
For (1), it would be helpful if you can post your full data set somewhere. (It should be sufficient to post shots[c('LOC_X','LOC_Y','SHOT_MADE_FLAG')], after checking that you still get the same issue with that subset of the data. We might not need all of the rows to reproduce, but we definitely need more than we have already.)
For (2), you could give us the results of sessionInfo() (or devtools::session_info()). It would be helpful to start from a clean R session to minimize the number of additional packages you have loaded, and to minimize the possibility that you have weird function definitions masking the ones in the package. (The results of find("ti"), find("gam") would also be useful; they should both be package:mgcv ...)
I know this question has been asked multiple times but I've run out of ideas to get the model working. The first 50 rows of the train data:
> train[1:25]
a b c d e f g h i j k l m
1: 0 148.00 27 16 0 A 0 117 92 0 13 271 2
2: 0 207.00 37 8 0 C 0 46 29 0 29 555 5
3: 0 1497.00 44 1 0 A 1 3754 2119 1 1961 5876 6
4: 0 463.00 44 1 0 A 0 287 202 0 105 1037 4
5: 0 19.00 82 1 0 A 0 301 186 0 344 2116 3
6: 0 204.00 41 1 0 A 0 92 76 0 290 1608 10
7: 0 79.00 69 16 0 B 0 48 29 0 1 27 3
8: 0 256.75 71 16 1 A 0 131 112 0 36 1183 0
9: 0 256.75 71 16 1 A 0 131 112 0 36 1183 2
10: 1 49.00 13 13 0 C 0 5 4 0 0 11 1
11: 0 19.00 76 1 0 A 0 897 440 0 575 2674 3
12: 0 49.00 100 100 0 C 0 6 6 0 0 0 1
13: 0 107.00 65 1 0 A 3 334 212 0 421 2773 6
14: 0 79.00 28 16 0 B 0 42 49 0 13 345 2
15: 0 1742.00 61 1 0 A 0 589 340 0 444 3853 8
16: 0 187.00 20 16 0 A 0 123 99 0 70 841 4
17: 0 68.00 73 1 0 A 0 757 507 0 359 773 3
18: 0 157.00 32 16 0 B 0 33 27 0 4 144 2
19: 0 49.00 52 16 0 C 0 10 7 0 2 51 3
20: 0 79.00 53 16 0 B 0 20 9 0 0 40 4
21: 0 68.00 45 1 0 A 0 370 245 0 298 1826 3
22: 0 1074.00 46 1 0 A 0 605 220 0 280 1421 7
23: 0 19.00 84 1 0 A 0 357 214 0 104 1273 3
24: 0 68.00 42 1 0 A 0 107 97 0 224 1526 3
25: 0 226.00 39 1 0 A 0 228 162 0 139 559 3
26: 0 49.00 92 16 0 C 0 4 3 0 0 0 3
27: 0 68.00 46 1 0 A 0 155 104 0 60 1170 3
28: 1 98.00 29 2 0 C 0 15 13 0 1 659 3
29: 0 248.00 44 1 0 A 0 347 204 0 281 1484 4
30: 0 19.00 84 1 0 A 0 302 166 0 170 2800 3
31: 0 444.00 20 16 0 A 0 569 411 1 369 1095 4
32: 0 157.00 20 16 0 B 0 38 30 0 18 265 3
33: 0 208.00 71 16 0 B 0 22 22 0 1 210 3
34: 1 84.00 27 13 0 A 0 37 24 0 1 649 1
35: 1 297.00 17 7 0 A 0 26 21 0 0 0 1
36: 1 49.00 43 16 1 C 0 4 4 0 0 0 2
37: 0 99.00 36 1 0 A 0 614 432 0 851 2839 4
38: 0 354.00 91 2 1 C 0 74 48 0 102 1005 9
39: 0 68.00 62 16 0 A 0 42 32 0 0 0 3
40: 0 49.00 78 16 0 C 0 12 10 0 0 95 3
41: 0 49.00 57 16 0 C 1 9 8 0 1 582 3
42: 0 68.00 49 1 0 A 0 64 47 0 49 112 3
43: 0 583.00 70 2 1 A 0 502 293 0 406 2734 9
44: 0 187.00 29 1 0 A 0 186 129 0 118 2746 5
45: 0 178.00 52 1 0 A 0 900 484 0 180 1701 4
46: 1 98.00 50 44 0 C 0 13 12 0 1 647 4
47: 1 548.00 21 14 0 A 0 19 14 0 0 0 1
48: 0 178.00 28 16 0 C 0 43 33 0 6 921 3
49: 1 49.00 20 20 0 C 0 8 6 0 0 0 1
50: 0 49.00 124 124 1 A 0 14 11 0 0 0 1
a b c d e f g h i j k l m
This data is not normalised, but it doesn't matter at this stage.
I can't get a simple gbm model work using the gbm package:
> require(gbm)
> gbm_model <- gbm(a ~ .
, data = train
, distribution="bernoulli"
, n.trees= 10
, shrinkage=0.001
, bag.fraction = 1
, train.fraction = 0.5
, n.minobsinnode = 3
, cv.folds = 0 # no cross-validation
, keep.data=TRUE
, verbose=TRUE
)
Iter TrainDeviance ValidDeviance StepSize Improve
1 nan nan 0.0010 nan
2 nan nan 0.0010 nan
3 nan nan 0.0010 nan
4 nan nan 0.0010 nan
5 nan nan 0.0010 nan
6 nan nan 0.0010 nan
7 nan nan 0.0010 nan
8 nan nan 0.0010 nan
9 nan nan 0.0010 nan
10 nan nan 0.0010 nan
Columns 'e' and 'f' are factors. Train data sample size is approximately 6,000. I've tried running gbm with various bag.fraction, train.fraction, n.tree, and shrinkage values but still get the same result of all NaNs. Trees and SVM work without any problem on the same data. I even tried converting column 'f' to character, as it was suggested in previous posts, and it didn't work.
Edit: data has no NAs or invalid values. I tried one-hot encoding the 'f' column and still same results.
In my case, this issue was resolved by converting the dependent variable to character.
gbm_model <- gbm(as.character(a) ~ .
, data = train
, distribution="bernoulli"
, n.trees= 10
, shrinkage=0.001
, bag.fraction = 1
, train.fraction = 0.5
, n.minobsinnode = 3
, cv.folds = 0 # no cross-validation
, keep.data=TRUE
, verbose=TRUE
)
I am using R's r2dtable function to generate contingency tables with given marginals. However, when inspecting the resulting tables values look somewhat too concentrated to the midpoints. Example:
set.seed(1)
matrices <- r2dtable(1e4, c(100, 100), c(100, 100))
vec.vals <- vapply(matrices, function(x) x[1, 1], numeric(1))
> table(vec.vals)
vec.vals
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
1 1 1 7 25 49 105 182 268 440 596 719 954 1072 1152 1048
52 53 54 55 56 57 58 59 60 61 62
1022 775 573 404 290 156 83 50 19 6 2
So the minimal upper left corner value is 36 and the max is 62 out of 10,000 simulations.
Is there a way to achieve somewhat less concentrated matrices?
You need to consider that it would be extremely unlikely that any given random draw would have a value with and upper left corner of 35. 1e4 attempts may not be sufficient to realize such an event. Look at the theoretic predictions (courtesy of P. Dalgaard on Rhelp list this morning.):
round(dhyper(0:100,100,100,100)*1e4)
[1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[18] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[35] 0 0 0 1 4 9 21 45 88 160 269 417 596 787 959 1081 1124
[52] 1081 959 787 596 417 269 160 88 45 21 9 4 1 0 0 0 0
[69] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[86] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
If you increase the number of draws the probability of a single value of 1 "widens":
vec.vals <- vapply(matrices, function(x) x[1, 1], numeric(1)); table(vec.vals)
vec.vals
33 34 35 36 37 38 39 40 41 42 43 44 45
1 3 8 47 141 359 864 2148 4515 8946 15928 27013 41736
46 47 48 49 50 51 52 53 54 55 56 57 58
59558 78717 96153 108322 112524 107585 96042 78054 60019 41556 26848 16134 8627
59 60 61 62 63 64 65 66 68
4580 2092 933 351 138 42 11 4 1
... as predicted:
round(dhyper(0:100,100,100,100)*1e6)
[1] 0 0 0 0 0 0 0 0 0 0 0 0
[13] 0 0 0 0 0 0 0 0 0 0 0 0
[25] 0 0 0 0 0 0 0 0 0 1 4 13
[37] 43 129 355 897 2087 4469 8819 16045 26927 41700 59614 78694
[49] 95943 108050 112416 108050 95943 78694 59614 41700 26927 16045 8819 4469
[61] 2087 897 355 129 43 13 4 1 0 0 0 0
[73] 0 0 0 0 0 0 0 0 0 0 0 0
[85] 0 0 0 0 0 0 0 0 0 0 0 0
[97] 0 0 0 0 0
To get less concentrated matrices, you will have to find a balance between the number of columns / rows, totals and number of matrices. Consider the following sets:
m2rep <- r2dtable(1e4, rep(100,2), rep(100,2))
m2seq <- r2dtable(1e4, seq(50,100,50), seq(50,100,50))
which gives differences in number of unique value:
> length(unique(unlist(m2rep)))
[1] 29
> length(unique(unlist(m2seq)))
[1] 58
plotting this with:
par(mfrow = c(1,2))
plot(table(unlist(m2rep)))
plot(table(unlist(m2seq)))
gives:
Now consider:
m20rep <- r2dtable(1e4, rep(100,20), rep(100,20))
m20seq <- r2dtable(1e4, seq(50,1000,50), seq(50,1000,50))
which gives:
> length(unique(unlist(m20rep)))
[1] 20
> length(unique(unlist(m20seq)))
[1] 130
plotting this with:
par(mfrow = c(1,2))
plot(table(unlist(m20rep)))
plot(table(unlist(m20seq)))
gives:
As you can see, playing with the parameters helps.
HTH
I have to matrices (or data frames) that contain the same column and rownames.
The rownames of matrix 1 has an ID called dataTissue, the rownames of matrix 2 has the ID dataSerum. I would like to combine the two matrices where identical rows are placed adjacent (on top of each other). Please see my desired output.
I was thinking of using rbind, but I am not sure how to get this structure.
matrix 1:
> head(TumorTissue3)
020 045 080 082 084 086 088 090 091 092 094 096 1018 102 1065
dataTissue.hsa-let-7a-2-3p 1 0 1 0 0 0 1 1 0 2 0 0 0 5 0
dataTissue.hsa-let-7a-3p 2 0 0 0 1 0 1 1 0 2 1 1 1 1 0
dataTissue.hsa-let-7a-5p 67 12 25 34 40 115 42 33 26 58 22 149 64 178 52
dataTissue.hsa-let-7b-3p 11 5 10 15 1 34 29 59 16 30 11 44 11 65 3
dataTissue.hsa-let-7b-5p 4289 689 902 3340 3947 7326 3146 6249 2032 5664 1657 6619 1577 21132 720
dataTissue.hsa-let-7c-3p 1 0 0 2 0 9 2 13 2 10 2 13 5 9 0
1068 1104 112 113 1167 1196 120 121 1222 1237 1241 1302 1304 1322 134
dataTissue.hsa-let-7a-2-3p 2 11 0 0 0 0 3 0 2 1 0 0 3 5 1
dataTissue.hsa-let-7a-3p 0 0 1 0 0 1 0 0 1 4 1 0 2 3 0
dataTissue.hsa-let-7a-5p 70 266 60 8 29 99 90 37 102 93 28 22 156 214 176
dataTissue.hsa-let-7b-3p 14 15 24 12 8 8 43 25 14 33 9 12 16 38 11
dataTissue.hsa-let-7b-5p 1780 4185 5797 1168 1039 1006 10818 3269 2893 8847 3136 4990 1798 10142 3248
dataTissue.hsa-let-7c-3p 5 7 5 2 1 3 3 3 1 10 27 1 17 11 3
1372 140 145 146 1474 1532 1540 157 158 1588 1604 161 1743 176
dataTissue.hsa-let-7a-2-3p 0 0 0 0 6 1 10 0 6 1 0 1 0 2
dataTissue.hsa-let-7a-3p 1 0 1 0 0 1 0 0 3 0 0 2 0 1
dataTissue.hsa-let-7a-5p 18 1 53 17 129 54 110 2 165 70 51 165 81 77
dataTissue.hsa-let-7b-3p 3 0 22 12 46 3 60 0 79 9 15 40 3 50
dataTissue.hsa-let-7b-5p 931 245 3707 3632 16730 2653 13619 93 27568 3485 6202 18206 3094 11185
dataTissue.hsa-let-7c-3p 1 0 12 0 10 0 5 0 20 10 8 7 2 9
1808 1809 185 1859 186 1894 192 201 204 21 215 2218 236 27 32
dataTissue.hsa-let-7a-2-3p 2 1 1 0 1 0 0 0 3 0 2 6 0 3 5
dataTissue.hsa-let-7a-3p 0 0 0 0 1 0 0 0 1 0 3 3 0 1 0
dataTissue.hsa-let-7a-5p 33 160 56 16 92 63 90 3 119 58 116 46 37 137 40
dataTissue.hsa-let-7b-3p 11 1 23 10 18 3 48 14 34 16 54 23 12 96 33
dataTissue.hsa-let-7b-5p 3497 548 5575 2886 6664 1030 5895 604 8151 4076 14150 11132 2154 24793 5654
dataTissue.hsa-let-7c-3p 3 3 5 6 6 2 4 2 6 9 20 6 2 11 6
38 39 45 46 bf33 d10 HEP014 HEP015 mm7 s26 TxHEP-014 TxHEP-015
dataTissue.hsa-let-7a-2-3p 0 0 6 0 0 0 1 2 2 4 0 1
dataTissue.hsa-let-7a-3p 0 0 0 2 0 2 2 0 1 2 0 1
dataTissue.hsa-let-7a-5p 18 75 192 41 41 88 55 119 24 112 223 25
dataTissue.hsa-let-7b-3p 6 16 56 11 17 24 8 29 12 29 7 18
dataTissue.hsa-let-7b-5p 1648 2805 19275 1769 4554 5316 1552 7605 2369 7495 33820 2144
dataTissue.hsa-let-7c-3p 3 1 14 2 2 3 6 18 1 23 3 2
TxHEP-018 vs29
dataTissue.hsa-let-7a-2-3p 1 1
dataTissue.hsa-let-7a-3p 0 0
dataTissue.hsa-let-7a-5p 13 50
dataTissue.hsa-let-7b-3p 23 47
dataTissue.hsa-let-7b-5p 1631 4990
dataTissue.hsa-let-7c-3p 1 11
matrix 2:
> head(Serum3)
020 045 080 082 084 086 088 090 091 092 094 096 1018 102 1065 1068
dataSerum.hsa-let-7a-2-3p 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
dataSerum.hsa-let-7a-3p 4 2 0 2 329 1 186 0 2 4 1 6 13 7 15 3
dataSerum.hsa-let-7a-5p 988 2033 587 1480 4035 1167 4641 761 668 4118 6040 2660 10368 5802 5668 2709
dataSerum.hsa-let-7b-3p 9 8 4 18 76 3 62 1 5 24 9 9 41 10 30 6
dataSerum.hsa-let-7b-5p 1499 849 108 868 3197 202 2411 273 224 1309 943 822 5819 1594 3335 1164
dataSerum.hsa-let-7c-3p 0 0 0 0 29 0 11 0 0 0 5 0 2 0 0 1
1104 112 113 1167 1196 120 121 1222 1237 1241 1302 1304 1322 134 1372
dataSerum.hsa-let-7a-2-3p 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
dataSerum.hsa-let-7a-3p 15 2 0 8 5 2 0 11 13 51 4 0 1 7 0
dataSerum.hsa-let-7a-5p 30222 1836 1518 3902 5122 4597 983 3809 6310 3165 4023 434 2489 1496 600
dataSerum.hsa-let-7b-3p 57 2 1 14 19 14 2 14 35 162 10 0 10 11 6
dataSerum.hsa-let-7b-5p 11314 329 354 2169 2277 747 256 1157 3328 3662 1057 274 1267 991 305
dataSerum.hsa-let-7c-3p 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0
140 145 146 1474 1532 1540 157 158 1588 1604 161 1743 176 1808 1809
dataSerum.hsa-let-7a-2-3p 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
dataSerum.hsa-let-7a-3p 27 10 0 2 2 5 0 40 0 0 19 0 1 4 4
dataSerum.hsa-let-7a-5p 5364 12731 670 473 1045 767 927 49689 535 8 78671 757 1502 1146 539
dataSerum.hsa-let-7b-3p 63 37 1 3 14 5 10 59 1 0 56 6 3 12 6
dataSerum.hsa-let-7b-5p 2262 3209 88 363 759 459 309 13482 234 3 15113 1064 545 587 569
dataSerum.hsa-let-7c-3p 6 0 0 0 0 0 1 0 0 0 0 0 1 0 0
185 1859 186 1894 192 201 204 21 215 2218 236 27 32
dataSerum.hsa-let-7a-2-3p 0 0 0 0 0 3 0 0 0 0 0 0 0
dataSerum.hsa-let-7a-3p 16 0 101 3 7 346 10 1 93 0 305 6 12
dataSerum.hsa-let-7a-5p 42694 528 18730 498 3410 20484 11907 1031 474051 2185 299085 14576 9218
dataSerum.hsa-let-7b-3p 24 7 164 14 16 85 29 12 111 9 145 32 12
dataSerum.hsa-let-7b-5p 5454 216 4647 182 1149 8973 2645 147 72681 807 46354 4672 2375
dataSerum.hsa-let-7c-3p 0 0 9 0 0 32 0 0 0 0 3 1 0
38 39 45 46 bf33 d10 HEP014 HEP015 mm7 s26 TxHEP-014 TxHEP-015
dataSerum.hsa-let-7a-2-3p 0 0 0 1 0 0 0 0 0 0 0 0
dataSerum.hsa-let-7a-3p 9 17 6 119 27 0 1 5 0 10 7 1
dataSerum.hsa-let-7a-5p 2395 4382 8361 9747 6440 616 2981 5851 291 1386 3709 2494
dataSerum.hsa-let-7b-3p 18 28 24 104 33 6 12 24 5 36 2 11
dataSerum.hsa-let-7b-5p 690 1756 3425 3972 2330 136 1035 2152 235 638 555 1409
dataSerum.hsa-let-7c-3p 1 3 2 10 1 0 0 0 0 2 0 0
TxHEP-018 vs29
dataSerum.hsa-let-7a-2-3p 0 0
dataSerum.hsa-let-7a-3p 0 2
dataSerum.hsa-let-7a-5p 397 266
dataSerum.hsa-let-7b-3p 0 9
dataSerum.hsa-let-7b-5p 67 182
dataSerum.hsa-let-7c-3p 0 0
Output:
dataTissue.hsa-let-7a-2-3p 23 24 35 ....
dataSerum.hsa-let-7a-2-3p 42 535 54 ....
dataTissue.hsa-let-7a-3p 234 224 35 ....
dataSerum.hsa-let-7a-3p 2 33 54 ....
We can rbind both the matrices and then order by the rownames after removing the prefix part 'dataTissue./dataSerum.' from the row names using sub.
res <- rbind(TumorTissue3, Serum3)
nm1 <- sub('^[.]+\\.', '', row.names(res))
res[order(nm1),]
I am doing factor analysis with 500 variables and 1316 records. I am getting the following error:
Error in solve.default(cv) :
system is computationally singular: reciprocal condition number = 2.5024e-18
The syntax I am using is:
factor.out1 <- factanal(covmat=cor(factor3, use="complete.obs"),factor=50,rotation="none")
Any suggestion will be of great help.
Thanks
com.kakao.talk com.android.phone com.gau.go.launcherex com.android.browser com.android.contacts com.nhn.android.search com.kakao.story
1 61 616 0 363 48 0 4
2 180 192 0 70 94 60 10
3 270 109 0 411 53 0 0
4 610 55 0 23 47 201 18
5 653 0 0 11 0 153 64
6 341 147 0 2 0 58 437
7 167 118 0 1 1 109 0
8 0 379 0 4 289 121 0
9 1254 93 1434 616 86 7 268
10 1907 111 328 0 119 1073 0
here is a sample of 10 obs and 7 rows