Calculate 90th percentile across columns in a data frame - r

I'm trying to calculate the 90th percentile of all station fecal samples by sample date, across columns in a data frame. It would be nice to be able to add this as a new column but not absolutely necessary.
I rearrange my data in the following way, although I don't know if this is necessary. It is easy for me to visualize this way.
library(dplyr)
FecalData <- RawData %>%
select(Station, SampleDate, FecalColiform)
#Rearange by station
library(reshape2)
FecalbyStation <- dcast(FecalData, SampleDate ~ Station, fun.aggregate = mean, na.rm = TRUE)
This leaves me with the following sturcture:
dput(FecalbyStation[1:5,])
structure(list(SampleDate = structure(c(6942, 6979, 7014, 7042,
7070), class = "Date"), `114` = c(114.5, 2, 17, 7.9, 1.8), `115` = c(41,
6.8, 33, 220, 4.5), `116` = c(64, 4, 14, 6.8, 1.8), `117` = c(33,
2, 4.5, 1.8, 2), `118` = c(81.5, 2, 6.8, 33, 1.8), `119` = c(28,
11, 4.5, 1.8, 2), `120` = c(64, 4.5, 11, 1.8, 1.8), `121` = c(31,
4.5, 3.6, 13, 2), `122` = c(41, 2, 33, 13, 1.8), `123` = c(28,
7.8, 2, 13, 1.8), `124` = c(NaN, 7.8, NaN, NaN, NaN), `125` = c(NaN,
NaN, NaN, NaN, NaN), `126` = c(NaN, NaN, NaN, NaN, NaN), `127` = c(NaN,
NaN, NaN, NaN, NaN), `128` = c(NaN, NaN, NaN, NaN, NaN), `129` = c(NaN,
NaN, NaN, NaN, NaN), `614` = c(NaN, NaN, NaN, NaN, NaN), `615` = c(NaN,
NaN, NaN, NaN, NaN), `639` = c(NaN, NaN, NaN, NaN, NaN), `758` = c(NaN,
NaN, NaN, NaN, NaN)), .Names = c("SampleDate", "114", "115",
"116", "117", "118", "119", "120", "121", "122", "123", "124",
"125", "126", "127", "128", "129", "614", "615", "639", "758"
), row.names = c(NA, 5L), class = "data.frame")
I have been able to find row.means() this way and have tweaked this code over and over again to try to get the 90th percentile instead. I have received several different errors along the way. Here is the code I have landed on:
library(psych)
Q90 <- sapply(FecalbyStation, -1, quantile, probs=c(.90), na.rm = TRUE)
This gives me the following error:
Error in match.fun(FUN) : '-1' is not a function, character or symbol
Ultimately I would like to make the resulting 90th percentiles a time series so that I can run a kendall or regression on it to investigate any trend in fecal levels for the region. Any suggestions or advice is much appreciated.
Thank you!


You can keep your data in long form and get the 90th percentile by date as follows:
library(dplyr)
RawData %>% group_by(SampleDate) %>%
summarise(p90 = quantile(FecalColiform, probs=0.9, na.rm=TRUE))

Related

Loop over several columns while appending results to new columns using the name of the exisiting columns to name the new columns

The following for-loop loops over columns 6:22 in poll_22 to calculate lower and upper confidence intervals for each political party. The lower and upper intervals are then saved in two new columns. However, the code I have written overwrites the calculated intervals resulting in poll_22$lowerinterval and poll_22$upperinterval containing calculated intervals for the last political party in column 22.
Is it possible to add the name of the column i.e. the letter of a political party when appending the confidence intervals in the last two lines? Intervals for column 6 named A would then be poll_22$A_upperinterval and poll_22$A_lowerinterval. Column 7 named B would be poll_22$B_upperinterval and poll_22$B_lowerinterval etc.
for(i in poll_22[, 6:22]) {
n <- poll_22$n # sample size
p <- i/100 # party
# calculate confidence interval
margin <- qnorm(0.975)*sqrt(p*(1-p)/n)
# calculate upper and lower intervals
lowerinterval <- (p - margin)*100
upperinterval <- (p + margin)*100
# append intervals
poll_22$lowerinterval <- lowerinterval
poll_22$upperinterval <- upperinterval
}
Head of poll_22
poll_22 <- structure(list(id = c(1555, 1556, 1557, 1558, 1559, 1560), pollingfirm = c("VOXMETER",
"VOXMETER", "MEGAFON", "VOXMETER", "VOXMETER", "VOXMETER"), year = c(2022,
2022, 2022, 2022, 2022, 2022), month = c(1, 1, 1, 1, 1, 2), day = c(8,
16, 20, 23, 30, 6), A = c(25.1, 25.8, 23.9, 26.8, 24.9, 24),
B = c(7.2, 7.5, 6.9, 6.9, 7.7, 7.2), C = c(15, 14.6, 18.8,
15.5, 16.4, 16.1), D = c(6, 5.9, 7.9, 6.8, 6, 5.4), E = c(NA,
NA, NA, NA, NA, NA), F = c(8.6, 8.5, 8.9, 8, 8.6, 9.2), G = c(NA,
NA, NA, NA, NA, NA), I = c(2.5, 3.3, 2.7, 2.9, 2.1, 2.8),
K = c(1.8, 1.4, 1.6, 1.2, 1.7, 1.8), M = c(NA, NA, 2.2, NA,
NA, NA), O = c(6.2, 5.3, 4.5, 5.5, 7.1, 6.2), P = c(NA, NA,
NA, NA, NA, NA), Q = c(0.1, 0.3, 0.2, 0.1, 0, 0.2), V = c(16.5,
15.1, 11.6, 14.4, 14.2, 14.8), Ø = c(8.9, 9.3, 9.4, 9.4,
8.3, 9.1), Å = c(1.2, 1.1, 0.7, 0.8, 0.9, 0.9), Æ = c(NA_real_,
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_), noparty = c(24.9,
23.5, NA, NA, 25, 25.8), n = c(1002, 1003, 2015, 1008, 1015,
1024)), row.names = c(NA, -6L), class = c("tbl_df", "tbl",
"data.frame"))

If I understood what you are looking for correctly, then here is a solution using dplyr:
library(dplyr)
poll_22 %>%
mutate_at(vars(6:22), ~./100) %>% # performs p <- i/100 for columns 6:22
mutate_at(vars(6:22), list(lowerinterval = ~100 * (. - qnorm(0.975)*sqrt(.*(1-.)/n),
upperinterval = ~100 * (. + qnorm(0.975)*sqrt(.*(1-.)/n))) %>%
mutate_at(vars(6:22), ~100*.) # reverts values in columns 6:22

Multi-column data.frame as list argument to purrr::pmap for iteration over a data frame

I am trying to run a function to calculate proportion of time within a specified range over each row in a data frame using purrr::pmap, however one of the arguments for the original function is 11 columns. This means that the function works fine over a single row but when iterating over every row, it fails:
df_test <- structure(list(duration = c(268, 264, 256, 200, 296, 60, 16,
396, 400, 388), d1 = c(22.4, 12.4, 15.6, 21.6, 15.2, 2, 1.2,
2.4, 2, 1.6), d2 = c(25.2, 16, 16.8, 28.4, 16.8, 2, 1.6, 2.4,
2, 2), d3 = c(24.8, 18.4, 16.4, 28, 16.4, 2, 1.6, 2.4, 2, 2),
d4 = c(21.6, 20.4, 18, 24.4, 16.4, 2, 1.6, 2.8, 2, 1.6),
d5 = c(24.4, 20.4, 19.2, 25.2, 17.6, 1.6, 1.6, 2.8, 2, 2),
d6 = c(24.8, 19.6, 18.8, 24, 19.2, 1.6, 1.6, 2.8, 2, 2),
d7 = c(22.8, 19.6, 18, 23.6, 18.8, 1.6, 2, 2.8, 2.4, 1.6),
d8 = c(17.6, 18.4, 17.6, 22.4, 20.8, 1.6, 2, 2.8, 2.4, 2),
d9 = c(20.4, 16.4, 16.8, 20.4, 23.2, 1.6, 2, 2.8, 2.4, 2),
d10 = c(NA_real_, NA_real_, NA_real_, NA_real_, NA_real_,
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_), d11 = c(NA_real_,
NA_real_, NA_real_, NA_real_, NA_real_, NA_real_, NA_real_,
NA_real_, NA_real_, NA_real_), bottom = c(18.02, 14.03, 14.03,
20.3, 18.95, 4.54, 4.54, 4.54, 4.54, 4.54)), row.names = c("5",
"6", "7", "9", "10", "17", "18", "19", "20", "21"), class = "data.frame")
prop_fun <- function(time, d, bottom, mini=5, maxi=25 ) {
#temp = df[i,]
Secs = seq(from=0, to=time, by=1) # create sequence of one second intervals
DT = seq(from = 0, to=time, by=time/10)
if(is.na(d[11])){
DD = data.frame(c(0, d[1:9], 0))
# interpolate d
d.reg = approx(DT, DD, xout=Secs)$y
d.regS = bottom-d.reg
output = length(d.regS[d.regS>mini & d.regS<maxi ])/time
} else {
DD = data.frame(c(0, d[1:11], 0))
d.reg = approx(DT, DD, xout=diveSecs)$y
d.regS = bottom-d.reg
output = length(d.regS[d.regS>mini & d.regS<maxi ])/time
}
return(output)
}
prop_fun(df_test$duration[1], df_test[1,2:12],df_test$bottom[1])
# [1] 0.1268657
argument_list <- list(time=df_test$duration, d=df_test[,2:12],bottom=df_test$bottom, mini=5, maxi=25 )
try <- purrr::pmap(argument_list, prop_fun)
#Error: Element 2 of `.l` must have length 1 or 10, not 11
Not quite sure how the input differs between the single run and the list argument to purrr::pmap. Do I need to just include all separate columns as separate arguments in prop_fun and concatenate them inside ?

length(df) = ncol(df), but I think you want to pass each row to your function. You can extract a list of rows using:
d = lapply(1:nrow(df_test[, 2:12]), function(x) df_test[x, 2:12])
Of course, for purrr::pmap you can also assemble everything into a data.frame or tibble and pass along:
tibble::tibble(
time = df_test$duration,
bottom = df_test$bottom,
d = lapply(1:nrow(df_test[, 2:12]), function(x) df_test[x, 2:12]),
mini = 5,
maxi = 25
) %>%
purrr::pmap(prop_fun)

How to avoid taking negative base of power function during the problem solving process using NLsolve in Julia?

I am a Julia beginner. I would like to solve the following non-linear equation using nlsolve.
#Variables
D= 200 #number of dimension
w= [0.17935458155165915; 0.02074763117110885; 0.429373018098153; 0.05169130596707894; 0.1268251892348001; 0.28504924497346273; 0.2653879531047568; 0.28907305874421907; 0.3211732529510658; 0.19640464810886332; 0.23743064655310908; 0.31271077337877673; 0.48784812550619117; 0.46951525260820676; 0.414672793999191; 0.2568327281950456; 0.04655836280984356; 0.3553694395191944; 0.0961544234898889; 0.36088980496765105; 0.37309711801193857; 0.47543051066838987; 0.27729500688542985; 0.3655300449541484; 0.16055046782127508; 0.4715786828535815; 0.2921904371516787; 0.0867716272042234; 0.02458893193755404; 0.40826853858704576; 0.13564939368933704; 0.08565749854632393; 0.06533558241443299; 0.06402050462910032; 0.3671937923414826; 0.1720370049889512; 0.5002983463084618; 0.4554957808473552; 0.46517364874849754; 0.1212216024211966; 0.04284454643266707; 0.37915786950647656; 0.17464050734498965; 0.21533246021336955; 0.05582083301947215; 0.01890049888476679; 0.06796265527558022; 0.13497310074129715; 0.20160681047548307; 0.30315821352156913; 0.3251250459650451; 0.07933939439789317; 0.05474492295189952; 0.04305583367874512; 0.4361224247290211; 0.16432217073350208; 0.31303482284024337; 0.4451670195690176; 0.19978224229965896; 0.005269209445990741; 0.5169890042928693; 0.49791983700852965; 0.017989427943135895; 0.20315428046276351; 0.06689193239618459; 0.3680050603415692; 0.04551910758173742; 0.01662604468426492; 0.06004114983952084; 0.4478453864302301; 0.39411123774712575; 0.3094838485413219; 0.33815112721353957; 0.4240423855967578; 0.12932548776099762; 0.29086619751681675; 0.4030396981865735; 0.5054293368932627; 0.47285161364574857; 0.3098801558764114; 0.16515650220976294; 0.503913510788379; 0.4088656755934767; 0.22986629914797763; 0.18001932518476532; 0.33046480913959986; 0.23212104534106723; 0.2803414905211751; 0.5049525882832877; 0.08521812980417476; 0.4069606921394297; 0.2181371065953835; 0.053727840369329705; 0.17617975359933452; 0.4694543012021901; 0.34756349021452404; 0.48518448761421923; 0.02368032680125083; 0.490453983772221; 0.29518522250300117; 0.029269655059924668; 0.10399922894246545; 0.36353617514053105; 0.03387829393121781; 0.025490776554460855; 0.4310812832361403; 0.028795585312659543; 0.4933248156252178; 0.3656192375750969; 0.4589261148594532; 0.3288070975147629; 0.040785776187926344; 0.5022914836438017; 0.5160150437151949; 0.5242657868366228; 0.34618308317731006; 0.1319908503735835; 0.47195419822851153; 0.05207014105918679; 0.19467771783330873; 0.49166830880087564; 0.13531702218739206; 0.1823017910366312; 0.030660444370685203; 0.3962884724121194; 0.25086570699970984; 0.3959784732876863; 0.1641317515418158; 0.07964603629529568; 0.26417075311032945; 0.393654266782284; 0.20930682065612094; 0.05653175390672894; 0.37420934462301003; 0.23460192018381498; 0.46636742061631675; 0.14086364242937774; 0.08161750625372813; 0.4862927691208331; 0.05356082693328593; 0.5055848873287434; 0.028812527866598325; 0.18581121132195139; 0.43018487478885936; 0.4473538614535335; 0.4583871013830452; 0.0412195586331303; 0.03141717976980104; 0.4676441420752301; 0.034581891245706556; 0.36771481416639373; 0.4515357316321425; 0.13797591008597196; 0.026251893538842566; 0.03795891476862359; 0.029165568365405; 0.105839838518907; 0.5078856492325493; 0.2576434906126355; 0.31916462500676; 0.05572200820553658; 0.5307195502657114; 0.35363316006975987; 0.3449622649355296; 0.04727417005006297; 0.20715891655792446; 0.3304136066693219; 0.34214236013075755; 0.13725527724782474; 0.4194480554053426; 0.43190606219458283; 0.4925351054148299; 0.0368272519895124; 0.037005236939482836; 0.05482024474064293; 0.4658322939791588; 0.04183361864909097; 0.2819231154532247; 0.3071668925286857; 0.203845596772382; 0.039560321686506805; 0.08843817736989967; 0.44467968454990625; 0.16610048417579049; 0.4144496080954068; 0.39129200825792365; 0.3821227311795474; 0.40997465387715437; 0.15714005965819433; 0.30726849530055533; 0.4465691501738437; 0.10298380183276976; 0.10168582441320018; 0.030435150474560184; 0.13526831068591066; 0.3811373848822741; 0.39583398602024567; 0.3564651897756602; 0.045102997790797995; 0.06510548733856235]
ε = 6.83
B= [1.9113880593260923; 8.883378033508055; 1.5206988306259004e-20; 0.12873546853181547; 0.037433247431133146; 5.02095391224869e-152; 1.4869904879595479e-49; 4.511864438683239e-21; 2.1083439669659596e-12; 1.573705822018202e-20; 1.3453053264248006e-77; 1.5127934067991649; 4.717077633858784e-15; 3.4700877943503913e-16; 1.3322445962118157e-8; 11.06063549629749; 22.527098400242068; 1.2856792580100785e-26; 18.906029147573; 0.000655510993151066; 10.817043057533661; 0.016479254141517467; 27.298101247774202; 9.148687139608766e-23; 28.441081060976; 5.827459408920411e-9; 0.015917609194167417; 5.921299277490823; 15.131095232259607; 4.5221452182504495e-101; 0.3745605564474358; 35.22790120220832; 26.472693215422034; 15.748913895092457; 13.900215150506208; 0.6656796732438512; 1.0610761914228794e-166; 0.02034815783200824; 6.94235044611682e-21; 3.024382319727409; 4.403762322109802; 19.642152609916533; 0.312733600677696; 7.0450348072031685; 2.9943360410894515; 0.020386834152167167; 5.019796883409017; 1.953805941833118e-42; 26.218829427864886; 1.1679909471382518e-12; 4.700747703804262e-158; 0.739539471301838; 0.03947830752271118; 8.191358248239432; 9.382504981038742e-54; 0.16010589376742393; 0.009015716607052703; 3.637358355198932e-20; 27.327796808642322; 0.8004499183860179; 8.461125192766686e-18; 8.95910135749446e-14; 0.46263795683695763; 3.724763235623522e-29; 0.4627651339879373; 2.1317843123017353; 19.72612759625619; 7.121514602205041; 0.7498244950119917; 7.82776792212211e-29; 2.058164763314051e-48; 6.656376841007709e-110; 1.0185480326011094e-8; 1.0966196183164407e-40; 3.81587425576929; 17.022148683535356; 2.8781418780364653e-226; 6.283831883355779e-60; 7.103441240122817e-17; 1.7971618832549945e-128; 18.528854800853065; 4.497936571326762e-42; 5.9611599404636495; 9.131948748136594; 2.138817581505318; 6.963672986053975e-141; 0.005445679519556468; 1.6679405650005341; 2.3337652049764904e-22; 1.419494251779325; 2.261935115923545e-46; 1.2376566162579915e-32; 4.328787591872048; 5.844233889688261e-7; 8.830924221173814e-34; 4.377778889817346e-81; 1.8878051058094012; 0.2998408805976853; 3.9392863709967796; 2.8084798567644255e-7; 0.14707594960663317; 38.59529055572117; 5.209103347927197e-21; 0.0025973623652913317; 1.5593014209459997; 1.3107443853139182e-21; 5.179647971121189; 3.236157265411051e-16; 7.093888840049452e-26; 1.5564589470829775e-71; 0.001741657884835898; 2.627144619341441; 0.04548803609276873; 2.4673234325112487e-29; 1.3507640402698248e-87; 4.38379124625374e-107; 19.316618329311616; 1.2877899257156492e-8; 22.33866240442435; 7.477065799186867; 1.7971426221661947e-49; 3.3345318772733243; 1.603348886130618; 9.19752238851656; 6.356526861225398e-67; 18.93162033259596; 1.2485391021861125e-98; 17.39152256819622; 0.891805611066936; 35.20596843419915; 1.6898318496500515; 5.638826369435781e-41; 1.2229640299979205; 1.6370815943900768e-74; 0.9756549709108758; 1.3904665924398121e-6; 1.8582909404291397; 1.9869657417523816; 9.819641204968615e-5; 11.71516179379171; 2.226782016861399e-7; 2.0750601055907887; 2.1347726595107657e-10; 1.683191425244726e-46; 2.3584667613037514e-16; 18.624750803153773; 17.12056336027302; 0.36126716256726055; 1.2788867483362755e-118; 1.532524744507334; 5.252933249817591e-44; 4.136636658945092e-32; 0.43950895342231955; 6.120212273528275; 0.987566032480816; 10.027807407458246; 0.34575101054546964; 4.261326022977069e-24; 9.140183928737398e-71; 7.162942951718289; 2.582956434666498; 2.9453506559779514e-10; 18.472793617032533; 7.085056386755511e-12; 36.40227058129134; 0.13079840647173033; 5.7395520820424e-69; 7.839955677157711e-62; 3.817437369660718e-33; 3.6404078004251237e-261; 1.252788930584791e-60; 1.8439518176020697e-42; 3.7077748286865955; 10.923507699176934; 4.398723765471304; 37.00277456879605; 7.163775800693194; 3.042070021429068e-36; 1.133732867786907e-27; 8.471981021660175; 17.313578618750533; 29.52963937643628; 1.6883607661890261e-18; 16.675675294485313; 8.083285501565638e-116; 1.7856881885430648e-71; 7.554472386121312e-6; 2.959280305979924e-54; 0.47485862974337695; 43.64815259851988; 0.00020435835048960367; 4.178015918890087; 0.7154468290561525; 19.95948484510194; 5.121764686065876; 1.4777202796969108e-44; 4.6531581853388824e-32; 4.262223219325753e-77; 10.910140454174218; 2.3511019348088893]
C= [0.02236367582581154; 9.58383778144876e-9; 10.622999409755542; 6.16241579726002e-6; 0.0018032815929029718; 0.5820246596178108; 0.46011206498448826; 0.7889991211217988; 1.4684689617091775; 0.05302456476377496; 0.10653991490875167; 0.8187437238615558; 21.42099755745115; 14.532608506948357; 7.130677101093516; 0.2695827025367284; 2.840960239678163e-6; 2.6149971669855403; 0.0002862249023265326; 2.206551045659344; 3.201701648734815; 15.84638486786215; 0.42036030490492404; 1.6441067874030098; 0.007690277824884048; 13.4229674661184; 0.776815090305774; 0.00017622198806910893; 4.047911673106026e-8; 8.52074610887776; 0.0015589533591112501; 0.00019850233659576524; 3.451312792433583e-5; 2.2765682832796673e-5; 3.4658361107945646; 0.02240030876897966; 21.24634498913638; 14.886659535117845; 20.331597580089422; 0.0015512506515751182; 1.1473872576179618e-6; 4.463818741209099; 0.024558108769859248; 0.08003730285262999; 1.0245101733923846e-5; 4.810690364785042e-9; 3.8449023058738855e-5; 0.004656842074317859; 0.05215675661535674; 0.8754064652759576; 1.3659241825975006; 0.00010536064903923549; 6.815257400995486e-6; 1.6332498205461194e-6; 14.838232003036559; 0.014148116381231153; 1.0747805759685072; 13.735602775324027; 0.05821913419275878; 7.072971977150658e-13; 36.085053287224845; 24.234945337082472; 3.775435520586117e-9; 0.06754539214330033; 3.5509190137439215e-5; 3.5579316268225694; 2.9411447208169765e-6; 2.5133421674066183e-9; 1.5062541187856675e-5; 14.694661779281075; 6.897268288709871; 0.7043989278476316; 1.9167230320590787; 10.073132706193185; 0.0028524971206745826; 0.6830477988007796; 6.521786417021922; 21.39217680761771; 17.141148981019942; 0.8465190498941498; 0.017625871401570122; 37.530744437384; 6.981342810181801; 0.12553899875906288; 0.02373963846881343; 1.344559849850729; 0.1690456323666673; 0.5512783866374767; 35.36197200806055; 0.0001408808253739378; 7.3508161565003585; 0.10860528902466145; 7.2717212309772544e-6; 0.016522549776560022; 16.473016905351493; 2.6127833707757824; 28.271280477163984; 3.3312692063423355e-8; 19.994267776549698; 0.6657430297670275; 1.0123785359436453e-7; 0.0006714142571638888; 2.4800026677881197; 3.382125591220219e-7; 4.7115287960519034e-8; 9.525619254857425; 8.271785513563928e-8; 21.490119694933856; 2.5062610399242327; 14.608813232400994; 1.8841651678257887; 8.421086239731675e-7; 28.750558264884454; 15.462293619736093; 29.46282215646338; 2.7066671596653933; 0.002592176417467211; 16.927305215283532; 5.655929890117654e-6; 0.04185013906013212; 23.089441205413877; 0.004178233766777603; 0.0278435050258412; 1.7453606686033147e-7; 4.569441678495397; 0.2937225655189823; 6.0288522810291365; 0.009970572371840814; 0.00010933874610531001; 0.48682413706500893; 6.207955072097215; 0.08854058360680914; 8.12806476837625e-6; 2.2342992920598417; 0.0711867387607783; 20.429841948532424; 0.0029902989914194706; 5.159006124218317e-5; 24.856022732116386; 5.7161295515373945e-6; 21.593135919447022; 1.034593572589091e-7; 0.03432570558664665; 11.8662784181379; 14.064150052406164; 16.58378730247712; 1.4641109475465026e-6; 1.513379889790401e-7; 13.486193224640694; 2.3415045686879677e-7; 1.7274421009808538; 17.664229193215167; 0.00401027321008269; 5.362881287338797e-8; 5.149019194174631e-7; 1.0329083624982976e-7; 0.0006822794649301988; 20.346835250844983; 0.2636146568801335; 1.1869695390218922; 9.279107770625925e-6; 19.249201278261733; 2.8466949459082866; 2.334280646591445; 2.5958954067544824e-6; 0.02604700360517058; 1.6444095893601727; 1.8524998416755938; 0.004535053868363867; 8.402808209415497; 10.821421853057345; 10.71048783445611; 5.708352034806126e-7; 4.4951717518111495e-7; 3.980711719757998e-6; 19.796380459889406; 1.2375725596404167e-6; 0.4824144273681875; 0.5732515483334024; 0.0646340619070183; 9.772350730146414e-7; 0.0002754947154589728; 12.308915197976354; 0.01706939966443881; 8.629220990605731; 5.029702735627101; 4.556678536635026; 6.743800116781311; 0.004807381626578681; 1.0724061894701296; 13.221076988100998; 0.0005047754753299702; 0.0004606181382141919; 1.1269545306708471e-7; 0.004833714336739734; 4.087813188584282; 3.054096162953476; 3.4826092556609756; 2.5451852316521397e-6; 2.664488740692585e-5]
#Function
function eq!(F,w)
numer = w.^ε
denomi = sum(numer,dims=2)
prob = numer./denomi
F[1:D]=(B'*prob)'-C
end
result=nlsolve(eq!,ones(S))
Then, I got the following error.
DomainError with -0.26327276428720575:
Exponentiation yielding a complex result requires a complex argument.
Replace x^y with (x+0im)^y, Complex(x)^y, or similar.
I assume that the error was occurred since some components of w became negative during the problem solving process.
So I modified the above equation as follows and make the components of w take positive values during the problem solving process.
function eq_new!(F,w)
if sum(w.<=0)==0
numer = w.^ε
denomi = sum(numer,dims=2)
prob = numer./denomi
F[1:D]=(B'*prob)'-C
else
F[1:D].= sum(abs.(w[w.<=0]))+100
end
end
result=nlsolve(eq_new!,ones(S))
However, the iteration didn't converge and all results became NaN.
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
* Zero: [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]
* Inf-norm of residuals: 1.203524
* Iterations: 1000
* Convergence: false
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: false
* Function Calls (f): 1000
* Jacobian Calls (df/dx): 172
How can I solve the above nonlinear equation without taking negative base of power function during the problem solving process?

Merging three vectors with NaN's

I have three vectors that look something like this:
x = c(NaN, 15, 16, 14, 14, NaN)
y = c(NaN, NaN, NaN, NaN, NaN, 11)
z = c(17, NaN, NaN, NaN, NaN, 12)
I need to merge them into one vector. I need these three to mesh together smoothly, and where there are overlapping values (as seen in y and z), priority should be given to the value from z. The end result should look like so:
xyz = c(17, 15, 16, 14, 14, 12)
I've looked into using rowSums to handle the na's, but this doesn't work in cases where there are multiple values on the same index. I'm trying to avoid for loops if possible.
Pretty sure I could do this by iterating through the vectors but I'm working with a fairly large data set.

This seems to work. NB It relies on z being the last column of the data frame, then flips it so that it's first (i.e., rev).
df <- data.frame(x = c(NaN, 15, 16, 14, 14, NaN),
y = c(NaN, NaN, NaN, NaN, NaN, 11),
z = c(17, NaN, NaN, NaN, NaN, 12))
do.call(dplyr::coalesce, rev(df))
You can also use zoo package:
df <- rbind(x, y, z)
#it replaces last row with latest non-NA value therefore z always has priority:
xyz <- zoo::na.locf(df)['z',]
#[1] 17 15 16 14 14 12
Data:
x <- c(NaN, 15, 16, 14, 14, NaN)
y <- c(NaN, NaN, NaN, NaN, NaN, 11)
z <- c(17, NaN, NaN, NaN, NaN, 12)

Or you can using base R function na.omit
x = c(NaN, 15, 16, 14, 14, NaN)
y = c(NaN, NaN, NaN, NaN, NaN, 11)
z = c(17, NaN, NaN, NaN, NaN, 12)
dt=data.frame(z=z,x=x,y=y)
unlist(lapply(apply(dt,1,na.omit), `[[`, 1))
[1] 17 15 16 14 14 12

plotting issue with autofitVariogram in automap

I am a newbie in R. I am using autofitVariogram to daily rainfall data of 50 stations.The sample data is provided below.Some of stations have missing values represented by "NaN" values.
My question is regarding the variogramfit. The variogram covers only a distance of 60,000m. Why are the points in bins beyond 60Km not plotted. I had seen from spatial correlation plot maximum distance from lon-lat information is >200Km.
The summary of latitide and longitude information is provided below.
summary(lonlat)
lon lat
Min. :74.78 Min. :15.77
1st Qu.:75.14 1st Qu.:16.04
Median :75.56 Median :16.33
Mean :75.54 Mean :16.37
3rd Qu.:75.94 3rd Qu.:16.66
Max. :76.31 Max. :17.23
$ Sample data given below:
dput(rain[140:145,])
structure(list(Col0 = c(0, 0, 1, 9, 6.5, 0), Col1 = c(1.5, 36,
21, 44, 4, 0), Col2 = c(0, 0, 24.5, 21.5, 7.5, 1), Col3 = c(0,
1, 45, 3, 0, 0), Col4 = c(2, 0, 5, 54.5, 13.5, 0), Col5 = c(0.5,
2, 0, 3.5, 13.5, 0), Col6 = c(0.5, 0, 0, 59, 15.5, 0), Col7 = c(0,
0, 2.5, 1, 0, 0), Col8 = c(0, 6, 24, 2, 5.5, 0), Col9 = c(0,
3, 6, 1, 0, 7), Col10 = c(0.5, 1, 64, 20, 1, 0.5), Col11 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col12 = c(0, 11, 75, 19, 15.5, 0),
Col13 = c(0, 4, 57.5, 50.5, 8.5, 0), Col14 = c(1.5, 0.5,
127, 33.5, 34.5, 0), Col15 = c(0, 7, 0.5, 13, 1, 0), Col16 = c(0,
0.5, 81.5, 15, 49, 0), Col17 = c(0, 0, 4.5, 17, 5.5, 1),
Col18 = c(0, 3, 2.5, 0.5, 0, 0), Col19 = c(NaN, NaN, NaN,
NaN, NaN, NaN), Col20 = c(0, 0, 0, 0, 7, 0), Col21 = c(0,
1, 0, 5, 3.5, 0), Col22 = c(0, 0, 11.5, 28, 3.5, 0), Col23 = c(0,
0, 48.5, 0, 24.5, 0), Col24 = c(0, 0, 0, 10, 0.5, 14), Col25 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col26 = c(0, 7.5, 16, 28.5, 20.5,
0), Col27 = c(1.5, 0.5, 38, 28.5, 50, 0), Col28 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col29 = c(NaN, NaN, NaN, NaN, NaN,
NaN), Col30 = c(2.5, 0, 0, 80.5, 28, 13.5), Col31 = c(1,
0, 17, 85.5, 3.5, 0), Col32 = c(0, 0.5, 8, 101, 20, 4), Col33 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col34 = c(4, 3, 17, 122, 2, 2),
Col35 = c(0, 15.5, 14.5, 20, 3.5, 0), Col36 = c(0, 6.5, 8.5,
21, 7, 0), Col37 = c(0, 0, 1.5, 14.5, 0, 1.5), Col38 = c(0,
28, 30, 4, 0, 73), Col39 = c(28.5, 0, 4.5, 9.5, 1, 0), Col40 = c(1.5,
11.5, 32.5, 55, 0, 1), Col41 = c(0, 14.5, 0, 19, 12.5, 47.5
), Col42 = c(0, 28, 29, 17, 0.5, 20.5), Col43 = c(NaN, NaN,
NaN, NaN, NaN, NaN), Col44 = c(0, 19, 3.5, 42, 0, 0), Col45 = c(0,
0, 85, 15.5, 1, 0), Col46 = c(0, 0.5, 8, 24, 0.5, 0), Col47 = c(0,
1.5, 7, 12, 8.5, 0), Col48 = c(0, 0, 0, 43.5, 0, 1.5), Col49 = c(0,
13.5, 1, 16, 1, 1)), .Names = c("Col0", "Col1", "Col2", "Col3",
"Col4", "Col5", "Col6", "Col7", "Col8", "Col9", "Col10", "Col11",
"Col12", "Col13", "Col14", "Col15", "Col16", "Col17", "Col18",
"Col19", "Col20", "Col21", "Col22", "Col23", "Col24", "Col25",
"Col26", "Col27", "Col28", "Col29", "Col30", "Col31", "Col32",
"Col33", "Col34", "Col35", "Col36", "Col37", "Col38", "Col39",
"Col40", "Col41", "Col42", "Col43", "Col44", "Col45", "Col46",
"Col47", "Col48", "Col49"), row.names = 143:148, class = "data.frame")
# Import the required libraries
library(rgdal)
library(maptools)
library(gstat)
library(sp)
library(automap)
library(XLConnect)
# Read the station data from xls file
stnrain = readWorksheetFromFile(path_fileName,"Sheet1", region = "D1:BA187", header = FALSE)
N = nrow(stnrain)
rain = stnrain[4:N,]
lat = as.numeric(t(stnrain[2,]))
lon = as.numeric(t(stnrain[3,]))
lonlat = cbind(lon,lat)
#Transform from GCS to UTM protection
sp = SpatialPoints(lonlat,proj4string = CRS("+proj=longlat"))
sp_utm = spTransform(sp, CRS("+proj=utm +zone=43N +datum=WGS84"))
krige_value = list() #prepare a list for storing the autokrige output
krige_stderr = list()
nRows = nrow(rain)
for (i in 1:nRows)
{
irain = rain[i,]
miss_indx = (irain == "NaN")
irain = irain[!miss_indx]
irain = as.numeric(irain)
isallZeros = (max(irain) == 0) # To take care of the cases of dry day(irain =0)
irain = as.data.frame(irain)
M = nrow(irain)
if ((M > 5) & (!isallZeros)) # To avoid cases of NaN across many stations
{
print(i)
foo_utm = sp_utm[!indx]# Removing the locations with NaN values
data = data.frame(foo_utm,irain)
names(data) = c("Easting","Northing","rain")
coordinates(data) = c("Easting","Northing")
variogram = autofitVariogram(rain~1,data,model = "Sph",fix.values=c(0,NA,NA))
p = plot(variogram, main="Semi-variogram (Spherical Model)",xlab="Distance(m)",ylab="Semi-Variance(mm2)", sub=paste("Range: ",variogram$var_model$range[2], "Day",i))
print(p)
png(p)
dev.off()
}
else
{
krige_value[[i]] = list(rep(0, L))
krige_stderr[[i]] = list(rep(0, L))
}
}
}
Q2) How can i save the variogram fit png file in a loop. I understand that dev.off() should be used after each saving the figure, which i had done, but I am not able to save the the figure.
Any help would be appreciated.
Thanks,
Any suggestions would be appreciated?

In regard to your first question, the sample variogram is built using points up to a maximum distance of around 1/3 of the diagonal of the area of interest. The assumption here is that points farther away form that are not related, and because they are not in the sample variogram or variogram model they are plotted. This is just a choice, and might not be the correct choice, but when I wrote autofitVariogram it seemed to work well for my data. The variogram model you show confirms this, the range is smaller than 60 km.
For saving your png's I have two suggestions. First, call the plot command inside the png() dev.off pair, so not:
print(p)
png()
dev.off()
but:
png()
print(p)
dev.off()
In addition, I would create meaningful names for the png files.
To create sets of variogram plots, I would use ggplot2. This uses geom_line and facet_wrap. ggplot2 cannot deal directly with gstat/automap variogram models, luckily you can create distance semivariance data using the function variogramLine from gstat. See for example figure 3.1, and the plots in appendix A of this report I wrote. This answer I wrote earlier does also include an example of using ggplot2 for spatial data, this time to plot a grid map.

Resources