Merging three vectors with NaN's - r

I have three vectors that look something like this:
x = c(NaN, 15, 16, 14, 14, NaN)
y = c(NaN, NaN, NaN, NaN, NaN, 11)
z = c(17, NaN, NaN, NaN, NaN, 12)
I need to merge them into one vector. I need these three to mesh together smoothly, and where there are overlapping values (as seen in y and z), priority should be given to the value from z. The end result should look like so:
xyz = c(17, 15, 16, 14, 14, 12)
I've looked into using rowSums to handle the na's, but this doesn't work in cases where there are multiple values on the same index. I'm trying to avoid for loops if possible.
Pretty sure I could do this by iterating through the vectors but I'm working with a fairly large data set.


This seems to work. NB It relies on z being the last column of the data frame, then flips it so that it's first (i.e., rev).
df <- data.frame(x = c(NaN, 15, 16, 14, 14, NaN),
y = c(NaN, NaN, NaN, NaN, NaN, 11),
z = c(17, NaN, NaN, NaN, NaN, 12))
do.call(dplyr::coalesce, rev(df))
You can also use zoo package:
df <- rbind(x, y, z)
#it replaces last row with latest non-NA value therefore z always has priority:
xyz <- zoo::na.locf(df)['z',]
#[1] 17 15 16 14 14 12
Data:
x <- c(NaN, 15, 16, 14, 14, NaN)
y <- c(NaN, NaN, NaN, NaN, NaN, 11)
z <- c(17, NaN, NaN, NaN, NaN, 12)


Or you can using base R function na.omit
x = c(NaN, 15, 16, 14, 14, NaN)
y = c(NaN, NaN, NaN, NaN, NaN, 11)
z = c(17, NaN, NaN, NaN, NaN, 12)
dt=data.frame(z=z,x=x,y=y)
unlist(lapply(apply(dt,1,na.omit), `[[`, 1))
[1] 17 15 16 14 14 12

Related

JS Line Chart - Null values \ Missing Data Points

I saw that in the .NET version of LightningChart that there is a way to add breaks in line charts. Is this possible in the JS version as well? I saw someone ask a similar question a couple years ago, but hoping that this has been implemented since then.
I've looked through the documentation and examples and didn't couldn't find anything. I'm hoping to replicate something that looks like this in ApexCharts:
https://apexcharts.com/javascript-chart-demos/line-charts/null-values/

Line breaks in LineSeries can be specified by using NaN.
At this time there is no official mention of supporting this, but this is just about to be changed with the next version release. Afterwards, please check the data input method documentation of each series type whether this is supported or not because it might be that not all features support it.
Here is a snippet with the same data as in your referenced ApexCharts example.
const {
lightningChart
} = lcjs
const chart = lightningChart().ChartXY()
const data = [
[5, 5, 10, 8, 7, 5, 4, NaN, NaN, NaN, 10, 10, 7, 8, 6, 9],
[10, 15, NaN, 12, NaN, 10, 12, 15, NaN, NaN, 12, NaN, 14, NaN, NaN, NaN],
[NaN, NaN, NaN, NaN, 3, 4, 1, 3, 4, 6, 7, 9, 5, NaN, NaN, NaN]
]
data.forEach(yValues => chart.addPointLineSeries().addArrayY(yValues))
<script src="http://unpkg.com/#arction/lcjs#3.3.0/dist/lcjs.iife.js"></script>

How to avoid taking negative base of power function during the problem solving process using NLsolve in Julia?

I am a Julia beginner. I would like to solve the following non-linear equation using nlsolve.
#Variables
D= 200 #number of dimension
w= [0.17935458155165915; 0.02074763117110885; 0.429373018098153; 0.05169130596707894; 0.1268251892348001; 0.28504924497346273; 0.2653879531047568; 0.28907305874421907; 0.3211732529510658; 0.19640464810886332; 0.23743064655310908; 0.31271077337877673; 0.48784812550619117; 0.46951525260820676; 0.414672793999191; 0.2568327281950456; 0.04655836280984356; 0.3553694395191944; 0.0961544234898889; 0.36088980496765105; 0.37309711801193857; 0.47543051066838987; 0.27729500688542985; 0.3655300449541484; 0.16055046782127508; 0.4715786828535815; 0.2921904371516787; 0.0867716272042234; 0.02458893193755404; 0.40826853858704576; 0.13564939368933704; 0.08565749854632393; 0.06533558241443299; 0.06402050462910032; 0.3671937923414826; 0.1720370049889512; 0.5002983463084618; 0.4554957808473552; 0.46517364874849754; 0.1212216024211966; 0.04284454643266707; 0.37915786950647656; 0.17464050734498965; 0.21533246021336955; 0.05582083301947215; 0.01890049888476679; 0.06796265527558022; 0.13497310074129715; 0.20160681047548307; 0.30315821352156913; 0.3251250459650451; 0.07933939439789317; 0.05474492295189952; 0.04305583367874512; 0.4361224247290211; 0.16432217073350208; 0.31303482284024337; 0.4451670195690176; 0.19978224229965896; 0.005269209445990741; 0.5169890042928693; 0.49791983700852965; 0.017989427943135895; 0.20315428046276351; 0.06689193239618459; 0.3680050603415692; 0.04551910758173742; 0.01662604468426492; 0.06004114983952084; 0.4478453864302301; 0.39411123774712575; 0.3094838485413219; 0.33815112721353957; 0.4240423855967578; 0.12932548776099762; 0.29086619751681675; 0.4030396981865735; 0.5054293368932627; 0.47285161364574857; 0.3098801558764114; 0.16515650220976294; 0.503913510788379; 0.4088656755934767; 0.22986629914797763; 0.18001932518476532; 0.33046480913959986; 0.23212104534106723; 0.2803414905211751; 0.5049525882832877; 0.08521812980417476; 0.4069606921394297; 0.2181371065953835; 0.053727840369329705; 0.17617975359933452; 0.4694543012021901; 0.34756349021452404; 0.48518448761421923; 0.02368032680125083; 0.490453983772221; 0.29518522250300117; 0.029269655059924668; 0.10399922894246545; 0.36353617514053105; 0.03387829393121781; 0.025490776554460855; 0.4310812832361403; 0.028795585312659543; 0.4933248156252178; 0.3656192375750969; 0.4589261148594532; 0.3288070975147629; 0.040785776187926344; 0.5022914836438017; 0.5160150437151949; 0.5242657868366228; 0.34618308317731006; 0.1319908503735835; 0.47195419822851153; 0.05207014105918679; 0.19467771783330873; 0.49166830880087564; 0.13531702218739206; 0.1823017910366312; 0.030660444370685203; 0.3962884724121194; 0.25086570699970984; 0.3959784732876863; 0.1641317515418158; 0.07964603629529568; 0.26417075311032945; 0.393654266782284; 0.20930682065612094; 0.05653175390672894; 0.37420934462301003; 0.23460192018381498; 0.46636742061631675; 0.14086364242937774; 0.08161750625372813; 0.4862927691208331; 0.05356082693328593; 0.5055848873287434; 0.028812527866598325; 0.18581121132195139; 0.43018487478885936; 0.4473538614535335; 0.4583871013830452; 0.0412195586331303; 0.03141717976980104; 0.4676441420752301; 0.034581891245706556; 0.36771481416639373; 0.4515357316321425; 0.13797591008597196; 0.026251893538842566; 0.03795891476862359; 0.029165568365405; 0.105839838518907; 0.5078856492325493; 0.2576434906126355; 0.31916462500676; 0.05572200820553658; 0.5307195502657114; 0.35363316006975987; 0.3449622649355296; 0.04727417005006297; 0.20715891655792446; 0.3304136066693219; 0.34214236013075755; 0.13725527724782474; 0.4194480554053426; 0.43190606219458283; 0.4925351054148299; 0.0368272519895124; 0.037005236939482836; 0.05482024474064293; 0.4658322939791588; 0.04183361864909097; 0.2819231154532247; 0.3071668925286857; 0.203845596772382; 0.039560321686506805; 0.08843817736989967; 0.44467968454990625; 0.16610048417579049; 0.4144496080954068; 0.39129200825792365; 0.3821227311795474; 0.40997465387715437; 0.15714005965819433; 0.30726849530055533; 0.4465691501738437; 0.10298380183276976; 0.10168582441320018; 0.030435150474560184; 0.13526831068591066; 0.3811373848822741; 0.39583398602024567; 0.3564651897756602; 0.045102997790797995; 0.06510548733856235]
ε = 6.83
B= [1.9113880593260923; 8.883378033508055; 1.5206988306259004e-20; 0.12873546853181547; 0.037433247431133146; 5.02095391224869e-152; 1.4869904879595479e-49; 4.511864438683239e-21; 2.1083439669659596e-12; 1.573705822018202e-20; 1.3453053264248006e-77; 1.5127934067991649; 4.717077633858784e-15; 3.4700877943503913e-16; 1.3322445962118157e-8; 11.06063549629749; 22.527098400242068; 1.2856792580100785e-26; 18.906029147573; 0.000655510993151066; 10.817043057533661; 0.016479254141517467; 27.298101247774202; 9.148687139608766e-23; 28.441081060976; 5.827459408920411e-9; 0.015917609194167417; 5.921299277490823; 15.131095232259607; 4.5221452182504495e-101; 0.3745605564474358; 35.22790120220832; 26.472693215422034; 15.748913895092457; 13.900215150506208; 0.6656796732438512; 1.0610761914228794e-166; 0.02034815783200824; 6.94235044611682e-21; 3.024382319727409; 4.403762322109802; 19.642152609916533; 0.312733600677696; 7.0450348072031685; 2.9943360410894515; 0.020386834152167167; 5.019796883409017; 1.953805941833118e-42; 26.218829427864886; 1.1679909471382518e-12; 4.700747703804262e-158; 0.739539471301838; 0.03947830752271118; 8.191358248239432; 9.382504981038742e-54; 0.16010589376742393; 0.009015716607052703; 3.637358355198932e-20; 27.327796808642322; 0.8004499183860179; 8.461125192766686e-18; 8.95910135749446e-14; 0.46263795683695763; 3.724763235623522e-29; 0.4627651339879373; 2.1317843123017353; 19.72612759625619; 7.121514602205041; 0.7498244950119917; 7.82776792212211e-29; 2.058164763314051e-48; 6.656376841007709e-110; 1.0185480326011094e-8; 1.0966196183164407e-40; 3.81587425576929; 17.022148683535356; 2.8781418780364653e-226; 6.283831883355779e-60; 7.103441240122817e-17; 1.7971618832549945e-128; 18.528854800853065; 4.497936571326762e-42; 5.9611599404636495; 9.131948748136594; 2.138817581505318; 6.963672986053975e-141; 0.005445679519556468; 1.6679405650005341; 2.3337652049764904e-22; 1.419494251779325; 2.261935115923545e-46; 1.2376566162579915e-32; 4.328787591872048; 5.844233889688261e-7; 8.830924221173814e-34; 4.377778889817346e-81; 1.8878051058094012; 0.2998408805976853; 3.9392863709967796; 2.8084798567644255e-7; 0.14707594960663317; 38.59529055572117; 5.209103347927197e-21; 0.0025973623652913317; 1.5593014209459997; 1.3107443853139182e-21; 5.179647971121189; 3.236157265411051e-16; 7.093888840049452e-26; 1.5564589470829775e-71; 0.001741657884835898; 2.627144619341441; 0.04548803609276873; 2.4673234325112487e-29; 1.3507640402698248e-87; 4.38379124625374e-107; 19.316618329311616; 1.2877899257156492e-8; 22.33866240442435; 7.477065799186867; 1.7971426221661947e-49; 3.3345318772733243; 1.603348886130618; 9.19752238851656; 6.356526861225398e-67; 18.93162033259596; 1.2485391021861125e-98; 17.39152256819622; 0.891805611066936; 35.20596843419915; 1.6898318496500515; 5.638826369435781e-41; 1.2229640299979205; 1.6370815943900768e-74; 0.9756549709108758; 1.3904665924398121e-6; 1.8582909404291397; 1.9869657417523816; 9.819641204968615e-5; 11.71516179379171; 2.226782016861399e-7; 2.0750601055907887; 2.1347726595107657e-10; 1.683191425244726e-46; 2.3584667613037514e-16; 18.624750803153773; 17.12056336027302; 0.36126716256726055; 1.2788867483362755e-118; 1.532524744507334; 5.252933249817591e-44; 4.136636658945092e-32; 0.43950895342231955; 6.120212273528275; 0.987566032480816; 10.027807407458246; 0.34575101054546964; 4.261326022977069e-24; 9.140183928737398e-71; 7.162942951718289; 2.582956434666498; 2.9453506559779514e-10; 18.472793617032533; 7.085056386755511e-12; 36.40227058129134; 0.13079840647173033; 5.7395520820424e-69; 7.839955677157711e-62; 3.817437369660718e-33; 3.6404078004251237e-261; 1.252788930584791e-60; 1.8439518176020697e-42; 3.7077748286865955; 10.923507699176934; 4.398723765471304; 37.00277456879605; 7.163775800693194; 3.042070021429068e-36; 1.133732867786907e-27; 8.471981021660175; 17.313578618750533; 29.52963937643628; 1.6883607661890261e-18; 16.675675294485313; 8.083285501565638e-116; 1.7856881885430648e-71; 7.554472386121312e-6; 2.959280305979924e-54; 0.47485862974337695; 43.64815259851988; 0.00020435835048960367; 4.178015918890087; 0.7154468290561525; 19.95948484510194; 5.121764686065876; 1.4777202796969108e-44; 4.6531581853388824e-32; 4.262223219325753e-77; 10.910140454174218; 2.3511019348088893]
C= [0.02236367582581154; 9.58383778144876e-9; 10.622999409755542; 6.16241579726002e-6; 0.0018032815929029718; 0.5820246596178108; 0.46011206498448826; 0.7889991211217988; 1.4684689617091775; 0.05302456476377496; 0.10653991490875167; 0.8187437238615558; 21.42099755745115; 14.532608506948357; 7.130677101093516; 0.2695827025367284; 2.840960239678163e-6; 2.6149971669855403; 0.0002862249023265326; 2.206551045659344; 3.201701648734815; 15.84638486786215; 0.42036030490492404; 1.6441067874030098; 0.007690277824884048; 13.4229674661184; 0.776815090305774; 0.00017622198806910893; 4.047911673106026e-8; 8.52074610887776; 0.0015589533591112501; 0.00019850233659576524; 3.451312792433583e-5; 2.2765682832796673e-5; 3.4658361107945646; 0.02240030876897966; 21.24634498913638; 14.886659535117845; 20.331597580089422; 0.0015512506515751182; 1.1473872576179618e-6; 4.463818741209099; 0.024558108769859248; 0.08003730285262999; 1.0245101733923846e-5; 4.810690364785042e-9; 3.8449023058738855e-5; 0.004656842074317859; 0.05215675661535674; 0.8754064652759576; 1.3659241825975006; 0.00010536064903923549; 6.815257400995486e-6; 1.6332498205461194e-6; 14.838232003036559; 0.014148116381231153; 1.0747805759685072; 13.735602775324027; 0.05821913419275878; 7.072971977150658e-13; 36.085053287224845; 24.234945337082472; 3.775435520586117e-9; 0.06754539214330033; 3.5509190137439215e-5; 3.5579316268225694; 2.9411447208169765e-6; 2.5133421674066183e-9; 1.5062541187856675e-5; 14.694661779281075; 6.897268288709871; 0.7043989278476316; 1.9167230320590787; 10.073132706193185; 0.0028524971206745826; 0.6830477988007796; 6.521786417021922; 21.39217680761771; 17.141148981019942; 0.8465190498941498; 0.017625871401570122; 37.530744437384; 6.981342810181801; 0.12553899875906288; 0.02373963846881343; 1.344559849850729; 0.1690456323666673; 0.5512783866374767; 35.36197200806055; 0.0001408808253739378; 7.3508161565003585; 0.10860528902466145; 7.2717212309772544e-6; 0.016522549776560022; 16.473016905351493; 2.6127833707757824; 28.271280477163984; 3.3312692063423355e-8; 19.994267776549698; 0.6657430297670275; 1.0123785359436453e-7; 0.0006714142571638888; 2.4800026677881197; 3.382125591220219e-7; 4.7115287960519034e-8; 9.525619254857425; 8.271785513563928e-8; 21.490119694933856; 2.5062610399242327; 14.608813232400994; 1.8841651678257887; 8.421086239731675e-7; 28.750558264884454; 15.462293619736093; 29.46282215646338; 2.7066671596653933; 0.002592176417467211; 16.927305215283532; 5.655929890117654e-6; 0.04185013906013212; 23.089441205413877; 0.004178233766777603; 0.0278435050258412; 1.7453606686033147e-7; 4.569441678495397; 0.2937225655189823; 6.0288522810291365; 0.009970572371840814; 0.00010933874610531001; 0.48682413706500893; 6.207955072097215; 0.08854058360680914; 8.12806476837625e-6; 2.2342992920598417; 0.0711867387607783; 20.429841948532424; 0.0029902989914194706; 5.159006124218317e-5; 24.856022732116386; 5.7161295515373945e-6; 21.593135919447022; 1.034593572589091e-7; 0.03432570558664665; 11.8662784181379; 14.064150052406164; 16.58378730247712; 1.4641109475465026e-6; 1.513379889790401e-7; 13.486193224640694; 2.3415045686879677e-7; 1.7274421009808538; 17.664229193215167; 0.00401027321008269; 5.362881287338797e-8; 5.149019194174631e-7; 1.0329083624982976e-7; 0.0006822794649301988; 20.346835250844983; 0.2636146568801335; 1.1869695390218922; 9.279107770625925e-6; 19.249201278261733; 2.8466949459082866; 2.334280646591445; 2.5958954067544824e-6; 0.02604700360517058; 1.6444095893601727; 1.8524998416755938; 0.004535053868363867; 8.402808209415497; 10.821421853057345; 10.71048783445611; 5.708352034806126e-7; 4.4951717518111495e-7; 3.980711719757998e-6; 19.796380459889406; 1.2375725596404167e-6; 0.4824144273681875; 0.5732515483334024; 0.0646340619070183; 9.772350730146414e-7; 0.0002754947154589728; 12.308915197976354; 0.01706939966443881; 8.629220990605731; 5.029702735627101; 4.556678536635026; 6.743800116781311; 0.004807381626578681; 1.0724061894701296; 13.221076988100998; 0.0005047754753299702; 0.0004606181382141919; 1.1269545306708471e-7; 0.004833714336739734; 4.087813188584282; 3.054096162953476; 3.4826092556609756; 2.5451852316521397e-6; 2.664488740692585e-5]
#Function
function eq!(F,w)
numer = w.^ε
denomi = sum(numer,dims=2)
prob = numer./denomi
F[1:D]=(B'*prob)'-C
end
result=nlsolve(eq!,ones(S))
Then, I got the following error.
DomainError with -0.26327276428720575:
Exponentiation yielding a complex result requires a complex argument.
Replace x^y with (x+0im)^y, Complex(x)^y, or similar.
I assume that the error was occurred since some components of w became negative during the problem solving process.
So I modified the above equation as follows and make the components of w take positive values during the problem solving process.
function eq_new!(F,w)
if sum(w.<=0)==0
numer = w.^ε
denomi = sum(numer,dims=2)
prob = numer./denomi
F[1:D]=(B'*prob)'-C
else
F[1:D].= sum(abs.(w[w.<=0]))+100
end
end
result=nlsolve(eq_new!,ones(S))
However, the iteration didn't converge and all results became NaN.
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
* Zero: [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]
* Inf-norm of residuals: 1.203524
* Iterations: 1000
* Convergence: false
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: false
* Function Calls (f): 1000
* Jacobian Calls (df/dx): 172
How can I solve the above nonlinear equation without taking negative base of power function during the problem solving process?

lm() breaks when using poly() with predictors set up as factors

I'm trying to model the relationship between a categorical predictor variable and a continuous outcome variable. I use lm() to this end. Since it's a categorical variable, the proper practice is to convert it to a factor variable type. However, when using poly() for the predictor's regression term and when setting up the predictor variable as a factor it causes lm() to break. On the other hand, if I run lm() without using poly() (but do keep the predictor as factor) or keep poly() but not convert the predictor to a factor (let it be numeric) -- then lm() doesn't break. I don't understand why it breaks and I don't understand if I can trust the results when it doesn't break.
Data
Data about 50 basketball players. One column (PosCode) is about player's position in the game, and the other (Height) is player's height.
data <-
structure(list(Player = c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,
44, 45, 46, 47, 48, 49, 50), PosCode = c(3, 3, 4, 1, 4, 1, 3,
1, 2, 2, 4, 1, 5, 5, 2, 1, 2, 5, 4, 4, 5, 4, 4, 4, 2, 3, 2, 3,
1, 1, 2, 4, 1, 2, 3, 1, 5, 4, 3, 4, 4, 1, 1, 4, 5, 1, 1, 1, 5,
2), Height = c(176.1, 179.1, 183.1, 169.7, 177.3, 179, 176.4,
174.9, 180.2, 176.5, 178.6, 167.9, 183.4, 166.2, 189.5, 171.9,
188.5, 172.6, 167.7, 172.6, 186.9, 163.8, 179.3, 165.4, 182.2,
166.1, 176.8, 171.9, 173.8, 163, 172.5, 184.9, 170.4, 170.6,
166.8, 172.6, 184.3, 163.3, 182.4, 165.8, 173.4, 182.1, 172.9,
184.9, 173.2, 185.8, 161.4, 186, 178.4, 170.7)), row.names = c(NA,
-50L), class = c("tbl_df", "tbl", "data.frame"))
> data
## # A tibble: 50 x 3
## Player PosCode Height
## <dbl> <dbl> <dbl>
## 1 1 3 176.
## 2 2 3 179.
## 3 3 4 183.
## 4 4 1 170.
## 5 5 4 177.
## 6 6 1 179
## 7 7 3 176.
## 8 8 1 175.
## 9 9 2 180.
## 10 10 2 176.
## # ... with 40 more rows
Modeling the data
I want to know whether I can predict players height from their position in the game. Since position is categorical (there are 5 possible positions), this variable should be of a factor type, with 5 levels.
library(tidyverse)
library(magrittr)
data %<>% mutate_at(vars(PosCode), ~ as.factor(.)) ## convert PosCode from dbl to fct
Modeling by using lm() without poly()
lm(Height ~ PosCode, data = data)
## Call:
## lm(formula = Height ~ PosCode, data = data)
##
## Coefficients:
## (Intercept) PosCode2 PosCode3 PosCode4 PosCode5
## 173.6714 4.9397 0.4429 0.1824 4.1857
Modeling by using lm() with poly()
lm(Height ~ poly(PosCode ,1), data = data)
## Error in qr.default(X) : NA/NaN/Inf in foreign function call (arg 1)
## In addition: Warning messages:
## 1: In mean.default(x) : argument is not numeric or logical: returning NA
## 2: In Ops.factor(x, xbar) : ‘-’ not meaningful for factors
If the predictor isn't a factor, there's no problem regardless of poly()
## convert PosCode from fct back to dbl
data %<>% mutate_at(vars(PosCode), ~ as.double(.))
## lm() without poly()
lm(Height ~ PosCode, data = data)
Call:
lm(formula = Height ~ PosCode, data = data)
## Coefficients:
## (Intercept) PosCode
## 174.3848 0.3112
## lm() with poly()
lm(Height ~ poly(PosCode ,1), data = data)
## Call:
## lm(formula = Height ~ poly(PosCode, 1), data = data)
## Coefficients:
## (Intercept) poly(PosCode, 1)
## 175.256 3.173
But clearly, treating PosCode as dbl rather than fct changes the model in a way that is wrong.
Bottom line
I don't understand why including poly() in lm() breaks it when the predictor is set up as a factor variable.

From help("poly"):
x a numeric vector at which to evaluate the polynomial.
Thus, you cannot use factors inside poly(). This is expected, because categorical variables (i.e., factors) have to be recoded, for example, into dummy variables. And it does neither make sense to have, say, a quadratic effect for the categorical variable as a whole nor for the coded (dummy) variables. (It does not make sense from a substantive perspective, but squaring a dummy variable that has only 0s and 1s does also not make much sense from a perspective blind to statistics.)
You can see that lm() recodes your factor because you get four coefficents (one less than the number of categories) for the variable PosCode in your first model.
In the end, poly() is not of much use unless you set its argument degree to a value > 1

Calculate 90th percentile across columns in a data frame

I'm trying to calculate the 90th percentile of all station fecal samples by sample date, across columns in a data frame. It would be nice to be able to add this as a new column but not absolutely necessary.
I rearrange my data in the following way, although I don't know if this is necessary. It is easy for me to visualize this way.
library(dplyr)
FecalData <- RawData %>%
select(Station, SampleDate, FecalColiform)
#Rearange by station
library(reshape2)
FecalbyStation <- dcast(FecalData, SampleDate ~ Station, fun.aggregate = mean, na.rm = TRUE)
This leaves me with the following sturcture:
dput(FecalbyStation[1:5,])
structure(list(SampleDate = structure(c(6942, 6979, 7014, 7042,
7070), class = "Date"), `114` = c(114.5, 2, 17, 7.9, 1.8), `115` = c(41,
6.8, 33, 220, 4.5), `116` = c(64, 4, 14, 6.8, 1.8), `117` = c(33,
2, 4.5, 1.8, 2), `118` = c(81.5, 2, 6.8, 33, 1.8), `119` = c(28,
11, 4.5, 1.8, 2), `120` = c(64, 4.5, 11, 1.8, 1.8), `121` = c(31,
4.5, 3.6, 13, 2), `122` = c(41, 2, 33, 13, 1.8), `123` = c(28,
7.8, 2, 13, 1.8), `124` = c(NaN, 7.8, NaN, NaN, NaN), `125` = c(NaN,
NaN, NaN, NaN, NaN), `126` = c(NaN, NaN, NaN, NaN, NaN), `127` = c(NaN,
NaN, NaN, NaN, NaN), `128` = c(NaN, NaN, NaN, NaN, NaN), `129` = c(NaN,
NaN, NaN, NaN, NaN), `614` = c(NaN, NaN, NaN, NaN, NaN), `615` = c(NaN,
NaN, NaN, NaN, NaN), `639` = c(NaN, NaN, NaN, NaN, NaN), `758` = c(NaN,
NaN, NaN, NaN, NaN)), .Names = c("SampleDate", "114", "115",
"116", "117", "118", "119", "120", "121", "122", "123", "124",
"125", "126", "127", "128", "129", "614", "615", "639", "758"
), row.names = c(NA, 5L), class = "data.frame")
I have been able to find row.means() this way and have tweaked this code over and over again to try to get the 90th percentile instead. I have received several different errors along the way. Here is the code I have landed on:
library(psych)
Q90 <- sapply(FecalbyStation, -1, quantile, probs=c(.90), na.rm = TRUE)
This gives me the following error:
Error in match.fun(FUN) : '-1' is not a function, character or symbol
Ultimately I would like to make the resulting 90th percentiles a time series so that I can run a kendall or regression on it to investigate any trend in fecal levels for the region. Any suggestions or advice is much appreciated.
Thank you!

You can keep your data in long form and get the 90th percentile by date as follows:
library(dplyr)
RawData %>% group_by(SampleDate) %>%
summarise(p90 = quantile(FecalColiform, probs=0.9, na.rm=TRUE))

plotting issue with autofitVariogram in automap

I am a newbie in R. I am using autofitVariogram to daily rainfall data of 50 stations.The sample data is provided below.Some of stations have missing values represented by "NaN" values.
My question is regarding the variogramfit. The variogram covers only a distance of 60,000m. Why are the points in bins beyond 60Km not plotted. I had seen from spatial correlation plot maximum distance from lon-lat information is >200Km.
The summary of latitide and longitude information is provided below.
summary(lonlat)
lon lat
Min. :74.78 Min. :15.77
1st Qu.:75.14 1st Qu.:16.04
Median :75.56 Median :16.33
Mean :75.54 Mean :16.37
3rd Qu.:75.94 3rd Qu.:16.66
Max. :76.31 Max. :17.23
$ Sample data given below:
dput(rain[140:145,])
structure(list(Col0 = c(0, 0, 1, 9, 6.5, 0), Col1 = c(1.5, 36,
21, 44, 4, 0), Col2 = c(0, 0, 24.5, 21.5, 7.5, 1), Col3 = c(0,
1, 45, 3, 0, 0), Col4 = c(2, 0, 5, 54.5, 13.5, 0), Col5 = c(0.5,
2, 0, 3.5, 13.5, 0), Col6 = c(0.5, 0, 0, 59, 15.5, 0), Col7 = c(0,
0, 2.5, 1, 0, 0), Col8 = c(0, 6, 24, 2, 5.5, 0), Col9 = c(0,
3, 6, 1, 0, 7), Col10 = c(0.5, 1, 64, 20, 1, 0.5), Col11 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col12 = c(0, 11, 75, 19, 15.5, 0),
Col13 = c(0, 4, 57.5, 50.5, 8.5, 0), Col14 = c(1.5, 0.5,
127, 33.5, 34.5, 0), Col15 = c(0, 7, 0.5, 13, 1, 0), Col16 = c(0,
0.5, 81.5, 15, 49, 0), Col17 = c(0, 0, 4.5, 17, 5.5, 1),
Col18 = c(0, 3, 2.5, 0.5, 0, 0), Col19 = c(NaN, NaN, NaN,
NaN, NaN, NaN), Col20 = c(0, 0, 0, 0, 7, 0), Col21 = c(0,
1, 0, 5, 3.5, 0), Col22 = c(0, 0, 11.5, 28, 3.5, 0), Col23 = c(0,
0, 48.5, 0, 24.5, 0), Col24 = c(0, 0, 0, 10, 0.5, 14), Col25 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col26 = c(0, 7.5, 16, 28.5, 20.5,
0), Col27 = c(1.5, 0.5, 38, 28.5, 50, 0), Col28 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col29 = c(NaN, NaN, NaN, NaN, NaN,
NaN), Col30 = c(2.5, 0, 0, 80.5, 28, 13.5), Col31 = c(1,
0, 17, 85.5, 3.5, 0), Col32 = c(0, 0.5, 8, 101, 20, 4), Col33 = c(NaN,
NaN, NaN, NaN, NaN, NaN), Col34 = c(4, 3, 17, 122, 2, 2),
Col35 = c(0, 15.5, 14.5, 20, 3.5, 0), Col36 = c(0, 6.5, 8.5,
21, 7, 0), Col37 = c(0, 0, 1.5, 14.5, 0, 1.5), Col38 = c(0,
28, 30, 4, 0, 73), Col39 = c(28.5, 0, 4.5, 9.5, 1, 0), Col40 = c(1.5,
11.5, 32.5, 55, 0, 1), Col41 = c(0, 14.5, 0, 19, 12.5, 47.5
), Col42 = c(0, 28, 29, 17, 0.5, 20.5), Col43 = c(NaN, NaN,
NaN, NaN, NaN, NaN), Col44 = c(0, 19, 3.5, 42, 0, 0), Col45 = c(0,
0, 85, 15.5, 1, 0), Col46 = c(0, 0.5, 8, 24, 0.5, 0), Col47 = c(0,
1.5, 7, 12, 8.5, 0), Col48 = c(0, 0, 0, 43.5, 0, 1.5), Col49 = c(0,
13.5, 1, 16, 1, 1)), .Names = c("Col0", "Col1", "Col2", "Col3",
"Col4", "Col5", "Col6", "Col7", "Col8", "Col9", "Col10", "Col11",
"Col12", "Col13", "Col14", "Col15", "Col16", "Col17", "Col18",
"Col19", "Col20", "Col21", "Col22", "Col23", "Col24", "Col25",
"Col26", "Col27", "Col28", "Col29", "Col30", "Col31", "Col32",
"Col33", "Col34", "Col35", "Col36", "Col37", "Col38", "Col39",
"Col40", "Col41", "Col42", "Col43", "Col44", "Col45", "Col46",
"Col47", "Col48", "Col49"), row.names = 143:148, class = "data.frame")
# Import the required libraries
library(rgdal)
library(maptools)
library(gstat)
library(sp)
library(automap)
library(XLConnect)
# Read the station data from xls file
stnrain = readWorksheetFromFile(path_fileName,"Sheet1", region = "D1:BA187", header = FALSE)
N = nrow(stnrain)
rain = stnrain[4:N,]
lat = as.numeric(t(stnrain[2,]))
lon = as.numeric(t(stnrain[3,]))
lonlat = cbind(lon,lat)
#Transform from GCS to UTM protection
sp = SpatialPoints(lonlat,proj4string = CRS("+proj=longlat"))
sp_utm = spTransform(sp, CRS("+proj=utm +zone=43N +datum=WGS84"))
krige_value = list() #prepare a list for storing the autokrige output
krige_stderr = list()
nRows = nrow(rain)
for (i in 1:nRows)
{
irain = rain[i,]
miss_indx = (irain == "NaN")
irain = irain[!miss_indx]
irain = as.numeric(irain)
isallZeros = (max(irain) == 0) # To take care of the cases of dry day(irain =0)
irain = as.data.frame(irain)
M = nrow(irain)
if ((M > 5) & (!isallZeros)) # To avoid cases of NaN across many stations
{
print(i)
foo_utm = sp_utm[!indx]# Removing the locations with NaN values
data = data.frame(foo_utm,irain)
names(data) = c("Easting","Northing","rain")
coordinates(data) = c("Easting","Northing")
variogram = autofitVariogram(rain~1,data,model = "Sph",fix.values=c(0,NA,NA))
p = plot(variogram, main="Semi-variogram (Spherical Model)",xlab="Distance(m)",ylab="Semi-Variance(mm2)", sub=paste("Range: ",variogram$var_model$range[2], "Day",i))
print(p)
png(p)
dev.off()
}
else
{
krige_value[[i]] = list(rep(0, L))
krige_stderr[[i]] = list(rep(0, L))
}
}
}
Q2) How can i save the variogram fit png file in a loop. I understand that dev.off() should be used after each saving the figure, which i had done, but I am not able to save the the figure.
Any help would be appreciated.
Thanks,
Any suggestions would be appreciated?

In regard to your first question, the sample variogram is built using points up to a maximum distance of around 1/3 of the diagonal of the area of interest. The assumption here is that points farther away form that are not related, and because they are not in the sample variogram or variogram model they are plotted. This is just a choice, and might not be the correct choice, but when I wrote autofitVariogram it seemed to work well for my data. The variogram model you show confirms this, the range is smaller than 60 km.
For saving your png's I have two suggestions. First, call the plot command inside the png() dev.off pair, so not:
print(p)
png()
dev.off()
but:
png()
print(p)
dev.off()
In addition, I would create meaningful names for the png files.
To create sets of variogram plots, I would use ggplot2. This uses geom_line and facet_wrap. ggplot2 cannot deal directly with gstat/automap variogram models, luckily you can create distance semivariance data using the function variogramLine from gstat. See for example figure 3.1, and the plots in appendix A of this report I wrote. This answer I wrote earlier does also include an example of using ggplot2 for spatial data, this time to plot a grid map.

Resources