geopandas dataframe to json - bigdata

I have a geodataframe called SchooolDistrictDf that has more than 19,814,822 rows and looks like the following:
FIPS SrcName crate_code geohash ncessch sLevel schnam shape stAbbrev
0 13820.0 NaN birmingh djfjrrw 010000700091 1 TRACE CROSSINGS ELEM SCH {u'type': u'Point', u'coordinates': [-86.85997... AL
1 13820.0 NaN birmingh djfjq2v 010000700091 1 TRACE CROSSINGS ELEM SCH {u'type': u'Point', u'coordinates': [-86.90528... AL
2 13820.0 NaN birmingh djfjx17 010000700091 1 TRACE CROSSINGS ELEM SCH {u'type': u'Point', u'coordinates': [-86.87507... AL
3 13820.0 NaN birmingh djfjx26 010000700091 1 TRACE CROSSINGS ELEM SCH {u'type': u'Point', u'coordinates': [-86.86546... AL
4 13820.0 NaN birmingh djfm8rd 010000700091 1 TRACE CROSSINGS ELEM SCH {u'type': u'Point', u'coordinates': [-86.82151... AL
I was told that I need to convert/save it as a json. I'm not familiar with json but I tried doing the following on a small subset of the data:
SchooolDistrictDf.geometry.__geo_interface__
but I got the following error:
AttributeError: No geometry data set yet (expected in column 'geometry'.
is this because my shape column is in the following format
`{u'type': u'Point', u'coordinates': [-86.8599700927734, 33.3084869384766]}`
moreover, how could I convert a large geodataframe to json? I'd greatly appreciate any help
Edit:
This is how I tried getting the SchooolDistrictDf['shape'] into normal geopandas geometry:
SchooolDistrictDf = pd.concat([SchooolDistrictDf.drop(['shape'], axis=1), SchooolDistrictDf['shape'].apply(pd.Series)], axis=1)
geometry = [Point(xy) for xy in SchooolDistrictDf['coordinates']]
crs = {'init': 'epsg:4269'}
SchooolDistrictDf = GeoDataFrame(SchooolDistrictDf, crs=crs, geometry=geometry)
SchooolDistrictDf = SchooolDistrictDf.drop(['coordinates','type'], axis=1)
SchooolDistrictDf =SchooolDistrictDf.to_crs(epsg=4269)

Related

Transposition Cipher: How to solve

I have a ciphertext as follows, of which I do not know the keylength:
wlna evesy ehudre thnma upbum w onaw-dino olsile tf hndcseoorl foouA. bnsst uho,et r,vweeirh teorf efer tsw lae tsutas sfeccsan,ul eytd hduu sbhe edtmel faut s,b bo nte oefrroeth ad ofhlea fl, in nthdan olwe hacpe lenbe euce rdo edt acsuhutobslinre ut,h tae a sv tmsoeedswitiny cl ardesro ndipipn ino,ng trat reeac edimanthf o chme ay einrh iwhccodha ur stoorn uftentuauac aqnctina dse oy.realgea Lrsea ms nos fl kiceofdan wi tndieer eroscvto edsindre oun a usht-out e,bcoo n wesin o retou befwh,nd mahic vehy alax ep teindre hepe nsecho oftul sebox kybhi eswav chhenbe eeal aref dyrd rereHo.to r ow uaudhyrencli erngie ba hdconee edenvym r fogaeth terdne to h wospt hrheecore ed rveeseshi menss hhigtreeav edimanevo fr m erarytysee e wrot itn to frof hesulmt ohi d,wol cht aud sy e vrn apli. ltaead Hehdev ei blntycanee d irre bwdono ty wonrpesne s,owhf o ad omhare rmy bkall asml aefethe ndtert ohsun uu llaly ogare Osne.e tn he,owhlwat i stms obar pothebl he atteni slglEt nanhisminb, eg le
How would I be able to decipher this transposed ciphertext in a non-manual way like on https://tholman.com/other/transposition/ ?
I believe that the punctuation and spaces matter as well in this ciphertext.

Extract date from a text document in R

I am again here with an interesting problem.
I have a document like shown below:
"""UDAYA FILLING STATION ps\na MATTUPATTY ROAD oe\noe 4 MUNNAR Be:\nSeat 4 04865230318 Rat\nBree 4 ORIGINAL bepas e\n\noe: Han Die MC DE ER DC I se ek OO UO a Be ten\" % aot\n: ag 29-MAY-2019 14:02:23 [i\n— INVOICE NO: 292 hee fos\nae VEHICLE NO: NOT ENTERED Bea\nss NOZZLE NO : 1 ome\n- PRODUCT: PETROL ae\ne RATE : 75.01 INR/Ltr yee\n“| VOLUME: 1.33 Ltr ae\n~ 9 =6AMOUNT: 100.00 INR mae wae\nage, Ee pel Di EE I EE oe NE BE DO DC DE a De ee De ae Cate\notome S.1T. No : 27430268741C =. ver\nnes M.S.T. No: 27430268741V ae\n\nThank You! Visit Again\n""""
From the above document, I need to extract date highlighted in bold and Italics.
I tried with strpdate function but did not get the desired results.
Any help will be greatly appreciated.
Thanks in advance.
Assuming you only want to capture a single date, you may use sub here:
text <- "UDAYA FILLING STATION ps\na MATTUPATTY ROAD oe\noe 4 MUNNAR Be:\nSeat 4 04865230318 Rat\nBree 4 ORIGINAL bepas e\n\noe: Han Die MC DE ER DC I se ek OO UO a Be ten\" % aot\n: ag 29-MAY-2019 14:02:23 [i\n— INVOICE NO: 292 hee fos\nae VEHICLE NO: NOT ENTERED Bea\nss NOZZLE NO : 1 ome\n- PRODUCT: PETROL ae\ne RATE : 75.01 INR/Ltr yee\n“| VOLUME: 1.33 Ltr ae\n~ 9 =6AMOUNT: 100.00 INR mae wae\nage, Ee pel Di EE I EE oe NE BE DO DC DE a De ee De ae Cate\notome S.1T. No : 27430268741C =. ver\nnes M.S.T. No: 27430268741V ae\n\nThank You! Visit Again\n"
date <- sub("^.*\\b(\\d{2}-[A-Z]+-\\d{4})\\b.*", "\\1", text)
date
[1] "29-MAY-2019"
If you had the need to match multiple such dates in your text, then you may use regmatches along with regexec:
text <- "Hello World 29-MAY-2019 Goodbye World 01-JAN-2018"
regmatches(text,regexec("\\b(\\d{2}-[A-Z]+-\\d{4})\\b", text))[[1]]
[1] "29-MAY-2019" "29-MAY-2019"

GLPK: access to data array with variable length

set ORIG;
set DEST;
set LINKS within {ORIG,DEST};
printf {i in ORIG, j in DEST} "LINKS[%s,%s] = %g \n",i,j,LINKS[i][j];
data;
set ORIG := GARY CLEV PITT;
set DEST := FRA DET LAN WIN STL FRE LAF;
set LINKS :=
(GARY,*) DET LAN STL LAF
(CLEV,*) FRA DET LAN WIN STL LAF
(PITT,*) FRA WIN STL FRE;
end;
This stops with this error message:
LINKS cannot be subscripted
How can I define LINKS so that it can be accessed with LINKS[i][j]?
With some advice from gnu-help, this seems to be a better syntax
for this problem:
set ORIG;
set DEST;
param LINKS{ORIG, DEST} default 0;
for {i in ORIG, j in DEST} printf "LINKS[%s,%s] = %g \n",i,j,LINKS[i,j];
data;
set ORIG := GARY CLEV PITT;
set DEST := FRA DET LAN WIN STL FRE LAF;
param LINKS : FRA DET LAN WIN STL FRE LAF :=
GARY 1 2 3 4 5 6 .
CLEV . 2 . 4 . 6 .
PITT 1 2 3 4 5 6 7;
end;

"Residual" data after filter?

I have some data on UFO sightings
Date,Country,City,State,lat,lng
12/21/2016,USA,Waynesboro,VA,38.0652286,-78.90588756
12/21/2016,USA,Louisville,KY,38.2542376,-85.7594069
12/20/2016,USA,Santa Rosa,CA,38.4404675,-122.7144313
12/20/2016,USA,Fresno,CA,36.7295295,-119.7088612
12/19/2016,USA,Reymert,AZ,33.2297793,-111.2092898
12/19/2016,USA,Redding,CA,40.5863563,-122.3916753
12/19/2016,USA,Gilbert,AZ,33.294207,-111.7379465
12/19/2016,USA,Phoenix,AZ,33.4485866,-112.0773455
12/19/2016,USA,Huber Heights,OH,39.85902405,-84.11136285
12/19/2016,USA,Conway,AR,35.0886963,-92.442101
12/19/2016,USA,Anchorage,AK,61.2163129,-149.8948522
12/19/2016,USA,Town and Country,MO,38.6122751,-90.4634531
12/19/2016,USA,Salt Lake City,UT,40.7670126,-111.8904307
12/19/2016,USA,Richardson,TX,32.9481789,-96.7297205
12/18/2016,CANADA,Wetaskiwin,AB,52.968492,-113.3679199
12/18/2016,USA,Berryville,AR,36.364792,-93.5679666
12/18/2016,USA,Honolulu,HI,21.304547,-157.8556763
12/18/2016,USA,St. George,UT,37.104153,-113.5841312
12/18/2016,USA,Bend,OR,44.0581728,-121.3153095
12/18/2016,USA,Mission,KS,39.0277832,-94.6557913
12/18/2016,USA,Lancaster,OH,39.7136754,-82.5993293
12/17/2016,USA,San Pedro,CA,33.7358518,-118.2922933
12/17/2016,USA,Kahana,HI,21.5543942,-157.873405
12/17/2016,USA,San Diego,CA,32.7174209,-117.1627713
12/17/2016,USA,Waipio,HI,21.4172766,-157.9986758
12/17/2016,USA,Ojai,CA,34.4480495,-119.2428889
12/17/2016,USA,Weston,FL,26.103632,-80.40310188
12/17/2016,USA,Fairfield,CA,38.2493581,-122.0399662
12/16/2016,USA,Rio Rancho,NM,35.269381,-106.6328189
12/16/2016,USA,Hixson,TN,35.236207,-85.2982059
12/16/2016,USA,Dade City,FL,28.3647248,-82.1959177
12/16/2016,USA,La Veta,CO,37.5050118,-105.0077746
12/16/2016,USA,Kelso,WA,46.1420334,-122.9060317
12/16/2016,USA,Skiatook,OK,36.3684245,-96.0013846
12/16/2016,USA,Carson City,NV,39.1637984,-119.7674033
12/15/2016,USA,Syracuse,NY,43.0481221,-76.1474243
12/15/2016,USA,Johnson City,TN,36.3134398,-82.3534727
12/15/2016,USA,Davie,FL,26.075729,-80.28410888
12/15/2016,USA,Winchester,KS,39.3222209,-95.2669154
12/15/2016,USA,Middlefield,CT,41.717613,-81.2086884
12/15/2016,USA,Corbin,KY,36.9486986,-84.096876
12/15/2016,USA,Simpsonville,SC,34.7370639,-82.2542833
12/15/2016,USA,Panama City,FL,30.165156,-85.6605594
12/15/2016,USA,Chandler,AZ,33.3067132,-111.8408488
12/15/2016,USA,Ozark,AL,33.547741,-86.5591659
12/14/2016,USA,Cumming,GA,34.2073196,-84.1401925
12/14/2016,USA,North Chesterfield,VA,38.6560565,-90.5742028
12/14/2016,USA,Dabney,KY,37.1839682,-84.5499416
12/14/2016,USA,Clinton,CT,42.26306,-71.8052219
12/14/2016,USA,San Diego,CA,32.7174209,-117.1627713
12/14/2016,USA,South Burlington,VT,44.4669941,-73.1709603
12/14/2016,USA,Prescott Valley,AZ,34.6100243,-112.3157209
12/14/2016,USA,Monroe Twp,NJ,40.3183284,-74.42021822
12/14/2016,USA,Berthoud,CO,40.3083174,-105.0810923
12/13/2016,USA,Liberty Lake,WA,47.6631371,-117.0855724
12/13/2016,USA,Chicago,IL,41.8755546,-87.6244211
12/13/2016,USA,La Jolla,CA,32.8472711,-117.2742085
12/13/2016,USA,Fort Lauderdale,FL,26.1254381,-80.1381514
12/13/2016,USA,Cedar Rapids,IA,41.9758872,-91.6704052
12/13/2016,USA,Panama City,FL,30.165156,-85.6605594
12/13/2016,USA,Hale,MI,44.3777947,-83.8047086
12/13/2016,USA,Dubuque,IA,42.5006217,-90.6647966
12/13/2016,USA,St. Johns,FL,29.9032284,-81.4145467
12/13/2016,USA,West Des Moines,IA,41.5645337,-93.759528
12/13/2016,USA,Pasadena,CA,34.1476452,-118.1444778
12/12/2016,USA,Hagerstown,MD,39.6419219,-77.720264
12/12/2016,USA,Jacksonville,FL,30.3321838,-81.6556509
12/12/2016,USA,Taos,NM,36.4072485,-105.5730664
12/12/2016,USA,Stevens Pass,WA,47.7456352,-121.0891717
12/12/2016,USA,Marietta,GA,33.9528472,-84.5496147
12/12/2016,USA,West Collingswood,NJ,39.9062242,-75.0929516
12/12/2016,USA,South Lake Tahoe,CA,38.929125,-119.9878464
12/12/2016,USA,Salem,OR,44.9391565,-123.0331209
12/12/2016,USA,Eden Prairie,MN,44.8454356,-93.5297242
12/12/2016,USA,Smithville,MO,39.3869442,-94.5810658
11/12/2016,USA,Casey,IA,41.5049873,-94.5194148
11/12/2016,USA,St. George,UT,37.104153,-113.5841312
11/12/2016,USA,Fort Collins,CO,40.5508527,-105.0668084
11/12/2016,USA,Helena,MT,46.5927122,-112.0361089
11/12/2016,USA,Independence,LA,37.2242358,-95.708313
11/12/2016,USA,Chester,PA,39.849557,-75.3557457
11/12/2016,USA,Trabuco Canyon,CA,33.6626232,-117.5893799
11/12/2016,USA,Gallatin Gateway,MT,45.5915958,-111.1977303
10/12/2016,USA,Hot Springs,SD,43.431646,-103.4743629
10/12/2016,USA,Erving,MA,42.6000863,-72.3981415
10/12/2016,USA,Farragut,TN,35.8845238,-84.153526
10/12/2016,USA,Delta,CO,38.8368777,-107.8568293
10/12/2016,USA,Virginia Beach,VA,36.8529841,-75.9774182
10/12/2016,USA,Kayenta,AZ,36.717954,-110.2606012
10/12/2016,USA,Lahaina,HI,20.872684,-156.6762728
10/12/2016,USA,Navajo,NM,35.9040858,-109.0335346
10/12/2016,USA,Santa Fe,NM,35.6869996,-105.9377996
10/12/2016,USA,Arlington,VA,38.8903961,-77.0841584
10/12/2016,USA,Hickory,NC,35.7331895,-81.3412005
10/12/2016,USA,Pearland,TX,29.5639758,-95.2864298
9/12/2016,USA,Phoenix,AZ,33.4485866,-112.0773455
9/12/2016,USA,Portland,OR,45.5202471,-122.6741948
9/12/2016,USA,Oldsmar,FL,28.06906015,-82.6501914
9/12/2016,USA,Kingman,AZ,35.189443,-114.0530064
9/12/2016,USA,Fredericksburg,VA,38.3031837,-77.4605398
9/12/2016,USA,Lancing,TN,36.1206306,-84.6538307
9/12/2016,USA,Tewksbury,MA,42.6106479,-71.2342247
9/12/2016,USA,Newport,ME,44.8353424,-69.2739364
9/12/2016,CANADA,Truro,NS,45.366668,-63.3000059
8/12/2016,USA,Jerseyville,IL,39.1200471,-90.3284478
8/12/2016,USA,St. Clair,MO,38.0592942,-93.7945455
8/12/2016,USA,Cromwell,CT,41.2861336,-72.3557585
8/12/2016,USA,Atlanta,GA,33.7490987,-84.3901848
7/12/2016,USA,Saint Augustine,FL,29.8946952,-81.3145394
7/12/2016,USA,Anchorage,AK,61.2163129,-149.8948522
7/12/2016,USA,Napili,HI,20.9717546,-156.6756045
7/12/2016,USA,Bellingham,WA,48.754402,-122.4788601
7/12/2016,USA,Snellville,GA,33.857328,-84.0199107
7/12/2016,USA,Las Vegas,NV,36.1662859,-115.1492249
7/12/2016,USA,Riverdale,GA,33.5726113,-84.4132593
7/12/2016,USA,Plymouth,MA,41.9584367,-70.6672576
7/12/2016,USA,Orlando,FL,28.5479786,-81.4127841
6/12/2016,USA,High Point,NC,35.9556924,-80.0053175
6/12/2016,USA,Austin,NV,39.4932592,-117.0695385
6/12/2016,USA,Austin,NV,39.4932592,-117.0695385
6/12/2016,USA,Nairn,LA,29.4279955,-89.6108946
6/12/2016,USA,Holland,PA,40.1728871,-74.9926687
6/12/2016,USA,Manhattan,NY,40.7902778,-73.9597221
6/12/2016,USA,Lake Jackson,TX,29.0338575,-95.4343858
6/12/2016,USA,Union,IL,37.4616454,-89.2504792
6/12/2016,USA,Osterville,MA,41.6293398,-70.3866805
6/12/2016,USA,Sunrise,FL,26.1482449,-80.3288858
5/12/2016,CANADA,Regina,SK,50.4480951,-104.615818
5/12/2016,USA,Niantic,CT,32.7809195,-117.2524695
5/12/2016,USA,Ivins,UT,37.1685907,-113.6794056
5/12/2016,USA,Haskell,NJ,41.0284304,-74.2959822
5/12/2016,USA,Westmoreland,NH,42.9620253,-72.4423101
5/12/2016,CANADA,Keswick,ON,44.2278666,-79.46145
5/12/2016,USA,Elizabethtown,PA,40.153364,-76.604252
5/12/2016,USA,Webster,NY,43.2122851,-77.4299938
5/12/2016,USA,Stratford,CT,37.2584705,-79.9622598
5/12/2016,USA,La Pine,OR,43.6703995,-121.5036359
4/12/2016,USA,Sauk Rapids,MN,45.5919097,-94.166101
4/12/2016,USA,Huntington Beach,CA,33.6783336,-118.0000165
4/12/2016,USA,Freeport,ME,43.857307,-70.1037599
4/12/2016,USA,Sioux Falls,SD,43.5499749,-96.7003269
4/12/2016,USA,Arcade,GA,34.0778881,-83.5615535
4/12/2016,USA,Redwood Falls,MN,44.5393721,-95.1164477
4/12/2016,USA,Brinklow,MD,39.1659403,-77.0155329
4/12/2016,USA,Winter Harbor,ME,44.395523,-68.0836489
4/12/2016,USA,Severn,MD,39.127886,-76.6869129
4/12/2016,USA,Mission Viejo,CA,33.5965685,-117.6594049
4/12/2016,USA,Marana,AZ,32.4446988,-111.215709
4/12/2016,CANADA,London,ON,42.988576,-81.2466429
3/12/2016,USA,Matawan,NJ,40.41483,-74.229589
3/12/2016,USA,Morgantown,WV,39.6296809,-79.9559436
3/12/2016,USA,Corte Madera,CA,37.9254806,-122.5274754
3/12/2016,USA,Boone Grove,IN,41.3547602,-87.1294741
3/12/2016,USA,Rockville,MD,39.0840054,-77.1527572
3/12/2016,USA,North Snohomish,WA,47.9394115,-122.0779886
3/12/2016,USA,Whittier,CA,33.9748932,-118.0336974
3/12/2016,USA,Santa Cruz,CA,36.9735903,-122.0260569
3/12/2016,USA,Gorham,ME,43.6796943,-70.4429341
2/12/2016,USA,Philadelphia,PA,39.9523993,-75.1635898
2/12/2016,USA,Reidsville,NC,36.354859,-79.6644749
2/12/2016,USA,Raphine,VA,37.9373548,-79.2328101
2/12/2016,USA,Chester,VA,37.3569086,-77.4421817
2/12/2016,USA,Ashland,VA,37.7594012,-77.4806602
2/12/2016,USA,Snellville,GA,33.857328,-84.0199107
1/12/2016,USA,Plainville,CT,41.6745432,-72.8581557
1/12/2016,USA,Portland,TN,36.5817089,-86.5163832
1/12/2016,USA,Glendale,AZ,33.5389854,-112.1858156
1/12/2016,USA,Conway,SC,33.8360035,-79.0478142
1/12/2016,USA,San Bernardino,CA,34.1083449,-117.2897651
1/12/2016,USA,Amherst,NY,42.9783924,-78.7997615
1/12/2016,USA,Montgomery,AL,40.854156,-78.2711029
1/12/2016,USA,Asbury Park,NJ,40.2203907,-74.0120816
11/30/2016,USA,Edmonds,WA,47.8105738,-122.3774951
11/30/2016,USA,Camp Shelby,MS,31.1975317,-89.2078257
11/30/2016,USA,Riverton,WY,38.515529,-121.5321489
11/30/2016,USA,Radcliff,KY,37.8403456,-85.9491297
11/30/2016,USA,Kingsland,GA,30.7999563,-81.689826
11/30/2016,USA,Fayetteville,TX,29.9057817,-96.6727527
11/29/2016,USA,Belmont,MI,40.0057737,-81.0097515
11/29/2016,USA,Bridgeport,WV,39.2864787,-80.256198
11/29/2016,USA,American Canyon,CA,38.223457,-122.227043
11/29/2016,USA,Colorado Springs,CO,38.8339578,-104.8253484
11/29/2016,CANADA,Niagara Falls,ON,43.1089442,-79.0636192
11/29/2016,USA,Carlinville,IL,39.2797699,-89.8817661
11/29/2016,USA,Fairfield,CA,38.2493581,-122.0399662
11/29/2016,USA,Englewood,CO,39.6482059,-104.987964
11/29/2016,USA,Corona,CA,37.0066161,-121.9969062
11/29/2016,USA,Midland,MI,43.6155825,-84.2472116
11/29/2016,USA,Janesville,WI,42.7151854,-88.9907742
11/29/2016,USA,Plainfield,IL,41.623191,-88.2284325
11/29/2016,USA,Happy Valley,OR,43.1358923,-122.3804695
11/29/2016,USA,Anchor Point,AK,59.76826,-151.6775519
11/29/2016,USA,Fishers,IN,39.9555928,-86.0138728
11/28/2016,USA,Sutherlin,OR,43.3896628,-123.3123597
11/28/2016,USA,Tazewell,VA,37.1236041,-81.5684128
11/28/2016,USA,Athol,MA,42.5959203,-72.2267496
11/28/2016,USA,American Canyon,CA,38.223457,-122.227043
11/28/2016,USA,Ste. Genevieve,MO,37.9814415,-90.0417789
11/28/2016,USA,St. Petersburg,FL,27.77330515,-82.6469933
11/28/2016,USA,Waynesville,MO,37.8286516,-92.2007226
11/28/2016,USA,Seekonk,RI,41.8674548,-71.3797769
11/28/2016,USA,Vineland,NJ,39.473152,-75.0020264
11/28/2016,USA,Fairfield,ME,44.588511,-69.5990749
11/28/2016,USA,Athens,GA,33.94385375,-83.3972898
11/28/2016,USA,Centerville,IL,31.2579584,-95.9782919
11/28/2016,USA,Mooresville,NC,35.5848596,-80.8100723
11/28/2016,USA,Grand Junction,CO,39.063956,-108.5507316
11/27/2016,CANADA,Carrying Place,ON,43.8088119,-79.2334018
11/27/2016,USA,St. Petersburg,FL,27.77330515,-82.6469933
11/27/2016,USA,Bay Shore,NY,40.7250986,-73.2453945
11/27/2016,USA,Longmont,CO,40.1672117,-105.1019286
11/27/2016,USA,Kenmore,WA,47.7573202,-122.2440147
11/27/2016,USA,Maui,HI,20.8029568,-156.3106832
11/27/2016,USA,Allentown,PA,40.6022059,-75.4712793
11/27/2016,USA,Lake Charles,LA,30.2265949,-93.2173758
11/27/2016,USA,Raymond,NE,40.956282,-96.7834109
11/26/2016,USA,Bailey,CO,34.0349194,-102.8149371
11/26/2016,USA,Bergen,NY,43.085391,-77.9417139
11/26/2016,USA,Middletown,NY,41.4459271,-74.422934
11/26/2016,USA,Charlotte,NC,35.2270869,-80.8431267
11/26/2016,USA,Davenport,FL,28.1614046,-81.6017416
11/26/2016,USA,Allen Park,MI,42.2575385,-83.2110374
11/26/2016,USA,Naugatuck,CT,41.4860186,-73.0509431
11/26/2016,USA,Venice,FL,27.0998708,-82.4544131
11/26/2016,USA,Fairview,OR,45.5469302,-122.4370392
11/26/2016,USA,McDonough,GA,33.4473361,-84.1468615
11/26/2016,USA,Spearfish,SD,44.490817,-103.8593699
11/26/2016,USA,Shallotte,NC,33.9732275,-78.385837
11/26/2016,USA,Brooklyn,NY,40.64530975,-73.9550229
11/26/2016,USA,McChord AFB,WA,47.1377,-122.4764999
11/26/2016,USA,Vacaville,CA,38.3565773,-121.9877443
11/26/2016,USA,West New York,NJ,40.785529,-74.0083002
11/25/2016,USA,Olathe,KS,38.8843867,-94.8161126
11/25/2016,USA,Tyrone,GA,33.6723506,-82.8612562
11/25/2016,USA,Vero Beach,FL,27.6387163,-80.3975398
11/25/2016,USA,Bedford,IN,38.8611619,-86.4872148
11/25/2016,USA,Nevada,NV,39.5158825,-116.8537226
11/25/2016,USA,Brandon,FL,27.928464,-82.2880445
11/25/2016,USA,Port Orange,FL,29.10150985,-81.0105537
11/25/2016,USA,Torrance,CA,33.8358492,-118.3406287
11/25/2016,USA,Longmont,CO,40.1672117,-105.1019286
11/25/2016,USA,Corpus Christi,TX,27.8002542,-97.3955743
11/25/2016,USA,Warner Robins,GA,32.598313,-83.6256769
11/25/2016,USA,Newcastle,WA,47.5395736,-122.156333
11/24/2016,USA,Graham,NC,36.069026,-79.4005759
11/24/2016,USA,Annapolis,MD,38.9786401,-76.4927859
11/24/2016,USA,Balko,OK,36.6600752,-100.679207
11/24/2016,USA,Largo,FL,27.9094665,-82.7873243
11/24/2016,USA,Woodbridge,NJ,40.55418,-74.2860007
11/24/2016,USA,Woodbridge,NJ,40.55418,-74.2860007
11/24/2016,USA,Oldsmar,FL,28.06906015,-82.6501914
11/24/2016,USA,Zebulon,NC,35.824321,-78.3147199
11/24/2016,USA,Monrovia,MD,39.3720477,-77.2719278
11/24/2016,USA,Grand Junction,CO,39.063956,-108.5507316
11/24/2016,USA,Colorado Springs,CO,38.8339578,-104.8253484
11/24/2016,USA,D'iberville,MS,30.4263092,-88.8908637
11/23/2016,USA,Tinton Falls,NJ,40.3159438,-74.0851403
11/23/2016,USA,Beaverton,OR,45.4871723,-122.8037803
11/23/2016,USA,Central Square,NY,43.286736,-76.1460359
11/23/2016,USA,Carlsbad,CA,33.1580933,-117.3505938
11/23/2016,USA,Poteau,OK,35.0537094,-94.6235578
11/22/2016,USA,Mesa,AZ,33.436188,-111.5860661
11/22/2016,CANADA,Oshawa,ON,43.9172764,-78.8614873
11/22/2016,USA,Largo,FL,27.9094665,-82.7873243
11/22/2016,USA,Beecher,IL,41.34059,-87.6214285
11/22/2016,USA,Santa Clara,UT,37.1330355,-113.6541265
11/22/2016,USA,Santa Clara,UT,37.1330355,-113.6541265
11/22/2016,USA,Derby,KS,37.5455735,-97.268933
11/22/2016,USA,Cecil,PA,32.3026416,-86.0085693
11/22/2016,USA,Atkinson,NH,42.8384221,-71.1470036
11/22/2016,USA,Brooklyn,NY,40.64530975,-73.9550229
11/22/2016,USA,Floral City,FL,28.7211775,-82.3076844
11/22/2016,USA,Jarrettsville,MD,39.60455,-76.4777421
11/22/2016,USA,Gainesville,GA,34.2978794,-83.8240662
11/22/2016,USA,San Diego,CA,32.7174209,-117.1627713
11/22/2016,USA,Scarborough,ME,43.59622635,-70.3300556
11/22/2016,USA,Bismarck,ND,46.8083268,-100.7837391
11/21/2016,USA,Key Colony Beach,FL,24.7209744,-81.0186826
11/21/2016,CANADA,Victoria,BC,48.4283327,-123.3649268
11/21/2016,USA,Riverdale,GA,33.5726113,-84.4132593
11/21/2016,USA,Fountain Hill,AR,33.3578937,-91.8504083
11/21/2016,USA,Ventura,CA,34.364744,-119.3105822
11/21/2016,USA,Alexandria,VA,33.7237617,-116.2673236
11/21/2016,USA,Pearisburg,VA,37.3265112,-80.7350711
11/21/2016,USA,Miramar Beach,FL,30.38208845,-86.3640414
11/21/2016,USA,Honolulu,HI,21.304547,-157.8556763
11/21/2016,USA,Sherman Oaks,CA,34.1508718,-118.4489864
11/21/2016,USA,Burbank,CA,34.1816482,-118.3258553
11/20/2016,USA,McKinney,TX,33.1976496,-96.615447
11/20/2016,USA,Boise,ID,43.61656,-116.2008349
11/20/2016,USA,Springfield,IL,39.7989763,-89.6443687
11/20/2016,USA,San Jose,CA,37.3361905,-121.8905832
11/20/2016,USA,Charleston,TN,35.4959148,-89.5089637
11/20/2016,USA,Lincoln,NE,40.8,-96.6678209
11/20/2016,USA,Franklin,WI,44.2127373,-91.123762
11/20/2016,USA,Pamplin,VA,37.2634817,-78.6825049
11/19/2016,USA,Apollo Beach,FL,27.7729445,-82.3981418
11/19/2016,USA,Salt Lake City,UT,40.7670126,-111.8904307
11/19/2016,USA,Myrtle Beach,SC,33.6956461,-78.8900408
11/19/2016,USA,Oregon City,OR,45.3573429,-122.6067582
11/19/2016,USA,San Marcos,CA,33.1433723,-117.1661448
11/19/2016,USA,Jefferson City,MO,38.577359,-92.1724264
11/19/2016,CANADA,Victoria,BC,48.4283327,-123.3649268
11/19/2016,USA,Boynton Beach,FL,26.5253491,-80.0664308
11/19/2016,USA,Brandon,FL,27.928464,-82.2880445
11/19/2016,USA,Zephyrhills,FL,28.2336196,-82.1811946
11/19/2016,USA,Bristol,CT,37.9317492,-122.0327847
11/19/2016,USA,Woodbridge,VA,38.658172,-77.2497049
11/19/2016,USA,Fishers,IN,39.9555928,-86.0138728
11/19/2016,USA,Louisville,KY,38.2542376,-85.7594069
11/19/2016,USA,Woodbridge,VA,38.658172,-77.2497049
11/19/2016,USA,Blaine,MN,45.1607987,-93.2349488
11/19/2016,USA,Ferrisburgh,VT,44.205835,-73.2465359
11/19/2016,USA,Springfield,MA,42.1014831,-72.5898109
11/19/2016,USA,Newnan,GA,33.3806716,-84.7996572
11/19/2016,USA,Largo,FL,27.9094665,-82.7873243
11/19/2016,USA,Cape Coral,FL,26.6058996,-81.9807339
11/19/2016,USA,Wichita,KS,37.6922361,-97.3375447
11/19/2016,USA,Smithton,PA,40.1539613,-79.7411534
11/19/2016,USA,Obernburg,NY,41.8448104,-75.0071096
11/18/2016,USA,Columbus,OH,39.9622601,-83.0007064
11/18/2016,USA,Sioux Falls,SD,43.5499749,-96.7003269
11/18/2016,USA,Fletcher,NC,35.4306712,-82.5012315
11/18/2016,USA,Billings,MT,45.7874957,-108.4960699
11/18/2016,USA,Billings,MT,45.7874957,-108.4960699
11/18/2016,USA,Stone Creek,OH,40.3972892,-81.5620642
11/18/2016,USA,Carrollton,OH,40.5728404,-81.0856531
11/18/2016,USA,Hampton,VA,37.0300969,-76.3452056
11/18/2016,USA,Nashville,TN,36.1622296,-86.774353
11/18/2016,USA,Kittery,ME,43.1033619,-70.7851622
11/18/2016,USA,McMurray,PA,40.2790921,-80.1017976
11/18/2016,USA,Bronx,NY,40.85703325,-73.83669606
11/18/2016,USA,New York City,NY,40.7305991,-73.9865811
11/18/2016,USA,Farmington,NH,43.39008,-71.0657499
11/17/2016,USA,Buena Park,CA,33.870413,-117.9962164
11/17/2016,USA,Athens,OH,39.3292396,-82.1012554
11/17/2016,USA,Methuen,MA,42.7262016,-71.1908923
11/17/2016,USA,Benton City,WA,46.2631897,-119.4878019
11/17/2016,USA,Stevens Point,WI,44.5229223,-89.5741109
11/17/2016,USA,High Point,NC,35.9556924,-80.0053175
11/17/2016,USA,Caldwell,ID,43.6678736,-116.6894155
11/17/2016,USA,Balko,OK,36.6600752,-100.679207
11/17/2016,USA,Jordan,MT,47.32121,-106.9104609
11/17/2016,USA,Elizabethton,TN,36.3487196,-82.2106875
11/17/2016,USA,Lower Burrell,PA,40.5882821,-79.7298186
11/17/2016,USA,Hemet,CA,33.778562,-117.0357665
11/17/2016,USA,Nampa,ID,43.5737361,-116.5596312
11/17/2016,USA,Oshkosh,WI,44.0206919,-88.5408573
11/17/2016,USA,Avon,CO,39.6329025,-106.4711837
11/17/2016,USA,Stonington,ME,44.156593,-68.6672969
11/16/2016,USA,Elk Grove,CA,38.4087993,-121.3716177
11/16/2016,USA,Benson,AZ,31.9678731,-110.2945759
11/16/2016,USA,Schenectady,NY,42.8095455,-74.0216719
11/16/2016,CANADA,London,ON,42.988576,-81.2466429
11/16/2016,USA,Middleton,ID,43.7068282,-116.6201356
11/16/2016,USA,Danville,KY,37.6456488,-84.7721822
11/16/2016,USA,Maple Grove,MN,45.0759797,-93.4561051
11/16/2016,USA,Lake Havasu City,AZ,34.4742786,-114.3440009
11/16/2016,USA,Maricopa,AZ,33.34883,-112.4912299
11/16/2016,USA,Carlton,OR,45.2942822,-123.1764948
11/16/2016,USA,Wood Village,OR,45.5372449,-122.4178386
11/16/2016,USA,Green Garden Township,IL,46.4385488,-87.2793086
11/16/2016,CANADA,Banff,AB,51.1777781,-115.5682503
11/16/2016,USA,Frostproof,FL,27.7458626,-81.5306312
11/16/2016,USA,Egg Harbor Township,NJ,39.3851791,-74.6756877
11/16/2016,USA,Shady Cove,OR,42.6109085,-122.8218511
11/16/2016,USA,Fremont,WI,44.2597027,-88.8648343
11/16/2016,USA,Mosinee,WI,44.7927298,-89.7035958
11/16/2016,USA,Clarksville,TN,36.5277607,-87.3588702
11/16/2016,USA,Richland,WA,46.2778406,-119.2769066
11/15/2016,USA,Bettendorf,IA,41.5255466,-90.5081477
11/15/2016,USA,Phoenix,AZ,33.4485866,-112.0773455
11/15/2016,USA,Portland,OR,45.5202471,-122.6741948
11/15/2016,USA,Stevens Point,WI,44.5229223,-89.5741109
11/15/2016,USA,Seattle,WA,47.6038321,-122.3300623
11/15/2016,CANADA,Prince Albert,SK,53.201097,-105.7489009
11/15/2016,USA,Stevens Point,WI,44.5229223,-89.5741109
11/15/2016,USA,Naperville,IL,41.7729107,-88.1478669
11/15/2016,USA,Berlin,WI,43.9680365,-88.9434476
11/15/2016,USA,Tahlequah,OK,35.91537,-94.9699559
11/15/2016,USA,Glendale,AZ,33.5389854,-112.1858156
11/15/2016,USA,Old Washington,OH,40.0386822,-81.444562
11/15/2016,USA,Saltillo,MS,34.3764923,-88.68172
11/15/2016,USA,Portland,OR,45.5202471,-122.6741948
11/14/2016,USA,Hillsboro,OR,45.5228939,-122.9898269
11/14/2016,USA,Gardena,CA,33.888658,-118.3076479
11/14/2016,USA,Longmont,CO,40.1672117,-105.1019286
11/14/2016,USA,Deltona,FL,28.9005446,-81.2636737
11/14/2016,USA,Central City,NE,41.1158475,-98.0017248
11/14/2016,USA,Irving,TX,32.8629195,-96.97917017
11/14/2016,USA,Jacksonville,FL,30.3321838,-81.6556509
11/14/2016,USA,Blanchardville,WI,42.81032,-89.8622148
11/14/2016,USA,Columbus,OH,39.9622601,-83.0007064
11/14/2016,USA,Summerville,SC,33.0206179,-80.1747536
11/14/2016,USA,Waupaca,WI,44.3735635,-89.03185979
11/14/2016,USA,Wilkesboro,NC,36.145965,-81.16064
11/14/2016,CANADA,Pickering,ON,43.8356637,-79.0905385
11/14/2016,USA,Fresno,CA,36.7295295,-119.7088612
11/14/2016,USA,Glendale,AZ,33.5389854,-112.1858156
11/13/2016,USA,Benson,VT,43.71558845,-73.30815198
11/13/2016,USA,Palmyra,PA,40.3089798,-76.5933012
11/13/2016,USA,North Chesterfield,VA,38.6560565,-90.5742028
11/13/2016,USA,Coconut Creek,FL,26.2714628,-80.18180782
11/13/2016,USA,Chesapeake Bay,MD,38.5167886,-76.3830045
11/13/2016,CANADA,Ottawa,ON,45.4210328,-75.6900218
11/13/2016,USA,Chester,SD,43.894974,-96.9264449
11/13/2016,USA,Centerville,OH,39.6283928,-84.1593817
11/13/2016,USA,Gila Bend,AZ,32.9478267,-112.7168238
11/13/2016,USA,Selma,IN,32.4078632,-87.0207472
11/13/2016,CANADA,Grande Prairie,AB,55.171025,-118.7951659
11/13/2016,USA,Benicia,CA,38.049365,-122.1585776
11/13/2016,USA,Monroeville,PA,40.4211798,-79.7881024
11/13/2016,USA,Wheaton,MD,39.0398314,-77.0552554
11/13/2016,USA,Ottawa,IL,41.3516628,-88.8454359
11/13/2016,USA,Elwood,IL,41.4039201,-88.1117241
11/13/2016,USA,Bagley,IA,41.8460964,-94.4299693
11/13/2016,USA,Grand Rapids,MI,42.9632405,-85.6678638
11/13/2016,USA,Stockton,CA,37.9577016,-121.2907795
12/11/2016,USA,Maryville,TN,35.7564719,-83.9704592
12/11/2016,USA,Andersonville,TN,36.1986898,-84.0371391
12/11/2016,USA,Auburn,WA,47.3075369,-122.2301807
12/11/2016,USA,Baltimore,MD,39.2908816,-76.6107589
12/11/2016,USA,New Market,MD,39.3826031,-77.2694277
12/11/2016,USA,Chesterfield,MI,38.6560565,-90.5742028
12/11/2016,USA,Highland Park,IL,42.1816919,-87.8003437
12/11/2016,USA,Melbourne,FL,28.0836269,-80.6081088
12/11/2016,USA,Glendale,AZ,33.5389854,-112.1858156
12/11/2016,USA,Durham,NC,35.9940329,-78.8986189
12/11/2016,USA,Fountain Hill,AR,33.3578937,-91.8504083
12/11/2016,USA,Bethesda,MD,38.9848265,-77.0946458
12/11/2016,USA,Bristol,VA,36.5959685,-82.1885008
12/11/2016,USA,Twin Falls,ID,42.5704219,-114.460317
12/11/2016,USA,Bloomfield Hills,MI,42.583645,-83.2454882
12/11/2016,USA,Louisville,KY,38.2542376,-85.7594069
12/11/2016,USA,Freeport,NY,40.6576022,-73.5831834
11/11/2016,USA,Middletown,DE,39.449556,-75.7163206
11/11/2016,USA,Alabaster,AL,33.2442813,-86.8163772
11/11/2016,USA,Owatonna,MN,44.0839937,-93.2261075
11/11/2016,USA,Jasper County,IA,41.6791308,-93.0647785
11/11/2016,USA,Anchorage,AK,61.2163129,-149.8948522
11/11/2016,USA,Terra Linda,CA,38.0040893,-122.5496999
11/11/2016,USA,Terra Linda,CA,38.0040893,-122.5496999
11/11/2016,USA,Raleigh,NC,35.7803977,-78.6390988
11/11/2016,USA,Phoenix,AZ,33.4485866,-112.0773455
11/11/2016,USA,Boca Raton,FL,26.3586885,-80.0830983
11/11/2016,USA,Salt Lake City,UT,40.7670126,-111.8904307
11/11/2016,USA,West Palm Beach,FL,26.7153425,-80.0533745
11/11/2016,USA,Nashua,NH,42.7653662,-71.4675659
11/11/2016,USA,Moss Point,MS,30.4115881,-88.53446
11/11/2016,USA,Greeley,CO,38.4570355,-101.8185006
11/11/2016,USA,West Salem,OR,44.9482087,-123.0629462
11/11/2016,USA,Keizer,OR,44.9958075,-123.0197172
11/11/2016,USA,Herod,IL,37.5803268,-88.4361546
11/11/2016,USA,Leominster,MA,42.5250906,-71.7597939
The whole thing is too big to post but this should get us started.
I fix up the dates and filter the data for a country and state
ufodata <-read.csv("UFO.csv", header=TRUE)
new.date<- strptime(ufodata$Date, format="%m/%d/%Y")
UFO<-cbind(ufodata, new.date)
stateselect <- UFO %>% filter(State=="VA",Country=="USA")
statesorted<-stateselect[order(stateselect$new.date),]
So I now have a dataframe called stateselect that has just the sighting for Virginia. R Studio shows it as 108 observations of 7 variables...that jives with my 6 original columns plus the new date.
But if I run
summary(stateselect)
Date Country City State lat lng new.date
11/3/2016 : 4 CANADA: 0 Virginia Beach:11 VA :108 Min. :33.72 Min. :-116.27 Min. :2016-01-06 00:00:00
2/12/2016 : 3 USA :108 Fredericksburg: 5 AB : 0 1st Qu.:36.89 1st Qu.: -78.74 1st Qu.:2016-03-17 18:00:00
1/30/2016 : 2 Stafford : 4 AK : 0 Median :37.64 Median : -77.46 Median :2016-07-10 00:00:00
10/11/2016: 2 Woodbridge : 4 AL : 0 Mean :37.75 Mean : -78.79 Mean :2016-06-26 20:48:20
10/12/2016: 2 Arlington : 3 AR : 0 3rd Qu.:38.66 3rd Qu.: -77.08 3rd Qu.:2016-10-04 12:00:00
11/19/2016: 2 Portsmouth : 3 AZ : 0 Max. :39.61 Max. : -75.38 Max. :2016-12-21 00:00:00
(Other) :93 (Other) :78 (Other): 0
So why does Canada still show up under Country and all the other states show up under State? What I'd like to do next is get the names of the cities in Virginia. But if I run
unique(stateselect$City)
[1] Waynesboro North Chesterfield Virginia Beach Arlington Fredericksburg Raphine
[7] Chester Ashland Tazewell Alexandria Pearisburg Pamplin
[13] Woodbridge Hampton Bristol Ashburn Charlottesville Colonial Beach
[19] Reston Springfield Petersburg Prince William False Cape State Park Spotsylvania
[25] Manassas Falls Church Gainesville Newport News Williamsburg Comer's Rock
[31] Fairfax Big Stone Gap Roanoke Chincoteague Hiltons Midlothian
[37] Farmville Marion Hurt Salem Madison Heights Aldie
[43] Portsmouth Front Royal Occoquan Stanley Covington Richmond
[49] Lynchburg Chesapeake Vinton Buckingham Stafford Winchester
[55] Burke Centreville Martinsville Radford Culpeper Hillsville
[61] Route 66 West Manassas Park Ivor Danville Rockville Suffolk
[67] Stanardsville
2626 Levels: 495 Maryland Hwy Aberdeen Aberdeen Gardens Abilene Abiquiu Absecon Accord Acworth Adairsville Adrian Agoura Hills Aiken ... Zion
You can see the 67 city names from Virginia, but there's also the "2626 Levels" stuff. So if I try to store the output
names <- unique(stateselect$City)
names is now a factor with 2626 levels. I'm thoroughly confused....
Try:
ufodata <- read.csv("UFO.csv", header=TRUE, stringsAsFactors = FALSE)
You are seeing variables with zero values and levels because the states and countries were converted to factors during import. You probably want them to stay as strings (characters).

Array values being overwritten in gawk

Sample of File I'm reading in
011084,31.0581,-87.0547, 25.9 AL BREWTON 3 SSE
012813,30.5467,-87.8808, 7.0 AL FAIRHOPE 2 NE
013160,32.8347,-88.1342, 38.1 AL GAINESVILLE LOCK
013511,32.7017,-87.5808, 67.1 AL GREENSBORO
013816,31.8700,-86.2542, 132.0 AL HIGHLAND HOME
015749,34.7442,-87.5997, 164.6 AL MUSCLE SHOALS AP
017157,34.1736,-86.8133, 243.8 AL SAINT BERNARD
017304,34.6736,-86.0536, 187.5 AL SCOTTSBORO
GAWK Code
#!/bin/gawk
BEGIN{
FS=",";
OFS=",";
}
{
print $1,$2,$3,$4
station=""$1 #Forces to be string
#Save latitude
stationInfo[station][lat]=$2
print "lat",stationInfo[station][lat]
#Save longitude
stationInfo[station][lon]=$3
print "lon",stationInfo[station][lon]
#Now try printing the latitude again
#It will return the value of the longitude instead
print "lat",stationInfo[station][lat]
print "---------------"
}
Sample output
011084,31.0581,-87.0547, 25.9 AL BREWTON 3 SSE
lat,31.0581
lon,-87.0547
lat,-87.0547
---------------
012813,30.5467,-87.8808, 7.0 AL FAIRHOPE 2 NE
lat,30.5467
lon,-87.8808
lat,-87.8808
---------------
For some reason the value stored in stationInfo[station][lat] is being overwritten by the longitude. I'm at a loss for what in the world is going on.
I'm using GAWK 4.1.1 on Fedora 22
Your problem is the fact that lon and lat are variables and evaluate to the empty string so this assignment stationInfo[station][lat]=$2 and stationInfo[station][lon]=$3 are assigning to stationInfo[station]["].
You need to quote the lat and lon in those (and the other) lines to use strings instead of variables.
#!/bin/gawk
BEGIN{
FS=",";
OFS=",";
}
{
print $1,$2,$3,$4
station=""$1 #Forces to be string
#Save latitude
stationInfo[station]["lat"]=$2
print "lat",stationInfo[station]["lat"]
#Save longitude
stationInfo[station]["lon"]=$3
print "lon",stationInfo[station]["lon"]
#Now try printing the latitude again
#It will return the value of the longitude instead
print "lat",stationInfo[station]["lat"]
print "---------------"
}

Resources