Hello currently I am using Infiniband and testing the performance with IMB-benchmark, I'am currently testing the parallel transfer test
and was wondering the results indeed reflect the parallel performance of the 8 processes.
The explanation of the results is too vague for me to understand.
Since ( 6 additional processes waiting in MPI_Barrier) is mentioned in every result, I suspect that it only runs 2 process each?
The throughput column t_avg[usec] result seems to get the proper result, but I need to make it sure that I am understanding correctly.
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 8
#-----------------------------------------------------------------------------
Is this passage above mean that I am running 8 processes parallel?
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 4
# ( 4 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
and this passage means that 4 processes are running on parallel?
Help from someone who is familiar with the IMB-benchmark is greatly appreciated thanks
Here is the full result below
# np - 8
#------------------------------------------------------------
# Intel (R) MPI Benchmarks 2018, MPI-1 part
#------------------------------------------------------------
# Date : Mon Oct 16 14:14:20 2017
# Machine : x86_64
# System : Linux
# Release : 4.4.0-96-generic
# Version : #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017
# MPI Version : 3.0
# MPI Thread Environment:
# Calling sequence was:
# ./IMB-MPI1 Sendrecv Exchange
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Sendrecv
# Exchange
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
# ( 6 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 13.85 13.85 13.85 0.00
1 1000 12.22 12.22 12.22 0.16
2 1000 10.08 10.08 10.08 0.40
4 1000 9.43 9.43 9.43 0.85
8 1000 8.89 8.91 8.90 1.80
16 1000 8.70 8.71 8.71 3.67
32 1000 9.00 9.00 9.00 7.11
64 1000 8.82 8.82 8.82 14.51
128 1000 8.90 8.90 8.90 28.77
256 1000 8.98 8.98 8.98 56.99
512 1000 9.78 9.78 9.78 104.75
1024 1000 12.65 12.65 12.65 161.91
2048 1000 18.31 18.32 18.31 223.63
4096 1000 20.05 20.05 20.05 408.52
8192 1000 21.15 21.16 21.16 774.11
16384 1000 27.46 27.47 27.46 1193.05
32768 1000 36.93 36.94 36.93 1774.31
65536 640 60.56 60.59 60.57 2163.39
131072 320 117.62 117.63 117.63 2228.57
262144 160 202.67 202.68 202.67 2586.78
524288 80 323.86 324.28 324.07 3233.56
1048576 40 615.05 615.47 615.26 3407.42
2097152 20 1214.74 1216.89 1215.82 3446.74
4194304 10 2471.83 2488.45 2480.14 3371.02
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 4
# ( 4 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 11.14 11.15 11.15 0.00
1 1000 11.16 11.16 11.16 0.18
2 1000 11.11 11.12 11.12 0.36
4 1000 11.10 11.11 11.10 0.72
8 1000 11.03 11.04 11.03 1.45
16 1000 11.21 11.22 11.22 2.85
32 1000 11.81 11.81 11.81 5.42
64 1000 11.58 11.58 11.58 11.05
128 1000 11.77 11.78 11.78 21.72
256 1000 11.88 11.89 11.89 43.05
512 1000 13.03 13.03 13.03 78.57
1024 1000 14.73 14.74 14.74 138.92
2048 1000 19.37 19.39 19.38 211.24
4096 1000 21.31 21.34 21.33 383.96
8192 1000 26.19 26.22 26.20 624.84
16384 1000 32.65 32.69 32.67 1002.26
32768 1000 48.71 48.78 48.75 1343.52
65536 640 75.14 75.22 75.18 1742.63
131072 320 174.66 175.15 174.94 1496.65
262144 160 301.22 302.02 301.44 1735.95
524288 80 539.40 542.68 540.78 1932.21
1048576 40 1015.45 1026.34 1020.59 2043.32
2097152 20 1959.53 1985.57 1971.34 2112.39
4194304 10 3549.00 3641.61 3590.76 2303.55
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 8
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 12.81 12.83 12.82 0.00
1 1000 12.82 12.84 12.83 0.16
2 1000 12.73 12.75 12.74 0.31
4 1000 12.82 12.85 12.84 0.62
8 1000 12.87 12.88 12.87 1.24
16 1000 12.83 12.86 12.84 2.49
32 1000 13.25 13.28 13.26 4.82
64 1000 13.44 13.46 13.45 9.51
128 1000 13.49 13.51 13.50 18.94
256 1000 13.72 13.74 13.73 37.27
512 1000 13.69 13.71 13.70 74.72
1024 1000 15.73 15.75 15.74 130.07
2048 1000 20.72 20.76 20.74 197.28
4096 1000 22.68 22.74 22.72 360.28
8192 1000 29.48 29.52 29.50 555.04
16384 1000 39.89 39.95 39.92 820.31
32768 1000 57.38 57.48 57.43 1140.24
65536 640 95.23 95.34 95.29 1374.78
131072 320 214.61 215.16 214.83 1218.38
262144 160 365.75 368.39 367.28 1423.18
524288 80 679.82 687.10 683.13 1526.08
1048576 40 1277.18 1309.22 1295.65 1601.83
2097152 20 2292.99 2377.56 2339.35 1764.12
4194304 10 4617.95 4919.67 4778.37 1705.12
#-----------------------------------------------------------------------------
# Benchmarking Exchange
# #processes = 2
# ( 6 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 12.41 12.42 12.42 0.00
1 1000 12.47 12.48 12.47 0.32
2 1000 11.93 11.94 11.94 0.67
4 1000 11.95 11.96 11.95 1.34
8 1000 11.91 11.92 11.92 2.69
16 1000 11.97 11.98 11.97 5.34
32 1000 12.80 12.81 12.80 10.00
64 1000 12.84 12.84 12.84 19.93
128 1000 12.90 12.91 12.91 39.67
256 1000 12.90 12.91 12.91 79.34
512 1000 14.04 14.04 14.04 145.82
1024 1000 17.13 17.14 17.13 239.02
2048 1000 21.06 21.06 21.06 389.05
4096 1000 23.32 23.33 23.32 702.41
8192 1000 28.07 28.07 28.07 1167.45
16384 1000 37.81 37.82 37.82 1732.64
32768 1000 55.23 55.24 55.24 2372.75
65536 640 101.04 101.06 101.05 2593.84
131072 320 212.88 212.88 212.88 2462.84
262144 160 362.37 362.38 362.37 2893.62
524288 80 668.88 668.89 668.88 3135.26
1048576 40 1286.48 1287.81 1287.15 3256.92
2097152 20 2463.56 2464.13 2463.84 3404.29
4194304 10 4845.24 4854.75 4849.99 3455.83
#-----------------------------------------------------------------------------
# Benchmarking Exchange
# #processes = 4
# ( 4 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 16.46 16.46 16.46 0.00
1 1000 16.42 16.43 16.42 0.24
2 1000 16.17 16.17 16.17 0.49
4 1000 16.17 16.17 16.17 0.99
8 1000 16.19 16.20 16.20 1.98
16 1000 16.21 16.22 16.22 3.94
32 1000 17.20 17.21 17.20 7.44
64 1000 17.09 17.10 17.10 14.97
128 1000 17.24 17.25 17.25 29.68
256 1000 17.40 17.41 17.40 58.83
512 1000 17.59 17.61 17.60 116.32
1024 1000 21.43 21.45 21.44 190.95
2048 1000 29.49 29.50 29.49 277.71
4096 1000 31.63 31.66 31.64 517.58
8192 1000 36.70 36.72 36.71 892.41
16384 1000 49.50 49.53 49.52 1323.07
32768 1000 68.35 68.36 68.36 1917.38
65536 640 108.80 108.85 108.82 2408.31
131072 320 314.38 314.72 314.56 1665.91
262144 160 521.71 522.24 521.94 2007.84
524288 80 930.03 933.47 931.82 2246.62
1048576 40 1729.81 1738.30 1734.66 2412.87
2097152 20 3384.33 3414.99 3403.61 2456.41
4194304 10 6972.50 7058.12 7028.16 2377.01
#-----------------------------------------------------------------------------
# Benchmarking Exchange
# #processes = 8
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 18.91 18.93 18.92 0.00
1 1000 19.06 19.08 19.07 0.21
2 1000 18.91 18.92 18.92 0.42
4 1000 19.07 19.09 19.08 0.84
8 1000 18.81 18.83 18.82 1.70
16 1000 19.02 19.03 19.03 3.36
32 1000 19.85 19.85 19.85 6.45
64 1000 19.76 19.78 19.77 12.94
128 1000 19.94 19.96 19.95 25.65
256 1000 20.16 20.18 20.17 50.75
512 1000 20.50 20.51 20.50 99.86
1024 1000 24.52 24.55 24.54 166.83
2048 1000 36.35 36.39 36.37 225.14
4096 1000 38.77 38.81 38.79 422.20
8192 1000 44.79 44.82 44.81 731.12
16384 1000 59.28 59.33 59.31 1104.68
32768 1000 86.39 86.47 86.42 1515.87
65536 640 142.47 142.60 142.53 1838.29
131072 320 402.11 402.98 402.57 1301.04
262144 160 648.90 650.30 649.68 1612.44
524288 80 1209.17 1213.71 1211.74 1727.89
1048576 40 2332.69 2355.17 2344.35 1780.89
2097152 20 4686.88 4767.48 4733.77 1759.55
4194304 10 9457.18 9674.69 9567.31 1734.13
# All processes entering MPI_Finalize
The IMB benchmark test all at once
various MPI subroutines (MPI_Sendrecv and MPI_Exchange here)
various message sizes (from 0 to 4MB here)
various communicator sizes (2, 4 and 8 here)
Since mpirun is invoked once with -np 8, it means there 8 MPI tasks are created.
So when testing a size 2 communicator, an extra size 6 communicator is created under the hood, and its 6 MPI tasks are simply hanging in MPI_Barrier, hence the message
# #processes = 2
# ( 6 additional processes waiting in MPI_Barrier)
Related
i got problem how to delete several lines in txt file then convert into csv with R because i just want to get the data from txt.
My code cant delete propely because it delete lines which contain the date of the data
Here the code i used
setwd("D:/tugasmaritim/")
FILES <- list.files( pattern = ".txt")
for (i in 1:length(FILES)) {
l <- readLines(FILES[i],skip=4)
l2 <- l[-sapply(grep("</PRE><H3>", l), function(x) seq(x, x + 30))]
l3 <- l2[-sapply(grep("<P>Description", l2), function(x) seq(x, x + 29))]
l4 <- l3[-sapply(grep("<HTML>", l3), function(x) seq(x, x + 3))]
write.csv(l4,row.names=FALSE,file=paste0("D:/tugasmaritim/",sub(".txt","",FILES[i]),".csv"))
}
my data looks like this
<HTML>
<TITLE>University of Wyoming - Radiosonde Data</TITLE>
<LINK REL="StyleSheet" HREF="/resources/select.css" TYPE="text/css">
<BODY BGCOLOR="white">
<H2>96749 WIII Jakarta Observations at 00Z 02 Oct 1995</H2>
<PRE>
-----------------------------------------------------------------------------
PRES HGHT TEMP DWPT RELH MIXR DRCT SKNT THTA THTE THTV
hPa m C C % g/kg deg knot K K K
-----------------------------------------------------------------------------
1011.0 8 23.2 22.5 96 17.30 0 0 295.4 345.3 298.5
1000.0 98 23.6 22.4 93 17.39 105 8 296.8 347.1 299.8
977.3 300 24.6 22.1 86 17.49 105 8 299.7 351.0 302.8
976.0 311 24.6 22.1 86 17.50 104 8 299.8 351.2 303.0
950.0 548 23.0 22.0 94 17.87 88 12 300.5 353.2 303.7
944.4 600 22.6 21.8 95 17.73 85 13 300.6 352.9 303.8
925.0 781 21.2 21.0 99 17.25 90 20 301.0 351.9 304.1
918.0 847 20.6 20.6 100 16.95 90 23 301.0 351.0 304.1
912.4 900 20.4 18.6 89 15.00 90 26 301.4 345.7 304.1
897.0 1047 20.0 13.0 64 10.60 90 26 302.4 334.1 304.3
881.2 1200 19.4 11.4 60 9.70 90 26 303.3 332.5 305.1
850.0 1510 18.2 8.2 52 8.09 95 18 305.2 329.9 306.7
845.0 1560 18.0 7.0 49 7.49 91 17 305.5 328.4 306.9
810.0 1920 15.0 9.0 67 8.97 60 11 306.0 333.4 307.7
792.9 2100 14.3 3.1 47 6.06 45 8 307.1 325.9 308.2
765.1 2400 13.1 -6.8 24 3.01 40 8 309.0 318.7 309.5
746.0 2612 12.2 -13.8 15 1.77 38 10 310.3 316.2 310.6
712.0 3000 10.3 -15.0 15 1.69 35 13 312.3 318.1 312.6
700.0 3141 9.6 -15.4 16 1.66 35 13 313.1 318.7 313.4
653.0 3714 6.6 -16.4 18 1.63 32 12 316.0 321.6 316.3
631.0 3995 4.8 -2.2 60 5.19 31 11 317.0 333.9 318.0
615.3 4200 3.1 -3.9 60 4.70 30 11 317.4 332.8 318.3
601.0 4391 1.6 -5.4 60 4.28 20 8 317.8 331.9 318.6
592.9 4500 0.6 -12.0 38 2.59 15 6 317.9 326.6 318.4
588.0 4567 0.0 -16.0 29 1.88 11 6 317.9 324.4 318.3
571.0 4800 -1.2 -18.9 25 1.51 355 5 319.1 324.4 319.4
549.8 5100 -2.8 -22.8 20 1.12 45 6 320.7 324.8 321.0
513.0 5649 -5.7 -29.7 13 0.64 125 10 323.6 326.0 323.8
500.0 5850 -5.1 -30.1 12 0.63 155 11 326.8 329.1 326.9
494.0 5945 -4.9 -29.9 12 0.65 146 11 328.1 330.6 328.3
471.7 6300 -7.4 -32.0 12 0.56 110 13 329.3 331.5 329.4
453.7 6600 -9.6 -33.8 12 0.49 100 14 330.3 332.2 330.4
400.0 7570 -16.5 -39.5 12 0.31 105 14 333.5 334.7 333.5
398.0 7607 -16.9 -39.9 12 0.30 104 14 333.4 334.6 333.5
371.9 8100 -20.4 -42.6 12 0.24 95 16 335.4 336.3 335.4
300.0 9660 -31.3 -51.3 12 0.11 115 18 341.1 341.6 341.2
269.0 10420 -36.3 -55.3 12 0.08 79 20 344.7 345.0 344.7
265.9 10500 -36.9 75 20 344.9 344.9
250.0 10920 -40.3 80 28 346.0 346.0
243.4 11100 -41.8 85 37 346.4 346.4
222.5 11700 -46.9 75 14 347.6 347.6
214.0 11960 -49.1 68 16 348.1 348.1
200.0 12400 -52.7 55 20 349.1 349.1
156.0 13953 -66.1 55 25 352.1 352.1
152.3 14100 -67.2 55 26 352.6 352.6
150.0 14190 -67.9 55 26 352.9 352.9
144.7 14400 -69.6 60 26 353.6 353.6
137.5 14700 -72.0 60 39 354.6 354.6
130.7 15000 -74.3 50 28 355.6 355.6
124.2 15300 -76.7 40 36 356.5 356.5
118.0 15600 -79.1 50 48 357.4 357.4
116.0 15698 -79.9 45 44 357.6 357.6
112.0 15900 -79.1 45 26 362.6 362.6
106.3 16200 -78.0 35 24 370.2 370.2
100.0 16550 -76.7 35 24 379.3 379.3
</PRE><H3>Station information and sounding indices</H3><PRE>
Station identifier: WIII
Station number: 96749
Observation time: 951002/0000
Station latitude: -6.11
Station longitude: 106.65
Station elevation: 8.0
Showalter index: 6.30
Lifted index: -1.91
LIFT computed using virtual temperature: -2.80
SWEAT index: 145.41
K index: 6.50
Cross totals index: 13.30
Vertical totals index: 23.30
Totals totals index: 36.60
Convective Available Potential Energy: 799.02
CAPE using virtual temperature: 1070.13
Convective Inhibition: -26.70
CINS using virtual temperature: -12.88
Equilibrum Level: 202.64
Equilibrum Level using virtual temperature: 202.60
Level of Free Convection: 828.70
LFCT using virtual temperature: 909.19
Bulk Richardson Number: 210.78
Bulk Richardson Number using CAPV: 282.30
Temp [K] of the Lifted Condensation Level: 294.96
Pres [hPa] of the Lifted Condensation Level: 958.67
Mean mixed layer potential temperature: 298.56
Mean mixed layer mixing ratio: 17.50
1000 hPa to 500 hPa thickness: 5752.00
Precipitable water [mm] for entire sounding: 36.31
</PRE>
<H2>96749 WIII Jakarta Observations at 00Z 03 Oct 1995</H2>
<PRE>
-----------------------------------------------------------------------------
PRES HGHT TEMP DWPT RELH MIXR DRCT SKNT THTA THTE THTV
hPa m C C % g/kg deg knot K K K
-----------------------------------------------------------------------------
1012.0 8 23.6 22.9 96 17.72 140 2 295.7 346.9 298.9
1000.0 107 24.0 21.6 86 16.54 135 3 297.1 345.2 300.1
990.0 195 24.4 20.3 78 15.39 128 4 298.4 343.4 301.2
945.4 600 22.9 20.2 85 16.00 95 7 300.9 348.0 303.7
925.0 791 22.2 20.1 88 16.29 100 6 302.0 350.3 304.9
913.5 900 21.9 18.2 80 14.63 105 6 302.8 346.3 305.4
911.0 924 21.8 17.8 78 14.28 108 6 302.9 345.4 305.5
850.0 1522 17.4 16.7 96 14.28 175 6 304.4 347.1 307.0
836.0 1665 16.4 16.4 100 14.24 157 7 304.8 347.5 307.4
811.0 1925 15.0 14.7 98 13.14 123 8 305.9 345.6 308.3
795.0 2095 14.2 7.2 63 8.08 101 9 306.8 331.6 308.3
794.5 2100 14.2 7.2 63 8.05 100 9 306.8 331.5 308.3
745.0 2642 10.4 2.4 58 6.14 64 11 308.4 327.6 309.6
736.0 2744 11.0 0.0 47 5.23 57 11 310.2 326.7 311.1
713.8 3000 9.2 5.0 75 7.70 40 12 310.9 335.0 312.4
711.0 3033 9.0 5.6 79 8.08 40 12 311.0 336.2 312.6
700.0 3163 8.6 1.6 61 6.18 40 12 312.0 331.5 313.1
688.5 3300 8.3 -6.0 36 3.57 60 12 313.1 324.8 313.8
678.0 3427 8.0 -13.0 21 2.08 70 12 314.2 321.2 314.6
642.0 3874 5.0 -2.0 61 5.17 108 11 315.7 332.4 316.7
633.0 3989 4.4 -11.6 30 2.50 117 10 316.3 324.7 316.8
616.6 4200 3.1 -14.1 27 2.09 135 10 317.1 324.3 317.6
580.0 4694 0.0 -20.0 21 1.36 164 13 319.1 323.9 319.4
572.3 4800 -0.4 -20.7 20 1.29 170 14 319.9 324.5 320.1
510.8 5700 -4.0 -26.6 15 0.86 80 10 326.1 329.2 326.2
500.0 5870 -4.7 -27.7 15 0.79 80 10 327.2 330.2 327.4
497.0 5917 -4.9 -27.9 15 0.78 71 13 327.6 330.5 327.7
491.7 6000 -5.5 -28.3 15 0.76 55 19 327.9 330.7 328.0
473.0 6300 -7.6 -29.9 15 0.68 55 16 328.9 331.4 329.0
436.0 6930 -12.1 -33.1 16 0.54 77 17 330.9 333.0 331.0
400.0 7580 -17.9 -37.9 16 0.37 100 19 331.6 333.1 331.7
388.3 7800 -19.9 -39.9 15 0.31 105 20 331.8 333.1 331.9
386.0 7844 -20.3 -40.3 15 0.30 103 20 331.9 333.1 331.9
372.0 8117 -18.3 -38.3 16 0.38 91 23 338.1 339.6 338.1
343.6 8700 -22.1 -41.4 16 0.30 65 29 340.7 342.0 340.8
329.0 9018 -24.1 -43.1 16 0.26 73 27 342.2 343.2 342.2
300.0 9680 -29.9 -44.9 22 0.23 90 22 343.1 344.1 343.2
278.6 10200 -34.3 85 37 344.1 344.1
266.9 10500 -36.8 60 32 344.7 344.7
255.8 10800 -39.4 65 27 345.2 345.2
250.0 10960 -40.7 65 27 345.4 345.4
204.0 12300 -51.8 55 23 348.6 348.6
200.0 12430 -52.9 55 23 348.8 348.8
194.6 12600 -55.0 60 23 348.1 348.1
160.7 13800 -70.1 35 39 342.4 342.4
153.2 14100 -73.9 35 41 340.6 340.6
150.0 14230 -75.5 35 41 339.9 339.9
131.5 15000 -76.3 50 53 351.6 351.6
124.9 15300 -76.6 50 57 356.2 356.2
122.0 15436 -76.7 57 45 358.3 358.3
118.6 15600 -77.3 65 31 360.2 360.2
115.0 15779 -77.9 65 31 362.2 362.2
112.6 15900 -77.7 85 17 364.8 364.8
107.0 16200 -77.2 130 10 371.2 371.2
100.0 16590 -76.5 120 18 379.7 379.7
</PRE><H3>Station information and sounding indices</H3><PRE>
Station identifier: WIII
Station number: 96749
Observation time: 951003/0000
Station latitude: -6.11
Station longitude: 106.65
Station elevation: 8.0
Showalter index: -0.58
Lifted index: 0.17
LIFT computed using virtual temperature: -0.57
SWEAT index: 222.41
K index: 31.80
Cross totals index: 21.40
Vertical totals index: 22.10
Totals totals index: 43.50
Convective Available Potential Energy: 268.43
CAPE using virtual temperature: 431.38
Convective Inhibition: -84.04
CINS using virtual temperature: -81.56
Equilibrum Level: 141.42
Equilibrum Level using virtual temperature: 141.35
Level of Free Convection: 784.91
LFCT using virtual temperature: 804.89
Bulk Richardson Number: 221.19
Bulk Richardson Number using CAPV: 355.46
Temp [K] of the Lifted Condensation Level: 293.21
Pres [hPa] of the Lifted Condensation Level: 940.03
Mean mixed layer potential temperature: 298.46
Mean mixed layer mixing ratio: 16.01
1000 hPa to 500 hPa thickness: 5763.00
Precipitable water [mm] for entire sounding: 44.54
and here my data
data
and this is what i want to get
contoh
I need to find the average of every 6 months, starting from v1 to v15. Now that i know that there are v15 columns hence its working with my below code. But there will more than 15 columns and I need a generic code that can solve the purpose.
Logic i am using is: taking the average of columns - 1:6 and printing, then 2:7 and so on- till 15, as i know there are 15 columns. But there will more columns in actual.
csv file:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15
1 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
2 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.11 0.04 0.04 0.04 0.04 0.04 0.04 0.04
3 3.29 3.56 3.97 3.23 2.96 2.35 0.06 1.72 2.19 1.92 1.84 2.87 2.57 2.24 3.06
4 11.79 15.01 14.76 13.19 18.29 4.51 16.24 11.92 10.49 13.05 12.74 12.95 12.25 14.46 14.27
5 20.11 21.76 21.92 23.67 19.87 25.59 23.04 16.67 22.78 21.32 20.85 21.57 21.99 22.69 22.96
6 24.85 26.56 29.45 24.96 25.91 16.31 27.51 22.56 28.35 26.96 26.53 28.23 28.24 29.85 29.79
7 29.02 32.75 29.95 27.7 29.6 17.91 32.08 25.71 33.16 31.56 30.89 32.68 34.05 36.26 33.27
8 32.83 33.09 17.03 33.23 31.22 39.71 35.43 28.77 37.09 34.18 34.05 36.98 37.16 38.74 37.32
9 32.86 36.34 35.47 33.6 35 42.79 37.22 30.62 38.74 35.83 36.17 39.48 39.18 42.87 39.54
10 36.02 37.66 36.15 34.79 36.84 22.19 38.9 32.62 40.28 37.87 38.09 41.04 41.62 44.94 42.18
11 36.96 39.22 19.13 36.68 37.43 46.26 40.84 33.88 41.31 39.09 39.14 43.46 42.75 47.2 43.8
12 37.34 40.87 35.91 37.66 39.22 46.95 42.26 35.19 42.93 41 40.61 44.73 45.2 48.14 44.49
13 38.92 38.37 41.01 39.01 41 48.89 43.8 37.16 44.1 42.46 41.3 45.47 46.65 50.48 47.6
14 21.67 43.16 20.98 39.84 42 49.62 44.35 37.46 44.63 43.15 42.64 48.48 48.53 53.55 48.57
a <- t(apply(mat,1,function(x){ c(mean(x[1:6]),mean(x[2:7]),mean(x[3:8]),mean(x[4:9]),mean(x[5:10]),mean(x[6:11]),mean(x[7:12]),mean(x[8:13]),mean(x[9:14]),mean(x[10:15])) }))
Please help. thanks in Advance.
We can do this with a rolling mean (rollmean
library(zoo)
t(apply(df1, 1, function(x) rollmean(x, 6)))
Using base R:
n=6
d=lapply(1:(ncol(data)-(n-1)),function(x) x:(x+n-1))
sapply(d,function(w) rowMeans(data[,w]))
another base solution:
rowlingRowMeans <- function(matrix, n_meanrows){
out <- NULL
for(z in 1:(nrow(matrix)-n_meanrows+2)){
out <- cbind(out, rowMeans(matrix[,z:(z+n_meanrows-1)]))
}
return(out)
}
mat <- matrix(rnorm(15*14, 1,10), ncol=15, nrow=14)
rowlingRowMeans(mat, 6)
I'm trying to use cor() to return the most correlated elements in order of their correlation. I wrote this function adapting cor() to do it and it works perfectly, but only when I run it on a big input. When I try and run it on a small input, I get a missing value where TRUE/FALSE needed error and I don't understand why?
Here is an example of my input data:
This can be directly copied into R(printed via write.table):
"Col2" "Col3" "Col4" "Col5" "Col6"
"Market Capitalization" NA NA 17082.69 17879.8 16266.11
"Cash & Equivalents" NA NA 747 132 394
"Preferred & Other" NA NA 0 0 0
"Total Debt" NA NA 12379 11982 11309
"Enterprise Value" NA NA 28714.69 29729.8 27181.11
"Total Revenue" 2896.75 3461.25 2818 3184 2901
"Growth % YoY" -0.15 0.68 1.7 3.44 -0.48
"Gross Profit" NA NA 1874 2080 1981
"Margin %" NA NA 66.5 65.33 68.29
"EBITDA" 758 1074 641 777 699
"Margin %1" 26.17 31.03 22.75 24.4 24.1
"Net Income Before XO" 214.5 410 172 192 207
"Margin %2" 7.4 11.85 6.1 6.03 7.14
"Adjusted EPS" 0.7 1.42 0.59 1.07 0.69
"Growth % YoY1" 0.72 -1.67 -3.28 5.94 -6.76
"Cash from Operations" 375.79 812.21 991 -84 961
"Capital Expenditures" NA NA -660 -676 -608
"Free Cash Flow" NA NA 331 -760 353
"Adjusted Price" 2094.66 3689.2 3805.62 3588.42 3582.4
This is the mycor() function I wrote
mycor<-function(dataset, relative.to=19, neg.cor=0){
#This takes the dataset (as a matrix) and computes the best correleted value
#and returns the row (variable ID) that is the most strongly correlated
#to the variable row referenced by relative.to. Use neg.cor = 1 for neg correlation
if(neg.cor == 0){
best.cor <- -1.0 #Have to get better correlation then this
best.cor.row <- integer() #The row with the best correlation
all.cor <- numeric() #The correlation for everything else
index <- 1 #The index for the all.cor array
for(i in 1:nrow(dataset)){
if(i != relative.to){ #No self correlation
temp.cor <- cor(dataset[i,], dataset[relative.to,], use = "na.or.complete")
all.cor[index] <- temp.cor
index <- index+1 #I wish the ++ opperator worked in R...
cat(best.cor)
pause()
if(temp.cor > best.cor){ #This remembers the best seen cor value
best.cor <- temp.cor
best.cor.row <- i
} #End inner if
} #End outter if
} #End for loop
}else{
best.cor <- 1.0 #Have to get better correlation then this
best.cor.row <- integer() #The row with the best correlation
all.cor <- numeric() #The correlation for everything else
index <- 1 #The index for the all.cor array
for(i in 1:nrow(dataset)){
if(i != relative.to){ #No self correlation
temp.cor <- cor(dataset[i,], dataset[relative.to,], use = "na.or.complete")
all.cor[index] <- temp.cor
index <- index+1 #I wish the ++ opperator worked in R...
if(temp.cor < best.cor){ #This remembers the worst seen cor value
best.cor <- temp.cor
best.cor.row <- i
} #End inner if
} #End outter if
} #End for loop
} #End else
return(list(all.cor = all.cor, best.cor.row = best.cor.row))
)
When I try and run this I get: Error in if (temp.cor > best.cor) { : missing value where TRUE/FALSE needed. The part about this that is strange, is that the mycor function works perfectly and gives no error when I give it a larger chunk of the same data set.
This is the larger chunk of the same data set.
This can also be copied into R(printed via write.table):
"Col2" "Col3" "Col4" "Col5" "Col6" "Col7" "Col8" "Col9" "Col10" "Col11" "Col12" "Col13" "Col14" "Col15" "Col16" "Col17" "Col18" "Col19" "Col20" "Col21" "Col22" "Col23" "Col24" "Col25" "Col26" "Col27" "Col28" "Col29" "Col30" "Col31" "Col32" "Col33" "Col34" "Col35" "Col36" "Col37" "Col38" "Col39" "Col40" "Col41" "Col42" "Col43" "Col44" "Col45" "Col46" "Col47" "Col48" "Col49" "Col50" "Col51" "Col52" "Col53" "Col54" "Col55" "Col56" "Col57" "Col58" "Col59" "Col60" "Col61" "Col62" "Col63" "Col64" "Col65" "Col66" "Col67" "Col68" "Col69" "Col70" "Col71" "Col72" "Col73" "Col74" "Col75" "Col76" "Col77" "Col78" "Col79" "Col80" "Col81" "Col82" "Col83" "Col84" "Col85" "Col86" "Col87" "Col88" "Col89" "Col90" "Col91" "Col92" "Col93" "Col94" "Col95" "Col96" "Col97" "Col98" "Col99" "Col100" "Col101" "Col102" "Col103" "Col104" "Col105" "Col106" "Col107" "Col108" "Col109" "Col110" "Col111"
"Market Capitalization" NA NA 17082.69 17879.8 16266.11 17540.1 18214.39 17110.13 18167.87 16700.24 15592.71 14824.06 14455.42 13685.56 12168.31 12550.1 12771.45 11273.2 10284.48 10863.21 10655.99 11750.74 10671.37 10818.32 13288.42 12558.8 12221.79 13213.51 12375.92 11854.12 10942.65 10689.79 11364.1 11887.9 11426.1 10249.34 10609.99 10167.51 9600.1 10001.68 9713.38 9184.3 9730.33 8249.64 9160.61 8586.38 8894.55 8908.81 11887.9 11426.1 10249.34 10609.99 10167.51 9600.1 10001.68 9713.38 9184.3 9730.33 8249.64 9160.61 8586.38 8894.55 8908.81 8566.69 8641.04 8444.84 7867.83 8163.04 7238.2 6279.55 6173.33 7376.47 9048.75 10095.35 10351.52 12311.04 12006.02 10785.58 11009.16 9655.09 7990.1 6918.52 7050.24 6844.2 6520.75 6873.11 7489.61 7459.85 7136.58 6930.38 6401.43 6048.8 5843.01 6224.43 6840.76 7529.23 8452.46 8247.48 8132.72 7632.03 7339.11 6549.2 6165.26 6535.8 5793.52 5621.57 5877.31 5391.98 4792.51 5362.35
"Cash & Equivalents" NA NA 747 132 394 69 1381 769 648 398 492 516 338 198 178 87 260 75 311 651 74 68 1757 144 210 192 186 157 94 234 63 177 81 119 818 477 26 70 487 55 49 49 60 62 117.86 83.4 59.2 108.34 119 818 477 26 70 487 55 49 49 60 62 117.86 83.4 59.2 108.34 271.35 432.14 41.63 59.57 94.83 72.81 37.66 73.6 485.05 188.94 291.14 57.5 102.29 153.82 105.01 198.26 183.46 269.87 12.23 94.9 106.88 117.28 57.37 103.23 342.29 429.89 48.49 111.39 245.22 360.74 80.65 205.1 36.76 203.96 143.32 74.33 282.45 349.66 384.84 238.24 317.86 315.65 291.01 185.21 353.33 160.33 160.31
"Preferred & Other" NA NA 0 0 0 0 0 0 213 213 213 213 213 213 213 213 213 213 213 213 213 213 213 257 256 255 255 254 254 254 255 255 255 254 255 255 252 252 253 254 255 221 222 221 221.47 221.13 221.2 220.79 254 255 255 252 252 253 254 255 221 222 221 221.47 221.13 221.2 220.79 222.09 212.56 249.61 212.56 249.61 212.56 212.56 212.56 249.61 212.56 212.56 212.56 249.61 318.02 318.02 318.02 318.02 322.34 322.42 322.54 322.65 322.74 322.77 322.84 639.92 639.98 640.13 640.24 640.31 640.39 640.47 640.54 640.73 640.89 640.95 641.09 641.25 645.87 634.99 635.05 635.18 637.51 637.73 638.05 638.15 640.53 640.77
"Total Debt" NA NA 12379 11982 11309 11111 11873 11073 10675 10676 10678 11144 10683 11526 11020 11027 10599 10773 10366 10699 10094 9751 9480 9363 9282 9213 8653 8943 8815 8968 8487 8162 8205 7687 7868 7498 7219 7245 7336 7432 7094 6968 6682 7000 6841.23 6584.25 6374.14 6264.74 7687 7868 7498 7219 7245 7336 7432 7094 6968 6682 7000 6841.23 6584.25 6374.14 6264.74 6234.03 6249.6 6448.51 6100.6 6011.55 5693.56 5536.13 5276.01 5449.52 4792.08 4881.68 4471.08 4312.4 4410.61 4480.08 4437.33 4758.17 4432.04 4532.28 4466.59 4387.54 4313.86 4316.43 4316.66 4146.02 4175.36 4082.33 4085.09 4089.16 4116.98 3970.11 3972.46 3827.89 3850.12 3927.94 3722.68 3709.36 3804.58 3658.69 3885.52 3667.45 3734.29 3737 3615.16 3492.38 3374.62 3229.81
"Enterprise Value" NA NA 28714.69 29729.8 27181.11 28582.1 28706.39 27414.13 28407.87 27191.24 25991.71 25665.06 25013.42 25226.56 23223.31 23703.1 23323.45 22184.2 20552.48 21124.21 20888.99 21646.74 18607.37 20294.32 22616.42 21834.8 20943.79 22253.51 21350.92 20842.12 19621.65 18929.79 19743.1 19709.9 18731.1 17525.34 18054.99 17594.51 16702.1 17632.68 17013.38 16324.3 16574.33 15408.64 16105.45 15308.35 15430.68 15286 19709.9 18731.1 17525.34 18054.99 17594.51 16702.1 17632.68 17013.38 16324.3 16574.33 15408.64 16105.45 15308.35 15430.68 15286 14751.46 14671.06 15101.34 14121.44 14329.37 13071.51 11990.59 11588.31 12590.55 13864.46 14898.46 14977.66 16770.77 16580.82 15478.67 15566.25 14547.82 12474.62 11760.98 11744.46 11447.51 11040.07 11454.93 12025.88 11903.5 11522.02 11604.35 11015.38 10533.05 10239.65 10754.35 11248.66 11961.09 12739.51 12673.05 12422.15 11700.18 11439.9 10458.04 10447.58 10520.58 9849.67 9705.29 9945.31 9169.17 8647.34 9072.61
"Total Revenue" 2896.75 3461.25 2818 3184 2901 3438 2771 3078 2915 3629 2993 3349 3140 3707 3017 3462 3273 3489 2845 3423 2998 3858 3149 3577 3228 3579 2957 3357 2649 3441 2555 3317 3107 3337 2395 2800 2181 2734 2164 2685 2279 2801 2176 2570 2057.03 2539.49 1848 2056 3337 2395 2800 2181 2734 2164 2685 2279 2801 2176 2570 2057.03 2539.49 1848 2056 1942.6 2627.56 2112.22 2886.26 2250.13 2820.78 2041.89 2318.59 1963.38 2346.24 1479.08 1776.59 1617.34 2061.62 1561.04 1853.05 1720.06 2011.03 1504.01 1886.15 1632.3 1920.34 1539.73 1867.36 1528.38 1879.88 1459.85 1668.79 1461.25 1821.99 1392.09 1697.76 1483.61 1799.69 1396.01 1586.08 1478.81 1717.88 1280.11 1456.11 1342.73 1720.3 1330.65 1479.39 1367.21 1613.83 1263.27
"Growth % YoY" -0.15 0.68 1.7 3.44 -0.48 -5.26 -7.42 -8.09 -7.17 -2.1 -0.8 -3.26 -4.06 6.25 6.05 1.14 9.17 -9.56 -9.65 -4.31 -7.13 7.8 6.49 6.55 21.86 4.01 15.73 1.21 -14.74 3.12 6.68 18.46 42.46 22.06 10.67 4.28 -4.3 -2.39 -0.55 4.47 10.79 10.3 17.75 25 5.89 -3.35 -12.51 -28.77 22.06 10.67 4.28 -4.3 -2.39 -0.55 4.47 10.79 10.3 17.75 25 5.89 -3.35 -12.51 -28.77 -13.67 -6.85 3.44 24.48 14.6 20.23 38.05 30.51 21.4 13.81 -5.25 -4.13 -5.97 2.52 3.79 -1.75 5.38 4.72 -2.32 1.01 6.8 2.15 5.47 11.9 4.59 3.18 4.87 -1.71 -1.51 1.24 -0.28 7.04 0.32 4.76 9.05 8.93 10.13 -0.14 -3.8 -1.57 -1.79 6.6 5.33 -1.02 NA NA NA
"Gross Profit" NA NA 1874 2080 1981 2393 1934 1993 1846 2244 1794 2000 1942 2103 1723 1826 1700 1979 1558 1551 1459 1531 1420 1588 1478 1595 1317 1506 1273 1554 1202 1322 1179 1460 1097 1217 916 1285 980 1169 1066 1349 975 1157 1024.93 1317.57 980 1091 1460 1097 1217 916 1285 980 1169 1066 1349 975 1157 1024.93 1317.57 980 1091 1052.71 1368.8 1091.61 1236.41 991.8 1374.86 1043.29 1236.87 1129.87 1507.31 998.19 1190.69 1151.22 1475.08 1025.84 1170.8 1115.9 1438.56 981.96 1159.37 1094.25 1401.25 1001.2 1198.64 1079.65 1405.45 984.46 1196.22 1086.13 1415.37 998.06 1177.1 1086.53 1381.01 971.41 1118.91 1055.19 1331.37 947.22 1036.88 991.58 1301.1 921.48 994.97 967.89 1217.32 848.39
"Margin %" NA NA 66.5 65.33 68.29 69.6 69.79 64.75 63.33 61.84 59.94 59.72 61.85 56.73 57.11 52.74 51.94 56.72 54.76 45.31 48.67 39.68 45.09 44.39 45.79 44.57 44.54 44.86 48.06 45.16 47.05 39.86 37.95 43.75 45.8 43.46 42 47 45.29 43.54 46.77 48.16 44.81 45.02 49.83 51.88 53.03 53.06 43.75 45.8 43.46 42 47 45.29 43.54 46.77 48.16 44.81 45.02 49.83 51.88 53.03 53.06 54.19 52.09 51.68 42.84 44.08 48.74 51.09 53.35 57.55 64.24 67.49 67.02 71.18 71.55 65.72 63.18 64.88 71.53 65.29 61.47 67.04 72.97 65.02 64.19 70.64 74.76 67.44 71.68 74.33 77.68 71.7 69.33 73.24 76.74 69.58 70.55 71.35 77.5 74 71.21 73.85 75.63 69.25 67.26 70.79 75.43 67.16
"EBITDA" 758 1074 641 777 699 1091 711 794 684 978 617 844 708 916 640 696 625 885 569 611 567 586 520 702 596 715 510 694 547 670 467 564 423 717 411 533 274 624 367 497 458 669 334 485 388.44 693.3 384 487 717 411 533 274 624 367 497 458 669 334 485 388.44 693.3 384 487 445 695.27 439.32 538.75 377.16 666.39 492.65 526.86 446.87 748.34 331.51 492.91 430.87 760.5 313.33 474.78 434.79 751.92 280.96 463.41 390.79 712.97 313.14 490.27 368.26 711.24 307.36 506.85 383.64 721.41 317.3 474.34 363.04 678.27 279.09 400.41 320.03 637.82 281.47 340.21 297.39 610.07 247.48 300.27 305.15 561.67 203.06
"Margin %1" 26.17 31.03 22.75 24.4 24.1 31.73 25.66 25.8 23.46 26.95 20.61 25.2 22.55 24.71 21.21 20.1 19.1 25.37 20 17.85 18.91 15.19 16.51 19.63 18.46 19.98 17.25 20.67 20.65 19.47 18.28 17 13.61 21.49 17.16 19.04 12.56 22.82 16.96 18.51 20.1 23.88 15.35 18.87 18.88 27.3 20.78 23.69 21.49 17.16 19.04 12.56 22.82 16.96 18.51 20.1 23.88 15.35 18.87 18.88 27.3 20.78 23.69 22.91 26.46 20.8 18.67 16.76 23.62 24.13 22.72 22.76 31.9 22.41 27.74 26.64 36.89 20.07 25.62 25.28 37.39 18.68 24.57 23.94 37.13 20.34 26.25 24.09 37.83 21.05 30.37 26.25 39.59 22.79 27.94 24.47 37.69 19.99 25.25 21.64 37.13 21.99 23.36 22.15 35.46 18.6 20.3 22.32 34.8 16.07
"Net Income Before XO" 214.5 410 172 192 207 440 214 280 193 386 168 314 236 353 186 229 205 339 153 183 163 185 283 303 209 313 154 261 205 234 129 183 148 290 121 184 55 253 92 158 50 260 69 157 123.03 286.54 101 169 290 121 184 55 253 92 158 50 260 69 157 123.03 286.54 101 169 128.51 280.74 104.07 182.51 49.48 283.27 72.14 191.53 124.96 339.41 69.8 180.05 135.23 351.55 66.51 176.45 143.61 355.04 47.56 166.61 120.15 327.99 71.42 188.48 113.12 333.3 76.4 201.03 117.88 339.87 87.21 189.31 117.29 324.84 62.45 153.94 100.63 309.44 77.54 116.48 92.2 303.36 64.65 106.7 121.1 263.26 49.06
"Margin %2" 7.4 11.85 6.1 6.03 7.14 12.8 7.72 9.1 6.62 10.64 5.61 9.38 7.52 9.52 6.17 6.61 6.26 9.72 5.38 5.35 5.44 4.8 8.99 8.47 6.47 8.75 5.21 7.77 7.74 6.8 5.05 5.52 4.76 8.69 5.05 6.57 2.52 9.25 4.25 5.88 2.19 9.28 3.17 6.11 5.98 11.28 5.47 8.22 8.69 5.05 6.57 2.52 9.25 4.25 5.88 2.19 9.28 3.17 6.11 5.98 11.28 5.47 8.22 6.62 10.68 4.93 6.32 2.2 10.04 3.53 8.26 6.36 14.47 4.72 10.13 8.36 17.05 4.26 9.52 8.35 17.65 3.16 8.83 7.36 17.08 4.64 10.09 7.4 17.73 5.23 12.05 8.07 18.65 6.26 11.15 7.91 18.05 4.47 9.71 6.8 18.01 6.06 8 6.87 17.63 4.86 7.21 8.86 16.31 3.88
"Adjusted EPS" 0.7 1.42 0.59 1.07 0.69 1.44 0.61 1.01 0.74 1.33 0.57 0.99 0.69 1.32 0.51 0.93 0.67 1.16 0.48 0.78 0.72 0.98 0.42 0.87 0.71 1.2 0.58 1.03 0.78 0.92 0.51 0.86 0.59 1.17 0.48 0.75 0.49 1.08 0.38 0.69 0.65 1.16 0.29 0.72 0.56 1.33 0.46 0.78 1.17 0.48 0.75 0.49 1.08 0.38 0.69 0.65 1.16 0.29 0.72 0.56 1.33 0.46 0.78 0.59 1.3 0.48 0.84 0.52 1.4 0.33 0.88 0.57 1.5 0.3 0.76 0.56 1.49 0.26 0.73 0.59 1.49 0.18 0.69 0.49 1.38 0.28 0.78 0.44 1.38 0.29 0.82 0.47 1.41 0.33 0.77 0.46 1.35 0.23 0.62 0.39 1.3 0.3 0.47 0.36 1.29 0.24 0.43 0.49 1.11 0.18
"Growth % YoY1" 0.72 -1.67 -3.28 5.94 -6.76 8.27 7.02 2.02 7.25 0.76 11.76 6.45 2.99 13.79 6.25 19.23 -6.94 18.37 14.29 -10.34 1.41 -18.33 -27.59 -15.53 -8.97 30.43 13.73 19.77 32.2 -21.37 6.25 14.67 20.41 8.33 26.32 8.7 -24.62 -6.9 31.03 -4.17 16.07 -12.78 -36.96 -7.69 -5.08 2.31 -4.17 -7.14 8.33 26.32 8.7 -24.62 -6.9 31.03 -4.17 16.07 -12.78 -36.96 -7.69 -5.08 2.31 -4.17 -7.14 13.46 -7.14 45.45 -4.55 -8.77 -6.67 10 15.79 1.79 0.67 13.64 4.11 -5.08 -0.07 44.44 5.89 20.41 8.05 -34.72 -11.62 11.36 0 -3.45 -4.88 -6.38 -2.13 -12.12 6.49 2.17 4.44 43.48 24.19 17.95 3.85 -23.33 31.91 8.33 0.78 25 9.3 -26.53 16.22 33.33 -23.21 NA NA NA
"Cash from Operations" 375.79 812.21 991 -84 961 391 845 402 976 572 1227 362 1407 179 794 1 997 26 798 645 581 -1237 733 563 630 109 346 481 710 -162 224 593 177 581 -346 389 525 164 490 152 766 218 492 -58 735.49 285 369 146 581 -346 389 525 164 490 152 766 218 492 -58 735.49 285 369 146 490.18 387.73 254.59 141.41 215.82 279.84 489.5 199.17 -325.31 -66.66 280.22 256.65 718.82 438.66 302.05 244.37 -52.38 647.78 53.19 258.9 294.29 359.1 267.8 184.51 310.07 585.52 233.75 145.31 426.63 480.57 187.86 270.34 236.08 472.92 243.13 69.8 261.19 291.41 285.57 77.33 283.64 328.4 309.68 11.95 357.21 141.59 357.15
"Capital Expenditures" NA NA -660 -676 -608 -478 -635 -523 -542 -503 -629 -460 -599 -548 -551 -465 -719 -531 -595 -529 -785 -584 -608 -547 -638 -519 -485 -482 -583 -480 -537 -420 -619 -385 -426 -390 -431 -439 -308 -373 -448 -356 -404 -317 -593.69 -310 -392 -340 -385 -426 -390 -431 -439 -308 -373 -448 -356 -404 -317 -593.69 -310 -392 -340 -302.22 -394.08 -274.8 -228.02 -75.57 -274.36 -684.94 -207.41 -211.95 -218.98 -157.07 -127.56 -210.59 -156.81 -150.58 -127.3 -226.32 -145.55 -171.37 -140.37 -244.12 -167.92 -185.35 -142.94 -239.55 -165.98 -166.25 -147.38 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
"Free Cash Flow" NA NA 331 -760 353 -87 210 -121 434 69 598 -98 808 -369 243 -464 278 -505 203 116 -204 -1821 125 16 -8 -410 -139 -1 127 -642 -313 173 -442 196 -772 -1 94 -275 182 -221 318 -138 88 -375 141.79 -25 -23 -194 196 -772 -1 94 -275 182 -221 318 -138 88 -375 141.79 -25 -23 -194 187.96 -6.35 -20.21 -86.61 140.26 5.47 -195.45 -8.24 -537.26 -285.64 123.15 129.09 508.23 281.85 151.46 117.07 -278.7 502.23 -118.18 118.53 50.17 191.18 82.45 41.57 70.51 419.54 67.49 -2.08 426.63 480.57 187.86 270.34 236.08 472.92 243.13 69.8 261.19 291.41 285.57 77.33 283.64 328.4 309.68 11.95 357.21 141.59 357.15
"Adjusted Price" 2094.66 3689.2 3805.62 3588.42 3582.4 3885.75 3523.13 3554.9 3420.27 3141.36 2984.19 2838.81 2760.09 2517.44 2447.56 2403.89 2188.98 1960.8 1952.2 2033.87 2099.97 1993.98 2043.36 2296.42 2201.73 2277.15 2301.5 2203.47 2086.87 1938.95 2019.34 2002.47 2048.12 1881.97 1817.17 1807.02 1664.57 1659.78 1717.25 1585.27 1589.9 1506.13 1534.98 1531.24 1498.21 1528.96 1418.46 1431.1 1343.43 1244.04 1194.62 1076.93 1058.66 960.76 1112.69 1322.69 1414.59 1442.28 1545.6 1364.27 1305.46 1231.15 1022.23 869.37 796.9 820.22 762.84 715.9 756.11 816.37 731.97 705.73 657.84 628.55 571.47 624.67 651.89 676.63 759.77 742.27 734.39 657.44 619.61 569.84 524.2 510.26 475.43 449.8 441.27 409.34 383 413.34 441.72 435.71 419.07 385.87 356.85 346.15 326.97 318.45 323.72 314.18 313.22 300.88 329.3 315.1 312.34 279.11 163.47 NA
The larger chunk works perfectly, but I need to be able to check the correlation on the smaller sections. I'm really new to R so it might be easy, but I've read the boards here and the r manuals and can't find it.
In your example above, your code fails on the first (smaller) data set because row 3 consists only of 0's and NA's, so it has a standard deviation of 0 and so its correlation with any other row will return NA, since computing correlation involves dividing the sample covariance by the sample standard deviation of each vector. It doesn't happen in the larget example because row 3 has sufficient variation to have a non-zero standard deviation.
However, your approach seems a bit convoluted. If you want to compute the correlation between a single row in the matrix and all other rows, sorted by correlation, then you can use cor() on the transposed matrix and sort the result, for example:
mycor <- function(dataset, relative.to=19) {
mat <- t(dataset)
cors <- cor(mat, mat[, relative.to], use="na.or.complete")
cors[order(drop(cors)), ]
}
mycor(dataset)
Had anyone else had this problem, or even better, does anyone know why this is giving me an error?
I'm attempting to create an ARMA model of order 3, 3. I'm using the TSA package.
stocks_arma <- arma(stocks$close, order = c(3,3))
I'm getting this warning:
Warning message:
In arma(VIXts, order = c(3, 3)) : Hessian negative-semidefinite
I understand that a Hessian negative-semidefinite matrix is a bad thing because we usually want global mins/maxes. However, I don't understand why this is happening. I am unsure is this is a mathematical issue or a syntactial issue.
My data is a very modest vector of 1000 entries. Here is one-tenth of it:
15.14 15.31 15.08 15.24 16.41 17.99 17.92 16.65 16.68 18.61 18.49 19.08 17.58 18.42 17.59 16.69 18.60 17.81 18.12 18.33 18.83 16.62 16.97 15.03 15.07 15.22 15.27 16.14 15.59 16.29 16.37 15.11 14.33 14.55 15.43 15.71 16.32 15.73 14.84 16.81 15.43 14.15 13.98 14.07 13.88 14.18 14.59 14.51 14.05 15.80 16.41 16.28 14.38 15.63 17.74 17.98 17.47 17.83 17.06 16.49 16.35 15.18 15.96 15.11 15.02 14.02 13.45 14.29 14.63 14.85 13.70 14.74 15.28 15.32 15.99 15.95 15.64 17.57 18.96 18.93 18.03 16.70 17.53 19.34 20.47 18.62 16.27 15.45 16.16 16.48 17.11 16.74 18.36 17.95 18.72 18.05 17.10 17.50 16.66 16.80 17.08 19.71 19.45 19.72 20.38
There is nothing overtly fishy about the values at all.
Any insight is very much appreciated.
I have three tables, vehicle_service_invoice, vehicle_service_labor, vehicle_service_parts. All of these tables relate to each other using the invoice row on each table. I want to make a view of these three tables called vehicle_service that will show the total price for parts and labor as they are connected to the invoice table.
With a single table, I'd SUM the extended column and GROUP BY the invoice column on both the vehicle_service_labor and vehicle_service_parts. But how would I connect this information back to the vehicle_service_invoice's invoice column so that it would display correctly?
I'd want it to look something like this ...
vehicle_service
vehicle date po invoice labor parts sublet total
008 2013-01-07 1301070 1111 204.00 129.11 0.00 333.11
008 2013-01-21 1301210 1122 521.00 584.70 0.00 1105.70
003 2013-02-07 1302070 2211 34.00 0.00 0.00 34.00
004 2013-02-18 1302180 2222 51.00 70.14 0.00 121.14
003 2013-03-18 1303180 3311 51.00 70.14 0.00 121.14
003 2013-04-18 1204180 4411 51.00 412.83 0.00 463.83
008 2013-04-25 1304250 4422 68.00 269.82 0.00 337.82
006 2013-05-25 1305250 5511 204.00 17.85 0.00 221.85
007 2013-05-29 1305290 5522 442.00 299.35 0.00 741.35
006 2013-06-29 1306290 6611 136.00 0.00 0.00 136.00
003 2013-07-16 1307160 7711 136.00 0.00 0.00 136.00
004 2013-08-16 1308160 8811 187.00 172.53 0.00 359.53
003 2013-09-16 1309160 9911 136.00 140.47 0.00 276.47
007 2013-10-30 1310300 1011 34.00 29.83 0.00 63.83
008 2013-11-30 1311300 0000 136.00 175.25 0.00 311.25
These are the tables I'm sourcing the information from.
vehicle_service_invoice
invoice date unit odometer sublet sub po
1111 2013-01-07 008 34863 0 0.00 1301070
1122 2013-01-21 008 36435 0 1105.70 1301210
2211 2013-02-07 003 35594 0 34.00 1302070
2222 2013-02-18 004 49079 121.14 1531214
3311 2013-03-18 003 36158 0 121.14 1303180
4411 2013-04-18 003 0 463.83 1204180
4422 2013-04-25 008 36516 0 337.82 1304250
5511 2013-05-25 006 48807 0 221.85 1305250
5522 2013-05-29 007 37133 0 741.35 1305290
7711 2013-06-06 003 38535 0 136.00 1307160
8811 2013-06-16 004 51588 0 359.53 1308160
9911 2013-07-16 003 39302 0 276.47 1309160
1011 2013-07-30 007 39675 0 63.83 1310300
0000 2013-08-30 008 40027 0 311.25 1311300
vehicle_service_labor
invoice labor hours extended description
1111 Pick Up 0 0 Pick Up & Del...
1111 Fuel Leak 3 204 Locate fuel l...
1122 PM Inspection 5 340 Preventative ...
1122 NYS Inspection 0.5 45 NYS Inspection
1122 Tires 1 68 Breakdown, mo...
1122 Lights 0.5 34 Repair front ...
1122 Receptacle 0.5 34 Replace broke...
2211 Flat 0.5 34 Repair Rear Flat
2222 Door 0.75 51 Remove broken...
3311 Door 0.75 51 Remove broken...
4411 Mirror 0.75 51 Remove and Re...
4422 Batteries 1 68 Check batteri...
5511 Pick Up 0 0 Pick Up & Del...
5511 Brakes 3 204 Add fluid, br...
5522 PM Inspection 5 340 Preventative ...
5522 NYS Inspection 0.5 34 NYS Inspection
5522 Leak 1 68 Checkout Cab ...
7711 Flat Call 2 136 Service Call ...
8811 LOF 2 136 Change Oil in...
8811 Flat 0.75 51 Raise and Sup...
9911 LOF 2 136 Change Oil in...
1011 Door 0.5 34 Check Out Doo...
0000 Flats 2 136 Service call ...
vehicle_service_parts
invoice part qty sale extended description
1111 Fuel Treatment 1 28.73 28.73 Enertech
1111 Fuel Filter 1 76.66 76.66 PF7698
1111 Antifreeze 1 23.72 23.72 Glychol
1122 Engine Oil 16 4.48 71.68 15W40
1122 Oil Filter 1 34.76 34.76 FL1995
1122 Air Filter 2 31.26 62.52 FA1618
1122 Fuel Additive 1 14.33 14.33 Stanadyne
1122 Coolant Additive 1 12.95 12.95 DCA60L
1122 Recepticle Cover 1 35 35 KUS 091-3WH
1122 22575r16 10 P... 2 165 333
1122 Tire Disposal... 2 8 16 Tire Disposal
1122 Tire Waste Sa... 1 2.25 2.25 NYS Tire Fee
1122 Mini Lightbulb 1 3.13 3.13 3156
1122 Bulb 2 1.04 2.08 194
2222 Pivot-Sprint Bar 1 31.11 31.11 Brapivot
2222 Door Check Right 1 39.03 39.03 15721
3311 Pivot-Sprint Bar 1 31.11 31.11 Brapivot
3311 Door Check Right 1 39.03 39.03 15721
4411 Mirror Assemb... 1 412.83 412.83 9C2Z17683CA
4422 Battery 2 134.91 269.82 665MF
5511 Brake Fluid 1 7.95 7.95 Brake Fluid
5511 Brake Parts C... 1 9.9 9.9 Brake Clean
5522 Engine Oil 16 4.48 71.68 15W40
5522 Oil Filter 1 24.86 24.86 FL2016
5522 Fuel Filter 1 109.58 109.58 PF7852
5522 Air Filter 1 69.95 69.95 PA4171
5522 Fuel Additive 1 14.33 14.33 Stanadyne
5522 Silicone Sealant 1 8.95 8.95 Silicone
8811 Engine Oil 16 4.48 71.68 15W40
8811 Oil Filter 1 24.86 24.86 FL2016
8811 Antifreeze 1 23.8 23.8 FYA - Ford Yellow
8811 Fuel Treatment 1 28.73 28.73 Enertech
8811 Windshield Wa... 1 2.25 2.25 Windshield Fluid
8811 Coolant Additive 1 12.95 12.95 DCA60L
8811 Power Steerin... 1 8.26 8.26 PSF
9911 Engine Oil 16 4.48 71.68 15W40
9911 Oil Filter 1 24.86 24.86 FL2016
9911 Fuel Treatment 1 28.73 28.73 Enertech
9911 Windshield Wa... 1 2.25 2.25 Windshield
9911 Coolant Additive 1 12.95 12.95 DCA60L
1011 Magnetic Jamb... 1 29.83 29.83 19171
0000 10 Ply Tire 1 165 165 255/75r16lt
0000 Tire Disposal... 1 8 8 Tire Disposal
0000 Tire Waste Sa... 1 2.25 2.25 NYS Tire Fee
There is a bit of caution with this data, as it's there for illustrative purposes only to show you what I mean, and the type of data that I'm talking about.
You can use subqueries to get the totals from other tables.
To be able to add the two subtotals together, use another subquery:
SELECT *, labor + parts AS total
FROM (SELECT unit AS vehicle,
date,
po,
invoice,
(SELECT TOTAL(extended)
FROM vehicle_service_labor
WHERE invoice = vehicle_service_invoice.invoice
) AS labor,
(SELECT TOTAL(extended)
FROM vehicle_service_parts
WHERE invoice = vehicle_service_invoice.invoice
) AS parts
FROM vehicle_service_invoice)
ORDER BY date,
invoice