I am trying to display data by Species that has different values depending on group Letter. The best way I have found to display my data is by putting my categorical data on the y-axis and displaying the Total_Observed on the x-axis. Lemon allows me to have different y-axis labels. Unfortunately, the graph sorts by my y-axis labels instead of using my data as is, which is sorted by most abundant species to least abundant. Any suggestions?
Using libraries: dplyr, ggplot2, lemon
My data:
|Letter |Species | Total_Observed|
|:------|:------------------------|--------------:|
|A |Yellowtail snapper | 155|
|A |Sharksucker | 119|
|A |Tomtate | 116|
|A |Mutton snapper | 104|
|A |Little tunny | 96|
|B |Vermilion snapper | 1655|
|B |Red snapper | 1168|
|B |Gray triggerfish | 689|
|B |Tomtate | 477|
|B |Red porgy | 253|
|C |Red snapper | 391|
|C |Vermilion snapper | 114|
|C |Lane snapper | 95|
|C |Atlantic sharpnose shark | 86|
|C |Tomtate | 73|
|D |Lane snapper | 627|
|D |Red grouper | 476|
|D |White grunt | 335|
|D |Gray snapper | 102|
|D |Sand perch | 50|
|E |White grunt | 515|
|E |Red grouper | 426|
|E |Red snapper | 150|
|E |Black sea bass | 142|
|E |Lane snapper | 88|
|E |Gag | 88|
|F |Yellowtail snapper | 385|
|F |White grunt | 105|
|F |Gray snapper | 88|
|F |Mutton snapper | 82|
|F |Lane snapper | 59|
Then I run the code for my ggplot/lemon
ggplot(test,aes(y=Species,x=Total_Observed))+geom_histogram(stat='identity')+facet_wrap(.~test$Letter,scales='free_y')
And my graphs print like this:
Answered by Johan Rosa's shared blog (https://juliasilge.com/blog/reorder-within/): The solution is to use the library(tidytext). With the functions reorder_within and scale_x_reordered.
The corrected code:
test %>% mutate(Species=reorder_within(Species,Total_Observed,Letter)) %>% ggplot(aes(Species,Total_Observed))+geom_histogram(stat='identity')+facet_wrap(~Letter,scales='free_y')+coord_flip()+scale_x_reordered()
Will now generate the graphs ordered correctly
I have a knitr_kable object. I want to put parenthesis on every value in every consecutive (2,4,6....) row. How I can do this.
out %>% knitr::kable(digits = 2, "pipe")
| | 1| 2| 3| 4| 5| 6| 7| 8| 9| 10| 10-1|
|:-------------|-----:|-----:|-----:|----:|-----:|-----:|-----:|-----:|-----:|-----:|-----:|
|Excess Return | -0.20| 1.20| 0.97| 1.04| 0.77| 0.82| 0.79| 0.76| 0.76| 0.59| 0.79|
|NW t_Stat | -0.28| 2.04| 3.02| 3.17| 2.56| 3.16| 3.09| 3.45| 3.86| 3.48| 1.13|
|FF5 Alpha | -0.97| 0.43| -0.05| 0.02| -0.26| -0.17| -0.18| -0.19| -0.11| -0.11| 0.86|
|NW t_stats | -1.41| 1.03| -0.26| 0.14| -2.75| -2.10| -2.32| -2.85| -1.77| -5.01| 1.26|
|MktRF beta | 0.92| 1.00| 1.07| 1.08| 1.13| 1.11| 1.10| 1.08| 1.06| 0.99| 0.07|
|NW t_Stat | 1.34| 2.40| 5.44| 6.49| 11.98| 13.51| 14.58| 16.13| 16.43| 46.90| 0.10|
|SMB5 beta | 1.35| 1.34| 1.46| 1.26| 1.14| 1.03| 0.84| 0.69| 0.36| -0.16| -1.51|
|NW t_Stat | 1.97| 3.22| 7.41| 7.56| 12.11| 12.44| 11.05| 10.28| 5.61| -7.37| -2.20|
|HML beta | 0.52| 1.27| 0.48| 0.46| 0.55| 0.51| 0.36| 0.29| 0.22| 0.10| -0.42|
|NW t_Stat | 0.75| 3.05| 2.46| 2.75| 5.82| 6.23| 4.70| 4.38| 3.47| 4.53| -0.62|
|CMA beta | 0.43| -0.66| 0.15| 0.28| 0.11| -0.02| 0.10| 0.11| 0.13| 0.08| -0.34|
|NW t_stat | 0.62| -1.59| 0.77| 1.65| 1.14| -0.28| 1.38| 1.61| 1.96| 3.92| -0.50|
|RMW beta | -0.68| -0.25| 0.11| 0.08| 0.15| 0.24| 0.32| 0.35| 0.31| 0.17| 0.84|
|NW t_stat | -0.98| -0.60| 0.54| 0.47| 1.56| 2.96| 4.22| 5.18| 4.75| 7.87| 1.23|
|Adj_r_square | 0.14| 0.31| 0.64| 0.73| 0.87| 0.93| 0.94| 0.94| 0.93| 0.98| 0.09|
I am R programmer moving into the pyspark world and have gotten a lot of the basic tricks down but something I am still struggling about is things I would do applys or basic for loops for.
In this case I am trying to calculate the "anti-groupby" for an ID. Basically the idea is to look at a population for that ID and then the population for not this ID and have both those values on the same row. The getting the population for that ID is easy using a groupby and then joining it to a dataset with new_id as the only column.
This is how I would do it in R:
anti_group <- function(id){
tr <- sum(subset(df1, new_id!=id)$total_1)
to <- sum(subset(df1, new_id!=id)$total_2)
54 * tr / to
}
test$other.RP54 <- sapply(test$new_id, anti_group )
How would I do it in pyspark?
Thanks!
Edit:
#df.show()
#sample data
+---+-----+
| id|value|
+---+-----+
| 1| 40|
| 1| 30|
| 2| 10|
| 2| 90|
| 3| 20|
| 3| 10|
| 4| 2|
| 4| 5|
+---+-----+
Then some function that creates a final dataframe that looks like this:
+---+-------------+------------------+
| id|grouped_total|anti_grouped_total|
+---+-------------+------------------+
| 1| 70| 137|
| 2| 100| 107|
| 3| 30| 177|
| 4| 7| 200|
+---+-------------+------------------+
I think you can do that in two steps: first you sum by id then you take the total and substract by the value for this id.
My idea is a little bit like a group_by(id) %>% summarise(x = sum(x)) %>% mutate(y = sum(x) - x) in dplyr
The solution I propose is based on Window function. It is untested:
Let's first create the data
import pyspark.sql.functions as psf
import pyspark.sql.window as psw
df = spark.createDataFrame([(1,40),(1,30),(2,10),(2,90),(3,20),(3,10),(4,2),(4,5)], ['id','value'])
df.show(2)
+---+-----+
| id|value|
+---+-----+
| 1| 40|
| 1| 30|
+---+-----+
only showing top 2 rows
and then apply that approach:
w = psw.Window.orderBy()
df_id = df.groupBy("id").agg(psf.sum("value").alias("grouped_total"))
df_id = (df_id
.withColumn("anti_grouped_total",psf.sum("grouped_total").over(w))
.withColumn('anti_grouped_total', psf.col('anti_grouped_total') - psf.col('grouped_total'))
)
df_id.show(2)
+---+-------------+------------------+
| id|grouped_total|anti_grouped_total|
+---+-------------+------------------+
| 3| 30| 177|
| 1| 70| 137|
+---+-------------+------------------+
only showing top 2 rows
So there's no in-built function that would replicate that groupBy function, but you could easily do it by creating a new column using case(when/otherwise clause) to create your group and anti-group, and then groupBy on that new column.
#df.show()
#sample data
+---+-----+
| id|value|
+---+-----+
| 1| 40|
| 1| 30|
| 2| 10|
| 2| 90|
| 3| 20|
| 3| 10|
| 4| 2|
| 4| 5|
+---+-----+
from pyspark.sql import functions as F
df.withColumn("anti_id_1", F.when(F.col("id")==1, F.lit('1')).otherwise(F.lit('Not_1')))\
.groupBy("anti_id_1").agg(F.sum("value").alias("sum")).show()
+---------+---+
|anti_id_1|sum|
+---------+---+
| 1| 70|
| Not_1|137|
+---------+---+
UPDATE:
from pyspark.sql.window import Window
from pyspark.sql import functions as F
w1=Window().partitionBy("id")
w=Window().partitionBy()
df.withColumn("grouped_total",F.sum("value").over(w1))\
.withColumn("anti_grouped_total", (F.sum("value").over(w))-F.col("grouped_total"))\
.groupBy("id").agg(F.first("grouped_total").alias("grouped_total"),\
F.first("anti_grouped_total").alias("anti_grouped_total"))\
.drop("value").orderBy("id").show()
+---+-------------+------------------+
| id|grouped_total|anti_grouped_total|
+---+-------------+------------------+
| 1| 70| 137|
| 2| 100| 107|
| 3| 30| 177|
| 4| 7| 200|
+---+-------------+------------------+
Less verbose/concise way to achieve the same output:
from pyspark.sql import functions as F
from pyspark.sql.window import Window
w = Window().partitionBy()
df.groupBy("id").agg(F.sum("value").alias("grouped_total"))\
.withColumn("anti_grouped_total",F.sum("grouped_total").over(w)-F.col("grouped_total")).orderBy("id"),show()
For 2 value columns:
df.show()
+---+------+------+
| id|value1|value2|
+---+------+------+
| 1| 40| 50|
| 1| 30| 70|
| 2| 10| 91|
| 2| 90| 21|
| 3| 20| 42|
| 3| 10| 4|
| 4| 2| 23|
| 4| 5| 12|
+---+------+------+
from pyspark.sql.window import Window
from pyspark.sql import functions as F
w = Window().partitionBy()
df.groupBy("id").agg(F.sum("value1").alias("grouped_total_1"),F.sum("value2").alias("grouped_total_2"))\
.withColumn("anti_grouped_total_1",F.sum("grouped_total_1").over(w)-F.col("grouped_total_1"))\
.withColumn("anti_grouped_total_2",F.sum("grouped_total_2").over(w)-F.col("grouped_total_2")).orderBy("id").show()
+---+---------------+---------------+--------------------+--------------------+
| id|grouped_total_1|grouped_total_2|anti_grouped_total_1|anti_grouped_total_2|
+---+---------------+---------------+--------------------+--------------------+
| 1| 70| 120| 137| 193|
| 2| 100| 112| 107| 201|
| 3| 30| 46| 177| 267|
| 4| 7| 35| 200| 278|
+---+---------------+---------------+--------------------+--------------------+
csv-data
I wanted to delete the empty column from the dataframe
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Test_Parquet").master("local[*]").getOrCreate()
names = spark.read.csv("name.csv", header="true", inferSchema="true")
names.show()
This is the dataframe created from name.csv file
+-------+---+---+---+-----+----+
| Name| 1|Age| 3|Class| _c5|
+-------+---+---+---+-----+----+
|Diwakar| | 25| | 12|null|
|Prabhat| | 27| | 15|null|
| Zyan| | 30| | 17|null|
| Jack| | 35| | 21|null|
+-------+---+---+---+-----+----+
Spark by default gave name to the empty column as 1, 3,_c5 can we stop spark giving default names to the columns.
I wanted to have a dataframe like given below:
+-------+---+---+---+-----+----+
| Name| |Age| |Class| |
+-------+---+---+---+-----+----+
|Diwakar| | 25| | 12|null|
|Prabhat| | 27| | 15|null|
| Zyan| | 30| | 17|null|
| Jack| | 35| | 21|null|
+-------+---+---+---+-----+----+
and i wanted to remove the empty column in one go like:
temp = list(set(names.columns))
temp.remove(" ")
names = names.select(temp)
names.show
+-------+---+-----+
| Name|Age|Class|
+-------+---+-----+
|Diwakar| 25| 12|
|Prabhat| 27| 15|
| Zyan| 30| 17|
| Jack| 35| 21|
+-------+---+-----+
I need to know if my residuals are correlated or not. I didn't find a way to do it using Spark-Scala on Databricks.
And i conclude that i should export my project to R to use acf function.
Does someone know a trick to do it using Spark-Scala on Databricks ?
For those who need more information : I'm currently working on Sales Forecasting. I used a Regression Forest using different features. Then, I need to evaluate the quality of my forecast. To check this, i read on this paper that residuals were a good way to see if your forecasting model is good or bad. In any cases, you can still improve it but it's just to make my opinion on my forecast model and compared it to others models.
Currently, I have one dataframe like the one below. It's a part of the testing data/out-of-sample data. (I casted prediction and residuals to IntegerType, that's why at the 3rd row 40 - 17 = 22)
I am using Spark 2.1.1.
You can find correlation between columns using spark ml library function
Lets first import the classes.
import org.apache.spark.sql.functions.corr
import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.stat.Statistics
Now prepare the input DataFrame :
scala> val seqRow = Seq(
| ("2017-04-27",13,21),
| ("2017-04-26",7,16),
| ("2017-04-25",40,17),
| ("2017-04-24",17,17),
| ("2017-04-21",10,20),
| ("2017-04-20",9,19),
| ("2017-04-19",30,30),
| ("2017-04-18",18,25),
| ("2017-04-14",32,28),
| ("2017-04-13",39,18),
| ("2017-04-12",2,4),
| ("2017-04-11",8,24),
| ("2017-04-10",18,27),
| ("2017-04-07",6,17),
| ("2017-04-06",13,29),
| ("2017-04-05",10,17),
| ("2017-04-04",6,8),
| ("2017-04-03",20,32)
| )
seqRow: Seq[(String, Int, Int)] = List((2017-04-27,13,21), (2017-04-26,7,16), (2017-04-25,40,17), (2017-04-24,17,17), (2017-04-21,10,20), (2017-04-20,9,19), (2017-04-19,30,30), (2017-04-18,18,25), (2017-04-14,32,28), (2017-04-13,39,18), (2017-04-12,2,4), (2017-04-11,8,24), (2017-04-10,18,27), (2017-04-07,6,17), (2017-04-06,13,29), (2017-04-05,10,17), (2017-04-04,6,8), (2017-04-03,20,32))
scala> val rdd = sc.parallelize(seqRow)
rdd: org.apache.spark.rdd.RDD[(String, Int, Int)] = ParallelCollectionRDD[51] at parallelize at <console>:34
scala> val input_df = spark.createDataFrame(rdd).toDF("date","amount","prediction").withColumn("residuals",'amount - 'prediction)
input_df: org.apache.spark.sql.DataFrame = [date: string, amount: int ... 2 more fields]
scala> input_df.show(false)
+----------+------+----------+---------+
|date |amount|prediction|residuals|
+----------+------+----------+---------+
|2017-04-27|13 |21 |-8 |
|2017-04-26|7 |16 |-9 |
|2017-04-25|40 |17 |23 |
|2017-04-24|17 |17 |0 |
|2017-04-21|10 |20 |-10 |
|2017-04-20|9 |19 |-10 |
|2017-04-19|30 |30 |0 |
|2017-04-18|18 |25 |-7 |
|2017-04-14|32 |28 |4 |
|2017-04-13|39 |18 |21 |
|2017-04-12|2 |4 |-2 |
|2017-04-11|8 |24 |-16 |
|2017-04-10|18 |27 |-9 |
|2017-04-07|6 |17 |-11 |
|2017-04-06|13 |29 |-16 |
|2017-04-05|10 |17 |-7 |
|2017-04-04|6 |8 |-2 |
|2017-04-03|20 |32 |-12 |
+----------+------+----------+---------+
The values of residuals for row 2017-04-14 and 2017-04-13 don't match as i am subtracting amount - prediction for residuals
Now proceeding forward to calculate correlation between all the columns.
This method is used for calculating correlation if number of columns are more and you need correlation between each column to others.
First we drop the column whose correlation is not to be calculated
scala> val drop_date_df = input_df.drop('date)
drop_date_df: org.apache.spark.sql.DataFrame = [amount: int, prediction: int ... 1 more field]
scala> drop_date_df.show
+------+----------+---------+
|amount|prediction|residuals|
+------+----------+---------+
| 13| 21| -8|
| 7| 16| -9|
| 40| 17| 23|
| 17| 17| 0|
| 10| 20| -10|
| 9| 19| -10|
| 30| 30| 0|
| 18| 25| -7|
| 32| 28| 4|
| 39| 18| 21|
| 2| 4| -2|
| 8| 24| -16|
| 18| 27| -9|
| 6| 17| -11|
| 13| 29| -16|
| 10| 17| -7|
| 6| 8| -2|
| 20| 32| -12|
+------+----------+---------+
Since there are more than 2 column for correlation, we need to find correlation matrix.
For calculating correlation matrix we need RDD[Vector] as you can see in spark example for correlation.
scala> val dense_rdd = drop_date_df.rdd.map{row =>
| val first = row.getAs[Integer]("amount")
| val second = row.getAs[Integer]("prediction")
| val third = row.getAs[Integer]("residuals")
| Vectors.dense(first.toDouble,second.toDouble,third.toDouble)}
dense_rdd: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[62] at map at <console>:40
scala> val correlMatrix: Matrix = Statistics.corr(dense_rdd, "pearson")
correlMatrix: org.apache.spark.mllib.linalg.Matrix =
1.0 0.40467032516705076 0.782939330961529
0.40467032516705076 1.0 -0.2520531290688281
0.782939330961529 -0.2520531290688281 1.0
The order of column remains same but you loose out the column names.
You can find good resources about structure of correlation matrix.
Since you want to find the correlation of residuals with other two columns.
We can explore other options
Hive corr UDAF
scala> drop_date_df.createOrReplaceTempView("temp_table")
scala> val corr_query_df = spark.sql("select corr(amount,residuals) as amount_residuals_corr,corr(prediction,residuals) as prediction_residual_corr from temp_table")
corr_query_df: org.apache.spark.sql.DataFrame = [amount_residuals_corr: double, prediction_residual_corr: double]
scala> corr_query_df.show
+---------------------+------------------------+
|amount_residuals_corr|prediction_residual_corr|
+---------------------+------------------------+
| 0.7829393309615287| -0.252053129068828|
+---------------------+------------------------+
Spark corr function link
scala> val corr_df = drop_date_df.select(
| corr('amount,'residuals).as("amount_residuals_corr"),
| corr('prediction,'residuals).as("prediction_residual_corr"))
corr_df: org.apache.spark.sql.DataFrame = [amount_residuals_corr: double, prediction_residual_corr: double]
scala> corr_df.show
+---------------------+------------------------+
|amount_residuals_corr|prediction_residual_corr|
+---------------------+------------------------+
| 0.7829393309615287| -0.252053129068828|
+---------------------+------------------------+