I am trying implement hierarchical clustering in R : hclust() ; this requires a distance matrix created by dist() but my dataset has around a million rows, and even EC2 instances run out of RAM. Is there a workaround?
One possible solution for this is to sample your data, cluster the smaller sample, then treat the clustered sample as training data for k Nearest Neighbors and "classify" the rest of the data. Here is a quick example with 1.1M rows. I use a sample of 5000 points. The original data is not well-separated, but with only 1/220 of the data, the sample is separated. Since your question referred to hclust, I used that. But you could use other clustering algorithms like dbscan or mean shift.
## Generate data
set.seed(2017)
x = c(rnorm(250000, 0,0.9), rnorm(350000, 4,1), rnorm(500000, -5,1.1))
y = c(rnorm(250000, 0,0.9), rnorm(350000, 5.5,1), rnorm(500000, 5,1.1))
XY = data.frame(x,y)
Sample5K = sample(length(x), 5000) ## Downsample
## Cluster the sample
DM5K = dist(XY[Sample5K,])
HC5K = hclust(DM5K, method="single")
Groups = cutree(HC5K, 8)
Groups[Groups>4] = 4
plot(XY[Sample5K,], pch=20, col=rainbow(4, alpha=c(0.2,0.2,0.2,1))[Groups])
Now just assign all other points to the nearest cluster.
Core = which(Groups<4)
library(class)
knnClust = knn(XY[Sample5K[Core], ], XY, Groups[Core])
plot(XY, pch=20, col=rainbow(3, alpha=0.1)[knnClust])
A few quick notes.
Because I created the data, I knew to choose three clusters. With a real problem, you would have to do the work of figuring out an appropriate number of clusters.
Sampling 1/220 could completely miss any small clusters. In the small sample, they would just look like noise.
Related
I am currently doing a K-means cluster analysis for some customer data at my company. I want to measure the performance of this cluster, I just don't know the library packages used to measure performance of it and I am also unsure if my clusters are grouped too close together.
The data feeding my cluster is a simple RFM (recency, frequency, & monetary value). I also included average order value per transaction by customer. I used the elbow method to determine the optimal number clusters to use. Data consists of 1400 customers and 4 metric values.
Attached is also an image of the cluster plot & R Code
drop = c('CUST_Business_NM')
#Cleaning & Scaling the Data
new_cluster_data = na.omit(data)
new_cluster_data = data[, !(names(data)%in%drop)]
new_cluster_data = scale(new_cluster_data)
glimpse(new_cluster_data)
#Elbow Method for Optimal Clusters
k.max <- 15
data <- new_cluster_data
wss <- sapply(1:k.max,
function(k){kmeans(data, k, nstart=50,iter.max = 15 )$tot.withinss})
#Plot out the Elbow
wss
plot(1:k.max, wss,
type="b", pch = 19, frame = FALSE,
xlab="Number of clusters K",
ylab="Total within-clusters sum of squares")
#Create the Cluster
kmeans_test = kmeans(new_cluster_data, centers = 8, nstart = 1000)
View(kmeans_test$cluster)
#Visualize the Cluster
fviz_cluster(kmeans_test, data = new_cluster_data, show.clust.cent = TRUE, geom = c("point", "text"))
You probably do not want to measure the performance of cluster but the performance of the cluster algorithm, in this case kmeans.
First, you need to be clear what cluster distance measure you want to use. The result of the cluster computation is a dissimilarity matrix, thus the choice of the distance measure is critical, you can play with euclidean, manhattan, any kind of correlation or other distance measure, e.g., like this:
library("factoextra")
dis_pearson <- get_dist(yourdataset, method = "pearson")
dis_pearson
fviz_dist(dis_pearson)
This will give you the distance matrix and visualize it.
The output of kmeans has several bits of information. The most important with regard to your question are:
totss: the total sum of squares
withinss: vector of within-cluster sum of squares
tot.withinss: total within-cluster sum of squares
betweenss: the between-cluster sum of squares
Thus, the goal is to optimize these by playing with distances and other methods to cluster the data. Using cluster package, you can simply extract these measures by mycluster <- kmeans(yourdataframe, centers = 2) and then calling mycluster.
Side comment: kmeans requires the number of clusters defined by the user (additional effort) and it is very sensitive to outliers.
My data has 30,000 rows and 140 columns and I am trying to cluster the data. I am doing a pca and then using about 12 pc to use in the cluster analysis. I took a random sample of 3000 observations and ran it and it took 44 minutes to run both the pca and the hierarchal clustering.
A co-worker did the same in SPSS and it took significantly less time? Any idea why?
Here is a simplified version of my code which works fine but is really slow on anything over 2000 observations. I included the USArrest dataset which is really small so it doesn't really represent my problem but shows what I'm trying to do. I'm hesitant to post a large dataset as that seems rude.
I'm not sure how to speed the clustering up. I know I can do random samples of the data and then use a predict function to assign clusters to the test data. But optimally I'd like to use all of the data in the clustering since the data is static and isn't ever going to change or be updated.
library(factoextra)
library(FactoMineR)
library(FactoInvestigate)
## Data
# mydata = My data has 32,000 rows with 139 variables.
# example data with small data set
data("USArrests")
mydata <- USArrests
## Initial PCA on mydata
res.pca <- PCA(mydata, ncp=4, scale.unit=TRUE, graph = TRUE)
Investigate(res.pca) # this report is very helpful! I determined to keep 12 PC and start with 3 clusters.
## Keep PCA dataset with only 2 PC
res.pca1 <- PCA(mydata, ncp=2, scale.unit=TRUE, graph = TRUE)
## Run a HC on the PC: Start with suggested number of PC and Clusters
res.hcpc <- HCPC(res.pca1, nb.clust=4, graph = FALSE)
## Dendrogram
fviz_dend(res.hcpc,
cex = 0.7,
palette = "jco",
rect = TRUE, rect_fill = TRUE,
rect_border = "jco",
labels_track_height = 0.8
)
## Cluster Viz
fviz_cluster(res.hcpc,
geom = "point",
elipse.type = "convex",
#repel = TRUE,
show.clust.cent = TRUE,
palette = "jco",
ggtheme = theme_minimal(),
main = "Factor map"
)
#### Cluster 1: Means of Variables
res.hcpc$desc.var$quanti$'1'
#### Cluster 2: Means of Variables
res.hcpc$desc.var$quanti$'2'
#### Cluster 3: Means of Variables
res.hcpc$desc.var$quanti$'3'
#### Cluster 4: Means of Variables
res.hcpc$desc.var$quanti$'4'
#### Number of Observations in each cluster
cluster_hd = res.hcpc$data.clust$clust
summary(cluster_hd)
Any idea why SPSS is so much faster?
Any idea how to speed this up? I know clustering is labor intensive but I'm not sure what the threshold is for efficiency and my data of 30,000 records and 140 variables.
Are some of the other clustering packages more efficient? Suggestions?
HCPC is a hierarchical clustering on the principal components using the Ward criterion. You can use k-means algorithm instead for the clustering part, which is way faster: Hierarchical clustering has a time complexity of O(n³) whereas k-means has a complexity of O(n) where n is the number of observations.
Since the criterions optimized by k-means and hierarchical clustering with Ward are the same (minimize the total within-cluster variance), you can use first k-means with a high number of clusters (say 300 for instance) and then run hierarchical clustering on the centers of the clusters if you need to keep the hierachical aspect.
I would like to resample data with weighted bootstrap for constructing random forest.
The situation is like that.
I have the data which consist of normal subjects(N=20000) and patients(N=500).
I made new data set with normal subjects (N=2000) and patients (n=500) because I conducted a certain experiment with subjects (N=2500).
As you can see, normal subjects extracted 1/10 of original data and patients extracted all of them.
Therefore, I should give a weight to normal subjects to perform machine learning algorithm.
Please let me know how I can bootstrap with weight in R.
Thank you.
It sounds like you really need to stratified resampling rather than weighted resampling.
Your data are structured into two different groups of different sizes, and you would like to preserve that structure in your bootstrap. You didn't say what function you were applying to these data, so lets use something simple like the mean.
Generate some fake data, and take the (observed) means:
controls <- rnorm(2000, mean = 10)
patients <- rnorm(500, mean = 9.7)
mean(controls)
mean(patients)
Tell R we want to perform 200 bootstraps, and set up two empty vectors to store means for each bootstrap sample:
nbootraps <- 200
boot_controls <- numeric(nbootraps)
boot_patients <- numeric(nbootraps)
Using a loop we can draw resamples of the same size as you have in the original sample, and calculate the means for each.
for(i in 1:nbootraps){
# draw bootstrap sample
new_controls <- controls[sample(1:2000, replace = TRUE)]
new_patients <- patients[sample(1:500, replace = TRUE)]
# send the mean of each bootstrap sample to boot_ vectors
boot_controls[i] <- mean(new_controls)
boot_patients[i] <- mean(new_patients)
}
Finally, plot the bootstrap distributions for group means:
p1 <- hist(boot_controls)
p2 <- hist(boot_patients)
plot(p1, col=rgb(0,0,1,1/4), xlim = c(9.5,10.5), main="")
plot(p2, col=rgb(1,0,0,1/4), add=T)
I am doing cluster of some data in R Studio. I am having a problem with results of K-means Cluster Analysis and plotting Hierarchical Clustering. So when I use function kmeans, I get 4 groups with 10, 20, 30 and 6 observations. Nevertheless, when I plot the dendogram, I get 4 groups but with different numbers of observations: 23, 26, 10 and 7.
Have you ever found a problem like this?
Here you are my code:
mydata<-scale(mydata0)
# K-Means Cluster Analysis
fit <- kmeans(mydata, 4) # 4 cluster solution
# get cluster means
aggregate(mydata,by=list(fit$cluster),FUN=mean)
# append cluster assignment
mydatafinal <- data.frame(mydata, fit$cluster)
fit$size
[1] 10 20 30 6
# Ward Hierarchical Clustering
d <- dist(mydata, method = "euclidean") # distance matrix
fit2 <- hclust(d, method="ward.D2")
plot(fit2,cex=0.4) # display dendogram
groups <- cutree(fit2, k=4) # cut tree into 4 clusters
# draw dendogram with red borders around the 4 clusters
rect.hclust(fit2, k=4, border="red")
Results of k-means and hierarchical clustering do not need to be the same in every scenario.
Just to give an example, everytime you run k-means the initial choice of the centroids is different and so results are different.
This is not surprising. K-means clustering is initialised at random and can give distinct answers. Typically one tends to do several runs and then aggregate the results to check which are the 'core' clusters.
Hierarchical clustering is, in contrast, purely deterministic as there is no randomness involved. But like K-means, it is a heuristic: a set of rules is followed to create clusters with no regard to any underlying objective function (for example the intra- and inter- cluster variance vs overall variance). The way existing clusters are aggregated to individual observations is crucial in determining the size of the formed clusters (the "ward.D2" parameter you pass as method in the hclust command).
Having a properly defined objective function to optimise should give you a unique answer (or set thereof) but the problem is NP-hard, because of the sheer size (as a function of the number of observations) of the partitioning involved. This is why only heuristics exist and also why any clustering procedure should not be seen as a tool giving definitive answers but as an exploratory one.
One recommended method for getting a good cluster solution is to first use a hierarchical clustering method, choose a number of clusters, then extract the centroids, and then rerun it as a K-means clustering algorithm with the centres pre-specified. A toy example:
library(cluster)
data(animals)
ag.a <- agnes(agriculture, method = "ward")
ag.2 <- cutree(ag.a, k = 2)
This would give me two clusters. How can I extract the cluster centres in a format that I can then put into the kmeans() algorithm and reapply it to the same data?
You can use the clustering to assign cluster membership and then calculate the center for all the observations in a cluster. The kmeans function allows you to specify initial centers via the centers= parameter if you pass in a matrix. You can do that with
library(cluster)
data(animals)
ag.a <- agnes(agriculture, method = "ward")
ag.2 <- cutree(ag.a, k = 2)
# calculate means per group
cent<-aggregate(cbind(x,y)~ag.2, agriculture, mean)
# pass as initial centers
kmeans(agriculture, cent[,-1])