I was trying to plot a network with 200 thousands nodes and 8 millions edges. The code I used was here:
library(igraph)
file.choose()
xlist<-read.graph("/Users/quyao/Desktop/redstar_relation.txt", format="ncol", directed=TRUE)
xlist
png('my_png.png', width = 1600, height =900)
plot(xlist)
dev.off()
I got this:
It's hard to study the topology with this kind of picture. Do you have any suggestions?
As there are too many nodes and edges, change the layout through the parameter 'layout' may not be so helpful.
Constructing this picture took about one and half hours using my code, how could i make it faster?
Many thanks.
Consider using another software for visualization. I generally use Gephi for graphs with a magnitude up to 150 000 nodes.
However it really depends on how much RAM your machine has (mine has 8 GB). There is also the possibility to expand RAM allocation to Gephi by modifying their configuration file (it is not allocated automatically like in RStudio.
Related
I am trying to manually identify/correct trees using LiDAR data (1.7 GB object) and a tree tops object via the locate_trees function. Part of the problem is:
Rgl is rendering very slow even though the 4 GB Nvidia 3050 should be able to handle it.
The tree tops (red 3D dots) are not even showing in the rgl window. When I close the rgl window, the tree tops start popping up (red dots appear and disappear resulting in a blank white window) in a new rgl window. And if I close that window, a new tree top window opens up so I stop the process to prevent this from happening.
Does rgl automatically use the GPU or does it default to the integrated graphics on the motherboard? Is there a way to fasten up the rendering?
My other system specs are Corei9 (14 threads) and 64 GB RAM. Moreover, I am using R 4.2.1.
Code:
library(lidR)
# Import LiDAR data
LiDAR_File = readLAS("path/file_name.las")
# Find tree tops
TTops = find_trees(LiDAR_File , lmf(ws = 15, hmin = 5))
# Manually correct tree identification
TTops_Manual = locate_trees(LiDAR_File , manual(TTops)) # This is where rgl rendering becomes too slow if there are too many points involved.
There were two problems here. First, the lidR::manual() function which is used to select trees has a loop where one sphere is drawn for each tree. By default rgl will redraw the whole scene after each change; this should be suppressed. The patch in https://github.com/r-lidar/lidR/pull/611 fixes this. You can install a version with this fix as
remotes::install_github("r-lidar/lidR")
Second, rgl was somewhat inefficient in drawing the initial point cloud of data, duplicating the data unnecessarily. When you have tens of millions of points, this can exhaust all R memory, and things slow to a crawl. The development version of rgl fixes this. It's available via
remotes::install_github("dmurdoch/rgl")
The LiDAR images are very big, so you might find you still have problems even with these changes. Getting more regular RAM will help R: you may need this if the time to the first display is too long. After the first display, almost all the work is done in the graphics system; if things are still too slow, you may need a faster graphics card (or more memory for it).
rgl has trouble displaying too many points. The plot function in lidR is convenient and allows to produce ready to publish illustrations but cannot replace a real point cloud viewer for big point clouds. I don't have GPU on my computer and I don't know if and how rgl can take advantage of GPU.
In the doc of the lidR function your are talking about you can see:
This is only suitable for small-sized plots
Network visualizations become common in science in practice. But as networks are increasing in size, common visualizations become less useful. There are simply too many nodes/vertices and links/edges. Often visualization efforts end up in producing "hairballs".
Some new approaches have been proposed to overcome this issue, e.g.:
Edge bundling:
http://vis.stanford.edu/papers/divided-edge-bundling or
https://gephi.org/tag/edge-bundling/
Hierarchial edge bundling:
http://graphics.cs.illinois.edu/sites/graphics.dev.engr.illinois.edu/files/edgebundles.pdf
Group Attributes Layout:
http://wiki.cytoscape.org/Cytoscape_3/UserManual
How to make grouped layout in igraph?
I am sure that there are many more approaches. Thus, my question is:
How to overcome the hairball issue, i.e. how to visualize large networks by using R?
Here is some code that simulates an exemplary network:
# Load packages
lapply(c("devtools", "sna", "intergraph", "igraph", "network"), install.packages)
library(devtools)
devtools::install_github(repo="ggally", username="ggobi")
lapply(c("sna", "intergraph", "GGally", "igraph", "network"),
require, character.only=T)
# Set up data
set.seed(123)
g <- barabasi.game(1000)
# Plot data
g.plot <- ggnet(g, mode = "fruchtermanreingold")
g.plot
This questions is related to
Visualizing Undirected Graph That's Too Large for GraphViz?. However, here I am searching not for general software recommendations but for concrete examples (using the data provided above) which techniques help to make a good visualization of a large network by using R (comparable to the examples in this thread: R: Scatterplot with too many points).
Another way to visualize very large networks is with BioFabric (www.BioFabric.org), which uses horizontal lines instead of points to represent the nodes. Edges are then shown using vertical line segments. A quick D3 demo of this technique is shown at: http://www.biofabric.org/gallery/pages/SuperQuickBioFabric.html.
BioFabric is a Java application, but a simple R version is available at: https://github.com/wjrl/RBioFabric.
Here is a snippet of R code:
# You need 'devtools':
install.packages("devtools")
library(devtools)
# you need igraph:
install.packages("igraph")
library(igraph)
# install and load 'RBioFabric' from GitHub
install_github('RBioFabric', username='wjrl')
library(RBioFabric)
#
# This is the example provided in the question:
#
set.seed(123)
bfGraph = barabasi.game(1000)
# This example has 1000 nodes, just like the provided example, but it
# adds 6 edges in each step, making for an interesting shape; play
# around with different values.
# bfGraph = barabasi.game(1000, m=6, directed=FALSE)
# Plot it up! For best results, make the PDF in the same
# aspect ratio as the network, though a little extra height
# covers the top labels. Given the size of the network,
# a PDF width of 100 gives us good resolution.
height <- vcount(bfGraph)
width <- ecount(bfGraph)
aspect <- height / width;
plotWidth <- 100.0
plotHeight <- plotWidth * (aspect * 1.2)
pdf("myBioFabricOutput.pdf", width=plotWidth, height=plotHeight)
bioFabric(bfGraph)
dev.off()
Here is a shot of the BioFabric version of the data provided by the questioner, though networks created with values of m > 1 are more interesting. The inset detail shows a close-up of the upper left corner of the network; node BF4 is the highest-degree node in the network, and the default layout is a breadth-first search of the network (ignoring edge directions) starting from that node, with neighboring nodes traversed in order of decreasing node degree. Note that we can immediately see that, for example, about 60% of node BF4's neighbors are degree 1. We can also see from the strict 45-degree lower edge that this 1000-node network has 999 edges, and is therefore a tree.
Full disclosure: BioFabric is a tool that I wrote.
That's an interesting question, I didn't know most of the tools you listed, so thanks. You can add HivePlot to the list. It's a deterministic method consisting in projecting nodes on a fixed number of axes (usually 2 or 3). Look a the linked page, there're many visual examples.
It works better if you have a categorical nodal attribute in your dataset, so that you can use it to select which axis a node goes to. For instance, when studying the social network of a university: students on one axis, teachers on another and administrative staff on the third. But of course, it can also work with a discretized numerical attribute (eg. young, middle-aged and older people on their respective axes).
Then you need another attribute, and it has to be numerical (or at least ordinal) this time. It is used to determine the position of a node on its axis. You can also use some topological measure, such as degree or transitivity (clustering coefficient).
(source: hiveplot.net)
The fact the method is deterministic is interesting, because it allows comparing different networks representing distinct (but comparable) systems. For example, you can compare two universities (provided you use the same attributes/measures to determine axes and position). It also allows describing the same network in various ways, by choosing different combinations of attributes/measures to generate the visualization. This is the recommanded way of visualizing a network, actually, thanks to a so-called hive panel.
Several softwares able of generating those hive plots are listed in the page I mentioned at the beginning of this post, including implementations in Java and R.
I've been dealing with this problem recently. As a result, I've come up with another solution. Collapse the graph by communities/clusters. This approach is similar to the third option outlined by the OP above. As a word of warning, this approach will work best with undirected graphs. For example:
library(igraph)
set.seed(123)
g <- barabasi.game(1000) %>%
as.undirected()
#Choose your favorite algorithm to find communities. The algorithm below is great for large networks but only works with undirected graphs
c_g <- fastgreedy.community(g)
#Collapse the graph by communities. This insight is due to this post http://stackoverflow.com/questions/35000554/collapsing-graph-by-clusters-in-igraph/35000823#35000823
res_g <- simplify(contract(g, membership(c_g)))
The result of this process is the below figure, where the vertices' names represent community membership.
plot(g, margin = -.5)
The above is clearly nicer than this hideous mess
plot(r_g, margin = -.5)
To link communities to original vertices you will need something akin to the following
mem <- data.frame(vertices = 1:vcount(g), memeber = as.numeric(membership(c_g)))
IMO this is a nice approach for two reasons. First, it can in theory deal with any size graph. The process of finding communities can be continuously repeated on collapsed graphs. Second, adopting a interactive approach would yield very readable results. For example, one can imagine the user being able to click on a vertex in the collapsed graph to expand that community revealing all of its original vertices.
I have looked around and found no good solution. My approach has been to remove nodes and play with edge transparency. It is more of a design solution rather than a technical one, but I've been able to plot gephi-like networks of up to 50,000 edges without much complications on my laptop.
with your example:
plot(simplify(g), vertex.size= 0.01,edge.arrow.size=0.001,vertex.label.cex = 0.75,vertex.label.color = "black" ,vertex.frame.color = adjustcolor("white", alpha.f = 0),vertex.color = adjustcolor("white", alpha.f = 0),edge.color=adjustcolor(1, alpha.f = 0.15),display.isolates=FALSE,vertex.label=ifelse(page_rank(g)$vector > 0.1 , "important nodes", NA))
Example of twitter mentions network with 30,000 edges:
Yet another interesting package is networkD3. There are a myriad of means of representing graphs within this library. In particular, I find the forceNetwork an interesting option. It is interactive and therefore allows you to really explore your network. It is great for EDA, but it maybe too "wiggly" for final work.
I tired this pacakge. It's very fast.
if (!require("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("netbiov")
https://www.bioconductor.org/packages/release/bioc/html/netbiov.html
Examples:
https://www.bioconductor.org/packages/release/bioc/vignettes/netbiov/inst/doc/netbiov-intro.pdf
I am trying to build graphs using tree-like data, where nodes typically split into >2 edges. I have tried various layouts, and I see that the layout.reingold.tilford parameter will generate tree-like graphs with non-bifurcating data. However the outputs are not particularly attractive. I would rather use something like the layout.lgl or layout.kamada.kawai since these produce more radial structures. I cannot see how to change the parameters in R such that these trees have no overlapping edges though. Is this possible?
I imported a simple data file in Pajek format, with 355 nodes and 354 edges. I'm currently printing it using:
plot.igraph(g,vertex.size=3,vertex.label=NA,layout=layout.lgl)
This gives me an output like this, which is nice, but still has overlapping edges. I have read that you can manually fix this using tkplot, or another program like cytoscape, however I have quite a few of these to build, and the size of them makes manual correction a hassle.
Many thanks.
Just want to add a comment but my rep is too low. The method that #bdemarest posted does not work on igraph version > 0.7. The newer version does not support the area parameter, so I cannot get the same effect. And getting the old version to build took me a while, so I though I'd share some insights. You can manually install igraph 0.7 from source if you download it from igraph nightly builds. On my machine (Mac OS 10.10), I encountered some problems building it, due to gfortran, so I found this link that solved the problem. Hope that helps anyone who wants to create similar graphs in R.
You may want to try layout.fruchterman.reingold(). It seems to do a good job keeping the edges from crossing. I've tested it with a 355 node version of the barabasi graph suggested by #Tamás.
library(igraph)
g = barabasi.game(355, directed=FALSE)
png("plot1.png", height=6, width=12, units="in", res=200)
par(mfrow=c(1, 2))
plot.igraph(g,vertex.size=3,vertex.label=NA,
layout=layout.fruchterman.reingold(g, niter=10000))
mtext("layout.fruchterman.reingold, area = vcount^2", side=1)
plot.igraph(g,vertex.size=3,vertex.label=NA,
layout=layout.fruchterman.reingold(g, niter=10000, area=30*vcount(g)^2))
mtext("layout.fruchterman.reingold, area = 30 * vcount^2", side=1)
dev.off()
layout.reingold.tilford has a parameter called circular. Setting this to TRUE will convert the final layout into a radial one by treating the X coordinate as the angle (after appropriate rescaling) and the Y coordinate as the radius. Ironically enough, this does not guarantee that there will be no edge crossings in the end, but it works nicely if your subtrees are not exceedingly wide compared to the rest of the graph:
> g <- barabasi.game(100, directed=F)
> layout <- layout.reingold.tilford(g, circular=T)
> plot(g, layout=layout)
I'm looking for an algorithm to automatically visualise a large DAG. It needs to scale well to hundreds or even thousands of nodes and connections (without turning unreadable). Connections should avoid crossing over each other where possible, and should especially avoid crossing over nodes that they aren't connected to.
Is there any standard algorithm I can adapt for this purpose?
You could check out the scalable force-directed placement algorithm. Graphviz implements this, so if you'd like to preview it before implementing, create a Graphviz file and run sfdp my_dag.gv (or fdp which might be easier to implement).
If that doesn't work for you, you might want something like Circos or Hive Plots. Hive Plots work really well for thousands of nodes for both directed and undirected graphs. The algorithm is described at a high level on the homepage, but there's an accompanying journal article too.
You can try Gephi a graph viz software.
You can feed it with different file type (.gexf, .gdf).
As this is a open source software, you can look inside spatialization algorithms.
url: http://gephi.org/
I have graph of friendship of one social network with about 1.5 million of nodes and 17 million edges - not all network, but that's enough for me.
Of course, it's undirected and unweighted.
What is the way to render it really beautiful? (e.g. to make a poster ;-)
You should try with Gephi. This is a very large graph but I think you can handle it with a correct computer.
I advice you to try OpenOrd (here) or ForceAtlas in Parallel version (here)
You can render as SVG or PDF to export you graph for "graphic customization"
You can have a look at Protovis ( http://mbostock.github.com/protovis/) especially ForceDirectedLayout
http://mbostock.github.com/protovis/ex/force.html