Extending ggplot2 properly? - r

Question

Recently a few neat uses of ggplot2 have come up, and either partial or full solutions have been posted:
ggheat
Curly braces
position_dynamic
ggheat is notable because it rather breaks the ggplot metaphor by just plotting rather than returning an object.
The curly brace solutions are notable because none really fits in the ggplot2 high-level concept (e.g. you should be specifying a range of points you want to breaks, and then somewhere else be able to specify the geom of how you want that range displayed--brace, box, purple cow, etc.).
The ggplot2 book (which I will order soon and have read the 2 online chapters) seems to be about using the grammar and functions rather than writing new ones or extensively extending existing ones.
I would like to learn to add a specific feature or develop a new geom, and do it properly. ggplot2 may not be intended as a general graphics package in the same way that grid or base graphics are, but there are a great many graphs which are only a step or two extension from an existing ggplot2 geom. When these situations come up, I can typically put together enough objects to do something once, but what if I need the same plot a few dozen times? What if other people like it and want to use it--now they have to kludge through the same process each time they want that graph. It seems to me that the proper solution is to add in a stat_heatplot and geom_heatplot, or to add a geom_Tuftebox for Tufte box plots, etc. Yet I've never seen an example of actually extending ggplot2; just examples of how to use it.
What resources exist to dig deeper into ggplot2 and start extending it? I'm particularly interested in a high-level way to specify a range on an axis as described above, but general knowledge about what makes ggplot2 tick is welcome as well.
Absent a coherent guide (which rarely exists for sufficiently advanced tinkering and therefore may not exist here), how would one go about learning about the internals? Inspecting source is obviously one way, but what functions to start with, etc.

Answer 1

ggplot2 is gradually becoming more and more extensible. The development version, https://github.com/hadley/ggplot2/tree/develop, uses roxygen2 (instead of two separate homegrown systems), and has begun the switch from proto to simpler S3 classes (currently complete for coords and scales). These two changes should hopefully make the source code easier to understand, and hence easier for others to extend (backup by the fact that pull request for ggplot2 are increasing).
Another big improvement that will be included in the next version is Kohske Takahashi's improvements to the guide system (https://github.com/kohske/ggplot2/tree/feature/new-guides-with-gtable). As well as improving the default guides (e.g. with elegant continuous colour bars), his changes also make it easier to override the defaults with your own custom legends and axes. This would make it possible to draw the curly braces in the axes, where they probably belong.
The next big round of changes (which I probably won't be able to tackle until summer 2012) will include a rewrite of geoms, stats and position adjustments, along the lines of the sketch in the layers package (https://github.com/hadley/layers). This should make geoms, stats and position adjustments much easier to write, and will hopefully foster more community contributions, such as a geom_tufteboxplot.

Answer 2

I am not certain that I agree with your analysis. I'll explain why, and will then point you to some resources for writing your own geoms.
ggheat
As far as I can tell, ggheat returns an object of class ggplot. Thus it is a convenient wrapper around ggplot, customised for a specific use case. Although qplot is far more generic, it does in principle the same thing: It is a wrapper around ggplot that makes some informed guesses about the data and chooses sensible defaults. Hadley calls this plot functions and it is described briefly on page 181 of the ggplot2 book.
curly braces
The curly brace solution does exactly what the ggplot philosophy says, i.e. separate data from presentation. In this case, the data is generated by a little custom function and is stored in a data.frame. It is then displayed using a geom that makes sense, i.e. geom_line.
quo vadis?
You have noted (in the r chat room) that you would prefer to have a more generic approach to plotting the curly braces. Something along the following lines (and I paraphrase and extend at the same time):
Supply data in the form of a bounding box coordinates (i.e. x0, x1, y0 and y1)
Specify a "statistic", such as brace, box or whatever
Specify a geom, such as geom_custom_shape
This sounds like a nice generalisation and extension of the ideas behind the curly brace solution, and would clearly require writing a new geom. There is an official ggplot wiki, where you can find instructions for creating a new geom.

Answer 3

Why do you want to extend it? What is the motivation? As I see it ggplot2 is meant to be a high-level graphics package designed to produce nice figures from a particular data set. And do things right and make other things easy: like scales, legends etc. ggplot2 is not meant to be a general-purpose graphics tool-kit. Like lattice it has a particular paradigm in mind and you use it for that purpose.
grid is the underlying graphical toolkit you want to use to do general purpose, customised plotting. And IIRC, it is relatively easy to add grid grobs to lattice or ggplot2 plots/objects, for this sort of arbitrary notation/annotation etc.
What doesn't make too much sense is extending ggplot2 or lattice along the lines you are thinking. I don't see why the ggplot2 can't do heatplots as it is? Or am I missing something here?
What would be very useful would be if the data processing guts of ggplot2 or lattice were available for others to write actual plotting code on top of. Hadley has mentioned this somewhere before.
ggplot2, in particular, and lattice are quite difficult codes to get into to read/understand. ggplot2 uses the proto package for a version of OOP, which means you need to understand what that is doing as well as ggplot2 semantics. lattice is similar as there is a lot of computing on the language done there that, if you are not familiar with that sort of R programming, can by quite intimidating, daunting and impenetrable!
For grid, I suggest you look at Paul Murrell's R Graphics book, a second edition of which is with the publisher: http://www.stat.auckland.ac.nz/~paul/RG2e/
Edit: The point I was intending to get across was that the interfaces provided by packages like ggplot2 and lattice are necessarily high-level. Extending them is fine as long as they stick to the paradigm/philosophy in use. Heatplots can already be made by using existing geoms; part of the philosophy of the ggplot system is to separate the data from the display/presentation, and to use geoms in interesting ways to produce the desired display.
Wrapping base ggplot + geom calls into a more user friendly function is OK as long as i) it works like ggplot already does and returns an object, and ii) it doesn't have an interface that is too different from the way ggplot works. Developers are free to write whatever code they want, it just isn't helpful to the wider community to provide wrappers that move too far away from the original's workings. That leads to confusion on the part of the user and doesn't foster learning of ggplot2 itself.
The dynamic positioning idea is interesting; you could include these ideas in all plotting packages. You could bolt this into a geom, or alternatively as an external function that modified the input coordinates to produce a new data object that could be used by the relevant geom. That same function could be used for other plotting packages - it wouldn't need to be ggplot-specific.