TLDR
Does rigorous best practice recommend that a proactive R developer explicitly disambiguate all base functions — even the ubiquitously common functions like c() or cat() — within the .R files of their package, using the package::function() convention?
Context
Though a novice developer, I need to create a (proprietary) package in R. The text R Packages (by the authoritative authors Hadley Wickham and Jenny Bryan) has proven extremely helpful (if occasionally deprecated).
I am keen on following best practices from the start, so as to save myself time and effort down the road. As described in the text, the use of the :: operator can prevent current and
future conflicts, by disambiguating functions whose names are overloaded. To wit, the authors are careful to introduce each function with the package::function() convention, and they recommend its general use within the .R files of one's package.
However, their code examples often call functions that hail from the base package yet are unaccompanied by base::. Many base functions, like the ubiquitous c() or cat(), are used by R programmers in their sleep and (I imagine) are unlikely to ever be overloaded by a presumptuous developer. Nonetheless, it is confusing to see (for example) the juxtaposition of base::with() against (the base function) print(), all within a few lines of text.
...(These functions are inspired by how base::with() works.)
f <- function(x, sig_digits) {
# imagine lots of code here
withr::with_options(
list(digits = sig_digits),
print(x)
)
# ... and a lot more code here
}
I understand that the purpose of base::with() is to unambiguously introduce the with() function to the reader. However, the absence of base:: (within the code itself) seems to stick out like a sore thumb, when the package is explicitly named for any function called from any other package. Given my inexperience, I don't feel comfortable assuming the authors' intent.
Question
Are the names of base functions sufficiently unique that using this convention — of calling base::function() for every function() within the base package — would not be worth it? That the risk of overloading the functions (at some point in the future) is far outweighed by the inconvenience (and sheer ugliness) of
my_vector <- base::c(1, 2, 3)
throughout one's .R files? If not, is there an established convention that would balance unambiguity with elegance?
As always, I am grateful for any help, especially on this, my first post to Stack Overflow.
In Python we have chance to import a certain function from a library with a command "import function from library as smth. Do we have something similar in R?
I know that we can call the function like "library::function()", my question mostly refers to the "as" part.
It is not common and not necessary to do this in R. The assignment operator <- can be used to give a new name to an existing function. For example, one could define a function that does exactly the same as lubridate's, year() function with:
asYear <- lubridate::year
One could argue that, by doing so, the year() function has been "imported" from the lubridate package and that it is now called asYear(). In fact, the new function does just the same (which is no surprise, simply because it is the same):
asYear(Sys.Date())
#[1] 2016
So it is possible to construct an analogy to "from package import as", but it is not a good idea to do this. Here are a few reasons I can think of:
Debugging a code where library functions have been renamed will be
much more difficult.
The documentation is not available for the renamed function. In this example, ?asYear won't work, in contrast to ?lubridate::year or library(lubridate); help(year).
The function is not only renamed but it is copied, which clutters the environment and is inefficient in terms of memory usage.
The maintenance of the code becomes unnecessarily difficult. If another programmer (or the original programmer a few years later) looks at a code containing such a redefinition of a function, it will be harder for her or him to understand what this function is doing.
There are probably more reasons, but I hope that this is sufficient to discourage the use of such a construction. Different programming languages have different peculiarities, and as a programmer it is necessary to adapt to them. What is common in Python can be awkward in R, and vice versa.
A simple and commonly used way to handle such a standard situation in R is to use library() to load the entire namespace of the package containing the requested function:
library (lubridate)
year(Sys.Date())
However, one should be aware of possible namespace clashes, especially if many libraries are loaded simultaneously. Different functions could be defined with the same name in different packages. A well-known example thereof are the contrasting implementations of the lag() function in the dplyr and stats package.
In such cases one can use the double colon operator :: to resolve the namespace that should be addressed. This would be similar to the use of "from" in the case of "import", but such a specification would be needed each time the function is called.
lubridate::year(Sys.Date())
#[1] 2016
When creating an R package, there are at least two alternatives for referencing functions in imported packages.
Either,
Explicitly name the function using the double colon operator whenever you call it, package::function.
Add importFrom(package, function) to the NAMESPACE file, either directly or via an #' #importFrom package function roxygen tag.
What are the advantages and disadvantages of each method?
Are there any technical differences in what each syntax achieves?
Arguments in favour of using package::function
It makes it completely clear where the function has come from.
Arguments in favour of using #importFrom package function
It involves less typing, particularly when a function is used many times by your package.
Since it involves looking up the package and a call to the :: function, package::function has a small runtime performance penalty. See https://stackoverflow.com/a/7283511/134830.
On balance, what's the verdict?
Both methods do the job and arguments either way aren't overwhelming, so don't lose sleep over this. Just pick one method and stick to it.
The policy that has been adopted at my place of work is that for a few commonly used packages, #importFrom roxygen tags should be used. For example, developers are expected to know that ddply comes from plyr, or functions beginning str_ come from stringr. In this case, the explicit parentage of the function isn't as useful to know. For functions outside this core list, (or if there is any ambiguity) :: should be used to make it clear where it came from.
I realize there are generic functions like plot, predict for a list of packages. I am wondering how could I get the R script of these generic functions for a specific package, like the lme4::predict. I tried lme4::predict, but it comes with error:
> lme4::predict
Error: 'predict' is not an exported object from 'namespace:lme4'
Since you state that my suggestion above was helpful I will tell you my process. I used my own co-authored package called pacman. This package was developed because we had a hard time remembering all the obscurely named functions to get information on and work with add on packages.
I used this to figure out what you wanted:
library(pacman)
p_funs(lme4, all=TRUE)
I set all = TRUE as predict is a method for a specific class (like print, summary and plot). Generally, these methods are not exported so p_funs won't return them unless you set all = TRUE. Then I scrolled down to the p section and found only a single predict method: predict.merMod
Next I realized it wasn't exported so :: won't show me the stuff and extra colon power is needed, hence: lme4:::predict.merMod
As pointed out by David and rawr above, some functions can be suborn little snippets (methods etc.), thus methods and getAnywhere are helpful.
Here is an example of that:
library(tm)
dissimilarity #The good stuff is hid
methods(dissimilarity) #I want the good stuff
getAnywhere("dissimilarity.DocumentTermMatrix")
Small end note
And of course you don't need pacman to look at functions for packages, it's what I use and helpful but it just wraps base R things. Use THE SOURCE to figure out exactly what.
Much has been written here about developing a workflow in R for statistical projects. The most popular workflow seems to be Josh Reich's LCFD model. With a main.R containing code:
source('load.R')
source('clean.R')
source('func.R')
source('do.R')
so that a single source('main.R') runs the entire project.
Q: Is there a reason to prefer this workflow to one in which the line-by-line interpretive work done in load.R, clean.R, and do.R is replaced by functions which are called by main.R?
I can't find the link now, but I had read somewhere on SO that when programming in R one must get over their desire to write everything in terms of function calls---that R was MEANT to be written is this line-by-line interpretive form.
Q: Really? Why?
I've been frustrated with the LCFD approach and am going to probably write everything in terms of function calls. But before doing this, I'd like to hear from the good folks of SO as to whether this is a good idea or not.
EDIT: The project I'm working on right now is to (1) read in a set of financial data, (2) clean it (quite involved), (3) Estimate some quantity associated with the data using my estimator (4) Estimate that same quantity using traditional estimators (5) Report results. My programs should be written in such a way that it's a cinch to do the work (1) for different empirical data sets, (2) for simulation data, or (3) using different estimators. ALSO, it should follow literate programming and reproducible research guidelines so that it's simple for a newcomer to the code to run the program, understand what's going on, and how to tweak it.
I think that any temporary stuff created in source'd files won't get cleaned up. If I do:
x=matrix(runif(big^2),big,big)
z=sum(x)
and source that as a file, x hangs around although I don't need it. But if I do:
ff=function(big){
x = matrix(runif(big^2),big,big)
z=sum(x)
return(z)
}
and instead of source, do z=ff(big) in my script, the x matrix goes out of scope and so gets cleaned up.
Functions enable neat little re-usable encapsulations and don't pollute outside themselves. In general, they don't have side-effects. Your line-by-line scripts could be using global variables and names tied to the data set in current use, which makes them unre-usable.
I sometimes work line-by-line, but as soon as I get more than about five lines I see that what I have really needs making into a proper reusable function, and more often than not I do end up re-using it.
I don't think there is a single answer. The best thing to do is keep the relative merits in mind and then pick an approach for that situation.
1) functions. The advantage of not using functions is that all your variables are left in the workspace and you can examine them at the end. That may help you figure out what is going on if you have problems.
On the other hand, the advantage of well designed functions is that you can unit test them. That is you can test them apart from the rest of the code making them easier to test. Also when you use a function, modulo certain lower level constructs, you know that the results of one function won't affect the others unless they are passed out and this may limit the damage that one function's erroneous processing can do to another's. You can use the debug facility in R to debug your functions and being able to single step through them is an advantage.
2) LCFD. Regarding whether you should use a decomposition of load/clean/func/do regardless of whether its done via source or functions is a second question. The problem with this decomposition regardless of whether its done via source or functions is that you need to run one just to be able to test out the next so you can't really test them independently. From that viewpoint its not the ideal structure.
On the other hand, it does have the advantage that you may be able to replace the load step independently of the other steps if you want to try it on different data and can replace the other steps independently of the load and clean steps if you want to try different processing.
3) No. of Files There may be a third question implicit in what you are asking whether everything should be in one or multiple source files. The advantage of putting things in different source files is that you don't have to look at irrelevant items. In particular if you have routines that are not being used or not relevant to the current function you are looking at they won't interrupt the flow since you can arrange that they are in other files.
On the other hand, there may be an advantage in putting everything in one file from the viewpoint of (a) deployment, i.e. you can just send someone that single file, and (b) editing convenience as you can put the entire program in a single editor session which, for example, facilitates searching since you can search the entire program using the editor's functions as you don't have to determine which file a routine is in. Also successive undo commands will allow you to move backward across all units of your program and a single save will save the current state of all modules since there is only one. (c) speed, i.e. if you are working over a slow network it may be faster to keep a single file in your local machine and then just write it out occasionally rather than having to go back and forth to the slow remote.
Note: One other thing to think about is that using packages may be superior for your needs relative to sourcing files in the first place.
No one has mentioned an important consideration when writing functions: there's not much point in writing them unless you're repeating some action again and again. In some parts of an analysis, you'll being doing one-off operations, so there's not much point in writing a function for them. If you have to repeat something more than a few times, it's worth investing the time and effort to write a re-usable function.
Workflow:
I use something very similar:
Base.r: pulls primary data, calls on other files (items 2 through 5)
Functions.r: loads functions
Plot Options.r: loads a number of general plot options I use frequently
Lists.r: loads lists, I have a lot of them because company names, statements and the like change over time
Recodes.r: most of the work is done in this file, essentially it's data cleaning and sorting
No analysis has been done up to this point. This is just for data cleaning and sorting.
At the end of Recodes.r I save the environment to be reloaded into my actual analysis.
save(list=ls(), file="Cleaned.Rdata")
With the cleaning done, functions and plot options ready, I start getting into my analysis. Again, I continue to break it up into smaller files that are focused into topics or themes, like: demographics, client requests, correlations, correspondence analysis, plots, ect. I almost always run the first 5 automatically to get my environment set up and then I run the others on a line by line basis to ensure accuracy and explore.
At the beginning of every file I load the cleaned data environment and prosper.
load("Cleaned.Rdata")
Object Nomenclature:
I don't use lists, but I do use a nomenclature for my objects.
df.YYYY # Data for a certain year
demo.describe.YYYY ## Demographic data for a certain year
po.describe ## Plot option
list.describe.YYYY ## lists
f.describe ## Functions
Using a friendly mnemonic to replace "describe" in the above.
Commenting
I've been trying to get myself into the habit of using comment(x) which I've found incredibly useful. Comments in the code are helpful but oftentimes not enough.
Cleaning Up
Again, here, I always try to use the same object(s) for easy cleanup. tmp, tmp1, tmp2, tmp3 for example and ensuring to remove them at the end.
Functions
There has been some commentary in other posts about only writing a function for something if you're going to use it more than once. I'd like to adjust this to say, if you think there's a possibility that you may EVER use it again, you should throw it into a function. I can't even count the number of times I wished I wrote a function for a process I created on a line by line basis.
Also, BEFORE I change a function, I throw it into a file called Deprecated Functions.r, again, protecting against the "how the hell did I do that" effect.
I often divide up my code similarly to this (though I usually put Load and Clean in one file), but I never just source all the files to run the entire project; to me that defeats the purpose of dividing them up.
Like the comment from Sharpie, I think your workflow should depends a lot on the kind of work you're doing. I do mostly exploratory work, and in that context, keeping the data input (load and clean) separate from the analysis (functions and do), means that I don't have to reload and reclean when I come back the next day; I can instead save the data set after cleaning and then import it again.
I have little experience doing repetitive munging of daily data sets, but I imagine that I would find a different workflow helpful; as Hadley answers, if you're only doing something once (as I am when I load/clean my data), it may not be helpful to write a function. But if you're doing it over and over again (as it seems you would be) it might be much more helpful.
In short, I've found dividing up the code helpful for exploratory analyses, but would probably do something different for repetitive analyses, just like you're thinking about.
I've been pondering workflow tradeoffs for some time.
Here is what I do for any project involving data analysis:
Load and Clean: Create clean versions of the raw datasets for the project, as if I was building a local relational database. Thus, I structure the tables in 3n normal form where possible. I perform basic munging but I do not merge or filter tables at this step; again, I'm simply creating a normalized database for a given project. I put this step in its own file and I will save the objects to disk at the end using save.
Functions: I create a function script with functions for data filtering, merging and aggregation tasks. This is the most intellectually challenging part of the workflow as I'm forced to think about how to create proper abstractions so that the functions are reusable. The functions need to generalize so that I can flexibly merge and aggregate data from the load and clean step. As in the LCFD model, this script has no side effects as it only loads function definitions.
Function Tests: I create a separate script to test and optimize the performance of the functions defined in step 2. I clearly define what the output from the functions should be, so this step serves as a kind of documentation (think unit testing).
Main: I load the objects saved in step 1. If the tables are too big to fit in RAM, I can filter the tables with a SQL query, keeping with the database thinking. I then filter, merge and aggregate the tables by calling the functions defined in step 2. The tables are passed as arguments to the functions I defined. The output of the functions are data structures in a form suitable for plotting, modeling and analysis. Obviously, I may have a few extra line by line steps where it makes little sense to create a new function.
This workflow allows me to do lightning fast exploration at the Main.R step. This is because I have built clear, generalizable, and optimized functions. The main difference from the LCFD model is that I do not preform line-by-line filtering, merging or aggregating; I assume that I may want to filter, merge, or aggregate the data in different ways as part of exploration. Additionally, I don't want to pollute my global environment with lengthy line-by-line script; as Spacedman points out, functions help with this.