I am trying to write a function to extract package names from a list of R script files. My regular expression do not seem to be working and I am not sure why. For begginers, I am not able to match lines that include library. For example
str <- c(" library(abc)", "library(def)", "some other text")
grep("library\\(", str, value = TRUE)
grep("library\\(+[A-z]\\)", str, value = TRUE)
Why does my second grep do not return elements 1 and 2 from the str vector? I have tried so many options but all my results come back empty.
Your second grep does not return 1,2 for two reasons.
You used value=TRUE which makes it return the matching string instead of the
location. and
You misplaced the +. You wantgrep("library\\(\\w+\\)", str)
If you'd like something a bit more robust that will handle some edge cases (library() takes a number of parameters and the package one can be a name/symbol or a string and doesn't necessarily have to be specified first):
library(purrr)
script <- '
library(js) ; library(foo)
#
library("V8")
ls()
library(package=rvest)
TRUE
library(package="hrbrthemes")
1 + 1
library(quietly=TRUE, "ggplot2")
library(quietly=TRUE, package=dplyr, verbose=TRUE)
'
x <- parse(textConnection(script)) # parse w/o eval
keep(x, is.language) %>% # `library()` is a language object
keep(~languageEl(.x, 1) == "library") %>% # other things are too, so only keep `library()` ones
map(as.call) %>% # turn it into a `call` object
map(match.call, definition = library) %>% # so we can match up parameters and get them in the right order
map(languageEl, 2) %>% # language element 1 is `library`
map_chr(as.character) %>% # turn names/symbols into characters
sort() # why not
## [1] "dplyr" "foo" "ggplot2" "hrbrthemes" "js" "rvest" "V8"
This won't catch library() calls within functions (it could be expanded to do that) but if top-level edge cases are infrequent, there is an even smaller likelihood of ones in functions (those wld likely use require() as well).
Related
The syntax for using scales::label_percent() in a mutate function is unusual because it uses double parentheses:
label_percent()(an_equation_goes_here)
I don't think I have seen ()() syntax in R before and I don't know how to look it up because I don't know what it is called. I tried ?`()()` and ??`()()` and neither helped. What is double parentheses syntax called? Can someone recommend a place to read about it?
Here is an example for context:
library(tidyverse)
members <-
read_csv(
paste0(
"https://raw.githubusercontent.com/rfordatascience/tidytuesday/",
"master/data/2020/2020-09-22/members.csv"
),
show_col_types = FALSE)
members %>%
count(success, died) %>%
group_by(success) %>%
# old syntax:
# mutate(percent = scales::percent(n / sum(n)))
# new syntax:
mutate(percent = scales::label_percent()(n / sum(n)))
#> # A tibble: 4 × 4
#> # Groups: success [2]
#> success died n percent
#> <lgl> <lgl> <int> <chr>
#> 1 FALSE FALSE 46452 98%
#> 2 FALSE TRUE 868 2%
#> 3 TRUE FALSE 28961 99%
#> 4 TRUE TRUE 238 1%
Created on 2023-01-01 with reprex v2.0.2
Most functions return a value, whether something atomic (numeric, integer, character), list-like (including data.frame), or something more complex. For those, the single set of ()s (as you recognize) are for the one call.
Occasionally, however, a function call returns a function. For example, if we look at ?scales::label_percent, we can scroll down to
Value:
All 'label_()' functions return a "labelling" function, i.e. a
function that takes a vector 'x' and returns a character vector of
'length(x)' giving a label for each input value.
Let's look at it step-by-step:
fun <- scales::label_percent()
fun
# function (x)
# {
# number(x, accuracy = accuracy, scale = scale, prefix = prefix,
# suffix = suffix, big.mark = big.mark, decimal.mark = decimal.mark,
# style_positive = style_positive, style_negative = style_negative,
# scale_cut = scale_cut, trim = trim, ...)
# }
# <bytecode: 0x00000168ee5440e8>
# <environment: 0x00000168ee5501b8>
fun(0.35)
# [1] "35%"
The first call to scales::label_percent() returned a function. We can then use that function with as many arguments as we want.
If you don't want to store the returned function in a variable like fun, you can use it immediately by following the first set of ()s with another set of parens.
scales::label_percent()(0.35)
# [1] "35%"
A related question is "why would you want a function to return another function?" There are many stylistic reasons, but in the case of scales::label_*, they are designed to be used in places where the option needs to be expressed as a function, not as a static value. For example, it can be used in ggplot code: axis ticks are often placed conveniently with simple heuristics to determine the count, locations, and rendering of the ticks marks. While one can use ggplot2::scale_*_manual(values = ...) to manually control how many, where, and what they look like, it is often more convenient to not care a priori how many or where, and in cases where faceting is used, it can vary per faceting variable(s), so not something one can easily assign in a static variable. In those cases, it is often better to assign a function that is given some simple parameters (such as the min/max of the axis), and the function returns something meaningful.
Why can't we just pass it scales::label_percent? (Good question.) Even though you're using the default values in your call here, one might want to change any or all of the controllable things, such as:
suffix= defaults to "%", but perhaps you want a space as in " %"?
decimal.mark= defaults to ".", but maybe your locale prefers commas?
While it is feasible to have multiple functions for all of the combinations of these options, it is generally easier in the long run to provide a "template function" for creating the function, such as
fun <- scales::label_percent(accuracy = 0.01, suffix = " %", decimal.mark = ",")
fun(0.353)
# [1] "35,30 %"
scales::label_percent(accuracy = 0.01, suffix = " %", decimal.mark = ",")(0.353)
# [1] "35,30 %"
An Expression followed by an argument list in round parentheses (( / )) is called a Function Call in R.
There's no need to have a special name for two function calls in a row. They're still just function calls.
If we run a function and the value returned by the function is itself a function then we could call one that too.
For example, we first run f using f() assigning the return value to g but the return value is itself a function so g is a function -- it is the function function() 3 -- and we can run that too.
# f is a function which returns a function
f <- function() function() 3
g <- f() # this runs f which returns `function() 3`
g() # thus g is a function so we can call it
## [1] 3
Now putting that all together we can write it in one line as
f()()
## [1] 3
As seen there is only one meaning for () and the fact that there were two together was simply because we were calling the result of a call.
I have a base query that i am trying to add 14 other strings to, which are named query1:14
I am trying to combine them using paste and the following works for one query. However, i am trying to set this up in a function which will have the number of queries passed to it, to know how many it should loop/add into the final restuls
Here is some code to add one query, which works:
result<- paste(base_query, eval(as.name(paste0("query", 2)))
Here is something i tried to loop the queries added, to no success
range <- 14
result<- paste(base_query, while(loop<range){loop<-loop+1
eval(as.name(paste0("query", loop)))}
I'm not sure how to get the names generated in the while loop to be added to the paste, thanks
In general, it is preferable to store the queries 1-14 in a list, rather than store them separately in the global environment. That is why half of this solution entails getting the objects into a list, which is easily pasted together with the collapse argument of paste.
Also pay attention to stools::mixedsort, which sorts them in the order that I assume you want (incrementally 1 to 14).
# Generate the query objects
base_query = "The letters of the alphabet are:"
for (i in 1:14) {
assign(x = paste0("query",i), value = LETTERS[i])
}
# Find all the query names, and sort them in a meaningful order
query_names = gtools::mixedsort(ls(pattern = "query*"))
query_names
#> [1] "base_query" "query1" "query2" "query3" "query4"
#> [6] "query5" "query6" "query7" "query8" "query9"
#> [11] "query10" "query11" "query12" "query13" "query14"
# Now get the appropriate objects
query_list = lapply(query_names, function(x){
get(x)
})
# Paste and collapse the contents of the list
paste(query_list, collapse = "")
#> [1] "The letters of the alphabet are:ABCDEFGHIJKLMN"
Created on 2022-08-26 by the reprex package (v2.0.0)
I have a dataframe of nested list indices, which have been stored as strings. To give a simplified example:
df1 <- data.frame(x = c("lst$x$y$a", "lst$x$y$b"), stringsAsFactors = F)
These are then coordinates for the following list:
lst <- list(x=list(y=list(a="foo",b="bar",c="")))
I'd like replace values or assign new values to these elements using the indices in df1.
One attempt was
do.call(`<-`, list(eval(parse(text = df1[1,1])), "somethingelse"))
but this doesn't seem tow work. Instead it assigns "something" to foo.
I'm not too happy with using eval(parse(text=)) (maintaining code will become a nightmare), but recognise I may have little choice.
Any tips welcome.
Let's consider 3 situations:
Case 1
do.call(`<-`, list("lst$x$y$a", "somethingelse"))
This will create a new variable named lst$x$y$a in your workspace, so the following two commands will call different objects. (The former is the object you store in lst, and the latter is the new variable. You need to call it with backticks because its name will confuse R.)
> lst$x$y$a # [1] "foo"
> `lst$x$y$a` # [1] "somethingelse"
Case 2
do.call(`<-`, list(parse(text = "lst$x$y$a"), "somethingelse"))
You mostly get what you expect with this one but an error still occurs:
invalid (do_set) left-hand side to assignment
Let's check:
> parse(text = "lst$x$y$a") # expression(lst$x$y$a)
It belongs to the class expression, and the operator <- seems not to accept this class to the left-hand side.
Case 3
This one will achieve what you want:
do.call(`<-`, list(parse(text = "lst$x$y$a")[[1]], "somethingelse"))
If put [[1]] behind an expression object, a call object will be extracted and take effect in the operator <-.
> lst
# $x
# $x$y
# $x$y$a
# [1] "somethingelse"
#
# $x$y$b
# [1] "bar"
#
# $x$y$c
# [1] ""
From ?builtins:
builtins(TRUE) returns an unsorted list of the names of internal functions, that is those which can be accessed as .Internal(foo(args ...)) for foo in the list.
I don't understand which functions are being returned.
I thought it would be all the closure functions in the base package that call .Internal().
However, the two sets don't match up.
base_objects <- mget(
ls(baseenv(), all.names = TRUE),
envir = baseenv()
)
internals <- names(
Filter(
assertive.types::is_internal_function,
base_objects
)
)
builtins_true <- builtins(internal = TRUE)
c(
both = length(intersect(internals, builtins_true)),
internals_not_builtins_true = length(setdiff(internals, builtins_true)),
builtins_true_not_internals = length(setdiff(builtins_true, internals))
)
## both internals_not_builtins_true builtins_true_not_internals
## 288 125 226
I also thought that it might be the values listed in src/main/names.c in R's source code, and there definitely seems to be some overlap with this, but it isn't exactly this list of values.
What is builtins() doing when you pass internal = TRUE?
Stibu's comment is a specific example of the general problem. ?builtins says that it fetches the names of the objects it returns directly from the symbol table (this is the C symbol table).
And builtins(TRUE) returns all the built-in objects callable via .Internal. That, however, doesn't mean there must be any function that calls .Internal(foo(args, ...)) for any foo.
Stibu gave one example: the internal function may not be called by an R function with the same name, as is the case for many generic functions where the default method calls .Internal.
Another example is something like .addCondHands and .addRestart, which are called by withCallingHandlers and withRestarts, respectively.
It's also possible that one R function calls multiple .Internal functions. I don't know of an example of that off the top of my head though.
After more digging, it seems that the list of functions is everything in the R_FunTab[] object in src/main/names.c where the second digit of the eval column is 1.
Here's a script to retrieve them.
library(stringi)
library(magrittr)
library(dplyr)
names.c <- readLines("https://raw.githubusercontent.com/wch/r-source/56a1b08b7282c5488acb71ee244098f4fd94f7c7/src/main/names.c")
fun_tab <- names.c[92:974] %>%
stri_replace_all_regex("^\\{", "") %>%
stri_replace_all_fixed("{PP", "PP") %>%
stri_replace_all_fixed("}},", "") %>%
stri_replace_all_fixed("\\t", "")
funs <- read.csv(text = fun_tab, header = FALSE, comment.char = "/")
cols <- names.c[86] %>%
stri_sub(4) %>%
stri_split_regex("\\t+") %>%
extract2(1) %>%
stri_trim()
colnames(funs) <- cols
funs$eval <- formatC(funs$eval, width = 3, flag = "0")
# Internal fns have 2nd digit of eval col == 1. See names.c[62:71]
internals <- funs %>% filter_(~ substring(eval, 2, 2) == 1)
I see slight differences when examining
setdiff(internals$printname, builtins(TRUE))
setdiff(builtins(TRUE), internals$printname)
For example builtins(TRUE) doesn't include shell.exec() if you aren't running Windows; mem.limits() was only recently removed from the devel branch of R, so it shows up in builtins(TRUE) for the current release version of R.
I am trying to pass multiple file path arguments via command line to an Rscript which can then be processed using an arguments parser. Ultimately I would want something like this
Rscript test.R --inputfiles fileA.txt fileB.txt fileC.txt --printvar yes --size 10 --anotheroption helloworld -- etc...
passed through the command line and have the result as an array in R when parsed
args$inputfiles = "fileA.txt", "fileB.txt", "fileC.txt"
I have tried several parsers including optparse and getopt but neither of them seem to support this functionality. I know argparse does but it is currently not available for R version 2.15.2
Any ideas?
Thanks
Although it wasn't released on CRAN when this question was asked a beta version of the argparse module is up there now which can do this. It is basically a wrapper around the popular python module of the same name so you need to have a recent version of python installed to use it. See install notes for more info. The basic example included sums an arbitrarily long list of numbers which should not be hard to modify so you can grab an arbitrarily long list of input files.
> install.packages("argparse")
> library("argparse")
> example("ArgumentParser")
The way you describe command line options is different from the way that most people would expect them to be used. Normally, a command line option would take a single parameter, and parameters without a preceding option are passed as arguments. If an argument would take multiple items (like a list of files), I would suggest parsing the string using strsplit().
Here's an example using optparse:
library (optparse)
option_list <- list ( make_option (c("-f","--filelist"),default="blah.txt",
help="comma separated list of files (default %default)")
)
parser <-OptionParser(option_list=option_list)
arguments <- parse_args (parser, positional_arguments=TRUE)
opt <- arguments$options
args <- arguments$args
myfilelist <- strsplit(opt$filelist, ",")
print (myfilelist)
print (args)
Here are several example runs:
$ Rscript blah.r -h
Usage: blah.r [options]
Options:
-f FILELIST, --filelist=FILELIST
comma separated list of files (default blah.txt)
-h, --help
Show this help message and exit
$ Rscript blah.r -f hello.txt
[[1]]
[1] "hello.txt"
character(0)
$ Rscript blah.r -f hello.txt world.txt
[[1]]
[1] "hello.txt"
[1] "world.txt"
$ Rscript blah.r -f hello.txt,world.txt another_argument and_another
[[1]]
[1] "hello.txt" "world.txt"
[1] "another_argument" "and_another"
$ Rscript blah.r an_argument -f hello.txt,world.txt,blah another_argument and_another
[[1]]
[1] "hello.txt" "world.txt" "blah"
[1] "an_argument" "another_argument" "and_another"
Note that for the strsplit, you can use a regular expression to determine the delimiter. I would suggest something like the following, which would let you use commas or colons to separate your list:
myfilelist <- strsplit (opt$filelist,"[,:]")
In the front of your script test.R, you put this :
args <- commandArgs(trailingOnly = TRUE)
hh <- paste(unlist(args),collapse=' ')
listoptions <- unlist(strsplit(hh,'--'))[-1]
options.args <- sapply(listoptions,function(x){
unlist(strsplit(x, ' '))[-1]
})
options.names <- sapply(listoptions,function(x){
option <- unlist(strsplit(x, ' '))[1]
})
names(options.args) <- unlist(options.names)
print(options.args)
to get :
$inputfiles
[1] "fileA.txt" "fileB.txt" "fileC.txt"
$printvar
[1] "yes"
$size
[1] "10"
$anotheroption
[1] "helloworld"
After searching around, and avoiding to write a new package from the bottom up, I figured the best way to input multiple arguments using the package optparse is to separate input files by a character which is most likely illegal to be included in a file name (for example, a colon)
Rscript test.R --inputfiles fileA.txt:fileB.txt:fileC.txt etc...
File names can also have spaces in them as long as the spaces are escaped (optparse will take care of this)
Rscript test.R --inputfiles file\ A.txt:file\ B.txt:fileC.txt etc...
Ultimatley, it would be nice to have a package (possibly a modified version of optparse) that would support multiple arguments like mentioned in the question and below
Rscript test.R --inputfiles fileA.txt fileB.txt fileC.txt
One would think such trivial features would be implemented into a widely used package such as optparse
Cheers
#agstudy's solution does not work properly if input arguments are lists of the same length. By default, sapply will collapse inputs of the same length into a matrix rather than a list. The fix is simple enough, just explicitly set simplify to false in the sapply parsing the arguments.
args <- commandArgs(trailingOnly = TRUE)
hh <- paste(unlist(args),collapse=' ')
listoptions <- unlist(strsplit(hh,'--'))[-1]
options.args <- sapply(listoptions,function(x){
unlist(strsplit(x, ' '))[-1]
}, simplify=FALSE)
options.names <- sapply(listoptions,function(x){
option <- unlist(strsplit(x, ' '))[1]
})
names(options.args) <- unlist(options.names)
print(options.args)
I had this same issue, and the workaround that I developed is to adjust the input command line arguments before they are fed to the optparse parser, by concatenating whitespace-delimited input file names together using an alternative delimiter such as a "pipe" character, which is unlikely to be used as part of a file name.
The adjustment is then reversed at the end again, by removing the delimiter using str_split().
Here is some example code:
#!/usr/bin/env Rscript
library(optparse)
library(stringr)
# ---- Part 1: Helper Functions ----
# Function to collapse multiple input arguments into a single string
# delimited by the "pipe" character
insert_delimiter <- function(rawarg) {
# Identify index locations of arguments with "-" as the very first
# character. These are presumed to be flags. Prepend with a "dummy"
# index of 0, which we'll use in the index step calculation below.
flagloc <- c(0, which(str_detect(rawarg, '^-')))
# Additionally, append a second dummy index at the end of the real ones.
n <- length(flagloc)
flagloc[n+1] <- length(rawarg) + 1
concatarg <- c()
# Counter over the output command line arguments, with multiple input
# command line arguments concatenated together into a single string as
# necessary
ii <- 1
# Counter over the flag index locations
for(ij in seq(1,length(flagloc)-1)) {
# Calculate the index step size between consecutive pairs of flags
step <- flagloc[ij+1]-flagloc[ij]
# Case 1: empty flag with no arguments
if (step == 1) {
# Ignore dummy index at beginning
if (ij != 1) {
concatarg[ii] <- rawarg[flagloc[ij]]
ii <- ii + 1
}
}
# Case 2: standard flag with one argument
else if (step == 2) {
concatarg[ii] <- rawarg[flagloc[ij]]
concatarg[ii+1] <- rawarg[flagloc[ij]+1]
ii <- ii + 2
}
# Case 3: flag with multiple whitespace delimited arguments (not
# currently handled correctly by optparse)
else if (step > 2) {
concatarg[ii] <- rawarg[flagloc[ij]]
# Concatenate multiple arguments using the "pipe" character as a delimiter
concatarg[ii+1] <- paste0(rawarg[(flagloc[ij]+1):(flagloc[ij+1]-1)],
collapse='|')
ii <- ii + 2
}
}
return(concatarg)
}
# Function to remove "pipe" character and re-expand parsed options into an
# output list again
remove_delimiter <- function(rawopt) {
outopt <- list()
for(nm in names(rawopt)) {
if (typeof(rawopt[[nm]]) == "character") {
outopt[[nm]] <- unlist(str_split(rawopt[[nm]], '\\|'))
} else {
outopt[[nm]] <- rawopt[[nm]]
}
}
return(outopt)
}
# ---- Part 2: Example Usage ----
# Prepare list of allowed options for parser, in standard fashion
option_list <- list(
make_option(c('-i', '--inputfiles'), type='character', dest='fnames',
help='Space separated list of file names', metavar='INPUTFILES'),
make_option(c('-p', '--printvar'), type='character', dest='pvar',
help='Valid options are "yes" or "no"',
metavar='PRINTVAR'),
make_option(c('-s', '--size'), type='integer', dest='sz',
help='Integer size value',
metavar='SIZE')
)
# This is the customary pattern that optparse would use to parse command line
# arguments, however it chokes when there are multiple whitespace-delimited
# options included after the "-i" or "--inputfiles" flag.
#opt <- parse_args(OptionParser(option_list=option_list),
# args=commandArgs(trailingOnly = TRUE))
# This works correctly
opt <- remove_delimiter(parse_args(OptionParser(option_list=option_list),
args=insert_delimiter(commandArgs(trailingOnly = TRUE))))
print(opt)
Assuming the above file were named fix_optparse.R, here is the output result:
> chmod +x fix_optparse.R
> ./fix_optparse.R --help
Usage: ./fix_optparse.R [options]
Options:
-i INPUTFILES, --inputfiles=INPUTFILES
Space separated list of file names
-p PRINTVAR, --printvar=PRINTVAR
Valid options are "yes" or "no"
-s SIZE, --size=SIZE
Integer size value
-h, --help
Show this help message and exit
> ./fix_optparse.R --inputfiles fileA.txt fileB.txt fileC.txt --printvar yes --size 10
$fnames
[1] "fileA.txt" "fileB.txt" "fileC.txt"
$pvar
[1] "yes"
$sz
[1] 10
$help
[1] FALSE
>
A minor limitation with this approach is that if any of the other arguments have the potential to accept a "pipe" character as a valid input, then those arguments will not be treated correctly. However I think you could probably develop a slightly more sophisticated version of this solution to handle that case correctly as well. This simple version works most of the time, and illustrates the general idea.