I am having trouble trying to add environment variables to a REnviron file that have special characters. This is on a Debian machine with the file located at /usr/lib/R/etc/Renviron. If my value has a &, I get a weird error when installing packages (although the package installs fine):
REnviron file: TEST_KEY=HEY&X&THERE
Command: install.packages(futures)
Error:
/usr/lib/R/bin/Rcmd: 468: /usr/lib/R/etc/Renviron: THERE: not found
/usr/lib/R/bin/Rcmd: 468: /usr/lib/R/etc/Renviron: X: not found
Which seems like it's because & is a special character. I can fix this by putting quotes around the value like this: TEST_KEY="HEY&X&THERE". However at that point I can't figure out how to handle when a value itself has a " in it. For example if I wanted the value to be HEY&"&THERE I am not sure how to format that (a backlash in front of the quote didn't work). I tried "HEY&\"&THERE", but that left the \ in the string once loaded into R. Which leads me to my broader question:
How can I ensure that anything that satisfies linux environment variable styling rules works in an REnviron file?
Update: this seems to be a Debian specific issue. You can recreate it using the debian:bullseye-slim docker image, installing R, then editing the Renviron to have a & in it.
Okay I spent an hour looking into this and I think there is the answer.
In both Ubuntu and Debian (and maybe other systems too), the Renviron file gets executed within bash. So what you're typing in the file is exactly bash commands. You can see in lines 39-40 of RCmd the commands:
. "${R_HOME}/etc${R_ARCH}/Renviron"
export `sed 's/^ *#.*//; s/^\([^=]*\)=.*/\1/' "${R_HOME}/etc${R_ARCH}/Renviron"`
The first line runs the Renviron file in the shell, the second then exports the variable names based on lines that have a = in them.
So in our case the way to handle this is to put double quotations around all the values, and any double-quote within the string should get a \ before it. The reason why I didn't realize the solution before I posted the question is that I didn't use cat() when printing my text in R, which removes the leading \. So: "HEY&\"&THERE" would be the right way to do it.
To recap:
The Renviron file is executed on the shell
To handle special characters in strings you use the same logic you would in the OS (so double quotes with \ to escape actual double quotes).
If I invoke
Rscript -e "print('hello')"
It correctly prints out the answer
[1] "hello"
However, if I switch single and double quotes, it does not work, and it looks like the double quotes are removed:
Rscript -e 'print("hello")'
gives:
Error in print(hello) : object 'hello' not found
Execution halted
Note that it's not powershell doing the escaping incorrectly. Echoing only gives the expected results:
PS> echo 'print("hello")'
print("hello")
PS> echo "print('hello')"
print('hello')
And the same behavior is not observed on macOs or Linux, where both variants are correctly parsed.
Interestingly enough, it's even crazier for command.com:
C:>Rscript -e "print('hello')"
[1] "hello"
C:>Rscript -e 'print("hello")'
[1] "print(hello)"
I mean... what?!?
This has already been mentioned here:
Single line code to run R code from Windows command line
but there's no explanation about it. In my opinion it's a bug of Rscript on windows, but I want to hear other opinions.
Dabombber's helpful answer provides all the pointers, but let me try to boil it down conceptually:
The problem is not specific to RScript.exe and potentially affects calls to any external executable from PowerShell:
Up to at least PowerShell 7.1 (current as of this writing), passing arguments with embedded double quotes (") to external programs is fundamentally broken, as detailed in GitHub issue #1995; in short: behind the scenes, PowerShell constructs a command line for the target program (process) that uses "..."-quoting only, but neglects to escape embedded verbatim " chars. for their syntactically valid inclusion in such double-quoted strings; a fix may be coming in v7.2 - see this answer.
For now, you have to manually escape embedded " chars. as \".
However, if and when the bug gets fixed, this workaround will break, because the fix requires that this escaping be applied automatically, which would then escape a verbatim \" as \\\".
# WORKAROUND as of v7.0, which will break if and when the problem gets fixed.
PS> Rscript -e 'print(\"hello\")'
The third-party Native module (install with Install-Module -Scope CurrentUser Native, for instance) offers helper function ie, which compensates for the broken behavior; it is written in a forward-compatible manner so that it will simply defer to the built-in behavior if and when it should get fixed:
# Thanks to `ie`, no workarounds are required.
PS> ie Rscript -e 'print("hello")'
As for ad hoc workarounds - both of them work for Rscript.exe, but can't be expected to be a general solution:
For target programs that support both '...' and "..." quoting: Swap the quotes to use only embedded ' chars., as shown in your question, but note that '...' and "..." strings have different semantics in PowerShell ("..." strings are expandable (interpolating) strings), and may have different semantics in the target program too (not the case in Rscript):
Rscript -e "print('hello')"
For target programs that accept input via stdin, use the PowerShell pipeline, where the bug doesn't surface (though note that you may have to set the $OutputEncoding preference variable to the character encoding expected by the target program):
'print("hello")' | Rscript -
As for your observations and background information, including about cmd.exe and POSIX-compatible shells:
Note that it's not powershell doing the escaping incorrectly.
As Dabombber points out, it is PowerShell that is the problem, but the problem only occurs when calling external programs, whereas echo is a built-in alias for the PowerShell-native
Write-Output cmdlet (verify with Get-Command echo).
On Windows, you could see the problem with the flawed parameter passing as follows, by invoking choice.exe (ignore the [Y,N]?N suffix):
PS> 'n' | choice /m 'print("hello")'
print(hello) [Y,N]?N
choice.exe with /m can be used to echo an argument as it would be received by external programs, and as you can see the double quotes were effectively lost, because PowerShell mistakenly placed print("hello") verbatim on the process command line - without escaping the " chars. - which external programs parse as verbatim print(hello), because they allow a single argument to be composed of unquoted and double-quoted parts (print( + hello (stripped of the syntactic double quotes) + )).
If verbatim print(hello) is interpreted as an R script, it looks for a variable (object) named hello - which in this scenario doesn't exist and triggers the error message you saw.
On Unix-like platforms (macOS, Linux), using the cross-platform PowerShell [Core] edition, /bin/echo 'print("hello")' shows the same problem.
And the same behavior is not observed on macOs or Linux, where both variants are correctly parsed.
Yes, if you use a native, POSIX-compatible shell there, such as bash, you'll get the correct behavior (see below).
it's even crazier for command.com:
As an aside: You probably meant cmd.exe, the legacy command processor (Command Prompt) on NT-based Windows platforms up to the current Windows 10.
(command.com was the command processor on the extinct DOS-based Windows versions that ended with Windows ME).
cmd.exe only recognizes double-quoting ("...") to demarcate argument boundaries for itself, not also single-quoting ('...'); irrespective of that, it essentially passes the original quoting through to the target executable (after performing its own interpretation of the command line, such as environment-variable expansion).
This differs fundamentally from what PowerShell and POSIX-compatible shells do:
On Unix-like platforms - where POSIX-compatible shells recognize '...'-quoted arguments - the concept of a process command line doesn't exist, and whatever arguments a POSIX-like shell has itself parsed out of its command line are passed as-is - as an array of verbatim arguments - to the target executable; thus shell string literals "print('hello')" and 'print("hello")' are passed as verbatim print('hello') and print("hello"), respectively, which works as expected, given that R too recognizes both '...' and "..." string literals.
PowerShell too has '...' strings (it treats their content verbatim), but on Windows it translates them to "..." strings behind the scenes (if quoting is needed), which is where the aforementioned bug can surface as of v7.0. The bug aside, this translation makes sense, because only "..." quoting can be assumed to have syntactic meaning on the command line for other programs (see bottom section). Unfortunately, PowerShell does the same thing on Unix-like platforms, even though it shouldn't (it constructs a pseudo command line that the .NET API then parses into an array of verbatim arguments passed to the target process), so the bug surfaces there as well.
Because cmd.exe preserves the original quoting, RScript interprets 'print("hello")' in command line Rscript -e 'print("hello")' as a string literal rather than as a command, because it removes any " chars. with syntactic function on the command line first (whereas ' (single quotes) by convention do not have syntactic meaning on the command line), before the result is interpreted as an R script:
'print("hello")' is therefore parsed as 'print( + hello (the command-line " are stripped) + ), resulting in verbatim 'print(hello)' getting interpreted as R code, which is an R string literal that therefore prints as-is (the output uses "..." quoting, but that's just an artifact of output formatting; note that an explicit call to print() isn't necessary, the result of an expression - such as string literal 'print(hello)' in this case - is automatically printed).
By contrast, "print('hello')" is parsed as verbatim print('hello') (the command-line " are stripped), which - due to the absence of enclosing quoting - is then interpreted as a command, namely a print() function call, as intended.
Ultimately, there are no hard and fast rules in the anarchic world of process command-line parsing on Windows: it is ultimately up to each program to interpret its command line - this answer contains excellent background information.
Fortunately, however, there are widely adhered-to conventions, as implemented in the MS C/C++/.NET compilers and documented here.
Unfortunately, as of PowerShell 7.0, PowerShell doesn't adhere to these conventions, due to the aforementioned bug. Since the bug has been around since v1, users have learned to work around it, such as with manual \"-escaping, as shown above. The problem is that fixing the bug would break all workarounds. Implementing a fix as an experimental feature is now being considered, for v7.1 at the earliest - see this PR on GitHub and the associated discussion here, which suggests that, in addition to implementing the widely established conventions, accommodations be made for calls to batch files and msiexec.exe-style programs, which have non-conventional quoting requirements.
It might be worth taking a look through this PowerShell issue: Arguments for external executables aren't correctly escaped. The Native module by Michael Klement provides a workaround until the problem is fixed (and shouldn't be broken post-fix like many current workarounds).
Note that it's not powershell doing the escaping incorrectly. Echoing only gives the expected results
echo is a PowerShell function rather than an external program so you won't notice the broken behaviour when using it.
PS> Get-Command echo
CommandType Name Version Source
----------- ---- ------- ------
Alias echo -> Write-Output
A better test would be to use the EchoArgs.exe command line tool from PowerShell
Community Extensions (downloadable here).
PS> echoargs.exe 'print("hello")'
Arg 0 is <print(hello)>
Command line:
"E:\echoargs.exe" print("hello")
PS> echoargs.exe "print('hello')"
Arg 0 is <print('hello')>
Command line:
"E:\echoargs.exe" print('hello')
Note that it's not powershell doing the escaping incorrectly. Echoing
only gives the expected results:
In the case of using echo, its echo which is directly consuming the argument you are passing to it, so you get the same result for single quotes or double quotes.
In the case of Rscript, I believe Rscript is just a convenient way of calling R with some additional arguments. (see https://swcarpentry.github.io/r-novice-inflammation/05-cmdline/ for explanation). Specifically, it says that "From this output, we learn that Rscript is just a convenience command for running R scripts...."
So maybe what's happening is that when you call RScript, its passing the argument to a separate process, and because of this its trying to expand hello as a variable, leading to the error (in the Powershell case)
As for cmd it has its own behavior for handling single and double quotes.
See: What does single-quoting do in Windows batch files?
and
Differences between single and double quotes in CMD
So the problem may not be with RScript. The resulting output of your use case may just be a side effect of how powershell and cmd handle double quotes and single quotes.
This may also explain why the problem is there only on windows, and not in Linux or MacOS.
Check out this one! https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7
expressions in single-quoted strings are not evaluated. They are interpreted as literals.
in a double-quoted string, expressions are evaluated, and the result is inserted in the string.
same rules apply for cmd
I would like to find any files within a given root that contain arbitrary extensions in the filename.
I saw this post:
How to delete all files with certain suffix before file extension
Based on that information, I tried this:
find . -iregex ".*\.\(wav\|aif\|wave\|aiff\)"
This seems like it should work, but I don't get any results printed to the terminal window.
Can anyone offer advice? I'm on OSX 10.7
Thanks,
jml
You are looking for:
find . -regex ".*\.\(wav\|aif\|wave\|aiff\)"
You were missing escape, \, characters on the or, |, operators
Is that an emacs style regex?
If not, try using -regextype. From the find man page on Linux (archaic):
-regextype type
Changes the regular expression syntax understood by -regex and -iregex tests which occur later on the command line. Currently-implemented types are emacs (this is the default), posix-awk, posix-basic, posix-egrep and posix-extended.
On MacOS X, the manual page for find says:
-iregex pattern
Like -regex, but the match is case insensitive.
-regex pattern
True if the whole path of the file matches pattern using regular expression. To match a file named './foo/xyzzy', you can use the regular expression '.*/[xyz]*' or '.*/foo/.*', but not 'xyzzy' or '/foo/'.
Some experimentation shows that:
find pdf -iregex ".*/.*.pdf"
finds a whole lot of PDF files in my folder full of them, but none of these variants find anything:
find pdf -iregex ".*/.*\.(pdf|doc|docx)"
find pdf -iregex ".*/.*\.\(pdf|doc|docx\)"
find pdf -iregex ".*/.*.(pdf|doc|docx)"
find pdf -iregex ".*/.*.\(pdf|doc|docx\)"
Consequently, one is forced to assume that the regexes supported by MacOS X (BSD) find do not include alternation (parentheses and pipes) amongst the recognized characters. 'Tis a pity: man 7 re_format implies it might, but it doesn't. The -regextype option is not supported on MacOS X (BSD), it seems.
So, it may be simplest to install GNU find, or to do N separate searches for the N different file extensions, or do one search for files in general and use egrep '\.(aff|wave?|aiff)$' to catch the files you're interested in. That rather assumes you don't use newlines in file names (spaces etc are OK, but newlines are not).
I've looked this up a thousand times, and I always forget it, so, here for eternity:
Solaris has a bit of an awkward syntax for tail.
How do I do the equivalent of BSD's tail -nN?
What I want are the last N lines from tail's input.
Just remove the "n"
tail -100
Or you can use:
/usr/xpg4/bin/tail
which does behave like you want (tail -nN).
xpg4 = Xopen Portability Guide Issue 4, contains binaries strictly compliant with several POSIX and other standards. The differences with the former ones are usually details in options supported and behavior.
According to your distribution, there is also /usr/xpg6/bin, /usr/openwin/bin (OpenWindows commands), /usr/dt/bin (CDE desktop commands), /usr/sfw/bin (Solaris freeware) and various other.
For instance, Solaris Express is introducing /usr/gnu/bin to provide Gnu binaries with their custom extensions and specificities.
Cross-platform variant of tail -n 10 for scripts:
sed -e :a -e '$q;N;11,$D;ba' file
This works the same for Linux and Solaris.
It seems like the only way to do this is to pass the -i parameter in when you initially run less. Does anyone know of some secret hack to make something like this work
/something to search for/i
You can also type command -I while less is running. It toggles case sensitivity for searches.
You can also set the environment variable LESS
I use LESS=-Ri, so that I can pump colorized output from grep into it, and maintain the ANSI colour sequences.
Another little used feature of less that I found is starting it with +F as an argument (or hitting SHIFT+F while in less). This causes it to follow the file you've opened, in the same way that tail -f <file> will. Very handy if you're watching log files from an application, and are likely to want to page back up (if it's generating 100's of lines of logging every second, for instance).
Add-on to what #Juha said: Actually -i turns on Case-insensitive with SmartCasing, i.e if your search contains an uppercase letter, then the search will be case-sensitive, otherwise, it will be case-insensitive. Think of it as :set smartcase in Vim.
E.g.: with -i, a search for 'log' in 'Log,..' will match, whereas 'Log' in 'log,..' will not match.
It appears that you can summon this feature on a per search basis like so:
less prompt> /search string/-i
This option is in less's interactive help which you access via h:
less prompt> h
...
-i ........ --ignore-case
Ignore case in searches that do not contain uppercase.
-I ........ --IGNORE-CASE
Ignore case in all searches.
...
I've not extensively checked but the help in less version 487 on MacOS as well as other Linux distros lists this option as being available.
On MacOS you can also install a newer version of less via brew:
$ brew install less
$ less --version
less 530 (POSIX regular expressions)
Copyright (C) 1984-2017 Mark Nudelman
References
less is always case-insensitive
When using -i flag, be sure to enter the search string completely in lower case, because if any letter is upper case, then its an exact match.
See also: the -I (capital i) flag of less(1) to change this behavior.