How do I run %%magic in R inside Jupyter?
%%javascript
IPython.OutputArea.auto_scroll_threshold = 99999;
The auto scroll feature on longer output is vastly annoying as I have several functions and scripts that spit out a lot of output.
The above Javascript works fine in python notebooks but not in R notebooks.
When I run the %% magic command in R, it barfs:
Error in parse(text = x, srcfile = src): <text>:1:1: unexpected SPECIAL
1: %%
Any suggestions?
According to this post disable_autoscroll.py, it may be possible to put that Javascript into a profile_dir/static/js/custom.js file. Pray tell, where is the profile_dir on a Windows box?
I found: c:/Anaconda2/Lib/site-packages/notebook/static/custom/custom.js but that is the central custom.js file.
References:
Auto-scrolling of long output should be configurable in the UI
In my anaconda install of the notebook, the custom.js file is in %USERPROFILE%\.jupyter\custom\custom.js.
For the "magics": magics are a thing of the python kernel, not the notebook. The R kernel does not implement a magic system and so these don't work. As mentioned above, use IRdisplay::display_javascript('IPython.OutputArea.auto_scroll_threshold = 99999;') for your usecase.
Related
When I use Python with colab, you can access the underlying operating system using things like ! and %%shell, e.g., !ls to list the files.
When I use colab with the R kernel, the ! and %%shell tricks don't work. Is there a way to call the shell from colab in this case?
https://colab.research.google.com/#create=true&language=r
There is the base R function system which allows you to call the shell behind the Notebook. Since colab suppresses the stdout, one needs to set the option intern = TRUE to see the result as an R character vector. To properly display line breaks, I defined a function called shell_call which is analog to ! in ipython:
shell_call <- function(command, ...) {
result <- system(command, intern = TRUE, ...)
cat(paste0(result, collapse = "\n"))
}
shell_call("ls -lah / | head")
The implementation of magics is kernel specific as explained here:
To Jupyter users: Magics are specific to and provided by the IPython
kernel. Whether Magics are available on a kernel is a decision that is
made by the kernel developer on a per-kernel basis. To work properly,
Magics must use a syntax element which is not valid in the underlying
language. For example, the IPython kernel uses the % syntax element
for Magics as % is not a valid unary operator in Python. However, %
might have meaning in other languages.
R does not offer good conditions to implement magics this way as for example %% is used as the modulo operator and ! as the logical NOT operator.
This is why the developers of the IRkernel have chosen not to support %%cell magic - see this or this.
As already outlined in the comments when working with the IRkernel you'll need to find solutions in the R language itself e.g. using built-in functions to access the shell system2("ls", stdout = TRUE) or search for files list.files() (you might also want to wrap them in a custom function as suggested by #danlooo).
If you still want to use magics another approach would be to use R magics (extension rmagic) along with the IPython kernel as shown here or here:
I am making a tutorial using Jupyter and I would like to display the content of an external Python script. Printing the content of the file is trivial, but I am interested in a color-coded/syntax-highlighted text (either in a markdown cell or as an output).
Use Ipython's Markdown module:
from IPython.display import Markdown as md
script = """
x = 2
if x*2 > 2:
print('x > 2')
else:
x = None
"""
md("```Python" + script + "```")
Will output:
One method is to use the magic command: %load
%load testLoad.py
This is assuming the external file is in your Jupyter Notebook starting directory
If you only want specific lines (say between lines 5 and 10 and also line 15) of your python code to be displayed then:
%load -r 5-10,15 testLoad.py
You can find out the options of a magic command by adding a '?' at the end of the magic command:
%load?
Magic commands are shortcuts that are used to help do things much faster in Jupyter Notebook. They are great for beginners as they generally contain everything a beginner needs to display and test Jupyter Notebook
Here is a link to all magic commands in IPython for Jupyter Notebook:
https://ipython.readthedocs.io/en/stable/interactive/magics.html
The following magic command (starting with %) can be used to output the contents of a python script with highlighting.
%pycat script.py
Alternatively, it is possible to run shell commands by prepending them with !. If you have pygmentize installed, it is possible to use that (though I recommend %pycat for notebooks):
!pygmentize -g script.py
You can also use the old but good UNIX tool cat on all POSIX compliant systems (Unix, Mac, Linux, BSD). Then you won't get highlighting, but it might be more suitable for text files:
!cat data.txt
On POSIX compliant systems notebooks also come with an alias to cat in the form of a magic command:
%cat data.txt
I can't work out how to get the %edit command working in Jupyter Notebook.
I type
%edit
and it returns with
IPython will make a temporary file named: /var/folders/dk/.../ipython_edit_JbS9ZC.py
My %EDITOR environment variable is
'EDITOR': '/usr/local/bin/subl -w'
I must be doing something wrong.
This feature issue appears to be well-known : https://github.com/ipython/ipython/issues/5879
A possible solution might be to use : Edit IPython cell in an external editor
Or you can embed the notebook in emacs (I don't use emacs) : https://github.com/tkf/emacs-ipython-notebook
What is the most productive and quickest way to debug shared objects that are loaded into R, in particular on OS X Mavericks? I'm primarily interested in debugging compiled Rcpp code.
I have read the R externals on debugging compiled code (http://cran.r-project.org/doc/manuals/R-exts.html#Debugging-compiled-code) which favours using gdb, but gdb isn't officially supported on Mavericks. However, it seems that lldb is a viable alternative? I found the most useful resource for working out how to debug compiled code in R from Dirk's response to this post (Thanks Dirk!) (Debugging (line by line) of Rcpp-generated DLL under Windows).
I can successfully debug compiled Rcpp code by following the steps which I outline below explicitly step by step, which other Rcpp novices may find this useful (apologies for the length, but I figure it's better to be clear than to omit and be vague). But this development process is a bit tedious and time consuming.
Questions:
is there a faster and/or easier way to debug compiled Rcpp code, compared to what I do below?
I'm a big fan of Rstudio, and would love to incorporate debugging shared objects created from that IDE, so if anyone knows how to do this, I would be interested to know? Rstudio seems to use attributes, and it appears that step 4 below needs modification, because there doesn't seem to be a .cpp file in the temporary directory after compiling (in my example below, there is no "file5156292c0b48.cpp" file)
The steps:
1) (One off) Go to the directory ~/.R (a hidden directory with the .). Create a new file called "Makevars" and in it add the line CXXFLAGS=-g -O0 -Wall.
2) In the terminal, type R -d lldb to launch R. lldb will now start.
3) Type run at the lldb command line. This will start R.
4) Compile the Rcpp code and find the location of the compiled objects. Dirk's response to the above mentioned post shows one way to do this. Here is an example I'll use here. Run the following commands in R:
library(inline)
fun <- cxxfunction(signature(), plugin="Rcpp", verbose=TRUE, body='
int theAnswer = 1;
int theAnswer2 = 2;
int theAnswer3 = 3;
double theAnswer4 = 4.5;
return wrap(theAnswer4);
')
This creates a compiled shared object and other files which can be found by running setwd(tempdir()) and list.files() in R. There will be a .cpp file, like "file5156292c0b48.cpp" and .so file like "file5156292c0b48.so"
5) Load the shared object into R by running dyn.load("file5156292c0b48.so") at the R command line
6) Now we want to debug the C++ code in this .so object. Go back to lldb by hitting ctrl + c. Now I want to set a break point at a specific line in the file5156292c0b48.cpp file. I find the correct line number by opening another terminal and looking at the line number of interest in file5156292c0b48.cpp. Say it's line 31, which corresponds to the line int theAnswer = 1; above in my example. I then type at the lldb command line: breakpoint set -f file5156292c0b48.cpp -l 31. The debugger prints back that a break point has been set and some other stuff...
7) Go back to R by running cont in lldb (the R prompt doesn't automatically appear for me until I hit enter) and call the function. Run fun() at the R command line. Now I am debugging the shared object (hit n to go to next line, p [object name] to print variables etc)....
To debug simple Rcpp scripts like this, one thing you can do is create a .cpp application (with a main) that embeds R. This way you can debug it directly with Xcode which will give you a good debugging experience.
It gets more complicate when you start to want to debug packages.
This is tough. I tried Xcode and Eclipse with a standalone C++ application, but it was so difficult to get all the headers and libraries working. Furthermore, the RcppExport code calls your real R function through a pointer which seemed to really confuse Xcode, and I couldn't step into my function.
I ended up with (gdb or lldb):
In R:
R -d lldb
In debugger, set breakpoint:
b functionName
run
In R:
.Call(etc) # or just call your R code which invokes compiled C/C++ code
Then back in debugger once break happens, you can step, examine frames, etc.
This lldb/gdb command quick reference helped a lot.
Forget about trying to do this in a GUI at this point. Hopefully Rstudio will get this going.
I've now got everything to work properly on my Mac OS X 10.6 machine so that I can create decent looking LaTeX documents with Sweave that include snippets of R code, output, and LaTeX formatting together. Unfortunately, I feel like my work-flow is a bit clunky and inefficient:
Using TextWrangler, I write LaTeX code and R code (surrounded by <<>>= above and # below R code chunk) together in one .Rnw file.
After saving changes, I call the .Rnw file from R using the Sweave command
Sweave(file="/Users/mymachine/Documents/Assign4.Rnw",
syntax="SweaveSyntaxNoweb")
In response, R outputs the following message:
You can now run LaTeX on 'Assign4.tex'
So then I find the .tex file (Assign4.tex) in the R directory and copy it over to the folder in my documents ~/Documents/ where the .Rnw file is sitting (to keep everything in one place).
Then I open the .tex file (e.g. Assign4.tex) in TeXShop and compile it there into pdf format. It is only at this point that I get to see any changes I have made to the document and see if it 'looks nice'.
Is there a way that I can compile everything with one button click? Specifically it would be nice to either call Sweave / R directly from TextWrangler or TeXShop. I suspect it might be possible to code a script in Terminal to do it, but I have no experience with Terminal.
Please let me know if there's any other things I can do to streamline or improve my work flow.
I use a Makefile of the following form for my Sweave documents:
pdf: myfile.tex
R CMD texi2pdf myfile.tex
myfile.tex: myfile.Rnw
R CMD Sweave myfile.Rnw
Then I can build the document in one step in the Mac OS Terminal by running the command make pdf
I'm sure there is a way to bring this closer to your one-click goal in Mac OS X, but this works well enough for me.
One-click Sweaving is easy to do in TeXShop using the Sweave.sh script by Gregor Gorjanc.
Get it from http://cran.r-project.org/contrib/extra/scripts/Sweave.sh and put it in your ~/Library/TeXShop/bin/ folder.
Then add the following files to your ~/Library/TeXShop/engines/ folder:
As Sweave.engine:
#!/bin/bash
~/Library/TeXShop/bin/Sweave.sh -ld "$1"
As SweaveNoClean.engine:
#!/bin/bash
~/Library/TeXShop/bin/Sweave.sh -nc -ld "$1"
You'll have to set the permissions on Sweave.sh and the two engine files to allow execution.
To Sweave with one click, restart TeXShop after adding these files, open the Sweave document (with Rnw extension) and in the dropdown menu above the document window, change it from LaTeX to Sweave or SweaveNoClean.
BEWARE: The "Sweave" option wll clean up after itself, deleting all the extra files LaTeX and Sweave creates. If your file is called myfile.Rnw, this will include files called myfile.R and myfile.tex. So a word to the wise: make sure the basename of your Rnw file is unique; then nothing unexpected will be written over and then deleted.
The SweaveNoClean option does not clean up after itself. This makes sure you don't delete anything unexpected; though it could still write over a file called myfile.tex if you Sweave a myfile.Rnw. This also doesn't delete any graphics that have been created, in case you want to have them separate from your full typeset document.
On the bash shell command line:
R CMD Sweave foo.Rnw && pdflatex foo.tex
Runs Sweave, and if that succeeds it goes on to do pdflatex. Out pops a pdf. If you've got this in a bash Terminal then just hit up-arrow to get it back and do it again. And again. And Again.
Makefile solution also good.
RStudio has a button that does this in one go. One caveat is that it runs in its own session, so any workspace variables you may have set are ignored.
Just a note: you can actually call things like pdflatex etc. directly from R using texi2dvi (in the tools package). For example:
Sweave(file="/Users/mymachine/Documents/Assign4.Rnw")
texi2pdf("Assign4.tex")
would compile your Rnw file into a pdf. Thus, no need to leave R to handle the tex->pdf step.
I use these (saved as sweave.engine and sweavebibtex.engine) for custom engines in texshop. I usually work up a code chunk in R, then copy the block into the rnw file I have open in texshop. I'd love a solution that lets me do syntax highlighting and spelling correction of R and tex in the same document (that isnt emacs).
#!/bin/bash
echo 'SWEAVE | PDFLATEX. Custom engine--Akasurak-16Nov2009'
export PATH=$PATH:/usr/texbin:/usr/local/bin
R CMD Sweave "$1"
pdflatex "${1%.*}"
and the second, for doing bibtex as well:
#!/bin/bash
date
before="$(date +%s)"
echo 'SWEAVE | PDFLATEX | BIBTEX | PDFLATEX | PDFLATEX. Custom engine--Akasurak-16Nov2009'
#Updated 20Jul2010 for auto including Sweave.sty
export PATH=$PATH:/usr/texbin:/usr/local/bin
R CMD Sweave "$1"
R CMD pdflatex "${1%.*}"
bibtex "${1%.*}.aux"
R CMD pdflatex "${1%.*}"
R CMD pdflatex "${1%.*}"
after="$(date +%s)"
elapsed_seconds="$(expr $after - $before)"
date
echo Elapsed time '(m:s)': $(date -r $elapsed_seconds +%M:%S)
Can't say they are the best way of doing things, but they do work.
I use either Aquamacs or Eclipse to do the editing of the .Rnw file, then I use the following shell function to compile & view it:
sweaveCache () {
Rscript -e "library(cacheSweave); setCacheDir(getwd());
Sweave('$1.Rnw', driver = cacheSweaveDriver)" &&
pdflatex --shell-escape $1.tex &&
open $1.pdf
}
Notice that I'm using the cacheSweave driver, which helps avoid constantly re-executing code sections that take a long time to run.
BTW, I'm also trying to switch over to Babel instead of Sweave; not sure which I'll end up using more often, but there are definitely aspects of Babel that I like.
The best solution is here: you create a new *.engine for TeXShop to use, then typeset using the shortcut or the 1 button.
http://cameron.bracken.bz/sweave-for-texshop
Cameron is also very responsive, so I highly recommend his solution.
If you are open to switching to a (paid) solution, TextMate has a Sweave plugin that takes you from .Rnw to PDF in one step: Sweave, typeset, and view. Combined with Skim, which can be configured to reload PDFs, it makes tweaking files pretty easy.
I had this same issue (I use Mac OSX) and I opted to download Eclipse Classic 3.6.2. and then installed the StatET plugin. It's a bit hairy to get set up but once you do this environment is nice because you can one-click compile your .Rwn Sweave document using pdflatex and set options for your favorite viewer so the .pdf automatically pops up when you compile like it does in TeXShop. You can do this in TeXShop as well, but TeXShop is lousy for debugging .Rnw files and it doesn't highlight the R-code in the .Rwn file. In Eclipse you can customize the syntax highlighting (not the greatest from the Texclipse end, but ok) so that you can easily distinguish between your R and LaTeX code. You can also launch the R console from within Eclipse and it has a graphical object browser. Anyway, I could go on. If you want details about how to get it all installed, message me.
Guess I'm late to the party on this, but I put together a webpage that documents my Sweave workflow based on Eclipse (with one-touch sweave):
http://www.stanford.edu/~messing/ComputationalSocialScienceWorkflow.html