Azure databricks %run magic command unable to find file path - runtime-error

I have 2 notebooks under the root folder in my workspace. Calling one notebook from the other with %run magic cmd returns error saying file path not found. This is my command:
%run /Users/name#comp.com/notebookB $arg1=val1 $arg2=val2

Found the solution, turns out the run command needs to be in a cell all by itself - no other code and no comments. I have seen some solutions online say that the cell should not contain any comments, but turns out it should not contain any other code either.

It's really better to use relative names instead of absolute (starting with /):
If both notebooks are in the same folder, then you can refer to another using syntax ./notebook_name
If another is in the folder above, then use ../notebook_name
if another is in the subfolder, then use subfolder/notebook_name

Related

How to run R projects / use their relative paths from the terminal without setwd() resp. cd

I'm kinda lost on that one:
I have set up an R project, let's call it "Test Project.Rproj". The beauty of R projects is the possibility to use relative paths (relative to the .Rproj file). My project consists of a "main.R" script, which is saved on the same level as the .Rproj file.
Additionally I have a directory called 'Output', where I want my plots and exported data to be saved. My "main.R" file looks like the following:
my_df <- data.frame(A = 1:10, B = 11:20)
my_df |>
writexl::write_xlsx(here::here("Output",
paste0("my_df_",
stringr::str_replace_all(as.character(Sys.time()), ":", ""),
".xlsx")))
My final goal is to automate the execution of the 'main.R' file using the Windows Task Scheduler. But in order to do so, I have to be able to run the script from the terminal. The problem here is the working directory. When opening an R project, all the paths are relative to .Rproj file. But in the terminal the current working directory is <C:\Users\my_name>. Of course I could manually set the working directory via cd "path\to\my\project. But I would like to avoid that.
My current call for the execution of the main.R file in the terminal is the following:
"C:\Program Files\R\R-4.1.0\bin\Rscript" -e "source('C:/Users/my_name/path/to/my/project/main.R')"
My two ideas for a solution are the following, but I am happy for other suggestions as well.
In order to replicate the usual use of a project: Is there a way to execute the .Rproj
file from the terminal? In order to create a similar environment as in RStudio, where all the relative paths are working, when executing scripts from the project afterwards?
There are two packages adressing the problem of relative paths: rprojroot and here, where the former is the basis for the latter. I am pretty sure that here does not provide the needed functionality. I tried adding here::i_am("main.R) to my main.R file, but the project root directory still is not found when executing in the terminal from a working directory outside the project.
For rprojroot to work, I think it is also necessary to have your current working directory somewhere within the project. But this package offers a lot of functionality, so I am not sure wheter I am overlooking something.
So I would be happy about any help. Maybe it is impossible and I have to change the working directory manually - then I would be glad to know that as well.
Some links I used in my research:
https://www.tidyverse.org/blog/2017/12/workflow-vs-script/
https://malco.io/2018/11/05/why-should-i-use-the-here-package-when-i-m-already-using-projects/
http://jenrichmond.rbind.io/post/how-to-use-the-here-package/
Thanks a lot!
Edit: My current implementation is an additional R script, where I manually set the working directory via setwd() and source the main.R file. However it is always suggested to avoid setwd, which is why this whole question exists.

Sourcing in Rprofile.site: How to find the location of the sourced file

I have a "copy deployed" installation of R with no R-specific environment variables set.
I want to source a file from within Rprofile.site:
source("how/to/find/this/file/my_settings.R")
Where could I place my my_settings.R file so that it can be found from within Rprofile.site no matter in which path Rprofile.site is installed?
BTW: I want to avoid an absolute path to my_settings.R this would work indeed. I'd prefer to use the same folder as Rprofile.site or a path relative to it to support copy deployment of R.
Edit 1: The problem is that getwd is always different depending on the current folder from which you start R
If you can't use absolute paths and that your working directory is not stable, one way is to use .libPaths or .Library.
By default your Rprofile should be in directory paste0(.Library,"/../etc/") or paste0(.libPaths()[2],"/etc/") so you can put your file there and source it with :
source(paste0(.Library,"/../etc/my_settings.R"))
source(paste0(.libPaths()[2],"/etc/my_settings.R"))
As far as I understand the first option is stable (I don't think one can change the value of .Library).
If you use the second option just make sure that if in the future you alter your .libPaths() you do it after sourcing your file.
See ?.libPaths for more info on default folders.

How to change working directory in Julia Jupyter?

How to change working directory in Julia Jupyter?
I tried and read everything, still have no idea how to do that. It only allows me to select directories under my home ~/ dir. I can't find any button to go up to /.
I'm pretty sure once you have started the server you cannot then go up directories, I may be wrong though.
So best thing to do is start the jupyter notebook server somewhere that contains all of the folders you might need -i.e. the root dir if you want to make sure you have access to everything.
You can use the --notebook-dir flag for this. Or you can set defaults in the config.
you need to create the config file, using cmd :
jupyter notebook --generate -config
Then, search for C:\Users\your_username\.jupyter folder (Search for that folder), and right click edit the jupyter_notebook_config.py.
Then, Ctrl+F: #c.NotebookApp.notebook_dir ='' . Note that the quotes are single quotes. Select your directory you want to have as home for your jupyter, and copy it with Ctrl+C, for example: C:\Users\username\JuliaProjects.
Then on that line, paste it like this : c.NotebookApp.notebook_dir = 'C:\\Users\\username\\JuliaProjects'
Make sure to remove #, as it is as comment.
Make sure to double slash \\ on each name of your path.
Ctrl+S to save the config.py file !!!
Go back to your cmd and run jupyter notebook. It should be in your directory of choice. Test it by making a folder and watch your directory from your computer.
I use Jupyter Lab and start it from the Julia REPL (1.4) like this:
using IJulia
jupyterlab(dir=pwd(), detached=true)

How can I set the latex path for sweave in R?

I would like to know how I can set the pdflatex path in R to use sweave. Because I have 2 different MikTeX installations and one is working properly. Please take into account that I am using R (RStudio) in Windows. I found some suggestion however, for Linux or Unix users.
thanks in advance
If you have multiple installs of LaTeX (i.e. MikTeX) and you want to use a specific one of these, then you need to make sure that R finds the one you need first. This means that you have to add the location of your preferred version of pdfLaTeX at the front of your PATH system environment variable.
If you do not have administrator rights in Windows, then you can use R's environment file to change the PATH variable for R only. See ?Startup in R for details on this process. Follow the following steps:
in R, check the output of Sys.getenv("R_ENVIRON"). This will return the full path to an existing environment file, but will be empty in most cases. If a file exists, skip to step 3 below.
if no path is returned in step 1, create a file Renviron.site in the folder R_HOME/etc where R_HOME is the path returned by Sys.getenv("R_HOME").
add a line to the environment file as follows: PATH=C:\\full\\path\\to\\the\\folder\\with\\pdflatex;"${PATH}" (the quotations marks are important)
restart R and check Sys.getenv("PATH") and Sys.which("pdflatex") returns the correct paths.
If you do not have rights to create a Renviron.site file in R_HOME\etc, then you can also create a .Renviron file in HOME (Sys.getenv("HOME")).

Get changed file path via Grunt and use it to pass as an argument for another task

It's interesting how can I get file path which file has changed via Grunt. Then use last changed file as argument for another task.
Actually I want to get file path and send it to grunt-shell to do some x [filename] command.
I've done working solution with this gruntfile: https://gist.github.com/nikoloza/515f4d4cac656cbe2594
Someone may find it useful.

Resources