I want to write an R script that will automatically send ESRI shapefiles into PostgreSQL using the shp2pgsql option in Bash.
# in Bash
shp2pgsql -I -s Port Shapefile schema.table | psql DB
# in R
Command<-c("-I","-s",Port,"shapefile","schema.table","|","psql","DB")
system2("shp2pgsql",Command,wait=FALSE)
/bin/sh: shp2pgsql: command not found
/bin/sh: psql: command not found
# Alternative in R using system
system("shp2pgsql -I -s 4326 shapefile schema.table | psql DB")
/bin/sh: shp2pgsql: command not found
/bin/sh: psql: command not found
*I understand there are other ways to achieve this result, and would appreciate answers in that regard. However, I am particularly interested in why the system2 method does not work.
Need to change to the path within R
# Find the correct directory
# Importantly, you do not want "locate psql".
# You want the directory psql is in, i.e., bin, not the full path to psql
Directory<-system("locate bin | grep /Applications/Postgres.app",inter=TRUE)[[1]]
# Create a command to redirect the path
Export<-paste("export PATH=",Directory,":$PATH",sep="")
# Create a command listing your shp2psql command
Command<-"shp2pgsql -I -s 4326 shapefile schema.table | psql DB"
# Join them using && to ensure that Export finishes before executing command
FinalCommand<-paste(Export,"&&",Command,sep=" ")
# Export it to your system
system(FinalCommand)
Related
I'm trying to execute the following code in R
system('"C:/Program Files/PostgreSQL/9.5/bin/raster2pgsql" -s 32630 -a -f raster Y:/Sen2R_Download/prueba_sergio/raster3/SCL/S2B2A_20180731_137_sen2r_SCL_10.tif sentinel > Y:/Sen2R_Download/prueba_sergio/rastersql27.sql')
But it throws an error
ERROR: Unable to read raster file: sentinel
But this error shouldn't happen and when I execute the same in cmd it works well
C:\Users\Public\Documents>"C:/Program Files/PostgreSQL/9.5/bin/raster2pgsql" -s 32630 -a -f raster Y:/Sen2R_Download/prueba_sergio/raster3/SCL/S2B2A_20180731_137_sen2r_SCL_10.tif sentinel > Y:/Sen2R_Download/prueba_sergio/rastersql27.sql
Processing 1/1: Y:/Sen2R_Download/prueba_sergio/raster3/SCL/S2B2A_20180731_137_sen2r_SCL_10.tif
What can I do to make it work with R?
Try running the same command with system2 instead.
See https://stat.ethz.ch/R-manual/R-devel/library/base/html/system.html for more details. Specifically the section regarding differences between Unix and windows
I have created below .sh file to run R code saved in separate .R file.
cat EE.sh
#!/bin/bash
VARIABLES=( 20190719 20190718 )
for i in ${VARIABLES[#]}; do
VARIABLENAME=$i
/usr/lib/R/bin/Rscript -e 'source("/home/EER.R")'
Basically what it is expected to do is, take the dates from VARIABLE and pass to the /home/EER.R file, and R will do execution based on passed date (after correct formatting)
Then I ran below code
sudo chmod a+rx EE.sh
and
sudo bash EE.sh
But I then get below error message.
sudo bash EE.sh
EE.sh: line 2: $'\r': command not found
EE.sh: line 3: $'\r': command not found
EE.sh: line 4: $'\r': command not found
Can anyone help me to resolve this issue.
I am using Ubuntu 18 with R version 3.4.4 (2018-03-15)
This problem looks to be related to carriage returns related(which come when we copy text from windows machine to unix machine), so to identify them use:
cat -v Input_file
If you see carriage returns in your file then try:
tr -d '\r' < Input_file > temp && mv temp Input_file
Once they are removed then try to run your program.
I need to issue unix commands from an R session. I'm on Windows R2 2012 server using RStudio 1.1.383 and R 3.4.3.
The shell() function looks to be the right one for me but when I specify the path to my bash shell (from Git for Windows install) the command fails with error code 127.
shell_path <- "C:\\Program Files\\Git\\git-bash.exe"
shell("ls -a", shell = shell_path)
## running command 'C:\Program Files\Git\git-bash.exe /c ls -a' had status 127'ls -a' execution failed with error code 127
Pretty sure my shell path is correct:
What am I doing wrong?
EDIT: for clarity I would like to pass any number of UNIX commands, I am just using ls -a for an example.
EDIT:
After some playing about 2018-03-09:
shell(cmd = "ls -a", shell = '"C:/Program Files/Git/bin/bash.exe"', intern = TRUE, flag = "-c")
The correct location of my bash.exe was at .../bin/bash.exe. This uses shell with intern = TRUE to return the output as an R object. Note the use of single quote marks around the shell path.
EDIT: 2018-03-09 21:40:46 UT
In RStudio we can also call bash using knitr and setting chunk options:
library(knitr)
```{bash my_bash_chunk, engine.path="C:\\Program Files\\Git\\bin\\bash.exe"}
# Using a call to unix shell
ls -a
```
Two things stand out here. Bash will return exit code 127 if a command is not found; you should try running the fully qualified command name.
I also see that your shell is being run with a /c flag. According to the documentation, the flag argument specifies "the switch to run a command under the shell" and it defaults to /c, but "if the shell is bash or tcsh or sh the default is changed to '-c'." Obviously this isn't happening for git-bash.exe.
Try these changes out:
shell_path <- "C:\\Program Files\\Git\\git-bash.exe"
shell("/bin/ls -a", shell = shell_path, flag = "-c")
Not on Windows, so can't be sure this will work.
Perhaps you need to use shQuote?
shell( paste("ls -a ", shQuote( shell_path) ) )
(Untested. I'm not on Windows. But do read ?shQuote))
If you just want to do ls -a, you can use the below commands:
shell("'ls -a'", shell="C:\\Git\\bin\\sh.exe")
#or
shell('C:\\Git\\bin\\sh.exe -c "ls -a"')
Let us know if the space in "Program Files" is causing problems.
And if you require login before you can call your command,
shell('C:\\Git\\bin\\sh.exe --login -c "ls -a"')
But if you are looking at performing git commands from R, the git2r by ropensci might suit your needs.
I am trying to find the cpu utilization of a process from top.So before that I had to test the below command
top -b -c -d1 -n2
I am using -c option to print the full command name as the process name gets truncated without -c.
No when I run this as nohup sh test.sh & ,the output nohup.out contains truncated process name and because of which I am not able to grep on the process name
159 neutron 30 0 127620 22765 5479 S 0.0 0.6 399:02.56 /usr/bin/p+t
But when I run this as sh test.sh & its printing the full command name to terminal.
Why is the full command name not printed in spite of using -c in batch mode for top command ?
Whats the difference between command name with -c enabled and process name ?
Or to phrase it the process name and command name are different and the process name is picked from /proc/pid/status by commands like ps or top?
We can set the COLUMNS environment variable before the top command to increase the available width.
COLUMNS=1000 top -b -c -d1 -n2
The other way would be is to use ps to find the pid's of the processes by their names and specify the format of ps output.This output can be used to feed top to get the cpu usage for the process based on pid.
ps -eo pid,comms,args
comms = command name only and not the args
args = full argument list used to launch the process
Apologies, as i have not tried this earlier.
Hi,
I need to create oozie workflow that exeuctes a shell script. The shell script has curl command which downloads a specific file from client's repo.
As commands in shell scripts are only able to recognize hdfs directories, how could i execute the script.?
Lets say below is the Sample code:
curl -o ~/test.jar http://central.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
hdfs dfs -copyFromLocal ~/test.jar /user/sr/test2
How can i execute the script with above two commands using oozie.?
I found the answer...
data=curl http://central.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.csv
echo "$data" | hdfs dfs -appendToFile - /path/to/hdfs/directory/PPP.csv