I have an internal repository on Gitlab.
In my R script I want to source an .R file from that internal repository.
Normally I can source a publicly available R script with the following code
source("public-path-to-file")
But When I try to do that with a private file I get:
Error in source("") :
:1:1: unexpected '<'
1: <
^
I found a hacky way to do it that at least gets things done:
First you need to create a private token with api access. Then you can call gitlab's API directly to GET the file
Code:
cmd <- "curl -s --header 'PRIVATE-TOKEN: <your private token for gitlab with api access>' '<full gitlab link of the raw file that you want to source>'" # you directly access the API with your private token
output <- system(cmd, intern=TRUE) # capturing the output of the system call
writeLines(output, "functions.R") # creating a temporary file which contains the output of the API call
source("functions.R") # Sourcing the file the usual way
file.remove("functions.R") # removing the temporary file for housekeeping
Explanation:
We call the API directly with a GET request using the system command curl.
Then you call the system command through R, capturing the result.
And then you can create a temporary file with the respective contents and source it as you would normally do. Hope this may help someone
Related
So I have a collection on json files located on my local machine that I am reading in currently using the command
file <- tbl_df(ndjson::stream_in("path/to/file.json")
I have copied these files to a linux server (using WinSCP) and I want to stream them in to my R session just like I did in the above code with ndjson. When searching for ways to do this I came across one method using RCurl that looked like this
file <- scp(host = "hostname", "path/to/file.json", "pass", "user")
but that returned an error
Error in function (type, msg, asError = TRUE) : Authentication failure
but either way I want to avoid copying my passphrase into my Rscript as other will see this script. I also came across a method suggesting this
d <- read.table(pipe('ssh -l user host "cat path/to/file.json"'))
however this command returned the error
no lines available in input
and I believe read.table would cause me issues anyways. Does anyone know I way I could read new line delimited json files from a remote server into an R session? Thank you in advance! Let me know if I can make my question more clear.
I am trying to use lua to access redis values from nginx. When i execute lua files on command line there everything is ok i am able to read and write values to redis. But i when try to execute the same files from nginx by accessing a location in which access_by_lua directive is written the following error logged in error log file
no field package.preload['socket']
no file '/home/sivag/redis/redis-lua/src/socket.lua'
no file 'src/socket.lua'
no file '/home/sivag/lua/socket.lua'
no file '/opt/openresty/lualib/socket.so'
no file './socket.so'
no file '/usr/local/lib/lua/5.1/socket.so'
no file '/opt/openresty/luajit/lib/lua/5.1/socket.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
What is the reason for this and how can i resolve this?
In my case I just needed to install the lua-socket package, as the socket library is not built into the default Lua installation like it is in some other languages.
You get this error because your code executes the command require("socket")
This command will search for a file with that name in several directories. If successful the content will be executed as Lua code. If it is not successful you'll end up with your error message.
In order to fix this you have to add the path containing the file either to the system variable LUA_PATH or you have to add it to the global table package.path befor you require the file.
Lua will replace ? with the name you give to require()
For example
package.path = package.path .. ";" .. thisPathContainsTheLuaFile .. "?.lua"
Please read:
http://www.lua.org/manual/5.3/manual.html#pdf-require
https://www.lua.org/pil/8.1.html
I'm trying to convert word documents to PDF, via the commandline using unoconv via PHP.
I'm using the Symfony Process Component to run the command via the command line.
public function run()
{
$cmd = 'unoconv --listener & unoconv ' . $this->path;
//Tested this to check for permissions and this worked.
//$cmd = 'touch /vagrant/public/testfile.pdf';
$process = new Process($cmd);
$process->run();
return $process->getOutput();
}
This yields no output, and doesn't convert the file. However if I echo the $cmd and paste it into the CLI it converts the file as expected and logs output as it goes.
Any ideas what could be the problem?
Edit:
I've since tried: calling mustRun() & start() methods on the symfony class.
mustRun() gives the following error:
"The command '//command//' failed. Exit Code: 251(Unknown error) Output: ================ Error Output: ================
After adding the log code as suggested by Diego Ferri, I get Error: Unable to connect or start own listener. Aborting. in the log file; but I can't find much online that's helpful for that.
Please read this section but also check the troubleshooting section.
It is possible that the shell is missing some important environment variables for unoconv/LibreOffice to work properly (PATH, HOME, ...). And it is recommended you call the LibreOffice python binary with unoconv instead of leaving it up to unoconv to determine the location of LibreOffice and python.
Okay - so here is what I'm trying to do.
I've got this password protected CSV file I'm trying to import into R.
I can import it fine using:
read.csv()
and when I run my code in RStudio everything works perfect.
However, when I try and run my .R file using a batch file (windows .bat) it doesn't work. I want to use the .BAT file so that I can set up a scheduled task to run my code every morning.
Here is my .BAT file:
"E:\R-3.0.2\bin\x64\R.exe" CMD BATCH "E:\Control Files\download_data.R" "E:\Control Files\DailyEmail.txt"
And here is my .R file:
url <- "http://username:password#www.url.csv"
data <- read.csv(url, skip=1)
** note, I've put my username/password and the exact location of the CSV in my code. I've used generic stuff here, as this is work related and posting usernames and passwords is probably frowned upon.
As I've said, this code works fine when I use it in RStudio. But fails when I use the .BAT file.
I get the following error message:
Error in download.file(url, "E:/data/data.csv") :
cannot open URL 'websiteurl'
In addition: Warning message:
In download.file(url, "E:/data/data.csv") :
unable to resolve 'username'
Execution halted
** above websiteurl is the http above (I can't post links)
So obviously, the .BAT is having trouble with the username/password? Any thoughts?
* EDIT *
I've gone so far as trying this on Linux. Thinking maybe windows was playing silly bugger.
Just from the terminal, I run Rscript -e "download_data.r" and get the EXACT same error message as I did in Windows. So I suspect this may be a problem with where I'm getting the data? Could the provider be blocking data from the command line, but not from with Rstudio?
I have had similar problems which had to do with file permissions. The .bat file somehow does not have the same privileges as you running the code directly from Rstudio. Try using rscript (http://stat.ethz.ch/R-manual/R-devel/library/utils/html/Rscript.html) within your .bat file like
Rscript "E:\Control Files\download_data.R"
What is the purpose of the argument "E:\Control Files\DailyEmail.txt"? Is the program suppose to use it in any way?
So, I've found a solution, which is likely not the most practical for most people, but works for me.
What I did was migrated my project over to a Linux system. Running daily scripts, is easier on Linux anyways.
The solution makes use of the "wget" function in linux.
You can either run the wget right in your shell script, or make use of the system() function in R to run the wget.
code looks like:
wget -O /home/user/.../file.csv --user=userid --password='password' http://www.url.com/file.csv
And you can do something like:
syscomand >- "wget -O /home/.../file.csv --user=userid --password='password' http://www.url.com/file.csv"
system (syscommand)
in R to download the CSV to a location on your hard drive, then grab the CSV using read.csv()
Doing it this way gave me some more insight into the potential root cause of the problem. While the system(syscommand) is running, I get the following output:
Connecting to www.website.com (www.website.com)|ip.ad.re.ss|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Reusing existing connection to www.weburl.com:80.
HTTP request sent, awaiting response... 200 OK
Not sure why it has to send the request twice? And why I'm getting a 401 Unauthorized the first try?
I need some Powershell advice.
I need to install an application's MSP update file on multiple Win08r2 servers. If I run these commands locally, within the target machine's PS window, it does exactly what I want it to:
$command = 'msiexec.exe /p "c:\test\My Application Update 01.msp" REBOOTPROMPT=S /qb!'
invoke-wmimethod -path win32_process -name create -argumentlist $command
The file being executed is located on the target machine
If I remotely connect to the machine, and execute the two commands, it opens two x64 msiexec.exe process, and one msiexec.exe *32 process, and just sits there.
If I restart the server, it doesn't show that the update was installed, so I don't think it's a timing thing.
I've tried creating and remotely executing a PS1 file with the two lines, but that seems to do the same thing.
If anyone has advice on getting my MSP update installed remotely, I'd be all ears.
I think I've included all the information I have, but if something is missing, please ask questions, and I'll fill in any blanks.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
My process for this is:
Read a CSV for server name and Administrator password
Create a credential with the password
Create a new session using the machine name and credential
Create a temporary folder to hold my update MSP file
Call a PS1 file that downloads the update file to the target server
>>> Creates a new System.Net.WebClient object
>>> Uses that web client object to download from the source to the location on the target server
Call another PS1 file that applies the patch that was just downloaded –>> This is where I’m having issues.
>>> Set the variable shown above
>>> Execute the file specified in the variable
Close the session to the target server
Move to the next server in the CSV…
If I open a PS window and manually set the variable, then execute it (as shown above in the two lines of code), it works fine. If I create a PS1 file on the target server, containing the same two lines of code, then right click > ‘Run With PowerShell’ it works as expected / desired. If I remotely execute my code in PowerGUI, it returns a block of text that looks like this, then just sits there. RDP’d into the server, the installer never launches. My understanding of the “Return Value” value is that “0″ means the command was successful.
PSComputerName : xx.xx.xx.xx
RunspaceId : bf6f4a39-2338-4996-b75b-bjf5ef01ecaa
PSShowComputerName : True
__GENUS : 2
__CLASS : __PARAMETERS
__SUPERCLASS :
__DYNASTY : __PARAMETERS
__RELPATH :
__PROPERTY_COUNT : 2
__DERIVATION : {}
__SERVER :
__NAMESPACE :
__PATH :
ProcessId : 4808
ReturnValue : 0
I even added a line of code between the variable and the execution that creates a text file on the desktop, just to verify I was getting into my ‘executeFile’ file, and that text file does get created. It seems that it’s just not remotely executing my MSP.
Thank you in advance for your assistance!
Catt11.
Here's the strategy I used to embed an msp into a powershell script. It works perfectly for me.
$file = "z:\software\AcrobatUpdate.msp"
$silentArgs = "/passive"
$additionalInstallArgs = ""
Write-Debug "Running msiexec.exe /update $file $silentArgs"
$msiArgs = "/update `"$file`""
$msiArgs = "$msiArgs $silentArgs $additionalInstallArgs"
Start-Process -FilePath msiexec -ArgumentList $msiArgs -Wait
You probably don't need to use the variables if you don't want to, you could hardcode the values. I have this set up as a function to which I pass those arguments, but if this is more of a one-shot deal, it might be easier to hard-code the values.
Hope that helps!
using Start-Process for MSP package is not a good practice because some update package lockdown powershell libs and so you must use WMI call