I am currently trying to upload a CSV file to a proprietary service via httr::POST(). Unfortunately the Admins are not experienced in R and can give only little support.
This is an example on how it should look like in the command line:
curl -X POST --header 'Content-Type: multipart/form-data' \
-F file=#"member-1-test.csv.gz" 'https://some/api/endpoint'
So, in the following code I just try to stick with the example (and additionally provide a token).
> library(tidyverse)
> library(httr)
# Provide some test data with characters specifically quoted
> test_file <- tibble::tribble(
~keytype, ~key, ~action, ~segment,
6,"\"https://www.google.com\"", 0, 37372818,
6,"\"https://www.sport1.de\"" , 0, 37372818
)
> data.table::fwrite(test_file, "test.csv", quote = FALSE)
> file <- upload_file(path = "C:/R/projects/DefaultWorkingDirectory/test.csv")
> res <- POST(
url = "https://some/api/endpoint",
body = list(file = file),
add_headers(.headers = c('Content-Type' = "multipart/form-data", "Authorization" = token))
)
This gives me the follwing error:
> res
Response [https://some/api/endpoint]
Date: 2020-11-18 09:35
Status: 503
Content-Type: text/plain
Size: 95 B
> content(res, encoding = "UTF8")
"upstream connect error or disconnect/reset before headers. reset reason: connection termination"
Any help or guidance on how to move forward with this issue is very much appreciated. Thanks!
Directly after posting the question it was already solved by one of the Admins :)
The issue was that the encoding needs to be set to "multipart" and that a specific content type needs to be provided which is similar to JSON but needs to be added to the "Accept"-header field.
Here the answer for any future references:
> res <- POST(
url = "https://some/api/endpoint",
body = list(
file = upload_file("test.csv")
),
add_headers(c(
"Authorization" = token,
"Accept" = "specific_format_for_the_application"
)),
encode = "multipart",
verbose()
)
I am learning python, and these days I am trying to work with socket module. Below is the client side logic.
import socket
from threading import Thread
import ipaddress
lic_server_host = input("Please enter the hostname: ")
port = input("Please enter the License service port number: ")
client_socket = socket.socket()
client_socket.connect((lic_server_host, port))
def receive_data():
while True:
data = client_socket.recv(1000)
print(data.decode())
def sending_data():
while True:
user_input = input()
client_socket.sendall(user_input.encode())
t = Thread(target=receive_data)
t.start()
sending_data()
Here I am taking user's input as a hostname. However. The above porgram is not able to convert the hostname to integer. I am getting below error
client_socket.connect((lic_server_hostname, port))
TypeError: an integer is required (got type str)
I tried to use some python way to get rid of the issue by introducing for loop on the user's input as below
lic_server_host = input("Please enter the License server hostname: ")
for info in lic_server_hostname:
if info.strip():
n = int(info)
port = input("Please enter the License service port number: ")
client_socket = socket.socket()
client_socket.connect((n, port))
But now I get below error:
client_socket.connect((n, port))
TypeError: str, bytes or bytearray expected, not int
So based on the error I used str() function on "n". But when I do that I get below error:
n = int(info)
ValueError: invalid literal for int() with base 10: 'l'
I have also searched for the above error available on the internet but the solutions not helping me.
Please help me understand my mistake.
Thank you
input returns a string when connect requires port as int.
client_socket.connect((lic_server_host, int(port)))
I would like to send a mail using SMTPS in R. Currently, non of the available packages supports sending Mails via TLS (rmail & sendmaileR) or they have a hard to install Java dependency (mailr). I tried using curl and managed to send a mail using the following code snippet:
curl --url 'smtps://mail.server.com:465' --ssl-reqd --mail-from 'mail1#example.com' --mail-rcpt 'mail2#example.com' --upload-file mail.txt --user 'user:password'
Unfortunately, I could not translate that snippet into R using the brilliant curl package. While I managed to find all options, the curl statement crashes the R session every time. Furthermore, I was not able to add the mail.txt file to the request which I created in a temporary directory. Did someone manage sending mails using the curl package? Why does the program always crash? The goal should be to send mails on all platforms.
# input variables
to <- "mail1#example.com"
from <- Sys.getenv("MAIL_USER")
password <- Sys.getenv("MAIL_PASSWORD")
server <- Sys.getenv("MAIL_SERVER")
port <- 465
subject <- "Test Mail"
message <- c("Hi there!",
"This is a test message.",
"Cheers!")
# compose email body
header <- c(paste0('From: "', from, '" <', from, '>'),
paste0('To: "', to, '" <', to, '>'),
paste0('Subject: ', subject))
body <- c(header, "", message)
# create tmp file to save mail text
mail_file <- tempfile(pattern = "mail_", fileext = ".txt")
file_con <- file(mail_file)
writeLines(body, file_con)
close(file_con)
# define curl options
handle <- curl::new_handle()
curl::handle_setopt(handle = handle,
mail_from = from,
mail_rcpt = to,
use_ssl = TRUE,
port = port,
userpwd = paste(from, password, sep = ":"))
con <- curl::curl(url = server, handle = handle)
open(con, "r")
close(con)
# delete file
unlink(mail_file)
A password cannot be specified in unzip (utils) function. The other function I am aware of, getZip (Hmisc), only works for zip files containing one compressed file.
I would like to do something like this to unzip all the files in foo.zip in Windows 8:
unzip("foo.zip", password = "mypass")
I found this question very useful but saw that no formal answers were posted, so here goes:
First I installed 7z.
Then I added "C:\Program Files\7-Zip" to my environment path.
I tested that the 7z command was recognized from the command line.
I opened R and typed in system("7z x secure.7z -pPASSWORD") with the appropriate PASSWORD.
I have multiple zipped files and I'd rather not the password show in the source code or be stored in any text file, so I wrote the following script:
file_list <- list.files(path = ".", pattern = ".7z", all.files = T)
pw <- readline(prompt = "Enter the password: ")
for (file in file_list) {
sys_command <- paste0("7z ", "x ", file, " -p", pw)
system(sys_command)
}
which when sourced will prompt me to enter the password, and the zip files will be decompressed in a loop.
I found #Kim 's answer worked for me eventually but not first off. I thought I'd just add a few extra links/steps that helped me get there in the end.
Close and reopen R so that environment path is recognised
If you've already opened R when you do steps 1-3 you need to close and reload R for R to recognise the environment path for 7z. #wush978 's answer to this question r system doesn't work when trying 7zip was informative. I used Sys.getenv("PATH") to check that 7zip was included in the environment paths.
Step 4. I opened R and typed in system("7z x secure.7z -pPASSWORD") with the appropriate PASSWORD.
I actually found this didn't work so I modified it slightly following the instructions in this post which also explains how to specify an output directory https://stackoverflow.com/a/16098709/13678913.
If you have already extracted the files the system command prompts you to choose whether you want to replace the existing file with the file from the archive and provides options
(Y)es / (N)o / (A)lways / (S)kip all / A(u)to rename all / (Q)uit?
So the modified step 4 (Y allows replacement of files)
system("7z e -ooutput_dir secure.zip -pPASSWORD" Y)
Putting this altogether as a modified set of instructions
Install 7z.
Added "C:\Program Files\7-Zip\" to my environment path using menu options (instructions here https://www.opentechguides.com/how-to/article/windows-10/113/windows-10-set-path.html)
Closed and reopened R studio. Typed Sys.getenv("PATH") to check path to 7zip recognised in the environment (as per #wush978 's answer to question r system doesn't work when trying 7zip)
Typed in the console system("7z e -oC:/My Documents/output_dir secure.zip -pPASSWORD") with the appropriate PASSWORD (as per instructions here https://stackoverflow.com/a/16098709/13678913)
And here is a modified version of #Kim 's neat function (including specified output directory and check for existing files):
My main script
output_dir <- "C:/My Documents/output_dir " #space after directory name is important
zippedfiles_dir <- "C:/My Documents/zippedfiles_dir/"
file_list <- paste0(output_dir , zippedfiles_dir , list.files(path = zippedfiles_dir, pattern = ".zip", all.files = T))
source("unzip7z.R")
Code inside source file unzip7z.R
pw = readline(prompt = "Enter the password: ")
for (file in file_list) {
csvfile <- gsub("\\.zip", "\\.csv", gsub(".*? ", "", file)) #csvfile name (removes output_dir from 'file' and replaces .zip extension with .csv)
#check if csvfile already exists in output_dir, and if it does, replace it with archived version and if it doesn't exist, continue to extract.
if(file.exists(csvfile)) {
sys_command = paste0("7z ", "e -o", file, " -p", pw, " Y")
} else {
sys_command = paste0("7z ", "e -o", file, " -p", pw)
}
system(sys_command)
}
password <- "your password"
read.table(
text = system(paste0("unzip -p -P ", password, " yourfile.zip ", "yourfile.csv"),
intern = "TRUE"
), stringsAsFactors = FALSE, header = TRUE, sep = ","
)
password <- "your password"
system(
command = paste0("unzip -o -P ", password, " ", "yourfile.zip"),
wait = TRUE
)
I am attempting to work with Azure storage via the REST API in R. I'm using the package httr which overlays Curl.
Setup
You can use R-fiddle: http://www.r-fiddle.org/#/fiddle?id=vh8uqGmM
library(httr)
requestdate<-format(Sys.time(),"%a, %d %b %Y %H:%M:%S GMT")
url<-"https://preconstuff.blob.core.windows.net/pings?restype=container&comp=list"
sak<-"Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ=="
signaturestring<-paste0("GET",paste(rep("\n",12),collapse=""),
"x-ms-date:",requestdate,"
x-ms-version:2009-09-19
/preconstuff/pings
comp:list
restype:container")
headerstuff<-add_headers(Authorization=paste0("SharedKey preconstuff:",
RCurl::base64(digest::hmac(key=sak,
object=enc2utf8(signaturestring),
algo= "sha256"))),
`x-ms-date`=requestdate,
`x-ms-version`= "2009-09-19")
Trying to list blobs:
content(GET(url,config = headerstuff, verbose() ))
Error
Top level message
The MAC signature found in the HTTP request 'Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ==' is not the same as any computed signature.
Response content
[1] "<?xml version=\"1.0\" encoding=\"utf-8\"?><Error>
<Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:1ab26da5-0001-00dc-6ddb-15e35c000000\nTime:2015-03-26T17:51:42.7190620Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request 'NTM1ODZjMjhhZmMyZGM3NDM0YTFjZDgwNGE0ODVmMzVjNDhkNjBkNzk1ZjNkZjJjOTNlNjUxYTMwMjRhNzNlYw==' is not the same as any computed signature. Server used following string to sign:
'GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 26 Mar 2015 17:52:37 GMT\nx-ms-version:2009-09-19\n/preconstuff/pings\ncomp:list\nrestype:container'.
</AuthenticationErrorDetail></Error>"
Verbose output
-> GET /pings?restype=container&comp=list HTTP/1.1
-> User-Agent: curl/7.39.0 Rcurl/1.95.4.5 httr/0.6.1
-> Host: preconstuff.blob.core.windows.net
-> Accept-Encoding: gzip
-> Accept: application/json, text/xml, application/xml, */*
-> Authorization: SharedKey preconstuff:OTRhNTgzYmY3OTY3M2UzNjk3ODdjMzk3OWM3ZmU0OTA4MWU5NTE2OGYyZGU3YzRjNjQ1M2NkNzY0ZTcyZDRhYQ==
-> x-ms-date: Thu, 26 Mar 2015 17:56:27 GMT
-> x-ms-version: 2009-09-19
->
<- HTTP/1.1 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
<- Content-Length: 719
<- Content-Type: application/xml
<- Server: Microsoft-HTTPAPI/2.0
<- x-ms-request-id: 3d47770c-0001-0085-2313-6d466f000000
<- Date: Thu, 26 Mar 2015 17:56:27 GMT
<-
Resolving the error
Googling for this issue doesn't seem to yield a consistent cause, but it's likely due to bad formatting / request structure on my part. To that end I checked:
I've verified my key is correct (it's just c&p from portal)
I've made sure the date is correctly formatted
There was a recent documentDB SO that suggested it could be to be a clock skew issue and I do note that my x-ms-date is a second ahead of the Date in the response. I've tried sending a fixed value that was definitely in the past, but within the 15 mins tolerance. Didn't get a change in message.
Added encoding="Base64" in headerstuff further to an MSDN forum question but the same error message was returned
Further to #Serdar's answer, I incorporated the construction of a signature string (I've verified that this matches the one provided in the error message so it), then encoding in base64 a hmac-sha256 (using the secondary access key (sak) as the encryption key) version of the UTF8 converted signaturestring as the value to be used in the SharedKey authorisation.
Further to #Serdar's comment, the date used in the signature string and for the main request must be the same, so defined once and reused
Is there anything obviously wrong? Are there other things to check? Does the code work for others?
Looks like your problem is with the key. The string of the key you have provided is actually base64 encoded. You need to decode that to the raw vector before you use it to sign the request. For example:
url<-"https://preconstuff.blob.core.windows.net/pings?restype=container&comp=list"
sak<-"Q8HvUVJLBJK+wkrIEG6LlsfFo19iDjneTwJxX/KXSnUCtTjgyyhYnH/5azeqa1bluGD94EcPcSRyBy2W2A/fHQ=="
requestdate<-format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
signaturestring<-paste0("GET",paste(rep("\n",12),collapse=""),
"x-ms-date:",requestdate,"
x-ms-version:2009-09-19
/preconstuff/pings
comp:list
restype:container")
headerstuff<-add_headers(Authorization=paste0("SharedKey preconstuff:",
RCurl::base64(digest::hmac(key=RCurl::base64Decode(sak, mode="raw"),
object=enc2utf8(signaturestring),
algo= "sha256", raw=TRUE))),
`x-ms-date`=requestdate,
`x-ms-version`= "2009-09-19")
content(GET(url,config = headerstuff, verbose() ))
There are no more authentication errors this way, though no blobs are listed. Perhaps that's a different issue.
Also, I changed the way the date/time was created to more "safely" change the local time to GMT.
It looks like you are using the key of your account directly in the Authorization header. To authenticate a request, you must sign the request with the key for the account that is making the request and pass that signature as part of the request. Please see Authentication for the Azure Storage Services for more information on how to construct the Authorization header.
Please also note that the service returns StringToSign in the error response. So, what your code should have done is to apply the following formula to StringToSign="GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Wed, 25 Mar 2015 22:24:12 GMT\nx-ms-version:2014-02-14\n/preconstuff/pings\ncomp:list\nrestype:container" (without quotes):
Signature=Base64(HMAC-SHA256(AccountKey, UTF8(StringToSign)))
How the service calculates StringToSign is explained in detail in the link shared above.
Worked trough the example code by MrFlick above and to get it to work I had to change a few things.
The date string has to be set in an English locale, for example:
lct <- Sys.getlocale("LC_TIME")
Sys.setlocale("LC_TIME", "us")
requestdate <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
Sys.setlocale("LC_TIME", lct)
The 'signaturestring' should be formated with \n between parameters:
signaturestring <- paste0("GET", paste(rep("\n", 12), collapse=""),
"x-ms-date:", requestdate,
"\nx-ms-version:2009-09-19\n/preconstuff/pings\ncomp:list\nrestype:container")
EDIT: Following procedure works for me. Based on Steph Locke example.
library(httr)
library(RCurl)
azureBlobCall <- function(url, verb, key, requestBody=NULL, headers=NULL, ifMatch="", md5="") {
urlcomponents <- httr::parse_url(url)
account <- gsub(".blob.core.windows.net", "", urlcomponents$hostname, fixed = TRUE)
container <- urlcomponents$path
# get timestamp in us locale
lct <- Sys.getlocale("LC_TIME"); Sys.setlocale("LC_TIME", "us")
`x-ms-date` <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
Sys.setlocale("LC_TIME", lct)
# if requestBody exist get content length in bytes and content type
`Content-Length` <- ""; `Content-Type` <- ""
if(!is.null(requestBody)) {
if(class(requestBody) == "form_file") {
`Content-Length` <- (file.info(requestBody$path))$size
`Content-Type` <- requestBody$type
} else {
requestBody <- enc2utf8(as.character(requestBody))
`Content-Length` <- nchar(requestBody, "bytes")
`Content-Type` <- "text/plain; charset=UTF-8"
}
}
# combine timestamp and version headers with any input headers, order and create the CanonicalizedHeaders
headers <- setNames(c(`x-ms-date`, "2015-04-05", unlist(headers)),
c("x-ms-date", "x-ms-version", unclass(names(unlist(headers)))))
headers <- headers[order(names(headers))]
CanonicalizedHeaders <- paste(names(headers), headers, sep=":", collapse = "\n")
# create CanonicalizedResource headers and add any queries to it
if(!is.null(urlcomponents$query)) {
components <- setNames(unlist(urlcomponents$query), unclass(names(unlist(urlcomponents$query))))
componentstring <- paste0("\n", paste(names(components[order(names(components))]),
components[order(names(components))], sep=":", collapse = "\n"))
} else componentstring <- ""
CanonicalizedResource <- paste0("/",account,"/",container, componentstring)
# create the authorizationtoken
signaturestring <- paste0(verb, "\n\n\n", `Content-Length`, "\n", md5, "\n", `Content-Type`, "\n\n\n",
ifMatch, "\n\n\n\n", CanonicalizedHeaders, "\n", CanonicalizedResource)
requestspecificencodedkey <- RCurl::base64(
digest::hmac(key=RCurl::base64Decode(key, mode="raw"),
object=enc2utf8(signaturestring),
algo= "sha256", raw=TRUE)
)
authorizationtoken <- paste0("SharedKey ", account, ":", requestspecificencodedkey)
# make the call
headers_final <- add_headers(Authorization=authorizationtoken, headers, `Content-Type` = `Content-Type`)
call <- httr::VERB(verb=verb, url=url, config=headers_final, body=requestBody, verbose())
print("signaturestring");print(signaturestring);
print(headers_final); print(call)
return(content(call))
}
## Tests. Replace 'key' and 'accountName' with yours
key <- "YowThr***********RDw=="
# Creates a container named 'test'
azureBlobCall("https://accountName.blob.core.windows.net/test?restype=container", "PUT", key)
# Creates a blob named 'blob' under container 'test' with the content of "Hej världen!"
azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "PUT", key,
headers = c("x-ms-blob-type"="BlockBlob"), requestBody = "Hej världen!") #upload_file("blob.txt"))
# List all blob in the container 'test'
azureBlobCall("https://accountName.blob.core.windows.net/test?comp=list&restype=container", "GET", key)
# deletes the blobl named 'blob'
azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "DELETE", key)
# Creates a blob named 'blob' under container 'test' with and upload the file 'blob.txt'
azureBlobCall("https://accountName.blob.core.windows.net/test/blob", "PUT", key,
headers = c("x-ms-blob-type"="BlockBlob"), requestBody = upload_file("blob.txt"))
# deletes the container named 'test'
azureBlobCall("https://accountName.blob.core.windows.net/test?restype=container", "DELETE", key)