Have an issue with asp.net s3 Object Copying using the asp.net s3 library.
I'm attempting to run the following code :-
CopyObjectRequest copyObjectRequest = new CopyObjectRequest()
{
SourceBucket = Bucket,
SourceKey = s3FileKey,
DestinationBucket = Bucket,
DestinationKey = archiveS3FileKey
};
CopyObjectResponse copyObjectResponse = s3Client.CopyObject(copyObjectRequest);
The request will just hang on the s3Client.CopyObject(copyObjectRequest); line.
Using fiddler/wireshark i extract the http header seen below (proper bucket and auth keys removed)
PUT https://bucket-name.s3-ap-southeast-2.amazonaws.com/Archive/test.zip HTTP/1.1
x-amz-copy-source: /bucket-name/App/test.zip
x-amz-metadata-directive: COPY
User-Agent: MyDotNetConsoleApp
Content-Type: application/x-amz-json-1.0
x-amz-date: Mon, 30 Jun 2014 05:34:53 GMT
Authorization: AWS <authorization code>
Host: bucket-name.s3-ap-southeast-2.amazonaws.com
Transfer-Encoding: chunked
When I alter the request by removing the Transfer-Encoding: chunked the request runs fine. See below alteration to the s3 library from aws.
And the object will be copied to the new key
To get around this I added the following to the s3 Client library as a temporary measure, obviously though I would like to understand properly, if I'm not setting something in the client request.
(line 281 in AmazonWebClientService.cs)
if (state.WebRequest.Headers.Get("x-amz-copy-source") != null)
{
state.WebRequest.SendChunked = false;
}
// this was the existing line above added.
httpResponse = state.WebRequest.GetResponse() as HttpWebResponse;
I can delete object from the same s3Client object leading up to this copyobject, and also full permissions have been granted to this access key on this bucket.
Anyone have any idea what would cause this or how to get around it without altering the standard library supplied from aws ?
Related
I'm trying to start a simple server in Julia. (Julia 1.7, HTTP v0.9.16, Windows 10).
By copying the snippet in the HTTP docs, I've written
using HTTP
using Sockets
HTTP.serve(Sockets.localhost, 8000) do request::HTTP.Request
#show HTTP.header(request, "Content-Type")
#show HTTP.payload(request)
return HTTP.Response("Helllllo")
end
in a julia terminal.
When i navigate to http://localhost:8000/ in Chrome, it Does show a page with the word "Helllllo".
But, when i try to send GET requests to it, it doesn't answer. Things I tried:
Opening a new julia terminal and writing
using HTTP
r = HTTP.request("GET", "https://127.0.0.1:8000");
What happens is that this command hangs, instead of producing a r.
Going in Chrome console and writing
fetch("https://127.0.0.1:8000")
.then(res => res.json())
.then(console.log)
Again, no answer.
What am I doing wrong?
Thanks!
A couple different issues:
You're trying to read from a https address, after having set up a plain http server. Once you fix this, Julia is able to get a request-response:
julia> r = HTTP.request("GET", "http://127.0.0.1:8000")
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Helllllo"""
But it still fails in the browser (in Firefox - I assume it will be the same in Chrome) with
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://127.0.0.1:8000/. (Reason: CORS request did not succeed).
This is simply because 127.0.0.1 != localhost for browser, for security purposes (so it doesn't allow you to read from the "different" host at 127.0.0.1 while you're on localhost). If you instead do:
>> fetch("http://localhost:8000")
.then(console.log)
Promise { <state>: "pending" }
Response { type: "basic", url: "http://localhost:8000/", redirected: false, status: 200, ok: true, statusText: "OK", headers: Headers, body: ReadableStream, bodyUsed: false }
I have an issue using an R script as a data source in Microsoft PowerBi. I think this is fundermentally an issue with PowerBi, but in the short term I'll need to find a solution in R.
Essentially, PowerBi doesn't appear to be able to handle the messages that would be sent to the console if I was using R Studio.
Within the R script I'm using a REST API to request data from a URL. The JSON message that is received is converted into an R data frame. When using the script as a datasource in PowerBi, this only works if I set the verbose settings to FALSE i.e. if I was using R Studio no messages (in particular data in) are sent to the console.
response <- GET(<url>,
body = list(),
add_headers(.headers = c('<identity token>' = ID_to_use)),
verbose(data_out = FALSE,
data_in = FALSE,
info = FALSE,
ssl = FALSE),
encode = "json")
However, I do not have the option to switch off the incoming/outgoing JSON header messages (which is going to come back to bite!).
<< {"identity":" <token>"}
* Connection #54 to <host> left intact
No encoding supplied: defaulting to UTF-8.
-> GET <URL request> HTTP/1.1
-> Host: <host>
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> <Identity>: <Identity>
->
<- HTTP/1.1 200 OK
<- X-Session-Expiry: 3599
<- Content-Type: application/json
<- Transfer-Encoding: chunked
<- Date: Thu, 06 Aug 2020 16:14:26 GMT
<- Server: <Server>
<-
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
No encoding supplied: defaulting to UTF-8.
From R help
.
.
verbose() uses the following prefixes to distinguish between different components of the http messages:
* informative curl messages
-> headers sent (out)
>> data sent (out)
*> ssl data sent (out)
<- headers received (in)
<< data received (in)
<* ssl data received (in)
.
.
Switching the verbose settings to FALSE works for a single request, however, I need to put the request into a loop and keep requesting more data until the API gateway indicates there is no more data to be received. PowerBi appears to fail when in the script five or more request/replies are sent/received.
Just from observation, I assume this is to do with the JSON Header messages piling up.
I've tried a number of approaches but nothing seems to work: sink('NUL'), invisible(), capture.output().
Any help would be appreciated.
I found a hacky solution, which at least solved the problem I had in R, but not in PowerBi.
By writing a "wrapper" R script (see below) which calls my main script THE_SCRIPT.R using a shell command. THE_SCRIPT dumps out a CSV file, which I then read in the wrapper script:
#Required by PowerBi
library(mice)
#set the directory, between R and the shell it's a pain to deal with spaces in the directories and quotes
setwd("C:/Program Files/R/R-3.6.2/bin/")
system("Rscript.exe C:\\Users\\<USER>\\Documents\\THE_SCRIPT.R > Nul 2>&1")
A_DATA_TABLE <- read.csv("C:\\Users\\<USER>\\Documents\\THE_FILE.csv")
However, this still didn't resolve the issue when running it in PowerBi.
Note, I tried sink('Nul 2>&1') in R, didn't work.
I am trying to get data from Amazon MWS API using GetMatchingProductForId operation.
When I use Amazon MWS Scratchpad it works perfectly fine.
I am now trying to replicate the urls that are sent in the HTTP POST request but I get a Signature error message.
I need to understand how the url request should be structured.
Below is the detail of the request in Amazon MWS Scratchpad, I ANONYMIZED private identifiers but that is the only thing I changed:
HTTP POST
POST /Products/2011-10-01?AWSAccessKeyId=ANONYMIZED
&Action=GetMatchingProductForId
&SellerId=ANONYMIZED
&SignatureVersion=2
&Timestamp=2018-09-28T05%3A45%3A43Z
&Version=2011-10-01
&Signature=ANONYMIZED
&SignatureMethod=HmacSHA256
&MarketplaceId=A13V1IB3VIYZZH
&IdType=EAN
&IdList.Id.1=9781933988665 HTTP/1.1
Host: mws.amazonservices.fr
x-amazon-user-agent: AmazonJavascriptScratchpad/1.0 (Language=Javascript)
Content-Type: text/xml
String to Sign
POST
mws.amazonservices.fr
/Products/2011-10-01
AWSAccessKeyId=ANONYMIZED&Action=GetMatchingProductForId&IdList.Id.1=9781933988665&IdType=EAN&MarketplaceId=A13V1IB3VIYZZH&SellerId=ANONYMIZED&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2018-09-28T05%3A45%3A43Z&Version=2011-10-01
=======
Now my question is, (and let's imagine my signature was created correctly), from the HTTP POST, what should the request look like ?
Here is my attempt:
https://mws.amazonservices.fr/Products/2011-10-01?AWSAccessKeyId=ANONYMIZED&Action=GetMatchingProductForId&SellerId=ANONYMIZED&SignatureVersion=2&Timestamp=2018-09-28T05%3A52%3A33Z&Version=2011-10-01&Signature=ANONYMIZED&SignatureMethod=HmacSHA256&MarketplaceId=A13V1IB3VIYZZH&IdType=EAN&IdList.Id.1=9781933988665
But what about '\n' escape characters that are in the scratchpad ? and what about 'HTTP/1.1' at the end, should I include that also ?
Thanks for your help.
I don't have an MWS account so I can't test the following, but this is one way you can do it:
# set this to your python2 binary; you'll need to do
# pip2 install boto
# from a command-line before using this code
Sys.setenv("RETICULATE_PYTHON"="/usr/bin/python2.7")
library(reticulate)
boto_mws_connection <- import("boto.mws.connection")
con <- boto_mws_connection$MWSConnection(
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = AWS_SECRET
Merchant = MERCHANT_ID
)
con$get_matching_product_for_id(
MarketplaceId = "A13V1IB3VIYZZH",
IdType = "EAN",
IdList = c("9781933988665")
)
The HTTP/1.1 is usually created by your http client library. I am not familiar with R, but I googled and it seems there is a CURL package for R. CURL is the standard http library for a lot of languages including PHP. My PHP code to send an XML feed through curl looks like this:
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'https://mws.amazonservices.fr/Products/2011-10-01?.....your data and signature here...');
curl_setopt($ch,CURLOPT_POST, 1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_POSTFIELDS, $xmlcontent);
curl_setopt($ch,CURLOPT_HTTPHEADER, array(
"Content-Type: text/xml",
"Content-MD5: ".base64_encode(md5($xmlcontent,true)),
"x-amazon-user-agent: TestScript/0.01")
);
$result = curl_exec($ch);
curl_close($ch);
By looking at this, it seems to me this should be easily translatable to the R interface for CURL.
Novice Artifactory user here so please bear with.
Trying to use curl on linux to deploy a file in a repo and failing. trying this running on the same server that's serving artifactory.
% --> curl -u my_idsid -X PUT
"http://localhost:8081/artifactory/test_repo/test_artifact_linux_01" -T
test_aftifact_linux_01
Enter host password for user 'my_idsid':<I entered the pw die my_user here>
HTTP/1.1 100 Continue
HTTP/1.1 403 Forbidden
Server: Artifactory/5.4.6
X-Artifactory-Id: 3444ab9991d26041:29864758:15e2f4940fa:-8000
Content-Type: application/json;charset=ISO-8859-1
Content-Length: 65
Date: Tue, 05 Sep 2017 21:00:20 GMT
Connection: close
{
"errors" : [ {
"status" : 403,
"message" : ""
} ]
}
I was able to put (deploy) a file (an artifact) in this same repo from windows with the drag-drop method for that same user. So I'm thinking that this confirms several things...
1) the repo exists
2) the permissions to allow that user write access is ok
3) the server is up and running ok
And the pw is OK, because when I intentionally enter something wrong, it comes up with a 401 instead.
I looked in artifactory.log and it has this for my attempt...
2017-09-05 17:15:00,332 [http-nio-8081-exec-7] [WARN ]
(o.a.w.s.RequestUtils:155) - Request /test_repo/test_artifact_linux_01
should be a repo request and does not match any repo key
Does not match a repo key ?
I'm using the example here...
https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-Example-DeployinganArtifact
and plugging in the repo name right after "artifactory" (replacing "my_repository") in the path.
I have a strong suspicion that I'm goofing up the path name, but I don't know what's wrong. How does one determine the proper path to use ?
Thanks in advance for any help.
UPDATE:
My bad. The repo was actually named "test-repo", not "test_repo".
Worked fine once I made that change.
Sorry for the false alarm.
I'm using this Mandrill package:
https://atmospherejs.com/wylio/mandrill
When I try to send an email I get this error:
I20150423-22:09:09.078(-7)? ====== BEGIN MAIL #0 ======
I20150423-22:09:09.078(-7)? (Mail not sent; to enable sending, set the MAIL_URL environment variable.)
I20150423-22:09:09.079(-7)? MIME-Version: 1.0
I20150423-22:09:09.079(-7)? From: admin#example.com
I20150423-22:09:09.079(-7)? To: victor#example.com
I20150423-22:09:09.079(-7)? Subject: Something something
I20150423-22:09:09.079(-7)? Content-Type: text/html; charset=utf-8
I20150423-22:09:09.079(-7)? Content-Transfer-Encoding: quoted-printable
I20150423-22:09:09.079(-7)?
I20150423-22:09:09.079(-7)? Some kind of message content.
I20150423-22:09:09.079(-7)? ====== END MAIL #0 ======
So I go into the Meteor Shell and I get this:
> process.env.MAIL_URL
'smtp://MY_MANDRILL_EMAIL:MY_MANDRILL_API_KEY#smtp.mandrillapp.com:587/'
Uh, so the MAIL_URL variable IS being set....
Just to be sure, I put this in a mailer.js file under the server folder.
Meteor.startup(function () {
process.env.MAIL_URL = 'smtp://MY_MANDRILL_EMAIL:MY_MANDRILL_API_KEY#smtp.mandrillapp.com:587/';
});
The same error happens. This is just running everything in localhost.
EDIT - I got it working, but I don't know why I'm doing this:
I noticed through some console logging that the email was trying to be shot off before the environmental variable was being set, despite setting the environmental variable inside of Meteor.startup()
What I ended up doing was creating a mailer_variable.js file inside server/lib
process.env.MAIL_URL = 'smtp://MY_MANDRILL_EMAIL:MY_MANDRILL_API_KEY#smtp.mandrillapp.com:587';
console.log('this is the mail_url: ', process.env.MAIL_URL);
This works. But I'm not sure why. It certainly isn't what the Mandrill package author says I should be doing...
I believe this has to do with the load order of Meteor. I don't recommend setting environment variables in the startup script precisely for this reason.
The correct way to set this is on Linux/Mac is to start Meteor like this:
MAIL_URL="smtp://.../" meteor
or
export MAIL_URL="smtp://.../"
meteor
More details are documented here: http://meteorpedia.com/read/Environment_Variables