I've been trying to use GAS's URLFetchApp to connect to an external website. However, the website redirects me to a "select country" page with a HTTP 302 error no matter what web address I use.
var options = {
'muteHttpExceptions':true
};
var response = UrlFetchApp.fetch("http://www.idtdna.com/site/account", options);
Logger.log(response.getAllHeaders());
Logger.log(response.getContentText());
var cookie = response.getAllHeaders()['Set-Cookie']+', CN=US&DateSet=7/28/2015 5:48:55 PM; Path=/; Domain=www.idtdna.com';
Logger.log(cookie);
var header = {'Cookie':cookie};
var o2 = {
'headers':header,
'muteHttpExceptions':true
};
var r2 = UrlFetchApp.fetch("https://www.idtdna.com/country.aspx", o2);
Logger.log(r2.getAllHeaders());
Logger.log(r2.getContentText());
Here is my code. I've been trying to send the cookie but either I've formatted it wrong or am sending it incorrectly because I keep getting redirected regardless.
After setting the muteHttpExceptions option to 'true' and further investigating the website you mentioned, I think it all happens because of the cookies for the website.
So when we first access the website, there is no valid cookie. SO it creates a session and sets the cookie. THis might store the session details with the country as well.
You can try setting the cookie while fetching the url by passing the cookie values in the request header. Check this page for details on how to send the cookie through header.
Hope that helps!
Related
I know this question has been asked a bunch of times, but I tried most of the answers and still can't get it to work.
I have a Golang API with net/http package and a JS frontend. I have a function
func SetCookie(w *http.ResponseWriter, email string) string {
val := uuid.NewString()
http.SetCookie(*w, &http.Cookie{
Name: "goCookie",
Value: val,
Path: "/",
})
return val
}
This function is called when the user logs in, and I expect it to be sent to all the other endpoints. This works as expected with Postman. However, when it comes to the browser, I can't seem to get it to remember the cookie or even send it to other endpoints.
An example of JS using an endpoint
async function getDataWithQuery(query, schema){
let raw = `{"query":"${query}", "schema":"${schema}"}`;
let requestOptions = {
method: 'POST',
body: raw,
redirect: 'follow',
};
try{
let dataJson = await fetch("http://localhost:8080/query/", requestOptions)
data = await dataJson.json();
}catch(error){
console.log(error);
}
return data;
}
I tried answers like setting SameSite attribute in Golang, or using credential: "include" in JS with no luck.
Thanks to the discussion in the comments, I found some hints about the problem.
Saving cookies (both API and frontend on the same host)
I used document.cookie to save the cookie. I set the options by hand since calling res.cookie on the response of the API fetch only returned the value. An example is document.cookie = `goCookie=${res.cookie}; path=/; domain=localhost;.
Sending cookies
This has been answered before in previous questions and answered again in the comments. The problem was that I used credential:'include' instead of the correct credentials:'include' (plural).
CORS and cookies
In case the API and the frontend are not on the same host you will have to modify both the API and the frontend.
frontend
The cookie has to have the domain of the API since it's the API that requires it, not the frontend. So, for security reasons, you can't set a cookie for a domain (API) from another domain (frontend). A solution would be redirect the user to an API endpoint that returns Set-Cookie header in the response header. This solution signals the browser to register that cookie with the domain attached to it (the API's domain, since the API sent it).
Also, you still need to include credentials:'include' in the frontend.
API
You will need to set a few headers. The ones I set are
w.Header().Set("Access-Control-Allow-Origin", frontendOrigin)
w.Header().Set("Access-Control-Allow-Credentials", "true")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type, withCredentials")
w.Header().Set("Access-Control-Allow-Methods", method) // use the endpoint's method: POST, GET, OPTIONS
You need to expose the endpoint where the frontend will redirect the user and set the cookie in the response. Instead of setting the domain of the API by hand, you can omit it, the browser will fill it with the domain automatically.
To handle the CORS and let JS send the cookie successfully, you will have to set the SameSite=None and Secure attributes in the cookie and serve the API over https (I used ngrok to make it simple).
Like so
func SetCookie(w *http.ResponseWriter, email string) string {
val := uuid.NewString()
http.SetCookie(*w, &http.Cookie{
Name: "goCookie",
Value: val,
SameSite: http.SameSiteNoneMode,
Secure: true,
Path: "/",
})
// rest of the code
}
I recommend you also read the difference between using localStorage and document.cookie, it was one of the problems I had.
Hope this helps.
I'm using Twilio API for getting the Twilio call logs. I want the recording for the corresponding call in .mp3 format. We are accessing recording URLs, but that is the .json format, so we replaced .json with .mp3 and added https://api.twilio.com at the beginning using the following codes
var recordings = RecordingResource.Read(
callSid: sid,
limit: 1
);,
foreach (var item in recordings)
{
recordUrl = "https://api.twilio.com" + item.Uri.Replace(".json", ".mp3");
}
But we can't play the recorded call.
We doubt in Enforce HTTP Auth on Media URLs currently its status is Enabled. If we disable it, is there any security issue? Are we able to play the audio after disabling it? What should Enforce HTTP Auth status on Media URLs if we access Twilio from the mobile app?
Disable the "Enforce HTTP Auth on Media URLs" option so you can use the URL to access the file without authentication.
When the "Enforce HTTP Auth on Media URLs" option is disabled, any person who has the Account SID and the Recording SID will be able to access the .mp3 file. On the other hand, when the option is enabled, you might have to use your Account SID and Auth Token to get access to the recording file.
If you need to keep your recordings secure with the HTTP basic auth, I suggest enable the "Enforce HTTP Auth on Media URLs" option and download the .mp3 file making an HTTP request; here is a code example of how to make the request.
using RestSharp;
using RestSharp.Authenticators;
using RestSharp.Extensions;
var client = new RestClient("https://api.twilio.com/2010-04-01/Accounts/ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/Recordings/RExxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.mp3");
var request = new RestRequest(Method.GET);
client.Authenticator = new HttpBasicAuthenticator("Account_SID", "Token_Auth");
IRestResponse response = client.Execute(request);
client.DownloadData(request).SaveAs("/your_path/recording_file.mp3");
i'm working with Pentaho Data Integration and i need a step which gives me a response header of a http request. What i would really need is an implementatione of the HEAD Method of a http call because i have to see the last-modified field of a resorce on internet. I tried using the REST Client step but it doesn't work. So i would like to write a script that makes everything i need.
I actually created a Modified Java Script Value and i put this code inside:
var req = new XMLHttpRequest();
req.open('GET', 'URL', false);
req.send(null);
var headers = req.getAllResponseHeaders().toLowerCase();
When i run it it doesn't gives me any error but there's anything in "headers".
Any suggestions? I would appreciate a different solution to
If the URL you're requesting is on a different domain then I can tell you that the server is not sending the correct headers to allow cross-domain resource sharing.
Access-Control-Allow-Origin: *
Without the server sending that header the only way to access any of the response (including headers) is if your script is running on the same domain (and protocol) or if you setup a proxy on your own server to get the data from the remote server.
If I open up the JS console on this page and run:
var req = new XMLHttpRequest();
req.open('GET', 'http://dati.toscana.it/it/storage/f/2012-07-26T160139/intoscana-arte-e-cultura.csv', false);
req.send(null);
console.log(req.getAllResponseHeaders());
I get the following output:
XMLHttpRequest cannot load http://dati.toscana.it/it/storage/f/2012-07-26T160139/intoscana-arte-e-cultura.csv. Origin http://stackoverflow.com is not allowed by Access-Control-Allow-Origin.
NetworkError: A network error occurred.
But if i put an alert in the end:
var req = new XMLHttpRequest();
req.open('GET', 'http://dati.toscana.it/it/storage/f/2012-07-26T160139/intoscana-arte-e-cultura.csv', false);
req.send(null);
var headers = req.getAllResponseHeaders().toLowerCase();
alert(headers);
it gives me the response.
It gives me this too:
Error: Failure
req.send(null);
I'm using Firebug.
I had a similar problem. In my case was to detect a redirection, something that could happens in services that uses third part storage (like Amazon S3). First time I tried with XMLHttpRequest but apparently that is not implemented in the JavaScript available in Kettle, just in browsers. To workaround this, I follow the instructions from http://type-exit.org/adventures-with-open-source-bi/2010/06/using-java-in-pentaho-kettle/ to reference another Java Class, mentioned in http://www.mkyong.com/java/how-to-get-http-response-header-in-java/, inside a Modified Javascript Value Step. The limitation is that you need to do a primary request to get the header, it is not a "just one request" solution.
//Get the location of a link that has a 301 permanent redirection
var pathurl=documentourl; //URL to lookup. It must be a valid address, otherwise will give error
var location; //final location in case of redirect
//fake import of the Java Classes
var HTTPRequest=javax.servlet.http.HttpServletRequest;
var url=java.net.URL;
//open a url connection
var conn= new url(pathurl).openConnection();
//if you yant to watch all the headers uncomment the following lines
//var headerobj= conn.getHeaderFields();
var status=conn.getResponseCode(); //get status code
if (status == 301){ //ie: permanent redirection
location=conn.getHeaderField("Location");
}
else{
//return the same url
location=pathurl;
}
I hope it helps to anyone with this problem!
I want to access html files which are protected by basic authentication.
I am trying to load this html page using htmlloader and sending the request by urlRequest.
I am using URLRequestDefaults class to set the credentials but when I see the request being send in the fiddler I don't see any authorization header being set by this.
Am I doing something wrong here. My code is below.
URLRequestDefaults.authenticate = false;
URLRequestDefaults.setLoginCredentialsForHost("www.xyz.com", "madhur", "sharma");
var req:URLRequest = new URLRequest("http://bazinga.xyz.com/MyHtml/index.html");
htmlControl.htmlLoader.load(urlRequest);
You should read the API more carefully:
Note that "example.com", "www.example.com", and "sales.example.com" are each considered unique hosts.
In this case, the domains don't match.
Has anyone suddenly encountered login errors from their users trying to connect to salesforce.com from a Flex app using as3salesforce.swc?
I get the following error... password removed to protect the innocent...
App Domain = null
Api Server name = na3.salesforce.com
_internalServerUrl = https://na3.salesforce.com/services/Soap/u/14.0
loading the policy file: https://na3.salesforce.com/services/Soap/cross-domain.xml
Your application must be running on a https server in order to use https to communicate with salesforce.com!
login with creds
loading the policy file: https://na3.salesforce.com/services/crossdomain.xml
Your application must be running on a https server in order to use https to communicate with salesforce.com!
invoke login
intServerUrl is null
intServerUrl = https://na3.salesforce.com/services/Soap/u/14.0
_invoke login
'5A5D3012-7717-E3C2-9B39-FFBBFF1F1B47' producer set destination to 'DefaultHTTPS'.
Method name is: login
'direct_http_channel' channel endpoint set to http://localhost/pm_server/pm/
'5A5D3012-7717-E3C2-9B39-FFBBFF1F1B47' producer sending message 'E32C7199-72C1-B258-B483-FFBC1641173D'
'direct_http_channel' channel sending message:
(mx.messaging.messages::HTTPRequestMessage)#0
body = "<se:Envelope xmlns:se="http://schemas.xmlsoap.org/soap/envelope/"><se:Header xmlns:sfns="urn:partner.soap.sforce.com"/><se:Body><login xmlns="urn:partner.soap.sforce.com" xmlns:ns1="sobject.partner.soap.sforce.com"><username>simon.palmer#dialectyx.com</username><password>******</password></login></se:Body></se:Envelope>"
clientId = (null)
contentType = "text/xml; charset=UTF-8"
destination = "DefaultHTTPS"
headers = (Object)#1
httpHeaders = (Object)#2
Accept = "text/xml"
SOAPAction = """"
X-Salesforce-No-500-SC = "true"
messageId = "E32C7199-72C1-B258-B483-FFBC1641173D"
method = "POST"
recordHeaders = false
timestamp = 0
timeToLive = 0
url = "https://na3.salesforce.com/services/Soap/u/14.0"
'5A5D3012-7717-E3C2-9B39-FFBBFF1F1B47' producer connected.
Method name is: login
Error: Ignoring policy file at https://na3.salesforce.com/crossdomain.xml due to meta-policy 'by-content-type'.
'5A5D3012-7717-E3C2-9B39-FFBBFF1F1B47' producer acknowledge of 'E32C7199-72C1-B258-B483-FFBC1641173D'.
responseType: Fault
Saleforce Soap Fault: sf:INVALID_LOGIN
INVALID_LOGIN: Invalid username, password, security token; or user locked out.
Comunication Error : sf:INVALID_LOGIN : INVALID_LOGIN: Invalid username, password, security token; or user locked out. : [object Object]
Obviously nobody else out there is building Flex apps on top of salesforce.com..
yippee, I'm first.
Anyhow, I just found out that this is a bug at salesforce.com as at 6th December 2008. The issue is that the scripts which handle login do not cope adequately with the redirect necessary because of load balancing on the salesforce.com servers.
It should be possible to go through the www front door of salesforce.com's api with a URL such as...
"https://www.salesforce.com/services/Soap/u/13.0";
where the 13 represents the version of their API you are targetting. However, all users are actually assigned to a specific server, so the front door should redirect the login request to the approriate place, and it doesn't if you are coming from Flex.
A workround is to specify your server in the URL, such as...
"https://na5.salesforce.com/services/Soap/u/13.0";
...which is what I was doing. That's fine if you are a single user accessing the same resources continually and your account remains attached to that server. However if...
You are distributing your app so anyone who has a salesforce.com enterprise account can log in OR
Your account gets moved because of some internal load balancing (which is what happened to me)
then the approach of providing a fixed server won't work.
The bug (as far as I understand it) is that the www route doesn't adequately redirect to your host server. Last intelligence was that it will be fixed "soon".
I wish I could mark this as the answer...