I'm trying to write a simple script that fetches text from a webpage and processes that string. But, that website requires me to be logged in. I was successful in logging in to that website. This is how I logged in:
var payload = {"name1":"val1","name2":val2"};
var opt ={"payload":payload,"method":"post"};
var respose = UrlFetchApp.fetch("http://website.com/login",opt);
After logging in, the website places me in http://website.com/home. I checked response.getContentText() and I can confirm that I have been logged in successfully as it contains the text from http://website.com/home.
Now I need to get the contents of http://website.com/page and process it.
I first assumed the script can handle cookies by itself and proceeded with
var pagedata = UrlFetchApp.fetch("http://website.com/page);//Did not work
That obviously didnt work and pagedata.getContentText() says me to login first, which indicates cookies were not successfully passed..
I then tried to extract cookies which the server responded during login and to send it along with this request.
var cookie = response.getAllHeaders()['Set-Cookie'];
// variable cookie now contains a legitimate cookie.
// It contains 'JSESSIONID=blabla;Path=/' and
// it is the ONLY cookie that server responds.
I tried to send that cookie in my page request.
var header = {'Cookie':cookie};
var opt2 = {"header":header};
var pagedata = UrlFetchApp.fetch("http://website.com/page",opt2);
I think even now cookies were not properly sent, as the content again says me to login.
Am I passing cookies correctly? I need help regarding the correct method of sending cookies in a request.
Here you can find cookies specification:
http://www.w3.org/Protocols/rfc2109/rfc2109
You have a potential issue in your code:
response.getAllHeaders()['Set-Cookie'] can return either a string or a table of string if multiple 'set-cookie' attributes are sent back from the server.
Eric is right, you cannot return the cookie without digesting it.
Second error in your code:
var opt2 = {"header":header};
should be
var opt2 = {"headers":header};
Be aware also that GAS uses Google IPs. It can happen that two consecutive fetch use different IPs.
The server your are connecting to may be session-IP dependant.
Are you sure the server only send you back one cookie after an authentification ?
It looks like you are setting the headers correctly in UrlFetchApp.fetch().
I believe that the data in the Set-Cookie header is in a different format than the data that is expected in Cookie header. For example, Set-Cookie contains information about expiration, etc.
Related
I'm trying to authenticate at a different domain as part of a Cypress test using cy.request. The authentication request needs to contain the value of the XSRF-TOKEN cookie as a header. This is easily solved when on the same baseURL as the authentication domain: visit the domain, read the cookie via cy.getCookie, then make the request. Since cypress does not allow visiting multiple domains in a single test a different workflow is needed.
My solution right now is to replace the initial cy.visit with a cy.reqeuest (this sets the cookie as verified by looking at the request headers of the second request). However, I can't figure out how to read this cookie before I make the second authentication request. cy.getCookies() is empty, and document.cookie is empty. The response of the cy.request only contains a "set-cookie" header the first time, and I can't figure out how to read the default headers of the cy.request since cy.intercept does not work with cy.request.
Sketch of my attempt (where c.value is null):
cy.request({url: "https://notbaseurl/login"}).then(res => {
cy.getCookie("XSRF-TOKEN").then(c => {
cy.request({url: "https://notbaseurl/auth", method: "POST", headers: {"X-XSRF-TOKEN": c.value}})
})
})
This is not currently possible. See https://github.com/cypress-io/cypress/issues/8956
I have the following rest endpoint that I would like to send a cookie along with my ResponseEntity. However after succesfully sending the response, the cookie is nowhere to be found.
#RequestMapping(value = "myPath", method = RequestMethod.POST)
public ResponseEntity<?> createToken(HttpServletResponse response)
final String token = "a1b2c3d4e";
Cookie cookie = new Cookie("token", token);
response.addCookie(cookie);
// Return the token
return ResponseEntity.ok(new MyCustomResponse(token));
}
MyCustomResponse
class MyCustomResponse {
private final String token;
public MyCustomResponse(String token) {
this.token = token;
}
}
I have also tried creating the ResponseEntity manually and setting the cookie in the headers with the "Set-Cookie" header but same thing, no cookie.
edit: I have confirmed that the Set-Cookie header is in fact present in the response, however it is not actually being stored in the browser. I am using a static web-app running in WebStorm to access my rest endpoint running on a different port. This web app is just using JQuery's $ajax method to call the REST endpoint, nothing fancy. When I run the web app in WebStorm I am able to see the cookie it creates in my browser, so that confirms my browser is allowing cookie storage.
I figured it out. My web application I am using to access my rest api is running on a different local port than my rest api. This causes the AJAX request to fail CORS requirements, thus the cookie doesnt actually get set.
I found the solution here What can cause a cookie not to be set on the client?
edit: I should add that it was adding the xhrFields snippet to JQuery's $ajax method that fixed it for me, the other parts weren't necessary.
(posting the answer below in case it gets deleted)
I think I found the solution. Since during development, my server is at "localhost:30002" and my web app at "localhost:8003", they are considered different hosts regarding CORS. Therefore, all my requests to the server are covered by CORS security rules, especially Requests with credentials. "Credentials" include cookies as noted on that link, so the returned cookie was not accepted because I did not pass
xhrFields: {
withCredentials: true
}
to jQuery's $.ajax function. I also have to pass that option to subsequent CORS requests in order to send the cookie.
I added the header Access-Control-Allow-Credentials: true on the server side and changed the Access-Control-Allow-Origin header from wildcard to http://localhost:8003 (port number is significant!). That solution now works for me and the cookie gets stored.
I am trying to login to a site, and then view user details.
The API documentation from the site is:
LogOn : All Calls with JSON and return JSON type
post - https://www.bit2c.co.il/Account/LogOn {UserName:'',Password:'',SecondFactor:[optional]}
return true if OK , error list of strings if not
Balance - GET https://www.bit2c.co.il/Account/Balance
return UserBalance as JSON
I've tried connecting to the site using
import requests
session=requests.session()
session.auth = ("username", "pass")
session.post("https//www.bit2c.co.il/Account/Balance")
but i am getting response 200 and the response content is "you must login".
What am I doing wrong ?
What kind of session? What is on server-side? Can you post the raw request/response pair(s)?
If PHP or ASP runs on server-side, you must capture the Set-Cookie header of the response to the login request, find the session cookie and in consequent requests you must set the Cookie header with the session cookie name and value captured previously.
I have been toiling over HttpURLConnection and setRequestProperty for 2 days so far and I cannot get this webpage to Post and return the page I desire. This is what I have so far...
...
String data = URLEncoder.encode("acctno", "UTF-8") + "=" + URLEncoder.encode("1991462", "UTF-8");
URL oracle = new URL("http://taxinquiry.princegeorgescountymd.gov");
HttpURLConnection yc = (HttpURLConnection) oracle.openConnection();
yc.setRequestMethod("POST");
yc.setRequestProperty("Content-Type", "text/html; charset=utf-8");
yc.setRequestProperty("Content-Length", "19004");
yc.setRequestProperty("Cache-Control", "private");
yc.setRequestProperty("Set-Cookie", "ASP.NET_SessionId=v5rdm145zv3jm545kdslgz55; path=/");
yc.setRequestProperty("X-AspNet-Version", "1.1.4322");
yc.setRequestProperty("X-Powered-By", "ASP.NET");
yc.setRequestProperty("Server", "Microsoft-IIS/6.0");
yc.setDoOutput(true);
yc.setDoInput(true);
OutputStreamWriter out = new OutputStreamWriter(yc.getOutputStream());
out.write(data);
out.flush();
//out.write(data);
out.close();
...
It returns the same page defined in URL. it doesn't send me the requested page which should have an ending /taxsummary.aspx
It looks as if the asp takes the post data and generates an HTML unique for each parameter given. How do I give it the correct parameters?
Your code looks fine. I believe it sends POST correctly. I think that the problem is not here. When you are using browser you first perform at least one HTTP GET to arrive to the form. When you are doing this the server creates HTTP session for you and returns its id in response header Set-Cookie. When you are submitting the form using browser it sends this header (Cookie) back, so the server can identify the session.
When you are working from java you are skipping the first phase (HTTP GET). So the first thing you are doing is POST while you do not have session yet. I do not know what is the logic of this ASP page but I think that it just rejects such requests.
So, first check this guess. You can use plugin to Firefox named LiveHttpHeaders. Install it and perform the operation manually. You will see all HTTP requests and responses. Save them. Check that the session ID is sent back from server to client. Now implement exactly the same in java.
BTW often the situation is event more complicated when server sends multiple redirect responses. Int this case you have to follow them. HttpConnection has method setFollowRedirects(). Call it with parameter true.
BTW2: Apache HttpClient is a perfect replacement to HttpConnection. it does everything and is very recommended when you are implementing such tasks.
That's all. Good luck. Sometimes it is not easy...
I want to post data to another server (JSP or ColdFusion).
Note: Post which means the data is required at the another server also the browser should be redirected automatically.
Is it better to use
form tag...input type hidden fields, values
...
and from javascript
form.submit();
or
HttpWebRequest myRequest =
(HttpWebRequest)WebRequest.Create("http://...");
myRequest.CookieContainer = new System.Net.CookieContainer(10000);
myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.ContentLength = data.Length;
Stream newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
return myRequest;
or
I should use WebClient class?
Please provide the points from Security view also.
If you want the browser to be correctly sent to the other server, then you should really do this client side - your second option will send the response from the remote server back down to the client, but any links in the HTML that are relative will appear broken, as the user will be attempting to request them from your server.
Also, making the request from the code-behind, you'll be sending the request from your server, without any of the client's cookies, headers, etc for that site (which you won't have access to).
The other issues to consider:
Client may have JavaScript disabled.
If the remote server supports SSL, then you should probably be posting to that.
Doing this client side, you'll be sending all form data to the client initially, and then sending it on to the 3rd party server.