I am attempting to scrape information from a website that uses oAuth2. After logging-in, I am able to find the bearer token through developer tools and use the following request to get the relevant info
url = 'somewebsite.com/api/getAccounts'
headers["Accept"] = "application/json"
headers['Authorization'] = 'Bearer mytoken'
response = requests.get(url, headers = headers)
json_obj = json.loads(response.content)
Obviously, the token expires and I want to scrape the site on a regular basis. Is there a way to automatically retrieve the bearer token while only having username/password in my disposition?
Related
Trying to get an access token for an API so I can use R to pull the data. The documentation I'm using has the following instructions:
The following POST request shows the format of an Axon Evidence token request:
POST /api/oauth2/token HTTP/1.1
Host: api.evidence.com
Content-Type: application/x-www-form-urlencoded
grant_type: client_credentials&partner_id={partner ID}&client_id={client ID}&client_secret={client secret}
Note: ensure that you use URL-encoded forms for all parameter values. The client secret is base64 encoded and therefore usually end in an equals sign (=), which must be encoded as %3D. Additionally, if you use a manual REST client for testing, ensure that you know whether the client is automatically performing URL encoding. For more information, see Resources.
I'm trying to request a token using the httr package with the instructions above and it doesn't seem to be working. I'm using:
Token_Response <- POST("https://api.evidence.com/", body = "POST /api/oauth2/token HTTP/1.1
Host: api.evidence.com
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials&partner_id={partner ID}&client_id={client ID}&secret={client secret}")
Unfortunately I can't make this question reproducible for security reasons, but I'm hoping it's just a syntax problem on my end.
You don't usually work directly with the body or headers when using HTTR. The function can do most of the formatting of the request for you. A proper translation would look more like
url <- "https://api.evidence.com/"
partnerId <- "my partner ID"
clientId <- "my client ID"
clientSecret <- "my client secret"
token_Response <- POST(url,
encode="form",
body=list(
grant_type = "client_credentials",
partner_id = partnerId,
client_id = clientId,
secret = clientSecret
)
)
I used a few variables for clarity but you could put the values directly in the function call if you like.
I'm trying to get information about a bot using the bot token, like the way to get user, passing on the header Authorization: Bearer ${userToken}. I'm sending a request to the route GET https://discord.com/api/oauth2/applications/#me passing on the header Authorization: Bearer ${botToken}, and the response is 401. But if I send a request to the route GET https://discord.com/api/oauth2/users/#me with my token in the header, I got my data. I don't have sure if the bot token can be used for this, I check on the docs, and I found this https://discord.com/developers/docs/topics/oauth2#get-current-application-information, but I don't understand what I need to pass in the header to get the data.
Yes, and you're very close to the correct solution.
The issue is with the "Authorization" header, instead of "Bearer" you should use "Bot" when using a token for a bot user.
In your case the header should be:
Authorization: Bot ${botToken}
CURL example:
curl --location --request GET 'https://discord.com/api/oauth2/applications/#me' \
--header 'Authorization: Bot <BOT TOKEN HERE>'
Regarding the question about "https://discord.com/api/oauth2/users/#me", try using "https://discord.com/api/users/#me" instead.
I've been trying to plug into the Toggl API for a project, and their examples are all using CURL. I'm trying to use the C# wrapper which causes a bad request when trying to create a report, so I thought I'd use Postman to try a simple HTTP request.
I can't seem to get the HTTP request to accept my API Token though. Here's the example that they give (CURL):
curl -u my-secret-toggl-api-token:api_token -X GET "https://www.toggl.com/reports/api/v2/project/?page=1&user_agent=devteam#example.com&workspace_id=1&project_id=2"
I've tried the following HTTP request with Postman with a header called
api_token with my token as the value:
https://www.toggl.com/reports/api/v2/project/?user_agent=MYEMAIL#EMAIL.COM&project_id=9001&workspace_id=9001
(changed ids and email of course).
Any help on how to use the CURL -u in HTTP would be appreciated, thanks.
The easy way is adding credential into url as user:pass# format.
https://my-secret-toggl-api-token:api_token#www.toggl.com/reports/api/v2/project/?page=...
<---------------------------------->
Alternately you can use credential with your http header like below:
Authorization: Basic XXXXXX
Here XXXXXX is base64(my-secret-toggl-api-token:api_token)
As explained to me in another post, you can pass the api token to the user property if you are using HttpWebRequest:
request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes($"my-secret-toggl-api-token:api_token")));
Using fetch
In node environment, your request might look something like so:
const url = 'https://www.toggl.com/reports/api/v2/project/page=1&user_agent=devteam#example.com&workspace_id=1&project_id=2';
const token = 'my-secret-toggl-api-token:api_token';
fetch(url, {
method: 'GET', // not required
headers: {
Authorization: `Basic ${Buffer.from(String(token)).toString('base64')}`,
},
})
I have a web app hosted on IIS 7 that is doing Http calls using the Google Fit API, I'm able to successfully send a POST and retrieve an access token, after which I do a GET for the following uri:
"https://www.googleapis.com/fitness/v1/users/me/dataSources/raw:com.google.weight:com.google.android.apps.fitness:user_input/datasets/00-1427209862000000000"
Here's how I build a request and look at the response:
request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "GET";
request.ContentLength = 0;
request.Accept = "application/json";
request.Headers.Add("Authorization", "Bearer " + dict["access_token"]);
response = (HttpWebResponse)request.GetResponse();
respStream = response.GetResponseStream();
sResponse = new StreamReader(respStream).ReadToEnd();
respStream.Close();
Response.Write(sResponse);
When I run this app on a browser on the host server, I successfully get a json object (it isn't the json I expect, but that's another issue). However, when I try to access the site on a remote client, I get a 403 error pointing to when I try to retrieve the response. Any ideas?
It probably depends how you got the access_token value.
How long after retrieving the access token are you making this request? It's only valid for an hour, so you may need to fetch a new one using the refresh token.
There's some more resources on access/refresh tokens this question:
Google OAuth2 Refresh_token expires when Access_token does
Update: This turned out to be a permissions issue. I was trying to access a data source that I didn't have scopes for, the fact that it didn't return a 403 when accessing on the host threw me off the trail. Still strange that it didn't return a 403 (it just returned an empty json object) when requested on the host server though...if you see this and have an idea why, feel free to comment. I'm curious.
Heloo
I am trying to get the response url from the backend. i am able to get request url, but not able to get response URL. Any suggestions/ ideas are mush appreciated.Thanks in advance.
Assuming your backend sends a redirect URL (as HTTP 302 Location Header) and the URL contains an auth code that you want to extract in the Apigee proxy-flow response path -
You can use a java script policy in the Response path of your proxy flow, like below:
url = context.getVariable("response.headers.Location");
var re = new RegExp("#.*[?&]" + "code" + "=([^&]+)(&|$)");
var code = url.match(re);
context.setVariable("authcode", code);