simulating login process with recaptcha (generating tokens) - web-scraping

I am trying to access a site that need login credentials and solving a recaptcha v2, I am using python requests library to do that.
I am using the anticaptcha api to solve the captcha, and I now have the "g-recaptcha-response", but when I get to the login request, using regular browser and network tab in the DevTools, I see the login request has payload with 2 extra tokens (beside other static data like username and password,..)
these tokens are:
__RequestVerificationToken
RecaptchaToken
__RequestVerificationToken (1yvl3JbNqqll88O3pg7-Qdn8_QC8hcQdUIyCLaAno7xNCg2ql-rycdgC75Xz1AKRlghiuOzE_XZOrJiBTUz25cLOf1M1)
RecaptchaToken (03AOLTBLT9huh3hlzUWneKS-LDVRr0U4_4N8OvsV2dAY6L-DduwTXTdHGFEaLJKzqyNW6w3SAFoKo_hQWq2qecRTsca0IrTYR0pExziAh4eWsDjCj-NdhwBnaVQwo97QaE6q0aKZAIMNx6efET9869Jv_x75KRZX9esLxVY0VT_Zo93ha-iX_4B7h1Si9aicVi59ldaCrCTCClb7SkAw7fYnf_0s1_uY1BfpkcylvHgoGeGXLUdJ91kn_wV0b2USCRkeFb8m9AAS5Cy6avDllc4cmZ8oDH5c9KedQ1kv3Tt03K0-q_4V3BRy0zPV25L6h7qt4PTqT9kRRIQfMI_AvoNbn78kD8mXAJNA)
what are those extra tokens? how I get them?
I searched in all process loaded while solving the captcha in the browser, no mention for these values or even the arguments names.

Related

Get Jwt Payload from Google Sign-In without default Google popup

I'm trying to use Google Sign-In from my web application (asp.net vb.net).
Following the tutorial that Google provides, and using Google API libraries for .NET this is an easy task.
https://developers.google.com/identity/sign-in/web/sign-in
Now the result of this approach is that I check on the client for a googleUser.getAuthResponse().id_token which is a JWT and send this to my server side code with ajax for validation and to get the payload with the Google unique user Id and other infos.
All this happens on the same page, with a popup from Google, triggered by the default button they provide in the tutorial.
What I'm trying to accomplish is to obtain the same JWT (id_token) without the popup, but actually issuing a redirect to Google, when the user clicks on a custom "Log in with Google" button.
Sadly all the example I found, even from Google itself, involve a much more complex interaction where you get a code from the server, that you then have to exchange for temporary and refresh tokens, and so on.
While the client side approach with the popup window they provide, gives you immediatly the JWT token in response, not that code to request the token, that you have to validate then (I do this with Google .Net APIs with GoogleJsonWebSignature.ValidateAsync(externalToken) and retrieve the payload that way).
Looking at the urls in the popup, what I noticed that differs from all the examples I found that serve you the "code" for token exchange, are those parameters: flowName=GeneralOAuthFlow and response_type=permission%20id_token
While the examples you find for server to server transaction all include calling Google with response_type=code
I tried tampering a bit with the popup url to let it open in a new full window, copying and modifying the url but with not much success. I'm redirected but without the id_token parameter.
Any hint would be much appreciated since I'm not able to find any documentation on response_type=permission%20id_token to query Google service.
Thanks in advance
For anyone interested... I found a solution. The url to redirect to is:
https://accounts.google.com/o/oauth2/auth?response_type=id_token&redirect_uri={0}&scope=https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile&client_id={1}&state={2}
Where
{0} is the redirect url on your server, registered inside google console for this client_id
{1} is your Google client_id
{2} is some querystring or variable you want back to your server when the redirect happens
It works... it gives you back directly the id_token that you can verify with Google .NET APi with GoogleJsonWebSignature.ValidateAsync(id_token) and get as a result a payload (you have the payload class in Google Api .NET as well).
Only issue is that when Google comes back to your redirect url with the id_token in the querystring, it uses hash (url fragment #) so nothing is passed to the server.
There are workarounds with js to get the value and send to the server with ajax or redirect to the same page replacing the hash with ? but this is very annoying.
I imagine there are serious security reason for google to do this but from a dev standpoint is really a pain.
Instead of all those hacks i resorted to the longer way requesting response_type=code instead of the id_token, which returns a canonical querystring with ?code=...
If anyone knows how to get beck the id_token without the hash in the url it would be great.

LinkedIn V2 Profile API not properly returning email

I am attempting to use the LinkedIn V2 Profile API to aid in signups for my website.
I have properly implemented the "Sign in with LinkedIn" button on my website, set up the OAuth 2.0 callback, and my server properly swaps the OAuth 2.0 Access Code for the user's Access Token.
To do this, I am using the Python linkedin_v2 library linked here.
I am then attempting to grab the user's profile, and obtain their first name, last name, and email, to store in my database as a method of signup. I have confirmed that my application requests r_emailaddress, r_liteprofile, and r_member_social accesses.
To obtain their profile, I have attempted the following methods:
Using the Python Library linked above to create an application with the users access token and request the profile as follows:
application = linkedin.LinkedInApplication(token=accessToken)
profile = application.get_profile()
Unfortunately, this method only returns the user's first name, last name, and id, even when I include selectors=['email-address'] as a parameter to the get_profile() function, as specified by the library documentation.
Sending a GET request using the Python requests library https://api.linkedin.com/v2/people/(id:{person ID}) with headers that include Authentication: Bearer {user access token}.
Unfortunately, this method results in a 403 (forbidden) error.
I am curious of the following things:
Why do I obtain a 403 when querying the Profile API using the request library, when the same access token works to query the API through the Python linkedin_v2 library?
Does anyone know of how to use the python_linkedin_v2 library to obtain an email address with the profile?
Does anyone know of a better library to use in order to accomplish my goal of obtaining profile information regarding users whom are logging into my product using linkedin?
Do I need any additional permissions in order to access my users' emails from LinkedIn?
Thank you so much for your help, and I look forward to discussing potential solutions with all of you.
-Rob

Meteor Restivus: keep user logged in if he goes to the main website

I have a Chrome extension that communicates with my Meteor app through a REST API created with the Restivus package.
The user authenticates to the REST API and then uses authenticated tokens to make any further requests.
So far, everything works fine, as long as he stays within the extension. However, from the chrome extension, I'd like to redirect the user to his profile page on my main website. When that happens, he's no longer authenticated, and must re-sign-in to access the profile page.
I figure this is because the REST API session and the webpage session are two completely different sessions on the server (even though both the API and the webpage run from the same server). My question is, is there a way to maintain the user's logged-in state as he moves from the extension to the main website?
I figure there are a few options:
I'm using the standard meteor accounts package. Is there a way to push whatever standard cookie / data that the accounts package uses, to the user's browser, so that when he goes to the website, he'll be considered logged in?
Push a custom cookie to the user, which I then check for and log him in when he first comes to the website. However, I don't know how to push a cookie through a REST API or generate one in the Chrome extension
Use DDP to communicate with the second session and transfer the login credentials.
I don't know if these are the best options (or even how to implement them if they are...). Has anyone figured out a way to do this already? Thanks!
I would suggest you to develop your own flow of authentification using a token as an URL parameter. You should achieve a similar experience that slack provides with magic authentification links
The idea is to generate a token and add it to the Meteor.users collection for the user logged in your chrome extension.
Then, redirect your user to an url with the token as a parameter. The app checks which user is linked with this token and log him in.
You can get inspiration on what is done in the account package to handle enrollment and reset links, or in the passwordless package

Google OAuth Always Showing Consent Screen

I'm building an installed application that will have features requiring the Google Drive REST API using Qt and C++. I understand Qt is now releasing new libraries to support OAuth flows but let's assume I'm a student and learning to use OAuth at this layer is a requirement for this project.
In my application, I have a working OAuth flow for installed applications that ends with an Access Token and Refresh Token being stored using QSettings (I'm open to input on whether this is a disastrously bad idea too). The application requires no authentication/login for its own sake/data, but it does need authentication to Google for calling API's using an Access Token. This application has no associated web backend being hosted; its simple and should be deployable completely locally (I've written and included a simple TCP server that will receive the authorization redirect_uri and will run and close when called from within the application).
As such, I'm curious about the best way to make sure that, when a user opens my application and wants to use the Google Drive features, they are appropriately authenticated on Google's end. Say, if I maintain an access token in the registry, and this access token is granted per-user/per-application basis (right?), then how can I make sure only the user the token belongs to is able to make calls to the API with it?
Here's my understanding and approach; feel free to correct me or educate me if I've got the wrong interpretation.
If an Access Token is found, perform the following:
Open a browser page to a Google login domain and have the user authenticate there (this could prohibit a user from being able to use a cached login session that would have access to a token they otherwise shouldn't have access to)
If user has correctly authenticated with a Google account, return control to the application and make a test call to an API using the stored token.
If the call fails (responds with an invalid_credentials) I should be able to be sure its because the access token has expired and the application will go through the flow to renew an Access Token from a Refresh Token.
If no Access Token is initially found:
Start a normal OAuth installed application flow
Get the tokens and store them so that when the user opens the application next time the former procedure is used
My issue then is the first two steps if the Access Token is found. Nominally this could be done by the typical OAuth flow but it appears that when using a localhost as the redirect uri, Google will always prompt for consent, regardless of settings for prompt and access_type authorization query parameters.
What can be done to accomplish these first two steps in a way that my application can control (i.e. not a solution that relies on a backend server being hosted somewhere)?
If this question is too open-ended for SO requirements I can make some more restrictions/assumptions to limit the problem domain but I'd rather not do that yet in case I unknowingly rope off a good viable solution.
Thanks for reading! Sorry if its a verbose; I wanted to ensure my problem domain was fully fleshed out!
If you are using an installed application, I wouldn't recommend using or storing refresh tokens. Storing refresh tokens on the client side means that if an intruder gains access to the client's application, they have infinite access to the user's application without ever having to enter the user's credentials. If you do insist on having a refresh token, ensure you follow the Google's installed app flow, with the code_verifier parameter included in your requests.
If the access token is found, you should try to verify it, and if verified then use it at the google api, otherwise force the user to login again (or refresh it if you chose to still use refresh tokens).
If no access token is found, your flow sounds fine.
A few notes on loggin in with Google:
Google will only return a refresh token if you specify access_type=offline in your auth request.
Google will only return a refresh token on the user's first authorization request, unless you always specify prompt=consent in your query params.
In my experience, when leaving out the prompt query param, the user is not prompted for their consent again. If they are logged in to google, you will get a new access token, but no refresh token, unless you have prompt=consent.
I think the idea is you use prompt=consent if you have no record of the user ever using your application. Otherwise if they have used it before, you may prefer to use prompt=select_account to allow the user to select which account he wants to use in case he has more then one, or you can just use prompt=none.
This is just my understanding of it all.
My approach I ended up using was just to deploy with an SQLite db that will be stored in the AppData roaming directory. The db schema includes a field for the user's Name (from the OpenID IDToken field if it exists), the user's picture URL (again from IDToken if it exists), the refresh and access token strings (will be stored as encrypted strings when I get around to it), the user's UID/sub string, and a field for a user name and password.
These latter two fields are authentication fields for within my own application, which, again, I wanted to avoid but it seems impossible to do so. So the user will be prompted to enter a username and password into a form, and these credentials will be checked against the existing SQLite db file mentioned previously.
If they exist and are correct, the user gets logged in and will have access to their respective access and refresh token.
If the user has forgotten their password, they'll be asked for reconsent (going through the installed app flow again), and whatever password they provided during initial login will be used as the reset password. It is considered, for my purposes, that logging into Google for the installed app flow is proof enough that the user account belongs to them and they should have authorization to reset the password.
If the user is a new user and doesn't have a record in the local SQLite db file, then they can also click a button to "Create New Account" - which effectively goes through the authorization flow as well but this time a whole new record is posted to the SQLite db with the appropriate fields filled.
There's still more optimization that could be done but at least I am getting closer to the level of security and control of access to Google user accounts that I want.
I'm not marking this as an answer because I feel like this solution is still not desired and that there should be an easier way. So if someone has evidence or experience of providing an equivalent level of authentication control without needing to maintain a local user account database then I would be more than happy to mark such a method as the solution!
Thanks again!

ASP.Net and Facebook: Logging-in via ASP.Net

I want to enable Facebook authentication and the FB-Graph in my website, which already has forms authentication. Using http://multitiered.wordpress.com/2010/08/05/getting-started-with-the-facebook-c-sharp-sdk/, I was able to figure out how to login server-side.
However, the problem with this approach is that a secure cookie will not be created, since the call returns the authentication code in the querystring via a callback. This means that the user will have to login every time.
I can see two ways around this:
Store the access token in a secure cookie manually
Instead of the above approach, use the FB JS API to login - this stores a secure cookie with the access token automatically
I would prefer not to use the second approach, as I would like the login code to be server-side.
Which would be the better approach? Am I missing something?
I use the JavaScript method to first authenticate the user, the JS SDK then writes an encrypted cookie (called "fbs_[YourAppID]") when a connected user hits your page; using one of the many Facebook c# SDKs, this cookie can be decoded using your application secret giving you the user ID, oAuth token, expiry date etc.
Then I hook into the AuthenticateRequest event of my .NET application, check the presence of the cookie, decode if it found, and then find a user who has been assigned this facebook ID (your user table must have a extra field for storing the ID of their facebook account).
If a match is found, I write a normal forms authentication cookie for this user, then .NET will recognise them for all future requests. If no user is found, then this is a brand new user who has just connected. Use the SDK again to query the graph API using their oAuth token, get things like their name/email etc and create a new account, then issue a authentication token as normal.
By writing a normal authetication cookie, the user will stay logged into to your site for all requests, just as if they were a normal user.
One side point, when using email address, check for duplicates, and check for the facebook cookie in all requests. For example, an existing registered logged in user may have just connected.

Resources