As part of a late night project I am working on out of interest, I am trying to scrape my Uber trip data off of their website.
I have had a look at the code of the login page on
https://login.uber.com/login
and have seen that they use POST method in their form setup as follows:
<form method="post" class="form" novalidate="">
<input type="hidden" name="_csrf_token" value="1452201446-01-hujzoBTxkYPrJessd6zQwnD2ZOFxMOVgIYN8iXntr6c=">
<input type="hidden" data-js="access-token" name="access_token">
<a href="#" class="btn btn--full btn--facebook" data-js="facebook-connect">
<span class="push--ends flush">Continue with Facebook</span>
</a>
<p class="primary-font primary-font--semibold text-uber-white background-line push--top push--bottom">
<span>or use email</span>
</p>
<div class="form-group push-tiny--top flush--bottom">
<input type="email" name="email" class="text-input square--bottom " placeholder="Email Address" value="" id="email">
</div>
<div class="form-group push--bottom">
<input type="password" name="password" class="text-input square--top " placeholder="Password" id="password">
</div>
What I have read up is that one needs to send csrf token along when trying to scrape
library(RCurl)
library(XML)
URL_str<-"https://login.uber.com/login"
URL_str2<-"https://riders.uber.com/trips"
email<-"exampleMail#gmail.com"
pass<-"thisisalong"
token<- "1452201446-01-hujzoBTxkYPrJessd6zQwnD2ZOFxMOVgIYN8iXntr6c="
params <- list('email' = email,
'password' = pass,
'_csrf_token'=token)
URL_doc = postForm(URL_str2, style="POST",
.params=params)
If I now try and scrape the site, I get
ERROR: FORBIDDEN
I have seen some examples in python with similar websites. Can the same be done in R?
The final answer ended up being quite simple:
library(rvest)
library(RCurl)
library(XML)
session <-html_session("https://login.uber.com/login")
email<-"exampleMail#gmail.com"
pass<-"thisisalong"
#Handling the html_form
form<-html_form(session)[[1]]
form<-set_values(form, email=email, password=pass)
#Found that unless I explicitly assigned the value to empty it created errors
form$url<-""
session_open<-submit_form(session,form)
session_open<-jump_to(session_open,URL_str2)
From here on in its straight forward pulling tables using html_table and isolating nodes using html_nodes. Hope this helps
Related
Upgrading site from 3 to 4. Existing code makes use of FullTextSearchable as per this guide that exists for 3, but not for 4. Google only results in unanswered questions of the same nature. Except for this one, which doesn't help. But I do find this guide for SS4, which is not geared towards the same topic really, but somewhat helpful.
Here is my code:
_config.php
\SilverStripe\ORM\Search\FulltextSearchable::enable();
Page.php
use SilverStripe\ORM\Connect\MySQLSchemaManager;
private static $create_table_options = [
MySQLSchemaManager::ID => 'ENGINE=MyISAM'
];
Page.ss
$SearchForm
Expected: Standard search form is displayed
Actual: Nothing is displayed
I went as far as to hard code the form from the existing site onto the template to see if the search function itself works:
Page.ss
<form id="SearchForm_SearchForm" action="/home/SearchForm" method="get" enctype="application/x-www-form-urlencoded">
<p id="SearchForm_SearchForm_error" class="message " style="display: none"></p>
<fieldset>
<input type="text" name="Search" value="Search" class="text nolabel" id="SearchForm_SearchForm_Search">
<div class="form-group">
<input type="submit" name="action_results" value="Go" class="action" id="SearchForm_SearchForm_action_results">
</div>
</fieldset>
</form>
Expected: Search results page displays with relevant search results for the entered term
Actual: 404 page not found
I have been given the following POST instruction, and I am trying to translate it into httr:
<form method="POST" action="https://this.website.com/foldername>
<input type="hidden" name="ExternalAction" value="AgetAscii">
File Type <input type="text" name="filechar" value="0">
<input type="submit" value="Click here to Retrieve the File"/>
</form>
I am having trouble getting the right syntax for httr. I would welcome suggestions. The initial input type, name and value seem straight forward, but I don't see how I bring the File Type arguments into httr, nor am I confident I am handling the final "submit" and value items properly.
I'd welcome suggestions.
Many thanks
If I paste that example html into an local html file and substitute the action to be http://bin.org/post like so:
<form method="POST" action="http://httpbin.org/post">
<input type="hidden" name="ExternalAction" value="AgetAscii">
File Type <input type="text" name="filechar" value="0">
<input type="submit" value="Click here to Retrieve the File"/>
</form>
Then the only form data reported back by httpbin.org as having been received are for ExternalAction and filechar. Therefore, to send this data via httr::POST() you would do:
library(httr)
action <- "https://this.website.com/foldername"
body <- list(ExternalAction = "AgetAscii",
filechar = "0")
POST(url = action, body = body)
I have an angular 7 reactive form, input field in this form is to let user see and modify time and then using Save() save it back to our db, the problem is that user can enter any none sense numbers in this field which dose not make sense in terms of Time.(like hours should be 1-24, Minutes 1-60)
My question is how can I validate what user entered into time field and if its valid then let Save btn be active?
Note: Based on my testing even though its none sense time entry but while trying to save its not saving to db and its probably sql not letting that happen.
I googled but i could not find anything, also angular has only email Validators which u can put while defining form.
this is my form defination
timeForm: FormGroup = new FormGroup({
Id: new FormControl(''),
Time: new FormControl(''),
});
and this is my HTML side
<form style="background-color: aliceblue;" [formGroup]="service.timeForm">
<mat-grid-list cols="1" rowHeight="120px">
<mat-grid-tile>
<div class="form-controles-container">
<input type="hidden" formControlName="Id" />
<mat-form-field>
<input formControlName="Time" matInput placeholder="Time" />
</mat-form-field>
<div class="button-row">
<button mat-raised-button color="primary" type="submit" (click)="Save()">Save</button>
<button mat-raised-button color="warn" (click)="Cancel()">Cancel</button>
</div>
</div>
</mat-grid-tile>
</mat-grid-list>
</form>
I need some sort of validation when user punch in any input which dose not make sense in Time my Save btn should not be activated.
I appreciate your help and guideline.
You can use the HTML type="time" or if you want more you can simply use a library like: JsDaddy/ngx-mask found on Github. Use it like this:
HTML with MatInput:
<input formControlName="Time" matInput placeholder="Time" type="time" />
With ngx-mas:
<input formControlName="Time" matInput placeholder="Time" mask="Hh:m0" />
Adding type="time" in all input time type use cases always help.
I'm facing a very weird problem. My error message is not showing
below is my codepen
https://codepen.io/cancerian73/pen/eoaYga
<form data-abide novalidate>
<div class="search-container">
<input type="text" id="seach_again" placeholder="Try once again" aria-required="true" required>
<span class="form-error" data-form-error-for="search_again">Amount is required.</span> <span class="form-error"> Valid email required.</span>
<input type="submit" class="go-btn" value="GO">
</div>
</form>
Your id has a typo in it (seach_again instead of search_again). Though these error messages will work even with the typo because they are adjacent siblings to the input element. I wonder if you did not enable the abide javascript plugin. You need to enable that in addition to the scss.
I am using Tuple to pass two models inside the view like code given below.
#model Tuple<AdvanceSearchModel, List<SearchUserModel>>
<form role="search" method="post" action="/Public/AdvanceSearch">
<div class="form-group">
<label>Name</label>
<input name="FullNames" type="text" class="form-control" value=""/>
</div>
<div class="form-group">
<label>Product</label>
<input name="Products" type="text" class="form-control" value="" />
</div>
<div class="form-group">
<label>Location:</label>
<input name="Location" type="text" class="form-control" value="" />
</div>
<div class="form-group">
<label>State</label>
<input name="States" type="text" class="form-control" value="" />
</div>
<div class="form-group">
<label>Country</label>
<input name="Countries" type="text" class="form-control" value=""/>
</div>
</form>
All the name attributes inside inputs are of AdvanceSearchModel. How do I use tag helper such as asp-for when passing multiple model to the views containing one or multiple forms? Also how do I retain values of the form after submitting the form in above scenario?
As you can see in the source code of InputTagHelper
You can see it creates the name attribute based on the (lambda) expression in html-tag:asp-for.
what you need
You need a form name tag like this SearchUserModel[0].Location
Where:
SearchUserModel is the property name on the model which is in the controller method you post to
[0] is the index in the list
Location is the property on the iten in the list the SearchUserModel instance
My suggestion
Not to do
Extend the InputTagHelper and add a prefix option (which adds a prefex to the name).
Use a view model Not a tuple!
Create a partial view that only takes SearchUserModel + a prefix (like an int for which row in the list it is for example: usermodel[1])
In your view loop over the list and call the partial.
result
#model SearchUserModel
<input asp-for="Location" my-prefix="ListItem[#Model.Id]" class="form-control" />
Better longterm option
Make a HTML template on how SearchUserModel part of the form should look.
Do ajax call to get the data or put the data as json in your view. (or take step 3 from what not to do)
Generate the form with well structured javascript.
On submit Instead of submitting the form, parse the from to json and send this as json ajax call.
Why do i say this? It is easier to debug if you get weird databindings in your controller.
That said, option 1 is perfectly fine but it might lead to problems later, as it is very static template, you wont be able to add or remove rows easily.
References for proper html name tags for lists:
http://www.hanselman.com/blog/ASPNETWireFormatForModelBindingToArraysListsCollectionsDictionaries.aspx
How does MVC 4 List Model Binding work?