I'm not sure if this is the appropriate place to ask this or not, but I'm looking for a library that wraps HTTP requests in a client structure and includes the ability to retrieve and store cookies for individual sessions (like Python Requests library).
I'm looking to create a small bot for scraping a website and wanted to use it as an excuse to learn Rust.
Teepee, hyper, and rust-http don't seem to have cookie support.
However, khodzha has some in-progress work for adding cookie support to Servo. This file from his branch might be able to help you.
Warning: The code is a work in progress, I suggest you contact the author on IRC or github before using it.
Related
I want use Google Assistant from my phone to send HTTP POST command to my server. I have a simple webnms app running over it, this server support REST API and now I want to use Google Assistant to shoot GET or POST command to that server and return my output.
Is it something possible? I am not full time developer.
Yes, as #Prisoner says it is possible. It is not what you asked - but have you seen these ways that Google provides to get skills published without requiring a lot of developer savvy?
https://developers.google.com/actions/content-actions/
https://developers.google.com/actions/templates/first-app
I don't speak for them, but IMO Google's target audience for Action building apart from the above is those who have at least some familiarity with the JavaScript language and its "run-time" Node.
There is also this - which I haven't tried by the way.
https://www.techadvisor.co.uk/how-to/digital-home/easy-actions-google-assistant-3665372/
In case it is not obvious, Google Actions are essentially websites that interact with Google's assistant running on a Home device or a smart phone, say. Think of the Assistant as a browser initiating requests and your Action as serving them. If you can (build and?) deploy a server that handles POSTS over HTTPS on a publicly addressable URL, and if you can understand the JSON payload that the Assistant sends and respond with appropriate JSON to carry out you application then you are good to go.
Where you don't have a public IP address - e.g. in testing - you can use a tool like ngrok ( https://ngrok.com/ ) to reverse proxy requests emanating from the Assistant to your server.
I have slides for a presentation I did targeting fledgling developers who had never built an Action here
https://docs.google.com/presentation/d/1lGxmoMDZLFSievf5phoQVmlp85ofWZ2LDjNnH6wx7UY/edit?usp=sharing
and the code that goes with it here
https://github.com/unclewill/parrot
On the upside the code is about as simple as it gets. On the downside it does almost nothing. In particular, it doesn't try to understand language. As #Prisoner says you'll likely need a tool like Dialog Flow for that.
Yes, it is possible.
Your server will need to implement the Actions on Google API. This is a REST API which will accept JSON containing what the user is intending to do and specific information about what they have said. Your server will need to send back JSON indicating the reply, along with additional information about how to continue the conversation.
You will likely also want to use a tool such as Dialogflow to handle building the conversational script and converting a user's phrases into something that makes sense to you. You'll also need to use the Actions on Google console to manage your Action and provide additional details about how users contact your Action. All of this is explained in the Actions on Google documentation.
Simple Actions are fairly easy to develop, and can certainly be done by a developer as a hobby. Good Actions, however, take a lot more thought and planning. Google offers you to the tools - it is up to you to best take advantage of them.
I've found the solution.
In the "Action" console https://console.actions.google.com/project/sandbox-csuite/scenes/Start
Go to menu "Webhook", click "Change fulfillment method", and then select "HTTPS endpoint"
Background: This is my first standalone web development project, and my only experience in Meteor is building the Discover Meteor app over the last summer. I come from about a year of CS experience as a side interest in school, and I am most comfortable with C and C++. I have experience in python and java.
Project so far: I'm creating a calendar management system (for fun). Using accounts-google, I have created user accounts that are authenticated through google. I have requested the necessary permissions that I need for my app, including 'identity' and 'calendar read/write access'. I've spent the last week or so trying to get over this next hurdle, which is actually getting data from google.
Goal: I'd like to be able to make an API call to Calendar.list using a GET request. I've already called meteor add http to add the GET request functionality, my issue comes with the actual implementation.
Problem: I have registered my app on the developer console and set up Accounts using the client ID and secret, but I have not been able to find/generate my 'API key' for use in the request. Here is the google guide for creating the access token by using my (already) downloaded private key. I'm having a hard time wrapping my head around an implementation on the server side using JS because I don't have a lot of experience with what is mentioned in the HTTP/REST portion of the implementation examples. I would appreciate some help on how to implement a handshake and receive an access token for use in my app. If there is a call I can make or some package that will handle the token generation for me, that would be even better than implementation help. I believe an answer to this would also benefit this other question
The SO answer that I've been referring to so far: https://stackoverflow.com/a/14543159/4259653 Some of it is in spanish but it's pretty understandable code. He has an API key for his request, which I asked this question to help me with. The accounts-google documentation isn't really enough to explain this all to me.
Also an unrelated small question: What is the easiest way to deal with 'time' parameters in requests. I'm assuming JS has some sort of built-in functionality that I'm just not aware of yet.
Thanks for your research. I have also asked a very similar question, and right now I am looking into the package you recommend. I have considered this meteor-google-api package, but it looks abandoned.
Regarding your question about time manipulation, I recommend MomentJS. There are many packages out there; I am using meteor add mrt:moment
EDIT: MomentJS now has an official package for Meteor, so use meteor add momentjs:moment instead of the mrt command above
Below is a snippet of what moment can do. More documentation here.
var startTimeUTC = moment.utc(event.startTime, "YYYY-MM-DD HH:mm:ss").format();
//Changes above formatting to "2014-09-08T08:02:17-05:00" (ISO 8601)
//which is acceptable time format for Google API
So I started trying to implement all of this myself on the server side, but was wary of a lot of the hard-coding I was doing and assumptions I was making to fill gaps. My security prof. used to say "never implement encryption yourself", so I decided to take another gander for a helpful package. Revising search criteria to "JWT", I found jagi's meteor-google-oauth-jwt on Atmosphere. The readme is comprehensive and provides everything I need. Following the process used in The Google OAuth Guide, an authorization request can be made and a key generated for making an API call.
Link to Atmosphere: https://atmospherejs.com/jagi/google-oauth-jwt
Link to Repo: https://github.com/jagi/meteor-google-oauth-jwt/
I will update this answer with any additional roadblocks I hit in the Google API process and how I solved them:
Recently, I've been running into problems with the API request result. I get an empty calendarlist back from the API call. I suspect this is becuase I make an API call to my developer account rather than to the subject user. I will investigate the problem and either create a new question or update this solution with the fix I find.
Fix: Wasn't including the 'sub' qualifier to the JWT token. Fixed by modifying JWT package token generation code to include delegationEmail: user.services.google.email after scope. I don't know why he used such a long designation for the option instead of sub: as it is in the google API, but I appreciate his package nontheless.
I'm quickly becoming proficient in this, so if people have meteor-related google auth questions, let me know.
DO NOT USE SERVICE ACCOUNTS AS POSTED ABOVE!
The correct approach is to use standard web access + requesting offline access. The documentation on the api page specifically states this:
Typically, an application uses a service account when the application uses Google APIs to work with its own data rather than a user's data.
The only exception to this is when you are using google apps domain accounts and want to delegate access to your service account for the entire domain:
Authorizing a service account to access data on behalf of users in a domain is sometimes referred to as "delegating domain-wide authority"
This makes logical sense as a user must be allowed to "authorise" your application.
Back to the posters original question the flow is simple:
1) Meteor accounts google package already does most of the work for you to get tokens. You can include the scope for offline access required.
2) if you are building your own flow, you will go through the stock standard process and calls as explained on auth
This will require you to:
1) HTTP call to make the original request or you can piggyback off some of the internal meteor calls : Package.oauth.OAuth.showPopup() -- go look at the source there are more nifty functions around there.
2) Then you need to create an Iron router server side route to accept the oauth response which will contain a code parameter that you will use to exchange for tokens.
3) Next use this code to make a final call to exchange the "code" for the token + refresh_token
4) Store these where ever you want - my requirement was to store them not at the user level but multiple per user
5) Use a package like GoogleAPI this wraps up Google API calls and refreshes when required - it only works when tokens are stored in user accounts so you will need to rip it apart a bit if your tokens are stored somewhere else (like in my case)
I am creating a client program that calls various APIs, one of which is the Evernote API, through purely HTTP calls, without the use of any SDKs provided by Evernote. I realize that this makes my life harder, but it makes the lives of the users of my product much easier.
So far I have been able to authenticate with oauth 1.0 to the Evernote server. However, it's not immediately clear how to make the HTTP call after I have the oauth_token.
Where is the endpoint to make API calls to create a note on an oauth-authenticated Evernote user account? What are the url parameters for such a call? Or, what are the requirements to include in such an HTTP request?
Does Evernote use HTTP to make such calls anyway? If not, what do they use?
Do I have to format the note in XML format, or does it accept JSON as well?
Evernote does not have a RESTful API. According to the CTO, it's due to a requirement to shuttle very large amounts of data and HTTP is not efficient. There is a project on github aiming to implement a RESTful API for Evernote.
https://github.com/ttddyy/evernote-rest-webapp
CTO's Reasoning:
http://blog.programmableweb.com/2013/10/03/is-evernotes-restless-api-approach-a-model-for-other-api-designs/
I haven't used Evernote in years and never used the API.
Evernote uses thrift which can be a bit tricky to deal with. This choice is explained in this blog post. I really advise you to use the SDK as it hides all the thrift complexity.
As mentioned above, a rest wrapper is available on github. It seems very cool but it's new and not official. Use it at your own risk.
Best I could find was this wiki entry
I I thought "surely there must be more to it than this".
Am I missing something?
From the doc:
What is WebHook?
The concept of a WebHook is simple. A WebHook is an HTTP callback: an
HTTP POST that occurs when something happens; a simple event-notification via HTTP POST.
A web application implementing WebHooks will POST a message to a URL
when certain things happen. When a web application enables users to
register their own URLs, the users can then extend, customize, and
integrate that application with their own custom extensions or even
with other applications around the web. For the user, WebHooks are a
way to receive valuable information when it happens, rather than
continually polling for that data and receiving nothing valuable most
of the time. WebHooks have enormous potential and are limited only by
your imagination! (No, it can't wash the dishes. Yet.)
Why should I care?
As integrated as we perceive the web, most web applications today
operate in silos. With the rise of API's we've seen mashups and some
degree of integration between applications. However, we have not seen
the vision of the programmable web: a web where you as the user can
"pipe" data between apps much like the Unix command line. Some say RSS
is the answer. They are wrong. The heart is in the right place, but
the implementation is wrong. RSS is still useful, but it is not going
to bring us the true programmable web.
We just need a simple way to get data out in real-time to let the user easily do whatever >they wantwith it. That means no polling, no content constraints, and no XML
parsing. That means no RSS. Using HTTP is simpler and easier to use.
PHP is a very popular and accessible programming environment, so it's
likely to be used often for writing hooklets... getting data from a
web POST in PHP is as simple as $_POST['something']. And making the
request to the user script is as simple as making an HTTP request,
something already built-in to most programming environments. In fact,
web hooks are easier to implement than an API.
I'm using this link https://www.google.com/reader/api/0/stream/contents/feed/FEEDHERE?output=json&n=20
to fetch feeds using Google's algorithm. As you can see I'm not adding any other parameters, just fetching the returned data in JSON format. My app will be heavily used hopefully and if I send a lot of requests to this link, will Google block my access or something?
Is there anything I can include, like userip, url for my app (so if they have problem to just contact me) or something else?
The most basic answer to your question is that Google will change its Terms of Service whenever it likes, and you've got no say in the matter. So if it's allowed today, it might not be allowed tomorrow, at Google's whim.
On this issue, though, you seem fairly safe. From the Terms of Service (these is the general document, since Reader doesn't seem to have a specific one):
Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide.
Google provides RSS and Atom. They provide these feeds, so I assume they expect that they'll be used. They don't say that it's a misuse to point someone else at those feeds, so it looks OK for now, but they could add such a clause at any time.
All online services are subject to the terms and conditions of the providers of those services. So, as others have said, they may be ok with your use today, but they can change their mind any time down the line. I doubt including a URL or email or contact info will help anything, because when these services change, they don't notify every user of the service, they just announce the change publicly, and usually they give several month's notice in order to give users a chance to adapt their applications, but this is not standardized or enforced so there is no guarantee. One example would be the fairly recent discontinuance of the Google Finance API (for which no replacement has been announced).
The safest approach would be to design your app such that this feature that uses google's functionality is decoupled as much as possible from the rest of your app, so that, when or if the availability of the service changes (ie it's no longer available at all) you can adapt your app to use some other source for the feeds with minimal impact to the rest of the app. Design for change and plan for the worst.