I am desperately looking for an online service for buffering rss feed items.
Basically, I have one rss feed that publishes approximately 40 items per hours; however the rss feed only exposes the latest 20. I would like to have access to a buffered rss feed that would for example expose the latest 250 items.
If such a service exists, and you know about it, let me know!
Ben.
Why not just consume the service and write it to DB? Then you have access as far back as you care to go. This is simple in just about every scripting language out there. And, as a bonus, it'll lessen the amount of http hits your site does (if you're displaying to the user from DB instead of on load from the RSS feed site)
Check out http://superfeedr.com (I'm the founder). We do not "cache" the feeds, but we poll them often enough that we should not miss any entry.
Also, check this CouchApp, as it can allow you to very easily store these updates in a couchdb app. You can be running in a few minutes, without even needing a server.
Related
Looking for some strategy here. We have a secure (subscription-based) website that points to another website for streaming video. Currently, a savvy user could potentially share the link to the streaming page thus bypassing the login. Looking for ideas on how to:
Prevent sharing the link
Cloak the link...or
Make the link from the origin page point to a dynamically-generated URL that can only be visited once.
Thanks in advance!
Unless there is a form of streaming the video from the third party to your application (by streaming I actually mean sending the video in chunks of data to your backend, assembling them back together, and serving the actual assembled video as if it was hosted directly on the same host as the web application, I have personally done this using Amazon S3 for my open-source subscription-based CMS called phpReel) I don't think you can securely do this.
If you do want to go down this route please note that it might get expensive if you have a lot of active subscriptions but if you are interested you could check out these files as they showcase how I have done it for phpReel. I am afraid though that you might need a developer for this job.
On a more optimistic note, may I ask what service do you actually use to stream your videos? For example, Vimeo with the cheapest paid plan offers an option that restricts access to your video outside a specific domain name. Meaning you can set that your videos must be streamed only on domain.com, and thus they will be accessible there and only there.
I am developing an Android app which basically does this: On the landing(home) page it shows a couple of words. These words need to be updated on daily basis. Secondly, there is an 'experiences' tab in which a list of user experiences (around 500) shows up with their profile pic, description,etc.
This basic app is expected to get around 1 million users daily who will open the app daily at least once to see those couple of words. Many may occasionally open up the experiences section.
Thirdly, the app needs to have a push notification feature.
I am planning to purchase a managed wordpress hosting, set up a website, and add a post each day with those couple of words, use the JSON-API to extract those words and display them on app's home page. Similarly for the experiences, I will add each as a wordpress post and extract them from the Wordpress database. The reason I am choosing wordpress is that it has ready made interfaces for data entry which will save my time and effort.
But I am stuck on this: will the wordpress DB be able to handle such large amount of queries ? With such a large userbase and spiky traffic, I suspect I might cross the max. concurrent connections limit.
What's the best strategy in my case ? Should I use WP, or use firebase or any other service ? I need to make sure the scheme is cost effective also.
My app is basically very similar to this one:
https://play.google.com/store/apps/details?id=com.ekaum.ekaum
For push notifications, I am planning to use third party services.
Kindly suggest the best strategy I should go with for designing the back end of this app.
Thanks to everyone out there in advance who are willing to help me in this.
I have never used Wordpress, so I don't know if or how it could handle that load.
You can still use WP for data entry, and write a scheduled function that would use WP's JSON API to copy that data into Firebase.
RTDB-vs-Firestore scalability states that RTDB can handle 200 thousand concurrent connections and Firestore 1 million concurrent connections.
However, if I get it right, your app doesn't need connections to be active (i.e. receive real-time updates). You can get your data once, then close the connection.
For RTDB, Enabling Offline Capabilities on Android states that
On Android, Firebase automatically manages connection state to reduce bandwidth and battery usage. When a client has no active listeners, no pending write or onDisconnect operations, and is not explicitly disconnected by the goOffline method, Firebase closes the connection after 60 seconds of inactivity.
So the connection should close by itself after 1 minute, if you remove your listeners, or you can force close it earlier using goOffline.
For Firestore, I don't know if it happens automatically, but you can do it manually.
In Firebase Pricing you can see that 100K Firestore document reads is $0.06. 1M reads (for the two words) should cost $0.6 plus some network traffic. In RTDB, the cost has to do with data bulk, so it requires some calculations, but it shouldn't be much. I am not familiar with the pricing small details, so you should do some more research.
In the app you mentioned, the experiences don't seem to change very often. You might want to try to build your own caching manually, and add the required versioning info in the daily data.
Edit:
It would possibly be more efficient and less costly if you used Firebase Hosting, instead of RTDB/Firestore directly. See Serve dynamic content and host microservices with Cloud Functions and Manage cache behavior.
In short, you create a HTTP function that reads your database and returns the data you need. You configure hosting to call that function, and configure the cache such that subsequent requests are served the cached result via hosting (without extra function invocations).
We are writing our custom scheduling app for our website.
Necessarily, it requests Google Calendar data to see when one of our 3 team members are available and then offers the visitor an array of available time slots.
Problem is, this takes too damn long to get the updated info.
I'm wondering if we could simply get all this data in the background and offer visitors to pick from data that is a few seconds old :)
So my question is, how often can we initiate this without getting banned by Google.
Here you go. The limit you're looking for depends on the type of google account you're using.
https://developers.google.com/apps-script/guides/services/quotas
Also you won't get banned, it just won't run. If you're on a consumer account you could ping it 1x every 18 seconds without it failing. That's as close as you can get to "Live Data".
I'm using this link https://www.google.com/reader/api/0/stream/contents/feed/FEEDHERE?output=json&n=20
to fetch feeds using Google's algorithm. As you can see I'm not adding any other parameters, just fetching the returned data in JSON format. My app will be heavily used hopefully and if I send a lot of requests to this link, will Google block my access or something?
Is there anything I can include, like userip, url for my app (so if they have problem to just contact me) or something else?
The most basic answer to your question is that Google will change its Terms of Service whenever it likes, and you've got no say in the matter. So if it's allowed today, it might not be allowed tomorrow, at Google's whim.
On this issue, though, you seem fairly safe. From the Terms of Service (these is the general document, since Reader doesn't seem to have a specific one):
Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide.
Google provides RSS and Atom. They provide these feeds, so I assume they expect that they'll be used. They don't say that it's a misuse to point someone else at those feeds, so it looks OK for now, but they could add such a clause at any time.
All online services are subject to the terms and conditions of the providers of those services. So, as others have said, they may be ok with your use today, but they can change their mind any time down the line. I doubt including a URL or email or contact info will help anything, because when these services change, they don't notify every user of the service, they just announce the change publicly, and usually they give several month's notice in order to give users a chance to adapt their applications, but this is not standardized or enforced so there is no guarantee. One example would be the fairly recent discontinuance of the Google Finance API (for which no replacement has been announced).
The safest approach would be to design your app such that this feature that uses google's functionality is decoupled as much as possible from the rest of your app, so that, when or if the availability of the service changes (ie it's no longer available at all) you can adapt your app to use some other source for the feeds with minimal impact to the rest of the app. Design for change and plan for the worst.
I want to create an in-house RSS feed (I work for 3 Mobile, Australia) for consumption on an INQ1 mobile phone, or any other RSS reader for that matter. However, testing it out on the phone's built-in RSS reader, I realize that without the ability to password protect the feed, or otherwise restrict access to it, I stand little chance of being able to develop this idea further.
One thing I thought of was to periodically change the Uri for the feed, so managers who had left the company couldn't continue to subscribe and see sensitive information, but the idea of making users do that would make it a harder sell, and furthermore is terribly inelegant.
Does anybody know how to make it so that prior to downloading a feed, a reader would have to authenticate the user? Is it part of the metadata within the feed, or something you would set in the reader software?
Update: I should have explained that I already have placed folder-level permissions on the parent folder, which brings up the normal authentication dialog when the feed is viewed in a browser, but which just results in a failed update with no explanation or warning in the phone's RSS reader, and is indistiguishable from the file being missing, when I next try and refresh the feed.
If the reader in the phone doesn't support HTTP Basic or Digest, your best bet is to create a unique url to the feed for each consumer. Have the customer login and generate a link with some token in it that is unique for that user. If the user ever leaves, you can then deny that token, shutting down access.
If you go this route, you probably want to investigate including the Feed Access Control bits in your feed. It's not perfect, but it is respected by the bigger aggregators, so if one of your clients decides to subscribe to the feed with Reader or Bloglines, things shouldn't show up in search results.
I believe you would set the permissions on the feed itself, forcing authentication, much like the Twitter feeds. The problem with this is that many readers (including Google Reader) don't yet support authenticated feeds.
The idea is to have authentication over a secure channel. These posts explain it pretty well:
RSS Security
Private RSS Feeds
Authentication by the webserver is probably the best solution, however to get round the issues of reader not supporting it (Google has been mentioned and I have issues with Safari) you could implement a simple value-key to append to the URL.
I.E.
http://www.mydomain/rss.php?key=value
Your system could then "authenticate" the key-value and output the RSS, invalid k-v could get a standard "invalid authenticate" message as single item RSS or return a 40x error.
It not very secure as you could see the k-v in the URL but it's a a trade off. An un-authenticated HTTPS would be slightly more secure.
Assuming your RSS feed is over HTTP then basic HTTP authentication would probably do the trick. This would either be done at the web server level (in IIS for example) or via whatever framework you're using to produce the feed (in ASP.NET for example).
The authentication scheme (basic username/password, NTLM, Kerberos etc) is up to you. If you're using WCF to produce the feed then these decisions are things you can make later and apply via config if needed.
Are you simply looking to authenticate consumers of the feed, or also encrypt it to prevent the information from being read by a "man in the middle". If you require encryption then SSL is probably the easiest to implement.
You should avoid simply "hiding" the RSS feed by changing it's name.
update:
Your question (with it's update) sounds like you're actually having issues with the RSS client on the device. You need to determine whether the phones RSS client understands how to deal with basic/digest authentication etc.
Assuming it doesn't, is there anything in the HTTP request that could allow you to associate a device with a user? Is there an HTTP Header that gives you a unique device ID? If so, you might be able to then perform a lookup against this data to perform your own weak-authentication, but you should remember that this sort of authentication could be easily spoofed.
Does the device have a client certificate that could be used for mutual SSL? If so, then that would be ideal.