Since everything is on-chain in Ethereum blockchain. Theoretically, each event is public visible and can be summed up as a block event. Is it possible to subscribe to events of non-contract addresses and create a feed page like RSS subscription?
It is possible, but honestly, I have never seen integration with the RSS protocol.
However, writing your script in the programming language is very easy. The procedure is well documented in the official geth documentation: https://goethereumbook.org/event-subscribe/
I would use this as a reference for other languages.
There is one big problem for you. To subscribe to Ethereum, you need access to the Ethereum node to get events from it.
There are three options from the best to the worst (in my opinion).
You can use the API from parties that provide access to ethereum networks. Example of it is INFURA, Alchemy and QuickNode. The huge disadventage is that requests are limited and you will use it very fast. Probably in minutes or hours.
You can create your own node connected to the Ethereum, but you need pretty fast computer/stable internet and 1TB SSD hard drive to keep it stable.
Find publicly available node. Usualy those nodes are not very stable and you will get ban soon. To discover ethereum nodes you can use Shodan. I have tried houndred times to use the public nodes to write my apps and those nodes are not stable. Each stable node is protected and does not allow to send any request to it...
If you need to read data from specific addresses you can use the Etherscan API - I love it as it is much easier than using the ETH API :)
There is an opensource protocol named RSS3 dedicated to RSS service on blockchain.
Its third-party API accesses the Ethereum network and creates a feed of any ENS address. The protocol not only displays transactions of the ENS, but also identifies and filter different types of on-chain transactions.
(check how more on RSS3 Docs and its Github)
The feed can be generated to a standard XML format RSS file, or import the RSS URL or that address directly to any RSS reader.
Take ETH founder Vitalik's ENS address (vitalik.eth) for example.
Access RSS3.io and type in the ENS
enter vitalik.eth
Generate RSS file
Click the RSS icon on the right and get the RSS file:
https://rss3.io/rss/0xd8da6bf26964af9d7eed9e03e53415d37aa96045/
Generate RSS URL
Go to https://rss3.io/manage and generate an RSS feed for an address/ENS.
Type in Vitalik.eth and get different types of RSS feed subscription:
All feeds: https://rss3.io/rss/vitalik.eth
These URLs should work in any RSS reader.
Related
I have many telegram channels, 24\7 they send messages in the format
"buy usdjpy sl 145.2 tp 167.4"
"eurusd sell sl 145.2 tp 167.4"
"eurusd sl 145.2 tp 167.4 SELL"
or these words in some order
My idea is to create app that checks every channel's message, and redirects it to my channel if it is in the above format.
Does telegram api allow it?
I have written a simple python code, using the telethon python module.
What the code basically does, is forwarding messages from various telegram channels through the telegram client api to a channel of your choosing. You can find it here.
Using the client api, one is able to read messages from groups and channels that your user is a part of. No bots required.
The telethon module makes it easy to filter messages that you want to be read. Feel free to fork the project and make the desired changes. You should look at the module documentation here.
You cannot scrape from a telegram channel with a bot, unless, the bot is an administrator in the channel, which only the owner can add.
Once that is done, you can easily redirect posts to your channel by listening for channel_post updates.
In order to be able to scrape messages from Telegram channels that you do not own, you need to develop you own Telegram client that is capable of:
Joining your desired channels by links
Forwarding messages, arriving to the channels your client is subscribed to, to your own Telegram channel
In order to develop your own Telegram client, you need to use some implementation of MTProto.
You can find a lot of implementations of MTProto on https://github.com using mtproto keyword.
A few examples of well-documented implementations:
In PHP: https://github.com/danog/MadelineProto
In Python: http://github.com/LonamiWebs/Telethon
But probably it would be an overkill to develop your own solution to this problem if the only thing you want is to have several redirections from existing Telegram channels to your own channel.
There are applications that provide such a service.
For example, there is MultiFeed Bot that allows you to setup forwarding of messages from any Telegram channels to your own Telegram channel.
This bot has a flexible filtering system so it should be pretty easy to setup filters to skip certain types of messages (ads, media content and etc.) and to leave only those messages that you want to see in your destination channel.
I solved a similar problem with TdLib. Their GitHub site has full C++, Java and C# examples that you can just modify.
I worked on the Java example, and applied most of my changes to the UpdatesHandler.onResult method (line 353). The C++ and C# examples have a similar structure. This method gets called by Td whenever any event occurs. Hence you can just intercept them there.
If you're not really sure where to begin, start by adding simple System.out.println statements (if using Java) to each of the case statements in the aforementioned method, and make sure you read the starting guide.
They actually have examples for many other languages (Python included), but from my point of view they are not as complete as the three I mentioned before.
This is very easy to do with Full Telegram API.
first on your mobile phone subscribe to all the interested channels
Next you develop a simple telegram client the receives all the updates from these channels
Next you build some parsers that can understand the channel messages and filter out what you are interested in
Finally you send the filtered content (re-formatted) to your own channel.
that's all that is required.
Is this what are you looking for? telegram-forward-bot
In the readme file:
Simple Telegram Bot for forwarding messages easily between various related channels and groups.
This bot allows you to automatically forward messages between different channels. We use it on our Student Comitee because we have like 15 different Telegram groups for each commission we are working on. Then, if we want some commission receives some important information, we can automatically forward to them using hashtags at the beggining of the message (or the caption of a media file).
I think I know your feeling, I'am trader and I follow various prediction channel. But not all of the information is usefull (sometimes ads). Hope this work for you :)
Got the solution to this problem.
Here is bot which automatically forwards messages from one channel to another without the forward tag.
Moreover the copying speed is legit!
#copythatbot
This is the golden tool everyone is looking for.
Depending on the language you want to use there are many libraries you can use to get the job done.
Let's take for example python, you can use libraries such as Telethon (for both user or bots) or "python telegram bot".
Both libraries are fantastic on what they do. Telethon is async so I kinda lean more towards it.
To do what your looking for you will need to catch the event.Message and use python regex re module for matching patterns from the messages.
Here's the code you want using Telethon:
import re
from telethon import TelegramClient, sync, events
# These example values won't work. You must get your own api_id and
# api_hash from https://my.telegram.org, under API Development.
api_id = 12345
api_hash = '0123456789abcdef0123456789abcdef'
client = TelegramClient('session_name', api_id, api_hash).start()
#client.on(events.NewMessage(chats=('TelethonChat', 'TelethonOffTopic')))
async def message_regex(event):
pattern = re.compile(".*145\.2 tp 167\.4.*", re.M)
raw_text = event.raw_text
if pattern.match(raw_text, raw_text):
## Pattern matched do something
pass
You just need to change the api keys and it should work properly. Now to add other things you will need knowledge about programming and python.
If you are looking for a simple solution you can always use this bot I've made #tg_feedbot
This is a bot used for forwarding messages from one/multiple groups to others. The way it works is by using your telegram account and when a message comes - if you have configured it so - it sees it and rewrites it to the channels you want. You can edit the way the message looks, filter it, delay and change words.
It's a free solution for automating Telegram User API and it has a simple to use interface together with documentation
If you are keen on learning yourself then I would suggest you to join Telegram groups such as Telethon or >>> telegram.Bot()
I recently created an application for the Gracenote OnConnect platform specifically to query Sport tv listings data. My project is an open source tool to allow users to easily find the channel for a given search event. So for example, if I wanted to find out when and on what channel the Yankees vs Red Sox game was tonight, I would go this site, search "Yankees vs Red Sox" and see that it is on ESPN at 7pm.
I've received an API key from the OnConnect API on the "Public Plan" and have made a few calls. It seems as though some of the required data is there, but I wanted to get some feedback before getting into the weeds. First of all, could I use the available API using the free public plan to get this kind of data? If so, do you have any suggestions for how to work with the API including specific endpoints to hit or consumption patterns?
Thank you,
Vance Faulkner
OnConnect API Public Plan is only for you to try out and test internally. Therefore, you cannot use it in your open source tool.
So I'm trying to figure out how much capabilities comes with Intersystems to send data to an XDS repository. Specifically with using the basic Ensemble package (NO HSF) Assume it's not the one Intersystems delivers, but an external XDS repository.
Is there a built-in way to send a large blob and wrap the ebRim around that blob?
As you can see at http://www.intersystemsbenelux.com/media/media_manager/pdf/1398.pdf, Ensemble does not natively support ebRIM, but it does support XML and XML schemas.
Maybe you could assemble an XML and use that to wrap your blob content.
You can send that over whatever protocol your XDS system provides (xDBC, SOAP, file system etc). Take a look at the items listed on sections "Ensemble Interoperability" and "Ensemble Adapter and Gateway Guides" of http://docs.intersystems.com/ens20122/csp/docbook/DocBook.UI.Page.cls for a full list of connectivity options.
Regards,
There is healthshare foundation product which has XDS connectivity
See this good answer on google groups https://groups.google.com/forum/m/?fromgroups#!topic/Ensemble-in-Healthcare/h7R300H68KQ
Or healthshare part of their website
HSF (HealthShare Foundation) XDS.b connectivity for query and retrieve and also the Provide and Register Operation.
Ok, so I re-read your question and have an answer for you. I think what you are trying to say is that you have Ensemble, not HSF, and you still want to be able to send documents (XDS provide and Register).
I did some testing with the Open Source Integration mirth and stumbled across an example channel of theirs, and it is doing a provide and register with straight up SOAP calls to the end point.
Basically, build the required soap envelope accordingly, then send a PDF or document to the repository using MTOM.
This is what makes HealthShare its money, encapsulating all that manual construction of objects that need to be sent to endpoints.
Anyway, a screenshot of the Mirth channel destination make give you an understanding:
http://www.integrationrequired.com/wp-content/uploads/2013/02/Capture.PNG
I need to take a few RSS feeds, and archive all the items that get added to them. I've never consumed or created RSS before, but I know xml, so the format seems pretty intuitive.
I know how to parse the feed: How can I get started making a C# RSS Reader?
I know I can't rely on the feed server to provide a complete history: Is it possible to get RSS archive
I know I'll have to have some custom logic around duplicates: how to check uniqueness (non duplication) of a post in an rss feed
My question is, how can I ensure I don't miss any items? My initial plan is to write a parser, where for each item in the feed:
1) Check to see if it's already in the archive database
2) If not, add it to the database
If I schedule this to run once a day, can I be confident I won't be missing any items?
It depends on the feed, some sites publish articles very frequently and may have their RSS feed configured to show only the 10 most recent articles. Some sites are going to do the opposite.
Ideally your app should 'learn' the frequency this from the sites and tune itself to ping those sites based on the learnt frequency. (Ex: If you see new unique articles every time you ping, you'll need to ping more often, on the other hand if you see the same set of articles on multiple attempts, you may back off the next time).
If you're open to relying on a service for this... I built my own RSS archival service (https://app.pub.center). You can access an RSS feed's data via our API.
Page 1 of The Atlantic
https://pub.center/feed/02702624d8a4c825dde21af94e9169773454e0c3/articles?limit=10&page=1
Page 2 of The Atlantic
https://pub.center/feed/02702624d8a4c825dde21af94e9169773454e0c3/articles?limit=10&page=2
The REST API is free. We have pricing plans for push notifications (email, SMS, your custom API endpoint)
Use a series of decisions based on the feed and the storage limitations. For example:
Connect to the Web site, and download the XML source of the feed. The Feed Download Engine downloads feeds and enclosures via HTTP or Secure Hypertext Transfer Protocol (HTTPS) protocols only.
Transform the feed source into the Windows RSS Platform native format, which is based on RSS 2.0 with additional namespace extensions. (The native format is essentially a superset of all supported formats.) To do this, the Windows RSS Platform requires Microsoft XML (MSXML) 3.0 SP5 or later.
Merge new feed items with existing feed items in the feed store.
Purge older items from the feed store when the predetermined maximum number of items have been received.
Optionally, schedule downloads of enclosures with Background Intelligent Transfer Service (BITS).
Use HTTP to its fullest to minimize wasted bandwidth:
To limit its impact on servers, the Feed Download Engine implements HTTP conditional GET combined with Delta encoding in HTTP (RFC3229) World Wide Web link. This implementation allows the server to transfer a minimal description of changes instead of transferring an entirely new instance of a resource cached on the client. The engine also supports compression using the HTTP gzip support of Microsoft Win32 Internet (WinInet).
A successful synchronization means that the feed was successfully downloaded, verified, transformed into the native format, and merged into the store. A server response of HTTP 304 Not Modified in response to a HTTP conditional GET (If-Modified-Since, If-None-Match, ETag, and so on) also constitutes success.
And define criteria for removal:
The following properties directly affect the number of items that remain after a synchronization operation.
PubDate—used to determine the "age" of items. If PubDate is not set, LastDownloadTime is used. If the feed is a list, the order of items is predetermined and PubDate (if present) is ignored.
MaxItemCount—a per-feed setting that limits the number of archived items. The feed's ItemCount will never exceed the maximum, even if there are more items that could be downloaded from the feed.
ItemCountLimit—the upper limit of items for any one feed, normally defined as 2500. The value of MaxItemCount may not exceed this limit. Set MaxItemCount to ItemCountLimit to retain the highest possible number of items.
References
Understanding the Feed Download Engine
Really simple question here:
For a PHP-driven RSS feed, am I just overwriting the same XML file every time I "publish" a new feed thing? and the syndicates it's registered with will pop in from time to time to check that it's new?
Yes. An RSS reader has the URL of the feed and regularly requests the same URL to check for new content.
that's how it works, a simple single xml rss file that gets polled for changes by rss readers
for scalability there there is FeedTree: collaborative RSS and Atom delivery but unlike another well known network program (bittorrent) it hasn't had as much support in readers by default
Essentially, yes. It isn't necessarily a "file" actually stored on disk, but your RSS (or Atom) is just changed to contain the latest items/entries and resides at a particular fixed URL. Clients will fetch it periodically. There are also technologies like PubSubHubbub and pinging for causing updates to get syndicated closer to real-time.
Yes... BUT! There are ways to make the susbcribers life better and also improve your bandwidth :) Implement the PubSubHubbub protocol. It will help any application that wants the content of the feed to be notified as soon as it's available. It'es relatively simple to implement on the publisher side as it only involves a ping.