Scrape, upload and notify - web-scraping

I am making an app which sends a notifications to users as soon as a new item is available on an external website
Currently this process is Manual, where I check the update myself and send out a notification. I then update my website to show the new item.
I am trying to use a web scraper to run on my website to extract the information and display it in the app but even that is run manually.
Are there any tools that I can use to automate this process?
So something that will:
check the external website for updates, periodically.
Get the change and update my website (WordPress)
Send out a notification to users

I have automated something similar (excluding the WordPress part).
It was also an automation for monitoring a website for new items (apartments for rent): when a new apartment was available - the automation would send me notification on my mobile.
As an automation tool, I used Easybots Studio (https://easybots.net) - simple, free, had all the functionality I needed out of the box and I could write my own 'bots'.
1. HTML Monitoring
In a nutshell, the automation solution looks something like:
On 15 minutes Elapsed
[HTML Bot] Get HTML of the page
[HTML Bot] GetAllBySelector(".someHtmlElementClass") (get all html elements in the list of apartments)
foreach new listing
Pushbullet Bot.Send Notification on my mobile
Click to see picture from the actual implementation in Easybots Studio
The automation is based on the their blog entry ("monitoring a web site for fast-selling theater tickets"), which might be helpful here as well.
So, if you go that path, you will need to explore the HTML of your external website, and see how are the items listed. That is needed for the HTML bot can get all items either by 'SelectAllByXPath' or by 'GetAllBySelector'.
2. WordPress Automation
As for automating the WordPress part - I can't find bot doing that on their web site, nut you can develop your own app (WordPress has REST API possibility), or even maybe contact the easybots support to see if they have such bot in pipeline.
If you are a .Net dev - here is a starting point to using WordPress Rest APIs: https://dev.to/yeisonpx/connecting-to-wordpress-using-c-5688
3. Sending Notificatons
In Easybots, there are Email bots that you can use to send emails, but also some mobile notifications bots (pushbullet, pushover, ..).
In the question it is not specified what kind of notification is needed, but this should be easy with Easybots.
Other tools out there: WinAutomation, Automation Anywhere, UiPath..

Related

Is there a way to get a developer account to avoid being confused with an automated tool?

I'm currently developing a Chrome extension to use with LinkedIn Sales, and I'm having issues while testing the front end.
Due to some style changes, I had to refresh the page multiple times and now my account is temporarily banned because they confused me with an automated tool.
Does anyone know a workaround for this? Or, alternatively, can I create some type of developer account to use?
TIA!
Due to some style changes, I had to refresh the page multiple times
This is really a suspicious action.
So isn't your "Chrome extension" also an automated tool?
Either get some API access (you find available APIs here https://developer.linkedin.com/product-catalog)
or if you just do some visual brush-up for users, you could try to make an offline copy of the page once (with a tool like HTTrack) and then use that copy for development.
For the Sales API it says
It's required that all integrations are built by approved partners for the SNAP program. To become a partner, visit this page.

Should I use Firebase Dynamic Links over Universal and App Links?

Im building an app for iOS and Android and need to open urls in the app on mobile and in the browser on desktop.
All links should open in the app if possible, so I cant generate one link for each possible url.
Can I do this in Firebase or do I have to create links in advance?
I'm not sure this question is related specifically to Firebase, or a general one.
I can try answering about the general case:
For opening in App:
In order to use Universal Links in iOS and AppLinks in Android, you'll need your click server to host Apple's AASA file or Android's asset file. This means that you'll need to serve that file for the operating system.
For opening in browser on desktop:
You'll need your click server to be able to detect desktop user agent and redirect you to store.
I didn't understand your question about: All links should open in the app if possible, so I cant generate one link for each possible url. Actually one link can fit all use cases, but you'll need to develop that capability in your own click server.

Crawl status saying it can't find any pages to index in my android app

Although I followed the below guide on my app (up to 7th step because I'm not interested in personal contents or on-device indexing):
Firebase appIndexing tutorial
And connected my app to firebase consol.
Also verified steps against "Get Started" guide:
Firebase AppIndex for Android Get Start guide
And followed steps included in Google guide to create deep link in app (except for reading data from coming intent section) as below link:
Google guide for creating deep link to app contents
I also verified the intent code in the manifest using adb command and got it successfully executed for specified url data (scheme, host, and pathPrefix) with added android:autoVerify="true".
The only thing I couldn't accomplish is to declare the "Digital Asset Links" file by uploading it to my website on Weebly, because I couldn't find a way through my search to do so.
Now, after I added the android app into my webmaster as a property and verified the ownership of my website on Weebly then associated the website to the app in the webmaster, I got the following message in the "Crawl status":
"We did not find any pages to index in your app
Your app pages must be found and indexed before Google can start showing them in search results. You can tell Google which pages to index in a sitemap or in web page markup, or if your website and app have identical page organization, Google can infer your app pages from their corresponding web pages."
I'm really stuck here, and don't know what else to do to index my app contents keywords into Google Play for users to find my app when they search these keywords.
Is there any steps I missed in order to have my app indexed? or is there any other method to make my app on Google Play found when searching for specific keywords (that's all what I want, and I created the website specially for this task)?
Thanks in advance for your help.
App Indexing is for finding app pages in Google search, not in Google Play.
Unfortunately, you do need to associate the Digital Asset file, but this is just a text file in a known location: you can see the details in https://developer.android.com/training/app-links/verify-site-associations

How to make Push Notification in Asp.Net Web App with Angular?

I need to make Push Notification for every user, when product's price is changed.
Just like this:
The thing is that I have a job, that runs in the background and checks all the prices and sends emails to everyone informing on changes. I want to add these desktop notifications, so javascript and html5 doesn't work for me. Is there any library for .net that implements this functionality, or does anyone know any links to tutorials that I could read and use them??

Unable to register my application with Google Developers

In my endeavour to gain use of the Youtube API I am required to follow the procedure defined here.
In this link there exists a Prerequisites section that states:
Register your application with Google so that it can submit API
requests.
This takes me here which is a page Titled: "Registering your Application". This page contains instructions stating:
To register a new application, do the following:
Enter the Google Cloud Console https: //cloud.google.com/console [hit my link limit LOL]
Select a project, or create a new one.
In the sidebar on the left, select APIs & auth. In the list of APIs, make sure the status is ON for the YouTube Data API v3.
In the sidebar on the left, select Registered apps.
I have done all of the above. I have also clicked on the project taking me to another page. I am however, unable to register the application.
My question to you is "Where do I find the "Registered apps" because it's not in the sidebar on the left.
Google even provide a video on the instructional page stepping you through it. Unfortunately the video appears to be for an older site.
I am running Firefox 25.0.1. and it doesn't work for me so any guidance would be appreciated.
Instead of Registered Apps, in new version you'll go Credentials under API's and Auth and create your keys there.
I tried everything to help you out but i couldn't find "Registered Apps" too. Have you ever tried "Credentials" ? I think it works for the same way.
They don't care much about updates i guess. That's why they did not renew the instructions and the video..

Resources