Is there any program/library available that can scrape the contents of an mobile apps' screen?
The goal is to have a nice data structure for the Instagram "Following" feed.
There is not complete solution for it, but it can be achieved by using different software like bruipsuit, postman, binaries, Amulators or rooted android devices and man in the middle attack technique.
If you need to achieve this you have to make a setup to interrupt the request to the app and decode data from the request.
I've done it for a few apps (not instagram) with screen-scraper.com. I set the phone to use wifi only, and logged the requests on the router. I used that to emulate the request the app makes. Usually it's a pretty easy HTTP request, and they can be touchy about the user agent. Then the data is almost always XML or JSON.
With appium you can write test for mobile app, using uiatomatorviewer find the elements address and then try to use norest=true capability you can scrap data.
https://www.softwaretestinghelp.com/automate-native-app-on-android-device/
Related
I have recently purchased a Nest Camera (Wired, Indoor) today and I am wanting to stream it to Youtube so I can embed a live-feed into my website and so I can share the stream with my friends so I do not have to add them to the Google Home application and give them access to all devices.
I've seen the api documents for it, but I do not understand them. I found the API Documents on google's developer site (https://developers.google.com/nest/device-access/api/camera-wired#extend_a_live_stream), however I do not understand how this works, or how to complete it.
In my personal opinion, there should be a feature to stream it to a platform such as Youtube, such a feature would make things a lot easier for many people.
I have so far found no documents explaining how to do this except from the api documents.
There is no simple "out of the box" way to do this. You would need to setup some kind of device as an intermediary. One one side it would connect to the SDM API, open a stream, and start receiving the data. On the other side it would need to connect to the YouTube API (or equivalent) and pass the data through. Unfortunately you would need some degree of programming skill to engineer such as system.
I've been experimenting with web scraping using Scrapy, and I was interested in retrieving all text messages from all chats on Whatsapp to use as training data for a Machine Learning project. I know there are websites that block web crawlers/scrapers, so I would like to know if it is possible to use Scrapy to obtain these messages, and if it isn't possible, what are some alternatives I can use?
I understand that I can click on the "Email chat" option for each chat, but this might not be feasible if I want to obtain a large amount of data, not just from my own chats, but from other people who are willing to let me use their chats for the project.
I think WhatsApp do not block crawlers and scrapers. You have access only to your web.whatsapp.com. It's your matter what will you do with your messages. When I write code to read/write WhatsApp messages I used Selenium WebDriver, which can fully automate any browser actions. It worked too stable for WhatsUpp. It was not fully automation, be course of QR code.
If you press F12 and go to "network" tab in web browser, you will notice XHR packets with messages inside. You can see it when you load new messages during scrolling or opening person. It look like byte data.
Thank you to Mohit Jindal. You are right there is a way to use browser profile like that:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('user-data-dir=selenium/')
driver = webdriver.Chrome(options=chrome_options)
It will crate Chrom profile in "selenium/" folder. This way allow you to login using your phone just initial time.
I want use Google Assistant from my phone to send HTTP POST command to my server. I have a simple webnms app running over it, this server support REST API and now I want to use Google Assistant to shoot GET or POST command to that server and return my output.
Is it something possible? I am not full time developer.
Yes, as #Prisoner says it is possible. It is not what you asked - but have you seen these ways that Google provides to get skills published without requiring a lot of developer savvy?
https://developers.google.com/actions/content-actions/
https://developers.google.com/actions/templates/first-app
I don't speak for them, but IMO Google's target audience for Action building apart from the above is those who have at least some familiarity with the JavaScript language and its "run-time" Node.
There is also this - which I haven't tried by the way.
https://www.techadvisor.co.uk/how-to/digital-home/easy-actions-google-assistant-3665372/
In case it is not obvious, Google Actions are essentially websites that interact with Google's assistant running on a Home device or a smart phone, say. Think of the Assistant as a browser initiating requests and your Action as serving them. If you can (build and?) deploy a server that handles POSTS over HTTPS on a publicly addressable URL, and if you can understand the JSON payload that the Assistant sends and respond with appropriate JSON to carry out you application then you are good to go.
Where you don't have a public IP address - e.g. in testing - you can use a tool like ngrok ( https://ngrok.com/ ) to reverse proxy requests emanating from the Assistant to your server.
I have slides for a presentation I did targeting fledgling developers who had never built an Action here
https://docs.google.com/presentation/d/1lGxmoMDZLFSievf5phoQVmlp85ofWZ2LDjNnH6wx7UY/edit?usp=sharing
and the code that goes with it here
https://github.com/unclewill/parrot
On the upside the code is about as simple as it gets. On the downside it does almost nothing. In particular, it doesn't try to understand language. As #Prisoner says you'll likely need a tool like Dialog Flow for that.
Yes, it is possible.
Your server will need to implement the Actions on Google API. This is a REST API which will accept JSON containing what the user is intending to do and specific information about what they have said. Your server will need to send back JSON indicating the reply, along with additional information about how to continue the conversation.
You will likely also want to use a tool such as Dialogflow to handle building the conversational script and converting a user's phrases into something that makes sense to you. You'll also need to use the Actions on Google console to manage your Action and provide additional details about how users contact your Action. All of this is explained in the Actions on Google documentation.
Simple Actions are fairly easy to develop, and can certainly be done by a developer as a hobby. Good Actions, however, take a lot more thought and planning. Google offers you to the tools - it is up to you to best take advantage of them.
I've found the solution.
In the "Action" console https://console.actions.google.com/project/sandbox-csuite/scenes/Start
Go to menu "Webhook", click "Change fulfillment method", and then select "HTTPS endpoint"
Is it possible to embed an external application inside the browser (IE, Chrome, Safari, Firefox) so it will look like a native web application but actually having access to the USB ports of the client machine? I have heard that I need to make an ActiveX control. I would like to use the .Net framework, but if that is not possible, maybe using Java or C++ will be fine.
I have to make an application that will allow to the users to connect an external device to an USB port, this device will take a backup of the information contained in a SIM card and send it to the user's account online agenda. So the user can restore it later using the same application. This should be a web application or at least look like one.
If the first is not possible. Is there any way to launch an external application from all the browsers, and then pass information to the browser window to allow it to refresh after the backup has been made?
Thanks for your help in advance.
First off this seems to be a big security issue and hence this is the reason why you might be finding it tricky.
What I would do is look at it from a different angle; what am I trying to achieve? How is the user going to use the data? Where is the user going to use the data?
From you question I have answered those questions with the following; I hope I've not miss interpretted anything.
I want to copy the data from an external sim card to a central location
I want the user to see this data from the central location; preferablly from a web application.
The user is going to see and use the data from the web app
Assuming all of these things are true; one design option is the following:
1 - Have a client based application which can read stuff from the usb device
2 - Have a secure webservice which the client based application can upload the data too
3 - Have a web application which can view this data and see refreshes
Let me go into bit more detail for each step.
1 - If you write a small client application it is installed or at least runs on the client computer. Due to this it can access the local client resources such as usb and interface with them. This will mean they can read the sim data throuogh this app, buut also potentially save it locally as well as upload the data. To access the web service they would enter their username/password so you could authenticate them for the upload.
2 - This web service would do the authentication from the client application, but also receive the data submitted from the client app. Acessing web services from .net now a days is really straight forward. Using this web service the client application could also do some checking to make sure the data has been updated and it could handle re-tries if the network dropped etc.
3 - The web front end of the system would interface to the same data source. This site would take the username / password to authenticate them on the site, but also let them see the uploaded data. As for the refreshes; if the user is logged in and looking at the data you could have a javascript timer polling an action/service to see when new records have been added etc. This could then display a message through jQuery or similar to notifiy the user. This could be similar to the notifications which StackOverflow gives when you visit for the first time or get a new badge etc.
Hope this helps :-)
I want to develop a web application that will have textbox and button. When the user enters a number and clicks on the submit button, I want to make a Skype call from the host. How can I do this?
Er... you're in for a world of pain. First, sure, your server could possibly make a skype call, but how would you handle the back-and-forth to your user's browser? Tons of custom programming with a rich client framework, that's how.
Better to let the user just use skype themselves. Look into the callto:// URI header for details.
Can't you setup your links with callto:// prefixes? If the user has the option enabled in Skype then the application will be associated with those links.
How to Use the Skype Callto:// Links
Perhaps you could build a link on the fly and initiate a call this way?
You can generate a link with HTML with a callto: address to launch the Skype client and make the call (if the client software is configured to do so), which is hopefully what you're looking for. See: http://chrisabraham.com/2005/05/05/how-to-hyperlink-to-a-skype-call-with-your-skype-contact-id-or-telephone-number/
If you want to actually get into the API, you'll need some kind of desktop C# application and interface with the Skype API, see following for details:
http://share.skype.com/sites/devzone/2006/12/c_example_project_for_skype4co.html
http://www.codeproject.com/KB/cs/skypecontrolapicsharp.aspx
So lets say you did launch a process on the server and automate the skype client on the web server. It then dials your customer via a phone located in the data center and the customer hears .... what? IIS isn't very talkative.
I know this is an old thread but we are getting a few visitors to our site from it. If you want to embed clickable links into your website that work with Skype please use the Skype URIs, see: https://dev.skype.com/skype-uri.
[Skype:echo123?call][1]