Form filling and web scraping from a website - web-scraping

Not a good way to start, but pardon me if this if off-topic, it seems like a programming question though...
From an ASP.NET website I want to open a page in a new browser that has a toolbar at top and an iframe-like window at the bottom. The frame-like window will support tabbed browsing and load a third party website. The toolbar will have buttons that allow the user to manipulate the HTML (form-fill and web-scrape). For example, toolbar buttons may be "Extract Webpage Data" or "Fill Form".
Ideally it would work with IE, Edge, Chrome and Safari, but an absolute minimum requirement is IE, a more preferable minimum requirement is Chrome and Edge.
I have seen this done, well, by other proprietary software. I do not know if they require a specific browser (like IE where they can install a plugin) or how they do it, that is my question.
So I have narrowed this problem down to three possibilities:
Use pure HTML, Javascript, et al. - Using an iFrame almost works perfectly but the content will not be in the same domain so I cannot access the iFrame's HTML.
Use (or write) a proprietary browser - I do not think you can (or want) to launch an EXE from a web page, plus this seems rather complex in itself.
Use (or write) a plug-in - Probably limits use to IE. I think an IE plugin could do what I want based on other plugins I've seen.
I have past desktop programming experience with a web automation and scripting product, while promising, I don't think they offer what I need:
They have an ASP.NET COM component that runs server side so it does not display an interface to the user but can be used to silently fill and scrape a website based on scripts.
They also have a proprietary browser that shows a user interface and runs scripts to fill and scrape. But this is an EXE, so cannot be launched from a web site.
They have an IE Plugin, that adds a companion popup window that attaches itself to IE. Similar to their browser and runs scripts.
Question - This can be done, I've seen it, but what is the mechanism? I'm leaning to an IE plugin.
If plugins are the answer, chrome has extensions, is that a possibility?

Related

Locating the url of every CSS-background image called by a web page

I have WordPress page whose theme seems to be loading an undesirable CSS background-image. I want to try to locate the code that is responsible for loading this image but first I have to find its name. I was wondering if there is some tool that allows coders to list and identify the urls of all css-background images that a page calls.
Obviously finding background-images is trivial if all of the page's CSS is controlled only by inline code and directly linked CSS-stylesheets: a text search "find" operation for "background-image" will allow you to find all bg images. But the task becomes complicated if styling is js dependent, and in this case, it was at times.
For those of you who immediately wonder why I need would want to go this route instead of simply using development tools in Chrome or Firefox, below is a list of reasons why.
Why I want a way to automatically extract the background-image urls:
The unwanted ghost images only loads on my mobile phone, so I can't inspect the element to find the image in using my desktop development environment. This is true even if I set the developer tools to "mobile".
I don't have a development environment for my phone that will let me inspect the relevant element.
I tried downloading the exact html loaded by the mobile phone in my browser, but the css ghost image will not appear on my desktop even when use the code my phone had loaded.
The ghost image is not from a virus in my mobile phone browser, because the ghost image loads on my phone even when I used a different browser.
Any help will be greatly appreciated!
Thanks
UPDATE: I figured out the cause of the ghost image using the free trial provided by Browserstack, a mobile emulator. At allows you to view interactive content and inspect it with dev tools. I learned about it from this question: test mobile website in desktop browser.
My problem turned out to be that the css-image in the theme was pointing to my local address, which different from what it should have been on my remote server. The issue turned out not to be a desktop vs. mobile problem, but rather local vs. remote. The emulator reproduced my issue, and it allowed me to inspect and find the problematic code. Still I would love to know if there is a such a css-crawling tool, so I will leave this question open.

Previewing Changes in Bigcommerce Mobile Template

Hopefully someone can help me out with this. I've been tasked with giving a mobile template an overhaul on a site that uses BigCommerce.
Does anyone know of an IDE or a windows based browser that would allow me to preview the changes that I make on the fly, preferably via Dreamweaver CC but this isn't a necessity.
Basically I want to completely revamp the homepage so that instead of having a mundane and dreary menu that take up the entire page, I'd like to change it so that I have responsive windows 8 like tiles that one could use to navigate the site.
Since BigCommerce is a paid service you are limited to what is available. You cannot simply download the site and run it on your localhost. I worked on one of those sites before, the best work around I found what to edit in google chrome's inspect element and console "F12" - this way you can make changes directly to the site to see how they render. But keep in mind, F12 is after chrome builds the page and will not match the actual code all the time.

Can I disable ad-ons from appearing on my website by modifying my xhtml code?

I developed a website for my husband’s company and just discovered that when I view these pages on my browser, ad-ons cause certain words to be underlined that link to pop-ups that direct users to other sites (potentially our competition). My research has turned up only ways to turn ad-ons off on my browser, but I’d like to fix my html (XHTML 1.0 Transitional) so that customers who view our pages don’t see these advertisements. Is there some way to modify the xhtml to disable ad-ons on a page by page basis?
Assuming the add-on is modifying your html as it's displayed to the browser (likely), you could use jquery to run after your page is loaded (and after add-ons run, so use a timer), to search the page for all links to external sites and if they're not links you put in (maybe identified by class), then remove them.
A bit of a cat and mouse game, but since the add-ons are not specifically targeted to your site, it should work.
Have you tried viewing the site using a different computer? I fix computers all the time and have noticed malware that makes advertising links on websites that have nothing to do with the malware. I would recommend scanning your computer, there are plenty of freeware programs and most anti-virus programs have a free version (AVG, Avast, Comodo, etc) though I would avoid McAffe, Norton and Windows Defender.
You said browser, but you did not specify which browser. In Firefox go to Tools menu at the top (you may have to press ALT or click on a big goofy combined menu at the top-right) or if you can't find the Addons menu press CTRL+SHIFT+a and then remove/disable anything that you did NOT intentionally install.
Since your question has limited scope of details please reply if this doesn't help/help enough and we can work on making sure you can ask the question in a way that can get you answers faster.

Is there any way to programmatically fix DOM/CSS on-the-fly in web browser?

For example, opening the front page of translate.ru, it contains a lot of banners. But by the Developer Tools in Chrome it's possible to go through the page and tweak CSS in-place (mostly adding 'display: none') and eventually the page looks this way:
(source: demin.ws)
So, a question: is there any APIs or existing extensions for Chrome allowing programmatically change DOM/CSS on the page? I saw similar Chrome extensions allowing to tweak Gmail and Google Reader, but it was specifically for those websites.
You can do this using Stylebot.

How do I make a mobile-friendly popup on my website?

Is there a simple way to trigger a mobile OS's native pop-up/alert/etc. from some form of web code? I'm writing an ASP.NET mobile web page and I'd like to, for example, have the iPhone's UIAlertView appear.
EDIT: What I'm looking for is not the method with which to detect which mobile browser is accessing the site (I already know how to do that). If the code to trigger a pop-up that will look nice in an Android browser is different than the code to trigger a pop-up that will look nice in an iPhone browser, I can simply throw in a switch statement that redirects the user to the pop-up that corresponds with their browser. I'm trying to find the html/javascript/asp.net code which will create a mobile-friendly pop-up, either in general or for the various popular mobile web browsers specifically.
Don't know whether there is any pre-built functionality in .NET that can achieve this, but you can surely write one yourself.
You can write a method, that returns the code for your popup, based on the user OS (simple switch statement should do).
EDIT after taking a short nap:
I believe you should reconsider using popups. They are quite annoying even on desktop browsers and many people block them automatically. Probably every blog about accessibility will tell you, that you should keep mobile version of your website as simple as possible because of various compatibility issues that you can run into.
Instead, try to think about some interesting way to incorporate messages for users in a different and appealing way, that won't disturb anybody.
What I do is use a div popup (that floats ontop of the page) and eighter make a big close button or set at timeout to remove it.
jquery mobile is a good place to start.

Resources