I'm currently using Splinter to build some web automation scripts and so far so good, but I've run into an issue where I can't actually fix it using the Splinter wrapper functions. But, I believe I might have found a solution that can be fixed using the Selenium Webdriver functions even though I'm primarily using Splinter.
I believe I've done this before years ago, but no matter where I've searched (Documentation, Google, Stackoverflow) I can't seem to find anything on this, so maybe it's not a feature anymore?
Anyways basically I need to access the Selenium Webdriver functions.
From memory, I believe the code was something like:
browser = splinter.browser("firefox")
brower.visit("google.com")
browser.webdriver.find_element_by_id('loginForm') #This is calling the selenium web driver direcetly, not the Splinter find_by_id function.
.webdriver
.driver
both do not seem to work.
Does anyone know how to correctly do this? Let me know if you need any more information. Thanks for your help.
The selenium is exposed by driver browser attribute:
>>> from splinter import Browser
b>>> b = Browser()
>>> b.driver
<selenium.webdriver.firefox.webdriver.WebDriver (session="e607ceaa-2c63-435e-9991-432376102bf5")>
>>>
Related
I've got a little problem with my scrapy spider. So I set up scrapy and all is working fine but everytime I want to scrape a website I have to start the spider by myself. But I want it to be full automated and doesn´t know how to do.
Actually I start the spider with cmdline.execute. I thought I could simply write a while True loop but turns out it doesn´t work. And i found out, that the spider doesn´t really quit. Hard to explain. Pycharm says "Finished with exit code 0" but if i put a print("End of program") after the cmdline.execute it doesnt print out anything.
And at this point I'm confused what to do. Can you help me?
Try using scrapyd.
Scrapyd is an application for deploying and running Scrapy spiders. It enables you to deploy (upload) your projects and control their spiders using a JSON API.
Some tutorials:
How to deploy scrapy spider using scrapyd?
Deploy, Schedule & Run Your Scrapy Spiders
There are many options for scheduling spiders.
CRON:
Like Alexander commented you can create a CRON Job, it think this is best suited for a situation where you have just a few spiders that you're not gonna change the schedule for often.
Scrapydweb: It's a web interface for managing scrapyd. You must host it yourself. Quite easy to use in my experience.
Zyte: Practially the same as scrapydweb but it's a SaaS app that you do not host yourself. Very easy to use but expensive.
Gerapy: I have not tried it but I believe it's similar to scrapydweb but seems to be built on some more modern frameworks.
Thank you all but I´ve found a solution. I wrote a BASH script and in that script I am able to loop.
Like the title says. I've recently gotten a hold of a Cypress codebase that I need to convert into Playwright. I'm new to both Cypress and Playwright but have experience using other automated testing systems. The last one I used made it pretty easy to set breakpoints on any line that would let me step through the code and see what each line was doing. I figured if I could do this, it would make my deciphering of the Cypress code so I could turn it into something that works with Playwright an easier prospect. Google has not been the most helpful here.
You can use Cypress' debugger to achieve something similar to setting breakpoints while executing via cypress open.
cy.get('foo').debug();
If you prefer VSCode, there is a similar SO question. It seems it require some manual setup to get debugger to work.
For IntelliJ (IDEA, WebStorm etc.), there is a paid Cypress Support Pro plugin (I'm the author). It fully integrates the IDE debugger with Cypress. See this video overview.
Xterm is used when running Corda locally on one computer using gradle.
Is there a way to specify your terminal editor when running as suggested by the following issue?
https://github.com/corda/corda/issues/2605
I completely share your pain on this. The way that runnodes has its tooling baked in makes it impossible for you to customize how the cordform plugin runs the nodes without digging into the internals.
Some other ideas for you
one thing you could do would be to stop using cordform altogether and run your corda network using dockerform (example here: https://github.com/corda/samples-java/blob/master/Features/dockerform-yocordapp/build.gradle#L93) so that the plugin doesn't need to actually create new terminals.
the much harder way would be to actually download the corda gradle plugins (https://github.com/corda/corda-gradle-plugins#installing-locally) and install it locally with your edits to the cordform task so that it opens the terminal of your choice. You may be able to PR them as the cordform task that's usually used to generate the runnodes script comes from here as far as I know.
As a separate note, I saw your github issue and I was disappointed by how that got handled. I'm sorry you had that experience and I'm going to dig into that issue internally to find out what's happening with that.
feel free to reach out to me (David Awad) on slack.corda.net and I can let you know what's going on there.
Thanks as always
I'm writing tests using webdriver in Java.
Is there a way to run browsers (ff / ie) in headless mode in a windows box?
I cant use a linux box and HtmlUnitDriver is not helping as our webapp has many ajax controls.
I have started a small project to supply a driver for WebDriver using SimpleBrowser. You may want to check it out: https://github.com/Teun/SimpleBrowser.WebDriver
It's still early days though. I would appreciate help in the form of contributions and testing.
At this point there is no headless version that I am aware of the Firefox or IE. Since Firefox is open source it's possible that you could get the source code for that browser and then remove the UI portion but there are a few problems with it that I can see. First you have to build a custom app to setup your own load testing engine. Second you are locked to one web render, though I think chrome is open source as well but it doesn't play as nicely as Firefox or IE (using the system .dlls vs. a custom executable). Third without a lot of testing I am sure there will be stability issues and potentially other unforseen issues with this option. That being said there is a .net headless browser that's fairly simple and would probably be decent for load testing called simple browser. However it depends on what you are wanting to load. If you just looking for bandwidth and server load then you should be find using multiple instances of the browser. I would be curious to know if using the headless browser in this fashion if you instantiated different WebDriver if it would play nicely. i.e.
WebDriver driver1 = new SimpleBrowser();
WebDriver driver2 = new SimpleBrowser();
.
.
.
.
.
WebDriver driver[n] = new SimpleBrowser();
and then you could run [n] parallel tests. Also of note in the above code SimpleBrowser() doesn't hook directly to the WebDriver interface you would have to Extend the interface to reference that .dll
Start the test program as a Windows service and the browsers will run just fine, even on a machine with no monitors. Technically speaking, they're running in a "session" that is disconnected, but it's essentially what you mean by "headless".
HtmlUnitDriver does support ajax, and can run in both java and c# (if ported via IKVM).
There is, however, no way to run IE and FF in headless mode. I can't tell you how many times I've wished for that...
Is it possible to run Google NaCl outside of the browser as a process sandbox?
Yes. The standalone build of Native Client contains a program called "sel_ldr" which runs the NaCl sandbox outside of the web browser. A lot of NaCl's test suite tests NaCl by running programs under sel_ldr rather than under the browser.
The Native Client documentation is probably a good starting point for trying to figure out how everything works.
As Mark Seaborn and Bennet Yee indicate, you are probably best off looking into using sel_ldr. More details about how sel_ldr is included in "The life of sel_ldr".
I haven't personally implemented anything that uses NaCl outside of a browser, so I can't unfortunately comment on the ease of doing so.
To add to Mark's answer, take a look at the sel_universal target. There are obviously some interfaces that cannot be supported, e.g., Pepper interfaces, since such a stand-alone run doesn't involve a browser. Furthermore, the stable ABI that we support is that which is exposed by the "integrated runtime" or irt, and the irt code thunk assumes that the browser is present. a standalone use of NaCl via sel_ldr would probably have to use the syscall interface initially, until an alternate irt is written. (NB: we make no guarantees about the stability of the syscall interface.)
You can try run.py which included in Native Client's source.
Here is an example that runs a hello world program.
make test_hello_world_nexe
python native_client/run.py out/Debug/hello_world_newlib_x64.nexe
This script can build, search and invoke sel_ldr, and pass proper arguments to it automatically.