I have launched the Scrapy shell and have successfully pinged Wikipedia.
scrapy shell http://en.wikipedia.org/wiki/Main_Page
I am confident that this step is correct, judging by the verbose nature of Scrapy's response.
Next, I'd like to see what happens when I write
hxs.select('/html').extract()
At this point, I get the error:
NameError: name 'hxs' is not defined
What is the problem? I know that Scrapy is installed fine, has accepted the URL for the destination, but why would there be an issue witht the hxs command?
I suspect you are using the version of Scrapy that doesn't have hxs on the shell anymore.
Use sel instead (deprecated after 0.24, see below):
$ scrapy shell http://en.wikipedia.org/wiki/Main_Page
>>> sel.xpath('//title/text()').extract()[0]
u'Wikipedia, the free encyclopedia'
Or, as of Scrapy 1.0, you should use the Selector object of the response, with it's .xpath and .css convenience methods:
$ scrapy shell http://en.wikipedia.org/wiki/Main_Page
>>> response.xpath('//title/text()').extract()[0]
u'Wikipedia, the free encyclopedia'
FYI, quote from Using selectors in the Scrapy documentation:
... after the shell loads, you’ll have the response available as response shell variable, and its attached selector in response.selector attribute.
...
Querying responses using XPath and CSS is so common that responses include two convenience shortcuts: response.xpath() and response.css():
>>> response.xpath('//title/text()')
[<Selector (text) xpath=//title/text()>]
>>> response.css('title::text')
[<Selector (text) xpath=//title/text()>]
You should use your verbose nature of Scrapy's response.
$ scrapy shell http://en.wikipedia.org/wiki/Main_Page
if your verbose looks like this:
2014-09-20 23:02:14-0400 [scrapy] INFO: Scrapy 0.14.4 started (bot: scrapybot)
2014-09-20 23:02:14-0400 [scrapy] DEBUG: Enabled extensions: TelnetConsole, CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState
2014-09-20 23:02:15-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-09-20 23:02:15-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-09-20 23:02:15-0400 [scrapy] DEBUG: Enabled item pipelines:
2014-09-20 23:02:15-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2014-09-20 23:02:15-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2014-09-20 23:02:15-0400 [default] INFO: Spider opened
2014-09-20 23:02:15-0400 [default] DEBUG: Crawled (200) <GET http://en.wikipedia.org/wiki/Main_Page> (referer: None)
[s] Available Scrapy objects:
[s] hxs <HtmlXPathSelector xpath=None data=u'<html lang="en" dir="ltr" class="client-'>
[s] item {}
[s] request <GET http://en.wikipedia.org/wiki/Main_Page>
[s] response <200 http://en.wikipedia.org/wiki/Main_Page>
[s] settings <CrawlerSettings module=None>
[s] spider <BaseSpider 'default' at 0xb5d95d8c>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
Python 2.7.6 (default, Mar 22 2014, 22:59:38)
Type "copyright", "credits" or "license" for more information.
your verbose will displays Available Scrapy objects
so hxs or sel depends on what you have display on your verbose. For your case hxs wasn't available so you'll need to use 'sel' (newer scrappy version). So for some hxs is ok and others sel is what they'll need to use
"sel" shortcut is deprecated , you should use response.xpath('/html').extract()
Related
I am trying to use the Flyway trial period for teams, but I can't seem to use it with any of the newer CLIs.
I've been able to use it with the enterprise-specific version from here, but I can't use it with any of the newer versions that are supposed to contain both community and teams code. I've tried both 9.9 (latest) and 9.2.
The license key is set in my conf file and I'm specifying the edition both through environment variables and flags, but it doesn't seem to pick it up.
I should also note that besides the batch and license key configuration settings, this validate works just fine with all three versions using community.
Command:
FLYWAY_EDITION=teams
$FLYWAY_PATH/flyway -teams -X validate -defaultSchema=$default_schema -url="$JDBC_CONNECTION_STRING" -configFiles=$CONF_FILE -placeholders.segment=${{ parameters.segment }}
Result:
DEBUG: Loading config file: /azp/_work/1/s/flyway-9.9.0/conf/flyway.conf
DEBUG: Unable to load config file: /root/flyway.conf
DEBUG: Unable to load config file: /azp/_work/1/s/infra/flyway/flyway.conf
DEBUG: Loading config file: /azp/_work/1/s/infra/flyway/./dev.conf
DEBUG: Using configuration:
DEBUG: flyway.baselineDescription -> Base Migration
DEBUG: flyway.baselineOnMigrate -> true
DEBUG: flyway.baselineVersion -> 0.0
DEBUG: flyway.batch -> true
DEBUG: flyway.cleanDisabled -> false
DEBUG: flyway.configFiles -> ./dev.conf
DEBUG: flyway.defaultSchema -> METADATA
DEBUG: flyway.ignoreMigrationPatterns -> *:pending
DEBUG: flyway.jarDirs -> /azp/_work/1/s/flyway-9.9.0/jars
DEBUG: flyway.jdbcProperties.authenticator -> snowflake_jwt
DEBUG: flyway.jdbcProperties.schema -> METADATA
DEBUG: flyway.jdbcProperties.tracing -> FINEST
DEBUG: flyway.jdbcProperties.user -> *****
DEBUG: flyway.licenseKey -> FL****************
DEBUG: flyway.locations -> filesystem:sql
DEBUG: flyway.placeholders.environment -> dev
DEBUG: flyway.placeholders.segment -> ***
DEBUG: flyway.schemas -> METADATA
DEBUG: flyway.url -> jdbc:snowflake://aza98716.us-east-1.snowflakecomputing.com/?(...)
DEBUG: flyway.user -> ******
ERROR: Unexpected error
org.flywaydb.core.internal.license.FlywayTeamsUpgradeRequiredException: Flyway Teams Edition upgrade required: batch is not supported by Flyway Community Edition
Try Flyway Teams Edition for free: https://rd.gt/2VzHpkY
at org.flywaydb.core.api.configuration.ClassicConfiguration.setBatch(ClassicConfiguration.java:1005)
at org.flywaydb.core.api.configuration.ClassicConfiguration.configure(ClassicConfiguration.java:1615)
at org.flywaydb.core.api.configuration.FluentConfiguration.configuration(FluentConfiguration.java:943)
at org.flywaydb.commandline.Main.getLegacyConfiguration(Main.java:281)
at org.flywaydb.commandline.Main.main(Main.java:122)
TIA!
For 8.5.10 onwards, Flyway Teams was moved into a separate package so those features can't be used in the Community version anymore.
You have to download the Teams version separately from https://download.red-gate.com/maven/release/org/flywaydb/enterprise/flyway-commandline/
I've been trying without success to connect to a local mySQL database, it keeps failing with the errors listed below.
[root#stargate drivers]# flyway info -X -url='"jdbc:mysql://localhost:3306/sms_database"'
DEBUG: Loading config file: /home/development/wip/flyway-9.1.3/conf/flyway.conf
DEBUG: Loading config file: /root/flyway.conf
DEBUG: Unable to load config file: /home/development/wip/flyway-9.1.3/drivers/flyway.conf
DEBUG: Using configuration:
DEBUG: flyway.jarDirs -> /home/development/wip/flyway-9.1.3/jars
DEBUG: flyway.locations -> service-sms/service-sms
DEBUG: flyway.password -> ********
DEBUG: flyway.url -> "jdbc:mysql://localhost:3306/sms_database"
DEBUG: flyway.user -> root
ERROR: Unexpected error
org.flywaydb.core.api.FlywayException: No database found to handle "jdbc:mysql://localhost:3306/sms_database"
at org.flywaydb.core.internal.database.DatabaseTypeRegister.getDatabaseTypeForUrl(DatabaseTypeRegister.java:54)
at org.flywaydb.core.internal.jdbc.DriverDataSource.<init>(DriverDataSource.java:109)
at org.flywaydb.core.internal.jdbc.DriverDataSource.<init>(DriverDataSource.java:89)
at org.flywaydb.core.api.configuration.ClassicConfiguration.configure(ClassicConfiguration.java:1528)
at org.flywaydb.core.api.configuration.FluentConfiguration.configuration(FluentConfiguration.java:943)
at org.flywaydb.commandline.Main.main(Main.java:157)
I've confirmed connectivity:
[root#stargate drivers]# mysql -h 127.0.0.1 -u root -ppassword
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 26
Server version: 8.0.23-vitess
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> status;
--------------
mysql Ver 8.0.26 for Linux on x86_64 (Source distribution)
Connection id: 26
Current database:
Current user: vt_app#localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 8.0.23-vitess
Protocol version: 10
Connection: 127.0.0.1 via TCP/IP
Server characterset: utf8mb4
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
TCP port: 3306
Binary data as: Hexadecimal
--------------
I've tried both the mariadb-java-client-2.7.3.jar and mysql-connector-java-8.0.30.jar drivers and error is the same. Any advice ?
This is fixed by removing the extra '' characters from the URL string.
I can't stream a video over network using vlc. When I look logfile I see these lines
What is the problem and how can I solve it? My operating system is windows 10 64 bit. Also I need to make a script to stream a video over network with vlc. I could not find any helpful document about this. Can somebody help me about this issue?
main debug: toggling resume
main debug: auto hiding mouse cursor
main debug: auto hiding mouse cursor
main debug: exiting
main debug: exiting
main debug: no exit handler
main debug: removing all interfaces
main debug: removing module "qt"
main debug: deactivating the playlist
main debug: incoming request - stopping current input
main debug: killing decoder fourcc `h264'
main debug: removing module "h264"
main debug: incoming request - stopping current input
main debug: removing a sout input (sout_input: 0000026d28625c20)
main debug: incoming request - stopping current input
main debug: killing decoder fourcc `h264'
main debug: removing module "avcodec"
main debug: incoming request - stopping current input
main debug: saving a free vout
main debug: incoming request - stopping current input
main debug: reusing provided vout
main debug: incoming request - stopping current input
mux_ts debug: removing input pid=100
mux_ts debug: freeing video PID 100
main debug: incoming request - stopping current input
main warning: no more input streams for this mux
main debug: incoming request - stopping current input
main debug: removing module "mp4"
mp4 debug: freeing all memory
main debug: incoming request - stopping current input
main debug: Program doesn't contain anymore ES
main debug: removing module "record"
main debug: incoming request - stopping current input
main debug: removing module "cache_read"
main debug: incoming request - stopping current input
main debug: removing module "filesystem"
main debug: incoming request - stopping current input
main debug: dead input
qt debug: IM: Deleting the input
main debug: nothing to play
main debug: destroying useless sout
main debug: destroying chain... (name=duplicate)
main debug: removing module "duplicate"
duplicate debug: closing a duplication
main debug: destroying chain... (name=display)
main debug: removing module "display"
main debug: destroying useless vout
main debug: removing module "direct3d11"
direct3d11 debug: Direct3D11 resources destroyed
direct3d11 debug: Direct3D11 device adapter closed
direct3d11 error: SetThumbNailClip failed: 0x800706f4
direct3d11 debug: Win32 Vout EventThread terminating
direct3d11 debug: Win32VoutCloseWindow
direct3d11 debug: WinProc WM_DESTROY
main debug: removing module "inhibit"
qt debug: releasing video...
qt debug: Video is not needed anymore
main debug: removing module "freetype"
main debug: removing module "yuvp"
main debug: removing module "swscale"
main debug: destroying chain done
main debug: destroying chain... (name=standard)
main debug: removing module "stream_out_standard"
main debug: removing module "mux_ts"
main debug: removing module "udp"
main debug: destroying chain done
main debug: destroying chain done
main debug: removing module "mmdevice"
qt debug: requesting exit...
qt debug: waiting for UI thread...
qt debug: QApp exec() finished
qt debug: Video is not needed anymore
qt debug: Killing extension dialog provider
qt debug: ExtensionsDialogProvider is quitting...
main debug: removing module "lua"
main debug: removing module "win32"
main debug: removing module "hotkeys"
main debug: removing module "win_msg"
main debug: destroying
main debug: saving media library to file C:\Users\gulbe\AppData\Roaming\vlc\ml.xspf.tmp30208
main debug: looking for playlist export module matching "export-xspf": 4 candidates
main debug: using playlist export module "export"
main debug: removing module "export"
main debug: deleting item `Media Library'
main debug: deleting item `videoOo.mp4'
main debug: deleting item `Playlist'
main debug: removing module "memory"
-- logger module stopped --
-- logger module started --
main: one instance mode ENABLED
I install ROS on my server, but when I called ros start and it will running at my server, here is the log:
login as: root
root#*.*.*.*'s password:
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-109-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Welcome to Alibaba Cloud Elastic Compute Service !
root#iZwz940pq66re8qvh8adzuZ:~# ros start
info: Loaded feature token capabilities=[Sync], expires=Wed Apr 19 2017 22:15:29 GMT+0800 (CST)
info: Realm Object Server version 2.5.1 is starting
info: [sync] Realm sync server started ([realm-core-4.0.4], [realm- sync-2.1.10])
info: [sync] Directory holding persistent state: /root/data/sync/user_data
info: [sync] Operating mode: master_with_no_slave
info: [sync] Log level: info
info: [sync] Download log compaction is enabled
info: [sync] Max download size: 131072 bytes
info: [sync] Listening on 127.0.0.1:35571 (sync protocol version 22)
info: Realm Object Server has started and is listening on http://0.0.0.0:9080
But when I entered the address in the browser, It told me that I could not connect.And I use Realm Studio to connect it also tell me that could not reach the server, did i forget something steps? Maybe my server's security policy forbide the port?
Per to the log description, Realm Object Server has started and is listening on http://0.0.0.0:9080.
Please ensure you've allowed TCP port 9080 in your ECS security group.
For detail steps, please refer the document
Add a security group rule
I installed openstack. All services are running successfully.
[root#test ~]# nova-manage service list
Binary Host Zone Status State Updated_At
nova-cert localhost.localdomain nova enabled :-) 2012-11-06 04:25:36.396817
nova-scheduler localhost.localdomain nova enabled :-) 2012-11-06 04:25:41.735192
nova-network compute nova enabled :-) 2012-11-06 04:25:42.109157
nova-compute compute nova enabled :-) 2012-11-06 04:25:43.240902
After that I change HOSTNAME in /etc/sysconfig/network to myhost.mydomain. Then restart the services.
Now I get the duplicate entry for the services.
[root#test ~]# nova-manage service list
Binary Host Zone Status State Updated_At
nova-cert localhost.localdomain nova enabled XXX 2012-11-06 04:25:36.396817
nova-cert myhost.mydomain nova enabled :-) 2012-11-06 05:25:36.396817
nova-scheduler localhost.localdomain nova enabled XXX 2012-11-06 04:25:41.735192
nova-scheduler myhost.mydomain nova enabled :-) 2012-11-06 05:25:41.735192
nova-network compute nova enabled :-) 2012-11-06 04:25:42.109157
nova-compute compute nova enabled :-) 2012-11-06 04:25:43.240902
From these services old services are not running.
I want to remove the services for host localhost.localdomain.
I check the nova-manage service --help but there is no option for the delete :(.
[root#test ~]# nova-manage service --help
--help does not match any options:
describe_resource
disable
enable
list
Looking at your example above, I suspect you're seeing a duplicate because you have two hosts with their hostnames set identically. If this is the case, the following code/answer isn't likely to help you out too much. There's an implicit assumption in that whole setup that hostnames of nodes upon which nova worker processes run will be unique.
In the latest branch, there isn't a command explicitly enabled for this, but the API exists underneath to do what you're after. Here's a snippet of code (untested!) that should do what you want; or at least point you to the relevant API if you're interested.
from nova import context
from nova import db
hostname = 'some_hostname'
service_name = 'nova_service_you_want_to_destroy'
ctxt = context.get_admin_context()
service = db.service_get_by_args(ctxt, hostname, service_name)
#... pick one of these services ...
#... assign it to 'service'
db.service_destroy(ctxt, service[id])
NOTE: this will remove the service from the database, or raise an exception if it doesn't exist (or something else goes wrong). If the service is running, expect that it will just "show up" again, as the service list is populated by the various nova worker agents processes reporting in.