reponse.css on not working - css

Question

I just started off with scrapy. I've loaded the page http://www.ikea.com/ae/en/catalog/categories/departments/childrens_ikea/31772/ with scrapy shell [url] and ran response.css(div.productTitle.Floatleft) to get product names but it gives me the following error:
Traceback (most recent call last): File "", line 1, in
NameError: name 'div' is not defined.
How can I fix this?

Answer 1

You have to use string: "div.productTitle.Floatleft". See " "
Now you try to use variable div.
EDIT: to get correct data you have to set User-Agent
Run shell
scrapy shell http://www.ikea.com/ae/en/catalog/categories/departments/childrens_ikea/31772/
In shell you can use web browser to see HTML from server and you will see error message.
view(response)
You get page again using different User-Agent (using url from previous response)
fetch(response.url, headers={'User-Agent': 'Mozilla/5.0'})
response.css('div.productTitle.floatLeft')
BTW: it has to be floatLeft, not Floatleft - see lower f and upper L
EDIT: the same as standalone script (doesn't need project)
import scrapy
class MySpider(scrapy.Spider):
name = 'myspider'
#allowed_domains = ['http://www.ikea.com']
start_urls = ['http://www.ikea.com/ae/en/catalog/categories/departments/childrens_ikea/31772/']
def parse(self, response):
print('url:', response.url)
all_products = response.css('div.product')
for product in all_products:
title = product.css('div.productTitle.floatLeft ::text').extract()
description = product.css('div.productDesp ::text').extract()
price = product.css('div.price.regularPrice ::text').extract()
price = price[0].strip()
print('item:', title, description, price)
yield {'title': title, 'description': description, 'price': price}
# --- it runs without project and saves in 'output.csv' ---
from scrapy.crawler import CrawlerProcess
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
'FEED_FORMAT': 'csv',
'FEED_URI': 'output.csv',
})
c.crawl(MySpider)
c.start()
Result in file output.csv:
title,description,price
BÖRJA,feeding spoon and baby spoon,Dhs 5.00
BÖRJA,training beaker,Dhs 5.00
KLADD RANDIG,bib,Dhs 9.00
KLADDIG,bib,Dhs 29.00
MATA,4-piece eating set,Dhs 9.00
SMASKA,bowl,Dhs 9.00
SMASKA,plate,Dhs 12.00
SMÅGLI,plate/bowl,Dhs 19.00
STJÄRNBILD,bib,Dhs 19.00