I've been trying to extract some text for a while now, and while everything works fine, there is something I can't manage to get.
Take this website : https://duproprio.com/fr/montreal/pierrefonds-roxboro/condo-a-vendre/hab-305-5221-rue-riviera-854000
I want to get the texts from the class=listing-main-characteristics__number nodes (below the picture, the box with "2 chambres 1 salle de bain Aire habitable (s-sol exclu) 1,030 pi2 (95,69m2)", there are 3 elements with that class in the page ( "2", "1" and "1,030 pi² (95,69 m²)"). I've tried a bunch of options in XPath and CSS, but none has worked, some gave back strange answers.
For example, with :
response.xpath('//span[#class="listing-main-characteristics__number"]').getall()
I get :
['<span class="listing-main-characteristics\_\_number">\n 2\n </span>', '<span class="listing-main-characteristics\_\_number">\n 1\n </span>']
For example, something else that works just fine on the same webpage :
response.xpath('//div[#property="description"]/p/text()').getall()
If I get all the spans with this query :
response.css('span::text').getall()
I can find my texts mentioned in the beginning in the. But from this :
response.css('span[class=listing-main-characteristics__number]::text').getall()
I only get this
['\n 2\n ', '\n 1\n ']
Could someone clue me in with what kind of selection I would need? Thank you so much!
Here is the xpath that you have to use.
//div[#data-label='#description']//div[#class='listing-main-characteristics__label']|//div[#data-label='#description']//div[#class='listing-main-characteristics__item-dimensions']/span[2]
you might have to use the above xpath. (Add /text() is you want the associated text.)
response.xpath("//div[#data-label='#description']//div[#class='listing-main-characteristics__label']|//div[#data-label='#description']//div[#class='listing-main-characteristics__item-dimensions']/span[2]").getall()
Below is the python sample code
url = "https://duproprio.com/fr/montreal/pierrefonds-roxboro/condo-a-vendre/hab-305-5221-rue-riviera-854000#description"
driver.get(url)
# get the output elements then we will get the text from them
outputs = driver.find_elements_by_xpath("//div[#data-label='#description']//div[#class='listing-main-characteristics__label']|//div[#data-label='#description']//div[#class='listing-main-characteristics__item-dimensions']/span[2]")
for output in outputs:
# replace the new line character with space and trim the text
print(output.text.replace("\n", ' ').strip())
Output:
2 chambres
1 salle de bain
1,030 pi² (95,69 m²)
Screenshot:
Using a fixed-width font (e.g. Consolas, Courier, etc), I am trying to
populate a dropdown menu (AjaxControlToolkit:DropDownList) that has 2
columns (in appearance). I have a product name and a category name
(neither of which I know until runtime). The appearance I'm looking
for is something like:
Chevy Cruz (gas)
Prius (hybrid)
Tesla Model S (electic)
My list can have over 300 entries and if I just append the category to the
product name, the menu is harder to read.
I've tried using a character array and copying in the category name at the
same index for each ListItem, but the spaces between disappear when the
dropdown list is opened. I've looked at the ListItem(Paragraph) constuctor
but it doesn't look to solve my problem to my understanding of it. I
haven't looked at the Telerik controls I have available because it
would mean a lot of coding changes.
I can't think of another AjaxControlToolkit control that might help.
string padding might be works for you
var _maxLengthOfProductName = 20; //number of space you need
var _productName = "Product Name";
var _type = "(type)";
var _ProductNameWithType = _productName.PadRight(_maxLengthOfProductName, ' ') + _type; //assign this to the dropdown item
_ProductNameWithType = _ProductNameWithType.Replace(" ", " ");
it will show
Product Name (type)
I've been using rvest to scrape pages, and I am aware of the benefits of selectorGadget. However, one page has data WITHOUT selectors. A snippet of the HTML is below. The page is here . I am trying to scrape the list of personnel on each jazz album listed. In the snippet of HTML below, the personnel data begins with "Sonny Rollins, tenor sax..." As you can see, that text is not surrounded by any CSS selectors. Any advice on scraping this out?
<h1>Blue Note Records Catalog: 4000 series</h1>
<div id="start-here"><!-- id="start-here" --></div>
<div id="catalog-data">
<h2>Modern Jazz 4000 series (12 inch LP)</h2>
<h3><a href="./album-index/#blp-4001" name="blp-4001">BLP 4001 Sonny
Rollins - Newk's Time <i>1959</i></a></h3>
Sonny Rollins, tenor sax; Wynton Kelly, piano #1,2,4-6; Doug Watkins, bass
#1,2,4-6; Philly Joe Jones, drums.
<div class="date">Van Gelder Studio, Hackensack, NJ, September 22,
1957</div>
<table width="100%">
<tr><td width="15%">1. tk.5<td>Tune Up
Etc...
Extract using xpath and using regular expression to filter out elements. Following script should work.
library(rvest)
library(stringr)
texts <- read_html("https://www.jazzdisco.org/blue-note-records/catalog-4000-series/") %>%
html_nodes(xpath = '//*[#id="catalog-data"]/text()') %>%
html_text()
texts[!str_detect(texts,"(^\\n$)|(^\\n\\*\\*)")] # I just notcie this line doesn't clean up the string entirely, you can figure out better regex.
About splitting the string, you may try following code:
sample_str <- "\nIke Quebec, tenor sax; Sonny Clark, piano; Grant Green, guitar; Sam Jones, bass; Louis Hayes, drums.\n"
str_trim(sample_str) %>%
str_split(",")
returns:
[[1]]
[1] "Ike Quebec" " tenor sax; Sonny Clark" " piano; Grant Green" " guitar; Sam Jones" " bass; Louis Hayes" " drums."
I have a site that had SetLocale(1033) [english united states] however I now had to change it to Chinese SetLocale(2052). That works great in that monthname etc returns Chinese, however my number format has gone to town. Instead of 0.33 the results are .33 for example and code crashes. I think that the date format has also changed.
This is an issue as there is a LOT of code that will need to be edited accordingly. Is there a way to setLocale to Chinese but keep other format as per United States?
Much appreciated.
Use the Session.LCID to capture your formatted information then set it back to normal at the end of processing.
oldsession = Session.LCID
Session.LCID = 2057
day_uk = WeekdayName(3)
Session.LCID = 1036
day_fr = WeekdayName(3)
Session.LCID = 1034
day_es = WeekdayName(3)
Session.LCID = oldsession
response.write day_uk & " | " & day_fr & " | " & day_es
I used UK English, French and Spanish locales as the Chinese was not loaded on my server.
So I'm new to coding in general, specifically Handlebars and I'm running into a problem. I want to make a breadcrumb trail but I don't know how to pull the right data with handlebars. A simplified version of my template is as follows:
<body>
<div class="container">
<div id="content" class="container-fluid" role="main">
</div>
</div>
<script id="animal-template" type="text/x-handlebars-template">
<div class="row finalrow">
<h1 class="text-center">{{name}}</h1>
<!-- start breadcrumb -->
<ol class="breadcrumb">
<li>Classes</li>
<li>{{?????}}</li>
<li>{{name}}</li>
</ol><!-- / breadcrumb -->
<div class="col-xs-12 col-sm-6">
<div class="animal-ex-1" data-id="{{#index}}">
<img class="crop-img" src="{{image1}}" alt=""/>
</div>
</div><!-- /col -->
<div class="col-xs-12 col-sm-6">
<div class="animal-ex-1" data-id="{{#index}}">
<img class="crop-img" src="{{image2}}" alt=""/>
</div>
</div><!-- /col -->
<div class="text-center col-xs-12">
<p class="description">{{description}}</p>
</div>
</div><!-- /row -->
</script><!-- /animal-template -->
</body>
Here's the object:
var animals_data = {
category : [
{
name : "Reptile",
thumbnail : "https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/Coast_Garter_Snake.jpg/500px-Coast_Garter_Snake.jpg",
animals : [
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/Coast_Garter_Snake.jpg/500px-Coast_Garter_Snake.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Nerodia_sipedon_shedding.JPG/440px-Nerodia_sipedon_shedding.JPG",
name : "Snake",
description : "Snakes are elongated, legless, carnivorous reptiles of the suborder Serpentes that can be distinguished from legless lizards by their lack of eyelids and external ears. Like all squamates, snakes are ectothermic, amniote vertebrates covered in overlapping scales. Many species of snakes have skulls with several more joints than their lizard ancestors, enabling them to swallow prey much larger than their heads with their highly mobile jaws. To accommodate their narrow bodies, snakes' paired organs (such as kidneys) appear one in front of the other instead of side by side, and most have only one functional lung. Some species retain a pelvic girdle with a pair of vestigial claws on either side of the cloaca. Living snakes are found on every continent except Antarctica, and on most smaller land masses — exceptions include some large islands, such as Ireland and New Zealand, and many small islands of the Atlantic and central Pacific.[3] Additionally, sea snakes are widespread throughout the Indian and Pacific Oceans. More than 20 families are currently recognized, comprising about 500 genera and about 3,400 species.[4][5] They range in size from the tiny, 10 cm-long thread snake to the reticulated python of up to 6.95 meters (22.8 ft) in length.[6] The fossil species Titanoboa cerrejonensis was 13 meters (43 ft) long.[7] Snakes are thought to have evolved from either burrowing or aquatic lizards, perhaps during the Jurassic period, with the earliest known fossils dating to between 143 and 167 Ma ago.[8] The diversity of modern snakes appeared during the Paleocene period (c 66 to 56 Ma ago). The oldest preserved descriptions of snakes can be found in the Brooklyn Papyrus. Most species are nonvenomous and those that have venom use it primarily to kill and subdue prey rather than for self-defense. Some possess venom potent enough to cause painful injury or death to humans. Nonvenomous snakes either swallow prey alive or kill by constriction."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/f/f9/Crocodilia_collage.jpg/600px-Crocodilia_collage.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3e/Caiman_crocodilus_Tropicario_2.JPG/440px-Caiman_crocodilus_Tropicario_2.JPG",
name : "Crocodilia",
description : "The Crocodilia (or Crocodylia) are an order of large, predatory, semiaquatic reptiles. They appeared 83.5 million years ago in the Late Cretaceous period (Campanian stage) and are the closest living relatives of birds, as the two groups are the only known survivors of the Archosauria. Members of the crocodilian total group, the clade Pseudosuchia, appeared about 250 million years ago in the Early Triassic period, and diversified during the Mesozoic era. The order Crocodilia includes the true crocodiles (family Crocodylidae), the alligators and caimans (family Alligatoridae), and the gharial and false gharial (family Gavialidae). Although the term 'crocodiles' is sometimes used to refer to all of these, a less ambiguous vernacular term for this group is crocodilians."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/1/18/Bartagame_fcm.jpg/500px-Bartagame_fcm.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Lizard_in_Yemen.JPG/440px-Lizard_in_Yemen.JPG",
name : "Lizard",
description : "Lizards are a widespread group of squamate reptiles, with approximately over 6,000 species,[1] ranging across all continents except Antarctica, as well as most oceanic island chains. The group, traditionally recognized as the suborder Lacertilia, is defined as all extant members of the Lepidosauria (reptiles with overlapping scales) that are neither sphenodonts (i.e., tuatara) nor snakes – they form an evolutionary grade.[2] While the snakes are recognized as falling phylogenetically within the Toxicofera clade from which they evolved, the sphenodonts are the sister group to the squamates, the larger monophyletic group, which includes both the lizards and the snakes."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/8/86/Chelonia_mydas_is_going_for_the_air.jpg/440px-Chelonia_mydas_is_going_for_the_air.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/Turtle3m.JPG/500px-Turtle3m.JPG",
name : "Turtle",
description : "Turtles are reptiles of the order Testudines (or Chelonii[3]) characterised by a special bony or cartilaginous shell developed from their ribs and acting as a shield.[4] \"Turtle\" may refer to the order as a whole (American English) or to fresh-water and sea-dwelling testudines (British English). The order Testudines includes both extant (living) and extinct species. The earliest known members of this group date from 157 million years ago,[1] making turtles one of the oldest reptile groups and a more ancient group than snakes or crocodilians. Of the 327 known species alive today, some are highly endangered."
},
]
},
{
name : "Mammal",
thumbnail : "https://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/AfricanWildCat.jpg/440px-AfricanWildCat.jpg",
animals : [
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/AfricanWildCat.jpg/440px-AfricanWildCat.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Kittyply_edit1.jpg/440px-Kittyply_edit1.jpg",
name : "Cat",
description : "The domestic cat (Felis catus or Felis silvestris catus) is a small, usually furry, domesticated, and carnivorous mammal. They are often called housecats when kept as an indoor pet or simply cats when there is no need to distinguish them from other felids and felines.[6] Cats are often valued by humans for companionship and their ability to hunt vermin."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/8/8c/Poligraf_Poligrafovich.JPG/340px-Poligraf_Poligrafovich.JPG",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/Cavalier_King_Charles_Spaniel_trio.jpg/440px-Cavalier_King_Charles_Spaniel_trio.jpg",
name : "Dog",
description : "The domestic dog (Canis lupus familiaris or Canis familiaris) is a domesticated canid which has been selectively bred for millennia for various behaviors, sensory capabilities, and physical attributes. Although initially thought to have originated as a manmade variant of an extant canid species (variously supposed as being the dhole, golden jackal, or gray wolf), extensive genetic studies undertaken during the 2010s indicate that dogs diverged from other wolf-like canids in Eurasia 40,000 years ago. Being the oldest domesticated animals, their long association with people has allowed dogs to be uniquely attuned to human behavior, as well as thrive on a starch-rich diet which would be inadequate for other canid species."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/6/6c/Diademed_ready_to_push_off.jpg/440px-Diademed_ready_to_push_off.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/Three_chimpanzees_with_apple.jpg/640px-Three_chimpanzees_with_apple.jpg",
name : "Primate",
description : "A primate is a mammal of the order Primates (Listeni/praɪˈmeɪtiːz/ pry-may-teez; Latin: \"prime, first rank\"). In taxonomy, primates include two distinct lineages, strepsirrhines and haplorhines. Primates arose from ancestors that lived in the trees of tropical forests; many primate characteristics represent adaptations to life in this challenging three-dimensional environment. Most primate species remain at least partly arboreal. With the exception of humans, who inhabit every continent, most primates live in tropical or subtropical regions of the Americas, Africa and Asia.[4] They range in size from Madame Berthe's mouse lemur, which weighs only 30 g (1 oz), to the eastern lowland gorilla, weighing over 200 kg (440 lb). Based on fossil evidence, the earliest known true primates, represented by the genus Teilhardina, date to 55.8 million years old.[5] An early close primate relative known from abundant remains is the Late Paleocene Plesiadapis, circa 55–58 million years old.[6] Molecular clock studies suggest that the primate branch may be even older, originating near the Cretaceous–Paleogene boundary or around 63-74 mya"
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/0/0d/%D0%9C%D1%8B%D1%88%D1%8C_2.jpg/500px-%D0%9C%D1%8B%D1%88%D1%8C_2.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/f/f5/Fancy_mice.jpg/500px-Fancy_mice.jpg",
name : "Mouse",
description : "A mouse (plural: mice) is a small rodent characteristically having a pointed snout, small rounded ears, a body-length scaly tail and a high breeding rate. The best known mouse species is the common house mouse (Mus musculus). It is also a popular pet. In some places, certain kinds of field mice are locally common. They are known to invade homes for food and shelter."
},
]
},
{
name : "Bird",
thumbnail : "https://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/American_Flamingo_-_Phoenicopterus_ruber.jpg/440px-American_Flamingo_-_Phoenicopterus_ruber.jpg",
animals : [
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/American_Flamingo_-_Phoenicopterus_ruber.jpg/440px-American_Flamingo_-_Phoenicopterus_ruber.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Caribbean_flamingo.jpg/440px-Caribbean_flamingo.jpg",
name : "American flamingo",
description : "The American flamingo (Phoenicopterus ruber) is a large species of flamingo closely related to the greater flamingo and Chilean flamingo. It was formerly considered conspecific with the greater flamingo, but that treatment is now widely viewed (e.g. by the American and British Ornithologists' Unions) as incorrect due to a lack of evidence. It has also been known as the Caribbean flamingo, but the species' presence in the Galápagos makes that name problematic. It is the only flamingo which naturally inhabits North America"
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/a/aa/Papageitaucher_Fratercula_arctica.jpg/600px-Papageitaucher_Fratercula_arctica.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4b/Atlantic_Puffin_Fratercula_arctica.jpg/440px-Atlantic_Puffin_Fratercula_arctica.jpg",
name : "Atlantic puffin",
description : "The Atlantic puffin (Fratercula arctica), also known as the common puffin, is a species of seabird in the auk family. It is the only puffin native to the Atlantic Ocean; two related species, the tufted puffin and the horned puffin, are found in the northeastern Pacific. The Atlantic puffin breeds in Iceland, Norway, Greenland, Newfoundland and many North Atlantic islands, and as far south as Maine in the west and the British Isles in the east. The Atlantic puffin has a large population and a wide range. It is not considered to be endangered although there may be local declines in numbers. On land, it has the typical upright stance of an auk. At sea, they swim on the surface and feed mainly on small fish, which they catch by diving underwater, using their wings for propulsion."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/1/19/%C3%81guila_calva.jpg/440px-%C3%81guila_calva.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Golden_Eagle_in_flight_-_5.jpg/440px-Golden_Eagle_in_flight_-_5.jpg",
name : "Eagle",
description : "Eagle is a common name for many large birds of prey of the family Accipitridae; it belongs to several groups of genera that are not necessarily closely related to each other. Most of the 60 species of eagles are from Eurasia and Africa.[1] Outside this area, just 14 species can be found – two in North America, nine in Central and South America, and three in Australia."
},
{
image1 : "https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Ara_ararauna_Luc_Viatour.jpg/440px-Ara_ararauna_Luc_Viatour.jpg",
image2 : "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4e/Rose-ringed_Parakeet_eating_leaves.JPG/440px-Rose-ringed_Parakeet_eating_leaves.JPG",
name : "Parrot",
description : "Parrots, also known as psittacines /ˈsɪtəsaɪnz/,[2][3] are birds of the roughly 372 species in 86 genera that make up the order Psittaciformes, found in most tropical and subtropical regions. The order is subdivided into three superfamilies: the Psittacoidea (\"true\" parrots), the Cacatuoidea (cockatoos) and the Strigopoidea (New Zealand parrots).[5] Parrots have a generally pantropical distribution with several species inhabiting temperate regions in the Southern Hemisphere as well. The greatest diversity of parrots is in South America and Australasia."
},
]
},
]
};
I know this is a lot and thank you for putting up with it. Finally the JS is next:
var classes_template, families_template, animal_template;
function showTemplate(template, data){
var html = template(data);
$('#content').html(html);
};
$(document).ready(function(){
var source = $("#classes-template").html();
classes_template = Handlebars.compile(source);
source = $("#families-template").html();
families_template = Handlebars.compile(source);
source = $("#animal-template").html();
animal_template = Handlebars.compile(source);
$("#classes-btn").click(function (){
$(".navbar-nav .active").removeClass("active");
$("#classes-btn").addClass("active");
showTemplate(classes_template, animals_data);
$(".classes-thumbnail").click(function (){
$(".navbar-nav .active").removeClass("active");
$("#families-btn").addClass("active");
var index = $(this).data("id");
current_category = animals_data.category[index];
showTemplate(families_template, current_category);
$(".animal-ex-1").click(function (){
var index = $(this).data("id");
current_animal = current_category.animals[index];
showTemplate(animal_template, current_animal);
});
});
});
$("#reptile-btn").click(function(){
$(".navbar-nav .active").removeClass("active");
$("#families-btn").addClass("active");
current_category = animals_data.category[0];
showTemplate(families_template, current_category);
$(".animal-ex-1").click(function (){
var index = $(this).data("id");
current_animal = current_category.animals[index];
showTemplate(animal_template, current_animal);
});
});
$("#mammal-btn").click(function(){
$(".navbar-nav .active").removeClass("active");
$("#families-btn").addClass("active");
current_category = animals_data.category[1];
showTemplate(families_template, current_category);
$(".animal-ex-1").click(function (){
var index = $(this).data("id");
current_animal = current_category.animals[index];
showTemplate(animal_template, current_animal);
});
});
$("#bird-btn").click(function(){
$(".navbar-nav .active").removeClass("active");
$("#families-btn").addClass("active");
current_category = animals_data.category[2];
showTemplate(families_template, current_category);
$(".animal-ex-1").click(function (){
var index = $(this).data("id");
current_animal = current_category.animals[index];
showTemplate(animal_template, current_animal);
});
});
$("#classes-btn").click();
});
All this, to just ask– in the template where I have {{?????}} as the third list item in the breadcrumb trail, what can I put in these handlebars in order to pull the name of the category rather than the animal. It's kinda like a parent element I guess? Is there a way to call data from the parent of the object you are pulling from?
Hope this makes sense and thanks for any help!
The template has access only to the data you pass it. To your animal_template you pass only the current_animal object, so there is no category data to make use of. In order to get the category data to our template, we would need to include it in the template data. We could change the calls to showTemplate with the animal_template as first parameter to the following:
showTemplate(animal_template, {
animal: current_animal,
category: current_category
});
We would then need to update our animal template to reflect the new data structure:
<div class="row finalrow">
<h1 class="text-center">{{animal.name}}</h1>
<ol class="breadcrumb">
<li>Classes</li>
<li>{{category.name}}</li>
<li>{{animal.name}}</li>
</ol>
<div class="col-xs-12 col-sm-6">
<div class="animal-ex-1" data-id="{{#index}}">
<img class="crop-img" src="{{animal.image1}}" alt=""/>
</div>
</div>
<div class="col-xs-12 col-sm-6">
<div class="animal-ex-1" data-id="{{#index}}">
<img class="crop-img" src="{{animal.image2}}" alt=""/>
</div>
</div>
<div class="text-center col-xs-12">
<p class="description">{{animal.description}}</p>
</div>
</div>