I am trying to scrape a table from test sites of webscaper but I am not able to. Please tell me next steps. link of website:https://webscraper.io/test-sites/tablesI am attaching a photograph.
Next step, select which web scrap library(or framework) is best for you.
If use javascript based (React, Angular, node.js), Axios and Cheerio or Puppeteer
If use python, Scrapy or BeautifulSoup
If use Java, Jsoup or HTMLUnit
Demo for your table scrapping by Puppeteer
#1 install node.js into your PC
https://nodejs.org/en/download/
#2 copy this code and save with 'get-table.js`
const puppeteer = require("puppeteer");
async function getData(url) {
try {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url);
titles = await page.evaluate(() => {
return Array.from(document.querySelectorAll("table tbody tr td"),
title => title.innerText.trim());
});
await browser.close();
return Promise.resolve(titles);
} catch (error) {
return Promise.reject(error);
}
}
getData('https://webscraper.io/test-sites/tables')
.then((titles) => {
console.log(titles);
})
#3 install puppeteer and run your code
npm install puppeteer
node get-table.js
#4 Get result for scrapping
$ node get-table.js
[
'1', 'Mark', 'Otto', '#mdo',
'2', 'Jacob', 'Thornton', '#fat',
'3', 'Larry', 'the Bird', '#twitter',
'4', 'Harry', 'Potter', '#hp',
'5', 'John', 'Snow', '#dunno',
'6', 'Tim', 'Bean', '#timbean',
'1', 'Mark', 'Otto', '#mdo',
'2', 'Jacob', 'Thornton', '#fat',
'3', 'Larry', 'the Bird', '#twitter',
'-', '-', '-', '-',
'1', 'Mark', 'Otto', '#mdo',
'2', 'Jacob', 'Thornton', '#fat',
'3', 'Larry', 'the Bird', '#twitter'
]
Related
I've been following this tutorial on how to use ELK stack for nginx logs.
I've created nginx.conf to configure how to get the logs but when i type: bin/logstash -f /etc/logstash/conf.d/nginx.conf
I get this error:
[ERROR] 2020-11-13 14:59:15.254 [Converge
PipelineAction::Create] agent - Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
[A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 9, column 8 (byte
135) after input{\n\t\n file{\n path =>
["/var/log/nginx/access.log" , "/var/log/nginx/error.log"]\n
type => "nginx"\n }\n filter{\n \n grok",
:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in
compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in
execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in block
in converge_state'"]}
and here's my nginx.conf file:
input{
file{
path => ["/var/log/nginx/access.log" , "/var/log/nginx/error.log"]
type => "nginx"
}
filter{
grok{
match => ["message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => ["message"]
}
mutate{
convert => ["response","integer"]
convert => ["bytes","integer"]
convert => ["responsetime","float"]
}
geoip{
source => "clientip"
target => "geoip"
add_tag => ["nginx-geoip"]
}
date {
match ⁼> ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
remove_field => ["timestamp"]
}
useragent {
source => "agent"
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "nginx-%{+yyyy.MM.dd}"
document_type => "nginx_logs"
}
}
}
I found similar question but the answer didn't help.
Is there anyone familiar with logstash syntax and help figure out my error
Thank you
You are missing a } to close the input section. Insert it before the filter keyword.
Also, remove the last } in the file.
I am using Angular and AngularFire2. I am trying to upload an image to firebase storage, then once that is done I am take that reference and get the download url and upload it to the database. For some reason even though the upload is complete and I have the snapshot, when I try to use that in order to get the URL it's giving me an error that the object does not exist. Any thoughts on what I might be doing wrong?
task.snapshotChanges().pipe(
concatMap(snap => {
return snap.ref.getDownloadURL()
}),
concatMap(url => this.db.collection('library').add({
name: this.image.name,
path: path,
largeUrl: url
}))
).subscribe(ref => {
this.completed = true;
}, error => console.log(error));
error:
Firebase Storage: Object 'library/1542515976022_lemonade-smoothie.jpg' does not exist.
Ok, so my issue was not really understanding concatMap. I thought it wasn't called until the last onNext() of the upload Observable. It was being called on the first onNext(), which means the file had not completely updated. Below is what I ended up doing, although it seems like there should be another way. What I would like is to only switch to the new Observable track if the downloaded bytes equals the total bytes. I'm not sure how to do this with RxJS though. If anyone had any thoughts let me know.
task
.snapshotChanges()
.pipe(finalize(() => this.uploadToDb(path)))
.subscribe();
uploadToDb(path: string) {
this.storage
.ref(path)
.getDownloadURL()
.pipe(
concatMap(url =>
this.db.collection('library').add({
name: this.image.name,
path: path,
largeUrl: url
})
)
)
.subscribe(
ref => (this.completed = true),
error => {
console.log(error);
this.error = true;
}
);
}
I'am sending push messages to application with react-native-firebase and it look's wonderful ! Also, I need to receive some portion of data, so sending data something like this
$request_body = [
'to' => $TOKEN_ID,
'notification' => [
'title' => 'Title',
'body' => 'Body',
'sound' => 'default',
],
'data' => [
'key' => 'value',
],
];
I'am trying to listen open push message event as bellow
this.notificationOpenedListener = firebase.notifications().onNotificationOpened((notificationOpen) => {
const notification = notificationOpen.notification;
const data = notificationOpen.data;
setTimeout(() => {
Alert.alert(data.key);
}, 5000);
});
After 5 second left, application closes without any error message. If changing Alert.alert(data.key); with Alert.alert(notification.title); application work`s fine and show an alert. Can someone explain to me, why retrieving data is not work properly?
After some research, I found a solution
const data = notification.data;
or
const data = notificationOpen.notification.data;
i have recently updated Firebase and AngularFire2 in my Ionic project
Versions:
Firebase: 5.0.3
AngularFire2: 5.0.0-rc.10
rxjs 6.2.0
now I tried to upgrade the project from the regular map to pipe using the migration guide:
Migration guide AngularFire2 version5
But if i use exact the same example for the following code block:
///my code
let dataBaseCollection = this.store.collection('items').snapshotChanges().pipe(
map(actions =>
actions.map(a => ({ key: a.key, ...a.payload.val() }))
)
).subscribe(items => {
return items.map(item => item.key);
});
///example
afDb.list('items').snapshotChanges().pipe(
map(actions =>
actions.map(a => ({ key: a.key, ...a.payload.val() }))
)
)
I get the following exceptions:
Argument of type 'OperatorFunction' is not assignable to parameter of type 'UnaryFunction, Observable<{ const: string; return: any; }[]>>'.
Types of parameters 'source' and 'source' are incompatible.
Type 'Observable' is not assignable to type 'Observable'. Two different types with this name exist, but they are unrelated.
Property 'source' is protected in type 'Observable' but public in type 'Observable'.
I already tried the two different operator from rxjs
import { map } from 'rxjs/operators';
import { map } from 'rxjs/operators/map';
I appreciate any help.
Well, I don't know if this is the issue, but a few observations:
let dataBaseCollection is receiving a Subscription from .subscribe and not the items. You cannot return inside the subscribe, so your code should be just this:
let dataBaseCollection = this.store.collection('items').snapshotChanges().pipe(
map(actions => actions.map(a => ({ key: a.key, ...a.payload.val() })))
)
If you want to map only the keys then just do:
let dataBaseCollection = this.store.collection('items').snapshotChanges().pipe(
map(actions => actions.map(a => a.key))
)
I get it running.
I do not have a real clue, what the actual problem was. But I completely deleted my local repository and get all files new.
Then I reinstalled all NPM files. Somewhere in this area was my problem. I would guess, that any of my npm packages was the problem.
Im sorry, that I can not clarify the issue more. Maybe someone else will have the same Problem in the future and can add it here.
I want to send an email when a new order is created in my firebase database, but nothing happens when I create an order. My function:
exports.sendEmailConfirmation = functions.database.ref('/orders').onCreate(event => {
const mailOptions = {
from: '"Someone." <noreply#firebase.com>',
to: 'someone#gmail.com',
};
// Building Email message.
mailOptions.subject = 'New order from mobile app!';
mailOptions.text = 'John Doe lorem ipsum';
return mailTransport.sendMail(mailOptions)
.then(() => console.log('¡¡¡ Enail sent !!!'))
.catch((error) => console.error('Error!!', error));
});
This code works using onWrite()....
Your function isn't triggering because /orders already exists. onCreate triggers will only run when the path you specify is newly created.
If you want to know when a child is newly added under /orders, you should use a wildcard in the path:
functions.database.ref('/orders/{orderId}')