Output console logs from page.evaluate to local terminal? - web-scraping

How can I output console logs from page.evaluate to local terminal?
await page.evaluate(() => {
console.log("test"); // <-- I want this message to show in my local terminal
I am running my code locally on a MacOS computer.
The following solutions I have found in github issues have not worked:
solution 1
const browser = await puppeteer.launch({
'args': ['--disable-dev-shm-usage', '--disable-software-rasterizer'],
dumpio: true
});
// output:
// ERROR:gpu_process_transport_factory.cc(967)] Lost UI shared context.
solution 2
const browser = await puppeteer.launch({
'args': ['--disable-dev-shm-usage', '--disable-software-rasterizer']
});
page.on('console', msg => {
for (let i = 0; i < msg.args.length; ++i)
console.log(`${i}: ${msg.args[i]}`);
});
// output:
// nothing

Just started with Puppeteer myself. Couldn't figure it out right away, but after trying a few things, this worked for me.
const names = await page.evaluate(() => {
let names = document.querySelectorAll("#lnkBEName");
console.log(names);
return names;
})
console.log(names);
Then console.log inside the function block doesn't work. However, the last line, outside the block works. Hope this helps someone.

Related

Unable to use array's method "find" in Vue 3

I am trying to get current location of a user and then push it into array. Before I do so, I check whether a city with the same name is already there. In that case, I won't push it into the array. However, when I am trying to check it, it says: Uncaught (in promise) TypeError: Cannot read properties of null (reading 'find').
const found = ref(false);
const weatherResponse = ref([]);
function getLocation() {
console.log("SETTING LOCATION");
navigator.geolocation.getCurrentPosition((position) => {
console.log(`Lat: ${position.coords.latitude}, Lon: ${position.coords.longitude}`);
if (position.coords.latitude && position.coords.longitude) {
axios.get(`https://api.weatherapi.com/v1/current.json?key=${API_KEY}&q=${Math.round(position.coords.latitude)},${Math.round(position.coords.longitude)}&aqi=no`)
.then((response) => {
found.value = weatherResponse.value.find((item) => item.location.name == response.data.location.name);
if (response.data?.error?.code != 1006 && !found.value) {
weatherResponse.value.push(response.data);
this.$store.commit("addToList", response.data);
console.log(weatherResponse.value);
}
})
}
},
(error) => {
console.log(error.message);
}
)
}
I've already tried using fetch, axios to grab the API, but the "find()" method is still not working. Regarding "found" variable, I tried using it in ref as well as declaring it as "let found".
After trying and testing, I've finally managed to get everything to work. My issue was in (weirdly) main.js. Because it was set out like this: createApp(App).use(cors, store).mount('#app') it, I guess, caused VueX.store not to load in properly because mounted hook was called and it was throwing all sorts of mistakes. Putting it like const app = createApp(App); app.use(store); app.use(cors); app.mount("#app"); actually made it work.

Watch console for errors

I have an app using canvas elements, which are difficult to test with javascript but it does throw messages to the console.
How can I watch for errors written to the console?
I tried monkey patching the window console, but it's not working.
const messages = []
window.console.error = (msg) => {
messages.push(msg)
})
// actions causing error
expect(messages.length).to.eq(0)
You can watch console messages with Cypress cy.spy(), ref docs
let spy;
Cypress.on('window:before:load', (win) => {
spy = cy.spy(win.console, "error")
})
// actions causing error
cy.then(() => {
expect(spy).not.to.be.called
})

Google Cloud Pub/Sub get or create subscription

I've got two working bits of code as follows.
This gets an existing subscription:
this.pubSubClient = new PubSub()
sub = this.pubSubClient.subscription(appConfig.pubSub.subscription)
The creates and gets a non-existing subscription:
this.pubSubClient = new PubSub()
this.topic = this.pubSubClient.topic(appConfig.pubSub.topic)
[sub] = await this.topic.createSubscription(appConfig.pubSub.subscription)
This all works great. However, the first bit of code causes problems if the subscription does NOT exist (on one environment) and the second bit of code causes problems if the subscription does exist (on another environment).
So I've tried to do this:
let sub
try {
sub = this.pubSubClient.subscription(appConfig.pubSub.subscription)
console.log('using existing subscription')
} catch (err) {
[sub] = await this.topic.createSubscription(appConfig.pubSub.subscription)
console.log('using created subscription')
}
But the above does not work because the first line of code never triggers an error. It merely fails to ever receive any messages. Is there a command to getOrCreateSubscription that I can use which will get the subscription and create it if necessary?
Best way seems to be to get subscriptions on the topic of interest and if its not listed then it does not exist so create it else just get it
const [subscriptions] = await this.topic.getSubscriptions()
const subs = subscriptions.map(subscription => last(subscription.name.split('/')))
const subExists = subs.includes(appConfig.pubSub.subscription)
let sub
if (subExists) {
sub = this.pubSubClient.subscription(appConfig.pubSub.subscription)
console.log('using existing subscription')
} else {
[sub] = await this.topic.createSubscription(appConfig.pubSub.subscription)
console.log('using created subscription')
}
FYI last is imported from lodash and returns the last item in an array
The right way
const subscription = this.topic.subscription("name");
const [exists] = await subscription.exists();
if (!exists) {
await subscription.create();
}

TypeOrm - Can't create a connection outside of main function

I have a problem with typeorm, function createConnection works only in index file, if I try to run it in any other file it gets stuck waiting for connection.
export async function saveKU(data: KUData) {
console.log("Foo");
let connection = await createConnection(typeormConfig);
console.log("Received!: " + connection);
const user = connection
.getRepository(KU)
.createQueryBuilder("ku")
.where("ku.ku_number = :ku_number", { ku_number: "54645" })
.getOne();
console.log(user);
}
Message received never gets logged, but then if I run the exact same script in the main function
const main = async () => {
// quit application when all windows are closed
app.on("window-all-closed", () => {
// on macOS it is common for applications to stay open until the user explicitly quits
if (process.platform !== "darwin") {
app.quit();
}
});
app.on("activate", () => {
// on macOS it is common to re-create a window even after all windows have been closed
if (mainWindow === null) {
mainWindow = createMainWindow();
}
});
app.allowRendererProcessReuse = true;
// create main BrowserWindow when electron is ready
app.on("ready", () => {
mainWindow = createMainWindow();
});
let connection = await createConnection(typeormConfig);
console.log("Received!: " + connection);
const user = connection
.getRepository(KU)
.createQueryBuilder("ku")
.where("ku.ku_number = :ku_number", { ku_number: "54645" })
.getOne();
console.log(user);
};
Everything works fine, no error is showing up in both cases, I have no idea what the problem could be, the closest to my example is the following link: https://senorihl.github.io/2019/03/electron-typescript-react-typeorm/
Except that I'm using electron-webpack and creating connections in an index.tsx file doesn't work for me, but rather index.ts

Using Google Cloud Speech to Text in Firebase Cloud Functions

Google Cloud Speech to Text documentation dictates that you can access it by:
const client = new speech.SpeechClient();
const [operation] = await client.longRunningRecognize({
config: {
encoding: 'LINEAR16',
sampleRateHertz: 16000,
languageCode: 'en-US'
},
audio: {
uri: `gs://${bucket}/${name}`
}
});
const [response] = await operation.promise();
response.results.forEach(result => {
console.log(`Transcription: ${result.alternatives[0].transcript}`);
});
Now, I wanna run this code in a Firebase Cloud Function. Unfortunately, Cloud Functions run on a version of Node that does not yet support async and await functions.
Some things I've tried:
Trying TypeScript, which supports async and await: Ran into a bunch of problems with some of the other APIs I'm using.
Upgrading all my functions to Node 8 (beta), which supports async and await: Again, ran into quite a bit of bugs from the Firebase side doing this.
"Translating" the code manually (is this even a thing?): I tried to treat the code to expect a promise.
That didn't work too well either, this is how it looks:
exports.onStorageObjectFinalize = functions.storage.object()
.onFinalize((object) => {
const client = new speech.SpeechClient();
return client.longRunningRecognize({
config: {
encoding: 'LINEAR16',
sampleRateHertz: 16000,
languageCode: 'en-US'
},
audio: {
uri: `gs://${object.bucket}/${object.name}`
}
})
.then(r1 => {
const [operations] = r1;
return operations.promise();
})
.then(r2 => {
const [response] = r2;
// response.results...
return true;
});
});
Edit: When the above function runs, it says there's no operations.promise(). In fact, after taking a look at the whole operations object, the structure doesn't look like its the same function. I did found there's a promise property in operations._callOptions, so I tried returning operations._callOptions.promise() but I got a strange error: TypeError: #<CallSettings> is not a promise at client.longRunningRecognize.then.r1.
Did I mess the translation code up or would this never work anyways?
Any other things I can try or are TypeScript and Node 8 my only two options here?
Thanks, much appreciated.

Resources