YouTube comment scraper not working/error? - web-scraping

I’m fairly new to coding/programming and I don’t know what I’m doing wrong here
I installed node.js and tried to run the following command:
youtube-comment-scraper -f json —outputFile C:\Users\P\Desktop\YTCS\Scrapes\file.json smn0HOvwoZ8
Cmd responses with „API response does not contain a „content_html“ field“
Node js responses with „Uncaught SyntaxError: Unexpected identifier“
How can I get this to work?
Link to the scraper:
https://github.com/philbot9/youtube-comment-scraper

Related

Deno 500 error from dev.jspm.io installing dependencies

I'm trying to run my first deno script which is pretty much from the denoDB docs, it just tries to connect to a database with a SQLite3 connector (I'm on a Macbook pro so it should be installed):
import { Database, SQLite3Connector } from 'https://deno.land/x/denodb/mod.ts';
const connector = new SQLite3Connector({
filepath: './db.sqlite',
});
export const db = new Database(connector);
I'm running deno run api/db.ts and I get this error after a few successful downloads:
Download https://deno.land/std#0.149.0/encoding/hex.ts
Download https://deno.land/std#0.149.0/hash/_wasm/lib/deno_hash.generated.mjs
error: Import 'https://dev.jspm.io/inherits#2.0' failed: 500 Internal Server Error
at https://raw.githubusercontent.com/Zhomart/dex/930253915093e1e08d48ec0409b4aee800d8bd0c/lib-dyn/deps.ts:4:26
I've deleted my /Users/<me>/Library/Caches/deno/deps/https and reran the script a few times but I still can't get past this. In my browser trying to follow the URL https://dev.jspm.io/inherits#2.0 does give me an error. What is going on here? There's not much code and I imagine it's not broken for everybody. What do I need to do to get this script to run without issues?
EDIT: it seems to be a library error https://github.com/eveningkid/denodb/issues/348
This is an error caused by a nested depedency, from a project that is not maintained.
See this for more info: [https://jspm.org/jspm-dev-release]
The point is dev.jspm.io is now jspm.dev
A way to fix this is to fork and update depedencies.
Another thing, if you're not using deno deploy, you can just use this as a replacement for your denodb: https://raw.githubusercontent.com/joeldesante/denodb/master/mod.ts
Just note that this script is no more maintained either, but it will fix your problem
Edit
I just made a dirty quick fix for deno deploy use this as a depedency isntead of denodb: https://raw.githubusercontent.com/ninjinskii/denodb/master/mod.ts
Again, i may not maintain this script forever.
The best thing that can happen is an update from these libs maintainers

Process ends with 100% progress and error while using pre-signed url for images

Using Mac, language .NET
Photoscene ID : krb1VCLBHop1AdbAdPtm96PWtCfsbncmfSXdSG5eUYM
I'm trying to upload images from mobile device to create 3d model of an object.
There were some issues uploading local images directly, so I uploaded images on AWS S3 and generated pre-signed url. These urls were used within file uploading curl command.
curl -v $BASE'photo-to-3d/v1/file'
-H 'Authorization: Bearer '$AUTH
-F "photosceneid="$PID
-F "type=image"
-F "file[0]="$URL
the result shows no error msg.
{"Usage":"2.1651990413666",
"Resource":"\/file",
"photosceneid":"krb1VCLBHop1AdbAdPtm96PWtCfsbncmfSXdSG5eUYM",
"Files":{"file":[{"filename":"YXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMTEwMDdUMDY0NTU0WiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmWC1BbXotU2lnbmF0dXJlPWUyMDgyYWE2YjgyNzFmNzk4NTFmNzc3MzkwYmFhN2ExN2FjM2Y5MjJjNjM3NGIyMzg5YWFmN2IwNzE3OTY3MjU=.jpg",
"fileid":"AALxKyQX1tAhcEgbM6w6lITCOHAleS1Dumu6ocy3qaY=-krb1VCLBHop1AdbAdPtm96PWtCfsbncmfSXdSG5eUYM",
"filesize":"187126",
"msg":"No error"}
Processing ends with error
{"Usage":"0.51469397544861",
"Resource":"\/photoscene\/krb1VCLBHop1AdbAdPtm96PWtCfsbncmfSXdSG5eUYM\/progress",
"Photoscene":{"photosceneid":"krb1VCLBHop1AdbAdPtm96PWtCfsbncmfSXdSG5eUYM",
"progressmsg":"ERROR","progress":"100"}}
File links were checked. Images were displayed right and can be downloaded.
Format is jpg.
checked photoscene properties and found error_msg_id
"error_msg_id":"262"
the meaning of error code was not found anywhere including error handling documentation
I don't know what to do from here since error message does not specify what exactly went wrong.

Uploading manifest file failing

After making some changes in a JSON manifest file, I was trying to update it following the Amazon documentation:
ask smapi update-skill-manifest -g development -s amzn1.ask.skill.xxxx --manifest "skillManifest.json" --debug
I kept getting this error:
The error was not pointing to what the error was but my guess was that it was related to the parameters, but that was strange as I was following the documentation to letter.
I then tried, instead of passing the json file, to cat the content of the file, which would be either:
For Powershell: --manifest "$(type skillmanifest.json)"
For Linux: --manifest "$(cat skillmanifest.json)"
I still kept getting the same error.
Firstly, for debugging and getting a more accurate error, I checked my ASK-CLi version, which was outdated.
After updating ASK to the latest version I was still getting the same error.
At that point it started including an error object, which was saying:
When looking into Parsing error due to invalid body. and INVALID_REQUEST_PARAMETER through the error codes, it just said the body of the request cannot be parsed.
After research and playing around, the problem was the manifest parameter, changing it to "file:FILENAME" solved the issue:
--manifest "file:skillmanifest.json"
The documentation is not stating this but it seems necessary for it to go through.
I hope this helps someone out there avoid spending a full day troubleshooting.

sh '<' syntax unexpected error

I am struggling with running new asp.net 5 on my qnap box. As far as I understand it is a strongly modified version of debian.
As part of running installation script I got this error:
-sh: /root/.dnx/dnvm/dnvm.sh: line 616: syntax error near unexpected token `<'
-sh: /root/.dnx/dnvm/dnvm.sh: line 616: read versionOrAlias downloadUrl < <(__dnvm_find_latest "$runtime" "$arch" "$os")'
I run my script bu using script command like:
script /root/.dnx/dnvm/dnvm.sh
as stated in documentation and previous installation script.
By commenting this line out I was able to run whole script but obviously dnvm command does not work properly.
My question is:
What does it do (line with < < syntax) and how do I fix it or rewrite so that my qnap box unix can understand it.
<(...) is Process substitution. /bin/sh doesn't support it, but /bin/bash does. Try changing the shell.

R Import - CSV file from password protected URL - in .BAT file

Okay - so here is what I'm trying to do.
I've got this password protected CSV file I'm trying to import into R.
I can import it fine using:
read.csv()
and when I run my code in RStudio everything works perfect.
However, when I try and run my .R file using a batch file (windows .bat) it doesn't work. I want to use the .BAT file so that I can set up a scheduled task to run my code every morning.
Here is my .BAT file:
"E:\R-3.0.2\bin\x64\R.exe" CMD BATCH "E:\Control Files\download_data.R" "E:\Control Files\DailyEmail.txt"
And here is my .R file:
url <- "http://username:password#www.url.csv"
data <- read.csv(url, skip=1)
** note, I've put my username/password and the exact location of the CSV in my code. I've used generic stuff here, as this is work related and posting usernames and passwords is probably frowned upon.
As I've said, this code works fine when I use it in RStudio. But fails when I use the .BAT file.
I get the following error message:
Error in download.file(url, "E:/data/data.csv") :
cannot open URL 'websiteurl'
In addition: Warning message:
In download.file(url, "E:/data/data.csv") :
unable to resolve 'username'
Execution halted
** above websiteurl is the http above (I can't post links)
So obviously, the .BAT is having trouble with the username/password? Any thoughts?
* EDIT *
I've gone so far as trying this on Linux. Thinking maybe windows was playing silly bugger.
Just from the terminal, I run Rscript -e "download_data.r" and get the EXACT same error message as I did in Windows. So I suspect this may be a problem with where I'm getting the data? Could the provider be blocking data from the command line, but not from with Rstudio?
I have had similar problems which had to do with file permissions. The .bat file somehow does not have the same privileges as you running the code directly from Rstudio. Try using rscript (http://stat.ethz.ch/R-manual/R-devel/library/utils/html/Rscript.html) within your .bat file like
Rscript "E:\Control Files\download_data.R"
What is the purpose of the argument "E:\Control Files\DailyEmail.txt"? Is the program suppose to use it in any way?
So, I've found a solution, which is likely not the most practical for most people, but works for me.
What I did was migrated my project over to a Linux system. Running daily scripts, is easier on Linux anyways.
The solution makes use of the "wget" function in linux.
You can either run the wget right in your shell script, or make use of the system() function in R to run the wget.
code looks like:
wget -O /home/user/.../file.csv --user=userid --password='password' http://www.url.com/file.csv
And you can do something like:
syscomand >- "wget -O /home/.../file.csv --user=userid --password='password' http://www.url.com/file.csv"
system (syscommand)
in R to download the CSV to a location on your hard drive, then grab the CSV using read.csv()
Doing it this way gave me some more insight into the potential root cause of the problem. While the system(syscommand) is running, I get the following output:
Connecting to www.website.com (www.website.com)|ip.ad.re.ss|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Reusing existing connection to www.weburl.com:80.
HTTP request sent, awaiting response... 200 OK
Not sure why it has to send the request twice? And why I'm getting a 401 Unauthorized the first try?

Resources