Web Crawling: get shares of Youtube Video from statistics tab - web-scraping

Question

Does anybody know a way to get the shares of youtube videos (not mine)? I would like to store them into a DB. It is not working with the yt api. Another problem ist that not every yt video has the statistics tab.
So far I tried the Youtube API, jsoup HTML Parser (the div showing the shares wasn't there, altough it is shown via inspect in firefox e.g) and import.io demo which was working but is definitely too expensive.

Answer 1

The best way is to look at the network logs, in this case it shows a POST on :
https://www.youtube.com/insight_ajax?action_get_statistics_and_data=1&v=$video_id
It sends a XSRF token in the body that is available in the original html body of the video page https://www.youtube.com/watch?v=$video_id in a javascript object like :
yt.setConfig({
'XSRF_TOKEN': "QUFFLUhqbnNvZUx4THR3eV80dHlacV9tRkRxc2NwSjlXQXxBQ3Jtc0ttd0JLWENnMjdYNE5IRWhibE9ZdDJTSk1aMktxTDR5d3JjSnkzVUtQWVcwdnp3X0tSOXEtM3hZdzVFdjNPeGpPRGtLVU5pVXV0SmtfdWJSUHNqTVg2WXBndjZpa3d6U25ja2FTelBBVWRlT0lZZkRDaDV6SU94VWE3cnpERHhWNVlUYWdyRjFqN1hvc0VLRmVwcEY3ZWdJMWgyUmc=",
'XSRF_FIELD_NAME': "session_token",
'XSRF_REDIRECT_TOKEN': "VlhMkn6F56dGGYcm4Rg7jCZR0vJ8MTQ5ODA1NzIwMkAxNDk3OTcwODAy"
});
It also needs some cookies set in this same video page.
Using python
with beautifulsoup & python-requests :
import requests
from bs4 import BeautifulSoup
import re
s = requests.Session()
video_id = "CPkU0dF4JKo"
r = s.get('https://www.youtube.com/watch?v={}'.format(video_id))
xsrf_token = re.search("'XSRF_TOKEN'\s*:\s*\"(.*)\"", r.text, re.IGNORECASE).group(1)
r = s.post(
'https://www.youtube.com/insight_ajax?action_get_statistics_and_data=1&v={}'.format(video_id),
data = {
'session_token': xsrf_token
}
)
metrics = [
int(t.text.encode('ascii', 'ignore').split(' ', 1)[0])
for t in BeautifulSoup(r.content, "lxml").find('html_content').find("tr").findAll("div", {"class":"bragbar-metric"})
]
print(metrics)
Using bash
with curl, sed, pup & xml_grep :
The following bash script will :
request the video page https://www.youtube.com/watch?v=$video_id with curl
store the cookies in a file called cookie.txt
extract the XSRF_TOKEN called session_token in the following request with sed
request the video statistic page https://www.youtube.com/insight_ajax?action_get_statistics_and_data=1&v=$video_id with curl with the cookies previously stored
parse the xml result extract the CDATA part with xml_grep
parse the html with pup to extract the bragbar-metric class div and convert the html result to json with json{}
use sed to remove unicode character
The script :
video_id=CPkU0dF4JKo
session_token=$(curl -s -c cookie.txt "https://www.youtube.com/watch?v=$video_id" | \
sed -rn "s/.*'XSRF_TOKEN'\s*:\s*\"(.*)\".*/\1/p")
curl -s -b cookie.txt -d "session_token=$session_token" \
"https://www.youtube.com/insight_ajax?action_get_statistics_and_data=1&v=$video_id" | \
xml_grep --text_only 'html_content' | \
pup 'div table tr .bragbar-metric text{}' | \
sed 's/\xc2\x91\|\xc2\x92\|\xc2\xa0\|\xe2\x80\x8e//' | \
sed 's/\s.*$//'
It gives number of views, time watched, subscriptions, shares:
120862
454
18
213