How to scrape new YouTube videos from a specific channel first? - web-scraping

I have a channel and I want to get a new uploaded video from it as fast as possible. What would be the best method to do it? Two option I know:
Use the YouTube API
Access the channel via url directly
With option 1 I would need to call the api in order to get a list of videos. Since there quotas in place, I guess I will run out of api calls I can make. I would assume that option 2 is the best bet, since I can call the url as often as I want.
Are new videos available via the api first? Or is a video accessed via url made available at different times to the users, depending on the region they are coming from? I build an url scraper myself. I access the url every minute. Still there are people having the video 8 minutes before I have it. I do not get why this is the case.

Just to build off a little of what MadRay wrote you can do some simple string substitution with this URL
Using Channel ID:
"https://www.youtube.com/feeds/videos.xml?channel_id=UCXuqSBlHAE6Xw-yeJA0Tunw"
Using Channel Name:
https://www.youtube.com/feeds/videos.xml?user=LinusTechTips
Took the liberty of parsing it for you.
from bs4 import BeautifulSoup
import requests
url="https://www.youtube.com/feeds/videos.xml?user=LinusTechTips"
html = requests.get(url)
soup = BeautifulSoup(html.text, "lxml")
for entry in soup.find_all("entry"):
for title in entry.find_all("title"):
print(title.text)
for link in entry.find_all("link"):
print(link["href"])
for name in entry.find_all("name"):
print(name.text)
for pub in entry.find_all("published"):
print(pub.text)
Response:
FINALLY Wireless Headphones that Sound GREAT
https://www.youtube.com/watch?v=rei5vMQmD4Q
Linus Tech Tips
2020-01-30T20:04:37+00:00
Don't give Apple your MONEY - Mac Pro Upgrade Adventure
https://www.youtube.com/watch?v=zcLbSCinX3U
Linus Tech Tips
2020-01-29T19:59:56+00:00
We got the Kick-Proof TV from China!
https://www.youtube.com/watch?v=4eSADWuZskk
Linus Tech Tips
2020-01-28T19:46:09+00:00
Everything went wrong... Water Cooled 8K Camera Final Test
https://www.youtube.com/watch?v=OEUCNh5g-2I
Linus Tech Tips
2020-01-27T20:08:27+00:00
I'm Returning my Mac Pro
https://www.youtube.com/watch?v=mIB389tqzCI
Linus Tech Tips
2020-01-26T19:59:45+00:00
The RGB HDMI cable ISN'T as dumb as you'd think...
https://www.youtube.com/watch?v=nva6oPszm60
Linus Tech Tips
2020-01-25T20:06:23+00:00
I am NOT Retiring... yet - WAN Show Jan 24, 2020
https://www.youtube.com/watch?v=cxjhTVR_dJw
Linus Tech Tips
2020-01-25T02:29:50+00:00
The Best VR Headset... got BETTER!?
https://www.youtube.com/watch?v=AGScX_8plYw
Linus Tech Tips
2020-01-23T19:52:00+00:00
I've been thinking of retiring.
https://www.youtube.com/watch?v=hAsZCTL__lo
Linus Tech Tips
2020-01-23T06:35:25+00:00
It’s time to upgrade your GPU - RX 5600 XT
https://www.youtube.com/watch?v=rKn-vWDMkwQ
Linus Tech Tips
2020-01-22T19:59:36+00:00
WE FINALLY DID IT!! - Water Cooling the 8K Camera!
https://www.youtube.com/watch?v=imJ9QgOJHzY
Linus Tech Tips
2020-01-21T19:59:47+00:00
We Water Cooled an SSD!!
https://www.youtube.com/watch?v=lQmI5A27Iv8
Linus Tech Tips
2020-01-20T20:17:22+00:00
Should you buy a $50 CPU??
https://www.youtube.com/watch?v=JISJ_YTI9s0
Linus Tech Tips
2020-01-19T20:19:02+00:00
Apple’s Pro Display XDR – A PC Guy’s Perspective
https://www.youtube.com/watch?v=X089oYPc5Pg
Linus Tech Tips
2020-01-18T19:59:29+00:00
The NSA is Giving Out It's Hacks for Free! - WAN Show Jan 17, 2020
https://www.youtube.com/watch?v=af6FBA-n7eA
Linus Tech Tips
2020-01-18T03:00:04+00:00
However, please keep in mind to use headers with your requests and watch out for hitting YouTube's backend too many times at once because your IP will receive a temporary 12hr suspension. Best of luck!

You can try RSS feed for interested channel. It contains fresh videos with UTC timestamp (so there're no issues with timezones you mentioned).
RSS link for channel's videos can be found at source of channel's page. Open source of page and search for "rssUrl":

Related

Which devices support which traits?

Is there a table somewhere of which devices support which traits? I'm particularly interested in humidity:
https://developers.google.com/nest/device-access/traits/device/humidity
But temp would be cool too, and I can convince myself ~most smart devices would have some awareness of temperature to shut down before they overheat etc.
I suspect humidity is just in the thermostats (not suitable for how we do HVAC here in AU) but I'm holding out hope it might also be in the smoke detectors... although they're not listed on the devices page. Even under the old API it looks like they didn't offer much (for example, it'd be awesome if you could get the motion detector state, or last motion detector trigger event from the nightlight feature).
Has anyone played around with the API enough to know? I'm only really interested in temp and humidity on the current list, so don't want to shell out the registration fee if they're not going to be available on my doorbell + smoke detectors + home hub.
The Device Access docs don't list devices per trait, but they do list traits per device:
Camera
Thermostat
Doorbell
Display
Today the only device type supporting the Humidity trait is Thermostats.

Reading data from car's ECU ABS wheel speed sensors

I'm currently working on a project that requires gathering data from a car's wheel speed sensors(4 hall efect speed sensors). Those sensors are connected to the car's ECU responsible for ABS/ESP/Stability control etc.
In order to extract the data from the ECU i need to make a request with a specific PID(parameter ID) AND know how to decode/compute the answer in order to extract any meaningful data. Unfortunately vehicle manufacturers don't seem to make such information public.
So far i've ordered an arduino CAN BUS shield and a OBD2 to RS232 cable in order to make the physical connection.
I have tried using a specialized hardware/software(that costs more than 1500euro) capable of extracting those parameters, but unfortunately it lacks logging functions. I tried using Wireshark to sniff the PIDs called, but had no luck there either.
If you guys have any ideas, questions or suggestions, please write them down.
I'm open to criticism and know that i might be missing something important.
Thanks.
P.S. This is a university project im working on. I need data samples from the wheel speed sensors and further computing of the data sample is done with the purpose of researching car safety and behavior in dynamic road scenarios.
You can only read the OBD data from the OBD-port. The OBD PIDs are generalized in ISO/CD 15765-5. You probably find non reliable descriptions also in Wikipedia. But in order to get the other PIDs, firstly you should know that those data are heavily under control by the car manufacturers and you have to hack them. One way to find them (but very unlikely to find one!) is the try and error method.
You should access the main CAN-BUS wires and the buy a connector device so you can sniff the packets. then monitor all the packets and make a small change. Monitor it again and compare these two. Maybe maybe maybe you have a chance to find some non-safety features with this method but finding security functionalities like ABS is heavily in doubt.
UNLESS you are some sort of genius hacker who can do weird stuff! If you can do it, then call the manufacturer and show them what you have so you would likely get a nice job and salary by them!
ONCE I saw a youtube clip that a guy could control a TOYOTA (if I remember correctly) with a laptop! and also maybe you can buy such info on the dark web which I advise highly against it!

GSM Modem AT commands memory "OM" "DM"

I am currently working on a project that involves automating sending and receiving text messages using a RAZR v3.
Usually, when choosing the memory for the phone, there is "ME", "SM",or "MT" which stands for phone, sim, and both respectively. However when inserting a specific sim card, its different. When I do a AT+CPMS=?, I get
("MT","IM","OM","BM","DM"), ("OM","DM"),("IM")
I know what MT is, and BM is for broadcast messages. What does "OM" and "DM" stand for and what are their relations to the other memory types?
I used this site for basic memory knowledge:
http://www.developershome.com/sms/cpmsCommand.asp
and according to this site for motorola commands, it seems as if "OM" and "DM" are unknown as well(Do a ctrl+f with the keyword "CPMS"): http://gatling.ikk.sztaki.hu/~kissg/gsm/at+c.html#27.07me
I have already tried searching the internet but to no avail. If there is an article that illustrates in depth on what I am asking, I would appreciate it if you would please point me to that location, otherwise please help with the question!
Thank you!
Thanks to reddit user 1991_vg, "OM" and "DM" stand for Outgoing Messages and Draft Messages respectively. On a RAZR phone, they represent folders where messages are stored, and thus should be able to send messages from those folders. Here is the motorola at commands for anybody who has this problem: http://read.pudn.com/downloads139/doc/599044/AT_Commands_Ref.pdf
In addition, here is a nice document posted by reddit user "farnz" for the specifications of modems: http://www.3gpp.org/DynaReport/27007.htm http://www.3gpp.org/DynaReport/27005.htm

PIXY camera modul: how to get image data?

i would like to know if there exists datasheet for pixy camera modul, official wiki pages are not worth much. For starter i am interested in getting image from this camera modul, in wiki pages there is only a hello world program that detects objects. How to get image data? (Arduino) (I would like to transfer this image data via UART to computer, i know about pixymoon)
I would also like to know if there exists port to stm discovery 32f4?
I think it's not possible. I saw this in the pixy forums, hope it's enlightening:
"is it possible to use an Arduino or any other device in order to decode/transform USB video into analog? What is the video format that is fed into USB? Is it possible to cut into Pixy hardware before the signal is encoded into USB and extract the signal? Thank you.
Hello ####,
I think analog video is a pretty high bandwidth signal with strict timing requirements. I doubt this is possible without adding significant complexity.
Edward (developer)"

Can't get good quality audio/video

I'm creating a video chat application, and no matter what combination of Camera/Microphone/NetStream properties and functions I use, I cannot get high quality video/audio. I get occasional audio latency, pixelated video, occasional frozen video and the degree of each depends on the combination of properties/functions I set/call.
Others such as TokBox, TinyChat, Chat Roulette, etc. have achieved great video/audio quality with FMS, what is the secret? At least point me in the right direction, because right now I'm not impressed with FMS ability to provide a good video/audio experience.
BTW, I'm using a P2P mesh using a group specifier, not NetStream.DIRECT_CONNECTIONS.
Thanks in advance!
Try Camera.setQuality function (on the client), particularly the 'bandwidth' parameter. You might also want to look at your own internet/server connection since by default the quality changes appropriately to the bandwidth. Those other sites have server farms with massive amounts of bandwidth, something I doubt you have yourself.
If you are using RTMFP, You cannot depend on the quality as it is a peer-2-peer technology. If you need quality and recording stuff, go for RTMP.

Resources