Get lyrics data from Spotify - web-scraping

I already figured out the URL is https://spclient.wg.spotify.com/color-lyrics/v2/track/${TRACK_ID}?format=json&vocalRemoval=false
And it requires two headers. app-platform: WebPlayer and authorization: Bearer TOKEN.
So using curl I am able to get the lyrics information like that:
$ TRACK_ID=3z8h0TU7ReDPLIbEnYhWZb
$ BEARER_TOKEN=xxxxxxxxxxxxxx
$ curl "https://spclient.wg.spotify.com/color-lyrics/v2/track/${TRACK_ID}?format=json&vocalRemoval=false" -H "app-platform: WebPlayer" -H "authorization: Bearer ${BEARER_TOKEN}"
{
"lyrics": {
"syncType": "LINE_SYNCED",
"lines": [
{
"startTimeMs": "110",
"words": "Is this the real life? Is this just fantasy?",
"syllables": [],
"endTimeMs": "0"
},
{
"startTimeMs": "6990",
"words": "Caught in a landslide, no escape from reality",
"syllables": [],
"endTimeMs": "0"
},
...
The actual question is how do I programmatically get the required bearer token? I've tried a token requested using Get Token button on this site: https://developer.spotify.com/console/get-track/
But that token only appears to work for the official API. For the lyrics API I always get the following response for that token:
{
"error": {
"status": 403,
"message": "Client not allowed"
}
}
Also, the bearer tokens from the link above are way shorter and contain no dashes.
I know I can just copy the Bearer Token that is used by the web client on https://open.spotify.com but the token always expires after a very short amount of time.
So I'm either looking for a manual way to get a permanent token or for a way to programmatically get short-lived tokens.
I'm not looking for a solution in a specific programming language. Any language will do or an abstract explanation.

This API is not designed nor documented for public use, and is additionally not licensed (see Musixmatch's API, where Spotify gets their lyrics from anyway), so using it (or bragging about finding it on StackOverflow) will likely only have Spotify further lockdown their API, and you (unlikely but possibly) getting into legal trouble (IANAL).
That being said, these Bearer xxx tokens are different to the ones created from Spotify's official API, and the easiest way to find these tokens is just by automating the process of logging in (probably in a browser), and making the request (these tokens may need to be refreshed either every few minutes/hours/days, so automation is key here). I'd suggest to start looking into Puppeteer or MS Playwright on a headless browser, and just retrieving Bearer tokens from there. If the API seems to break with those Bearer tokens, it may be best to perform the request from Puppeteer itself in the browser, by replicating the original request, and amending the song ID.
I am aware btw of how annoying Musixmatch can be, so I completely understand if the 2nd paragraph is the way you're going to go.

You can use the token from here: https://open.spotify.com/get_access_token
The only downside is, that it's not documented, but so is spclient.wg.spotify.com

Related

Why Doesn't my Authorization Header need "Bearer"?

I am currently working with a group of applications that are running on two separate, but equivalent, environments (Referred to as ENV1 and ENV2). I have been using OAuth 2.0 for Authorization and when I receive a response after requesting an access token from the OAuth service (I am making my requests through Postman) I get a response that looks like this from ENV1 and ENV2:
As far as I am aware, I believe that this "token_type": "Bearer" means that when I send in the access_token to my application I need to do so like this:
By sending in the token through the Authorization header, prefixed with "Bearer". This approach works fine on ENV1 but on ENV2 the request fails unless I send in the token alone with no "Bearer" prefix:
If I send in the Authorization header with the "Bearer" prefix, I get a 401 Unauthorized error as the response. This is the help tip that Postman provides(Emphasis mine):
Similar to 403 Forbidden, but specifically for use when authentication is possible but has failed or not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource.
The problem here is that there IS a WWW-Authenticate header field, and it contains "Bearer", which I would assume is a "challenge applicable to the requested resource" seeing as the token response contained "token_type": "Bearer":
Questions:
Why would this be different between environments?
How is this even possible? The documentation I have found on OAuth 2.0 shows that the "Bearer" prefix is needed for requests like the ones I am trying to make. (for example, in section 2.1 of the documentation here)
From your description, it seems that the environments are not actually identical. E.g. maybe ENV2 is behind a gateway that adds the Bearer prefix to the header. Or the API on ENV2 (or the gateway) is configured to read the header without the prefix.
When the OAuth Server returns the access token, it gives you the type - a bearer token. That type means, that the token is just this - a bearer token - as opposed to a Proof-of-Possession token. When you send a bearer token to an API, you don't have to provide any additional information that would prove that you are the owner of the token. (you can compare bearer with the DPoP standard)
The Bearer Token Usage standard does require you to use the prefix Bearer in the authorization header (as you pointed out), but it doesn't mean that all the APIs and gateways implement that standard correctly, or that they use that standard at all.
To summarize:
it's up to the gateway/API to decide in what format they want the Authorization header, and that has nothing to do with the type of the token (a bearer token). It's nice when they use standards, but they don't have to.
In your setup, there must be some kind of difference between the environments if the same request is treated differently between them. If you own the environments you should investigate what is configured differently. If you don't own them, you should contact the owner's support to solve the issue.

Bearer token invalid on here fuel prices API even though apikey is included as parameter

I am new to Here and am trying to make my first API call but I keep getting back this error:
{"Type":"Unauthorized","Message":["Bearer token invalid. Bearer missing or bearer value missing."]}
Here's the URL that I'm using to call the API:
https://fuel-v2.cc.api.here.com/fuel/stations.json?apiKey=${hereApiKey}&prox=${lat},${lng},1600
I have also tried including the api key in an authorization header and get the exact same thing. I've tried with apikey= and apiKey= and the results are the same. I've also tried moving the apikey param to after the prox param (though I know that shouldn't matter). I feel like I'm following the documentation when it says that I can use api key authentication for this call and that app code authentication is deprecated, so I'm not sure what I'm doing wrong.
I am currently on a Here Freemium plan and making this call from a Node JS server application.
The token expires in 24 hours, you need to generate a new token and test again
In order to use OAuth token authentication please use the below request
https://fuel-v2.cc.ls.api.here.com/fuel/stations.xml?prox=52.516667,13.383333,5000
And in header please include
Authorization = Bearer "Oauth token"
other way to include the token is -
https://developer.here.com/documentation/fuel-prices/dev_guide/topics/request-here-environments.html

HERE.com Places API, "ApiKey is missing" error message, but it isn't

I've tried the below request (with two different proper API keys), on the HERE Places API, but I get an auth error.
Request:
https://places.sit.ls.hereapi.com/places/v1/autosuggest
?at=40.74917,-73.98529
&q=chrysler
&apiKey=g51V9*****N4-_GTA
Response:
401
{
"error": "Unauthorized",
"error_description": "ApiKey is missing."
}
I've also tried OAuth, I can get the request token, but when I use this for the places request I get an HTML webpage (for HERE place API) as the response, rather than an expected JSON or XML reponse.
Is anyone able to get any of the example requests to work for Places API?
I had this issue before aswell. Trying the exact same request as you, in Postman with my own apiKey, made the request successfull.
Try making the URL a single line, there might be newline characters hidden somewhere.
This has solved this issue for me many times using this API.
Example:
https://places.sit.ls.hereapi.com/places/v1/autosuggest?at=40.74917,-73.98529&q=chrysler&apiKey=g51V9*****N4-_GTA
You need to check the key name it may wrong
you should check it out into the documentation of api. and see key name
try key name apikey or api_key or api-key instead of apiKey
https://places.sit.ls.hereapi.com/places/v1/autosuggest
?at=40.74917,-73.98529
&q=chrysler
&apikey=g51V9*****N4-_GTA
Please refer to this guide for the updated list of domains to use with apiKey.
For places, the domain is: places.ls.hereapi.com
Disclosure: I'm a product manager at HERE Technologies
In my case I was using : https://github.com/julianshen/gin-limiter with the provided example

Bit.ly bitlink creation always return forbidden even after email verification?

I have registered and done email verification. Then, I try the most baisc API trying to shorten a testing link. The following is my HTTP packet.
POST /v4/shorten HTTP/1.1
Host: api-ssl.bitly.com
Content-Type: application/json
Authorization: Bearer my-generic-access-token
Cache-Control: no-cache
{
"group_guid": "shorten_link_testing",
"domain": "mytesting.site.com",
"long_url": "http://mytesting.site.com/"
}
And the following is the return from bit.ly:
{
"message": "FORBIDDEN",
"resource": "bitlinks",
"description": "You are currently forbidden to access this resource."
}
I am using Postman to test, so I am quite sure it is not the problem of my code. And the message format meets that in the documentation of bit.ly, so at least the connection is success. As far as I know, bit.ly should have 10,000 quota for free shorten link per month for every account, right? What have I missed so that I am banned from the shorten link service?
The group_guid param needs to reference the id of a group associated with your bitly account. Hit this api endpoint with your generic access token to get a list of the groups/guids bitly has associated with your account:
GET https://api-ssl.bitly.com/v4/groups.
Apparently the domain param is optional, because your account may have a default domain associated with it. It worked for me after removing the domain param altogether.
https://groups.google.com/forum/#!topic/bitly-api/9RywN3ETLkg
Hope this helps!!
I have been looking around for these issues. Currently, my solution is by removing body group_guid and getting a response 201 (created). I hope my information helps.

Marketo REST API token keeps expiring

I followed the Quick Start Guide on Marketo's site for their REST API. This went well. I was able to get a successful response from their website.
Request:
curl https://ABC-DEF-123.mktorest.com/rest/v1/lists.json?access_token=123:ab
Response:
{"requestId":"123#abcf7aff","result":[],"success":true}
However, when I tried the same request the next day, I received:
{"requestId":"123#abc6731ab6f","success":false,"errors":[{"code":"601","message":"Access token invalid"}]}
I logged into the Marketo admin and noticed that the token I copied and pasted out of the dialog box was different. I tried this new one and it worked.
(This is taken from the guide)
I came across another guide on their site that describes a different authentication process. Marketo Authentication Guide
This guide mentions the token that is returned from the API endpoint has an expiration so I suspect that all Marketo tokens expire (or I need to disable this). However, I have not been able to successfully make requests to this endpoint with my client ID and client secret.
Request:
curl https://ABC-DEF-123.mktorest.com/identity/oauth/token?grant_type=client_credentials&client_id=ACLIENTID&client_secret=ACLIENTSECRET
Response:
{"error":"unauthorized","error_description":"An Authentication object was not found in the SecurityContext"}
Any help in the right direction would be appreciated. Thanks in advance.
REST API tokens expire. So you typically will need to request a token for each session.
from your specific installed URL, like: MARKETOURL/identity/oauth/tokengrant_type=client_credentials&client_id=abc&client_secret=xyz
Which will result in (example):
{
"access_token": "1234",
"token_type": "bearer",
"expires_in": 3599,
"scope": "email#email.com"
}
You will need to start out your sessions with this request to do subsequent calls. I have code that runs this first and then requests a new token if it expires. The expires_in field is in seconds.
Source: http://developers.marketo.com/documentation/rest/authentication/
I was not able to figure out why my curl requests were failing, but I was able to successfully call the Marketo API with mrkt, a Ruby Gem for Marketo.
I've looked through the logs of what the gem is generating for requests and they appear to be exactly the same. But at least I now successfully calling Marketo.
You need to use following sequences.
grant_type= client_credentials
client_id =you will get Marketo admin where you generate token
client_secret=you will get Marketo admin where you generate token
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
GET <Identity URL>/oauth/token?grant_type=client_credentials&client_id=<Client Id>&client_secret=<Client Secret>
Your first call failed because your token expires every hour currently, according to Marketo's docs.
Your curl call to get a new token failed because curl (or something) was stripping off the auth arguments. Try wrapping the url in quotes.
curl "https://ABC-DEF-123.mktorest.com/identity/oauth/token?grant_type=client_credentials&client_id=ACLIENTID&client_secret=ACLIENTSECRET"
You can get more information about what curl is sending with the -v flag. Running this would have given you enough information to at least know that your entire url wasn't being passed down to the request.
curl -v https://ABC-DEF-123.mktorest.com/identity/oauth/token?grant_type=client_credentials&client_id=ACLIENTID&client_secret=ACLIENTSECRET

Resources