I try to get my linkedin feed using this API :
https://linkedin.api-docs.io/v1.0/feed/42Hm9SaY2p2CGwPzp
I try to use this request : "GET /voyager/api/feed/updates" with this shell code :
curl --request GET \ --url
https://www.linkedin.com/voyager/api/feed/updates \ --data '{}'
But I get this response : "CSRF check failed". I understand why linledin respond this but how to avoid it ?
You missing headers, see API docs here: https://linkedin.api-docs.io/v1.0/feed and explanation how get headers here: https://towardsdatascience.com/using-browser-cookies-and-voyager-api-to-scrape-linkedin-via-python-25e4ae98d2a8
API docs a bit outdated, data output format might be different, this at least true for messaging/conversations, not sure about feed
In regards of headers I suggest to try apify.com and extract them in real time from browser instance (run puppeteer, login to LiN, get headers, save them)
Phantombuster will not allow you to use your own code so not very useful
Related
I'm trying to deploy a Google Apps Script as a web app, but while I have no problem doing GET requests, I'm having trouble with POST requests.
My code is very simple:
function doGet(request) {
var result = JSON.stringify({ data: 'Thanks, I received the GET request' });
return ContentService.createTextOutput(result).setMimeType(ContentService.MimeType.JSON);
}
function doPost(request) {
var result = JSON.stringify({ data: 'Thanks, I received the POST request' });
return ContentService.createTextOutput(result).setMimeType(ContentService.MimeType.JSON);
}
I deployed the web app with "Execute the app as: Me" and "Who has access to the app: Anyone, even anonymous". Every time I do some change I re-deploy it with a new version ("Project version: New").
After publishing, my curl GET request works perfectly:
> curl -L https://script.google.com/macros/s/$SCRIPT_ID/exec
{"data":"Thanks, I received the GET request"}
However, my POST request (curl -L -XPOST https://script.google.com/macros/s/$SCRIPT_ID/exec) just shows me a generic Google HTML page saying "Sorry, unable to open the file at this time. Please check the address and try again".
I tried sending some data and providing a content type, but nothing changed. I also tried changing the output type to just ContentService.createTextOutput("OK"), but it didn't work either. Curiously, deleting doPost changes the error message to "Script function not found: doPost" as expected. If it makes any difference, this script is attached to a Google spreadsheet.
Are there any special permissions I need to give to the script for POST requests specifically?
It seems that the problem was with my usage of curl, on subtle differences between using -XPOST and not using it. As suggested by Tanaike, changing from:
curl -L -XPOST https://script.google.com/macros/s/$SCRIPT_ID/exec
to
curl -L -d '' https://script.google.com/macros/s/$SCRIPT_ID/exec
Solved the issue. Even though curl helpfully says "Unnecessary use of -X or --request, POST is already inferred" when I do a request with -XPOST and a payload, its behavior is different in the presence of redirects. -XPOST forces all subsequent requests after a redirect to be made using POST as a method. On the other hand, if I don't specify -XPOST, the requests after the first POST are made as GET requests. I don't know if this is curl's intended behavior, but it's certainly unintuitive.
I'm following Clarifai's guide to make a cURL request and get the tags related to the image.
In the guide it says that I can do either this:
curl "https://api.clarifai.com/v1/tag/?url=https://samples.clarifai.com/metro-north.jpg" \
-H "Authorization: Bearer {access_token}"
or this:
curl "https://api.clarifai.com/v1/tag/" \
-X POST --data-urlencode "url=https://samples.clarifai.com/metro-north.jpg" \
-H "Authorization: Bearer {access_token}"
So what I do is that I type in the access token that I get when I create a new application and I change the link of "samples.clarifai.com" for a random link of a random image, but every time I want to do this I get the following message on terminal:
{"status_code": "TOKEN_INVALID", "status_msg": "Token is not valid. Please use valid tokens for a application in your account."}
Any idea why I keep getting this eben though my access token is right?
Thanks!
Just so there can have an official answer for this but Marcus Müller is totally right.
You should be sure to remove the braces with the Bearer access token. But you still want to be sure everything else is fine. This does assume though that you have generated a proper access token either by the Developer Documentation or within your Applications page once you have logged in.
http://snomedct.t3as.org/ This is a web service that will analyse English clinical text, and report any concepts that can be detected.
For e.g.- I have headache. It will identify headache as a Symptom.
Now what I would like to do is send the sentence to the web service through R, and get the table back from the web page to R for further analysis purpose.
If we take their example curl command-line:
curl -s --request POST \
-H "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "The patient had a stroke." \
http://snomedct.t3as.org/snomed-coder-web/rest/v1.0/snomedctCodes
that can be translated to httr pretty easily.
The -s means "silent" (no progress meter or error messages) so we don't really have to translate that.
Any -H means to add a header to the request. This particular Content-Type header can be handled better with the encode parameter to httr::POST.
The --data-urlencode parameter says to URL encode that string and put it in the body of the request.
Finally, the URL is the resource to call.
library(httr)
result <- POST("http://snomedct.t3as.org/snomed-coder-web/rest/v1.0/snomedctCodes",
body="The patient had a stroke.",
encode="form")
Since you don't do this regularly, you can wrap the POST call with with_verbose() to see what's going on (look that up in the httr docs).
There are a ton of nuances that one should technically do after this (like check the HTTP status code with stop_for_status(), warn_for_status() or even just status_code(), but for simplicity let's assume the call works (this one is their example so it does work and returns a 200 HTTP status code which is A Good Thing).
By default, that web service is returning JSON, so we need to convert it to an R object. While httr does built-in parsing, I like to use the jsonlite package to process the result:
dat <- jsonlite::fromJSON(content(result, as="text"), flatten=TRUE)
The fromJSON function takes a few parameters that are intended to help shape JSON into a reasonable R data structure (many APIs return horrible JSON and/or XML). This API would fit into the "horrible" category. The data in dat is pretty gnarly and further decoding of it would be a separate SO question.
I would like to stream changes to a Firebase location, but filter the results based on a query to some index, like so:
curl -i -L -H "Accept: text/event-stream" https://mydata.firebaseio.com/path.json?'orderBy="myIndexedField"&equalTo="desiredValue"'
What I observe is that Firebase appears to ignore my query and proceeds to stream all changes to that location whether they match the query or not. Is there any way to do this, other than writing code to perform my own client-side filtering?
EDIT
Frank's answer below shows that Firebase does indeed honor your query parameters. The problem I'm having still persists, and I simply misconstrued what was going on, as the situation turns out to be a little more complicated. I've reproduced my issue in Frank's Firebase, which he was kind enough to supply as a live example for this question.
Here are the steps to reproduce my issue:
Start a streaming query with the constraint type == 1:
$ curl -i -L -H "Accept: text/event-stream" 'https://stackoverflow.firebaseio.com/29265457/.json?print=pretty&orderBy="type"&equalTo=1'
In a separate terminal, post a new item with type==1:
$ curl -X POST -H "Content-Type: application/json" -d '{"type": 1}' https://stackoverflow.firebaseio.com/29265457/.json
{"name":"-JlY1nAmymCKw5lvLvMe"}
This object pops up in my ongoing curl stream as expected, since it matches the query of type==1:
event: patch
data: {"path":"/","data":{"-JlY1nAmymCKw5lvLvMe":{"type":1}}}
Now, here's the part I misinterpreted as Firebase ignoring queries. If I PUT that resource I just POSTed and change it to type==0, it still shows up in my stream! To perform the PUT:
$ curl -X PUT -H "Content-Type: application/json" -d '{"type": 0}' https://stackoverflow.firebaseio.com/29265457/-JlY1nAmymCKw5lvLvMe.json
And here's what I see pop up in my ongoing stream curl:
event: patch
data: {"path":"/","data":{"-JlY1nAmymCKw5lvLvMe":{"type":0}}}
If I PUT "type" to 0 again, it no longer shows up in my curl stream terminal. If I PUT type to 1, then it does pop up in my curl stream (expected). It's the transition from 1->0 where I get the unexpected event.
Also, using PATCH instead of PUT to modify "type" appears to result in the same behavior in the curl stream.
I think it's natural to expect that when I change a value to NOT match my query anymore, I wouldn't see it in my stream. However, it looks like perhaps the query is being matched before the value is edited, or something along those lines... I have no idea how this is implemented on the Firebase end.
So I guess the new question is: How can I avoid seeing changed values that don't match my query in the REST stream?
There are two things that could be wrong here:
you didn't define an index on myIndexedField
your URL is quoted incorrectly
Ad 1) When you access Firebase through the (JavaScript/iOS/Java) libraries, it will fall back to client-side ordering and filtering if you're ordering/filtering data that is not indexed. Given that the REST API doesn't come with a client-library, it cannot perform such a fallback and it will simply return the data unordered/unfiltered.
Ad 2) you need to put your single quotes around the entire URL, not just the query string. An example curl that works for me:
curl -i -L -H "Accept: text/event-stream" 'https://stackoverflow.firebaseio.com/29265457/.json?orderBy="type"&equalTo=0'
I push data to it with this JavaScript snippet:
new Firebase('https://stackoverflow.firebaseio.com/29265457/').push({ type: 0 });
And it shows up in the curl window; or it doesn't if I set type to 1.
I am learning REST API and URI design and I have found one here:
https://raw.githubusercontent.com/JeanVEGA/MI-MPR-DIP-Admission/master/examples/requests.sh
I have a few questions.
There is for example:
User.resetPassword, anonymous by User's {email}
curl -i -X POST http://localhost:9090/admission/services/user/person/email:{email}/reset_password
I do not understand construction email:{email}... what does it mean? It means that if I have String path param, I need to do it in this way?
The similar is here:
Term.get
curl -i -H "Accept: application/json" -H "X-CTU-FIT-Admission-Session: [session identifier from User.identity]" http://localhost:9090/admission/services/term/dateOfTerm:{dateOfTerm}/room:{room}
room:{room} - Is this because room should be for example 123ABC? So it is not a number so it need to be written in this way?
And my last question:
User.resetPassword for User by Admission Code, send notification to User's Email and this {email}
curl -i -H "X-CTU-FIT-Admission-Session: [session identifier from User.identity]" -X POST http://localhost:9090/admission/services/user/admission/{admissionCode}/person/email:{email}/reset_password
My poiont of question is "reset_password" ... I thought due to right design principles that no verb should be in URI... because if the verb is in URI, I thought that it means that resource is actually an operation.
That url can be only a resource identifier. So this is an url template which waits a unique email address as a parameter. A filled in template should look something like this:
./person/email:my#email.adr/reset_password
note:
The reset_password is not a valid REST resource (it describes a service not a resource) and the POST method is mostly for resource creation (not for update or partial update). Real REST requests look like this:
PUT ./person/email:{email}/password "newpass"
PUT ./person/{id}/password "newpass"
PUT ./person/email:{email}/identification_factors/password "newpass"
PATCH ./person/email:{email}/identification_factors {password: "newpass"}
and so on...