I'm using WSO2-APIM-2.1.0. Which internally uses solr 5.2.1.
Inside WSO2-APIM-2.1.0, I've added around 2087 applications. Wso2 internally generating the solr OR query which has around 2087 OR clauses of 121825 characters.
When I tried to get application list though the API call:
curl -k -H "Authorization: Bearer access_token" "https://localhost:9443/api/am/publisher/v0.11/subscriptions?apiId=apiId&limit=2200"
(It took around 6 minutes, which is too long.)
Using the link https://localhost:9443/store (by clicking on applications), it's not loading application list.(as browser is timing out)
I was getting exception of maxBooleanClauses. Then I made following changes:
I've modified the file repository/conf/solr/registry-indexing/conf/solrconfig.xml and modified the property: <maxBooleanClauses>16000</maxBooleanClauses>.
How can I speed-up the response ?
Related
Normally we are able to play around with REST APIs related to application, since the application has method to let us create a JWT Token for authentication.
But we are unable to create an application, don’t understand where and we can get the token to authorize us to let us create an application.
Let me tell step by step how to do that
Open the file {AMS_INSTALL_DIR}/webapps/root/WEB-INF/web.xml and change the following line
<filter-class>io.antmedia.console.rest.AuthenticationFilter</filter-class>
with this one
<filter-class>io.antmedia.console.rest.JWTServerFilter</filter-class>
Open the file {AMS_INSTALL_DIR}/conf/red5.properties and change the following lines
server.jwtServerControlEnabled=false
server.jwtServerSecretKey=
with these ones. You can use any 32 character alphanumeric key.
server.jwtServerControlEnabled=false
server.jwtServerSecretKey=WRITE_YOUR_32_CHARACTER_SECRET_KEY
For our sample we use cizvvh7f6ys0w3x0s1gzg6c2qzpk0gb9 as secret key
Restart the service
sudo service antmedia restart
Generate JWT Token. There are plenty of libraries that you can do programmatically. The easiest way for now is using JWT Debugger. So our generated token is eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.tA6sZwz_MvD9Nocf3Xv_DXhJaeTNgfsHPlg3RHEoZRk
Make the call to Create Application as follows
curl -X POST -H "Content-Type: application/json" -H "Authorization:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.tA6sZwz_MvD9Nocf3Xv_DXhJaeTNgfsHPlg3RHEoZRk" "https://ovh36.antmedia.io:5443/rest/v2/applications/testapp"
The result should be something like {"success":true,"message":null,"dataId":null,"errorId":0}
The app should be generated in a couple of seconds. You can get the list of the applications with the following command
curl -X GET -H "Content-Type: application/json" -H "Authorization:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e30.tA6sZwz_MvD9Nocf3Xv_DXhJaeTNgfsHPlg3RHEoZRk" "https://ovh36.antmedia.io:5443/rest/v2/applications"
References:
Web Panel REST Methods
Web Panel REST Methods JWT Documentation
Helllo,
I'm working on a scraper for this page : https://www.dirk.nl/
I'm trying to get in scrapy shell the 'row-wrapper' div class.
If I enter response.css('row-wrapper'), it gives me some random results, I think an anti scraping system is involved. I need the hrefs from this class.
Any opinions on how can I move forward ?
We would need a little bit more data, like the response you receive and any code if it's already set up.
But from the looks of it, it can be multiple things ( from 429 Response blocking the request because of the rate limit to sites internal API XHR causing data to not be rendered on page load etc. ).
Before fetching any website for scraping reasons, try curl, postman or insomnia software to see what type of the response are you going to receive. Some special servers and website architectures require certain cookies and headers while some don't. You simply have to do this research so you can make your scraping workflow efficient.
I ran curl https://www.dirk.nl/ and it returned data that's generated by Nuxt framework. In this case that data is unusable since Nuxt uses it's own functionality to parse data.
Instead, the best solution would be not to get the HTML based data but API content data.
Something like this:
curl 'https://content-api.dirk.nl/misc/specific/culios.aspx?action=GetRecipe' \
-H 'accept: application/json, text/plain, */*' \
--data-raw '{"id":"11962"}' \
--compressed
Will return:
{"id":11962,"slug":"Muhammara kerstkrans","title":"Muhammara kerstkrans","subtitle":"", ...Rest of the data
I don't understand this language but from my basic understanding this would be an API route for recipes.
I'm trying to deploy a Google Apps Script as a web app, but while I have no problem doing GET requests, I'm having trouble with POST requests.
My code is very simple:
function doGet(request) {
var result = JSON.stringify({ data: 'Thanks, I received the GET request' });
return ContentService.createTextOutput(result).setMimeType(ContentService.MimeType.JSON);
}
function doPost(request) {
var result = JSON.stringify({ data: 'Thanks, I received the POST request' });
return ContentService.createTextOutput(result).setMimeType(ContentService.MimeType.JSON);
}
I deployed the web app with "Execute the app as: Me" and "Who has access to the app: Anyone, even anonymous". Every time I do some change I re-deploy it with a new version ("Project version: New").
After publishing, my curl GET request works perfectly:
> curl -L https://script.google.com/macros/s/$SCRIPT_ID/exec
{"data":"Thanks, I received the GET request"}
However, my POST request (curl -L -XPOST https://script.google.com/macros/s/$SCRIPT_ID/exec) just shows me a generic Google HTML page saying "Sorry, unable to open the file at this time. Please check the address and try again".
I tried sending some data and providing a content type, but nothing changed. I also tried changing the output type to just ContentService.createTextOutput("OK"), but it didn't work either. Curiously, deleting doPost changes the error message to "Script function not found: doPost" as expected. If it makes any difference, this script is attached to a Google spreadsheet.
Are there any special permissions I need to give to the script for POST requests specifically?
It seems that the problem was with my usage of curl, on subtle differences between using -XPOST and not using it. As suggested by Tanaike, changing from:
curl -L -XPOST https://script.google.com/macros/s/$SCRIPT_ID/exec
to
curl -L -d '' https://script.google.com/macros/s/$SCRIPT_ID/exec
Solved the issue. Even though curl helpfully says "Unnecessary use of -X or --request, POST is already inferred" when I do a request with -XPOST and a payload, its behavior is different in the presence of redirects. -XPOST forces all subsequent requests after a redirect to be made using POST as a method. On the other hand, if I don't specify -XPOST, the requests after the first POST are made as GET requests. I don't know if this is curl's intended behavior, but it's certainly unintuitive.
We have a WordPress custom build and have integrated the Vimeo API to pull videos through to the website.
The setup is working but the API calls are taking 20 seconds. We have tested using Postman and they only take 1-2 seconds.
Is there a solution to this?
Use the fields parameter on your requests to tell the API to only return the metadata needed for your application. Because Vimeo API responses can be quite large, especially when retrieving a list of videos, the fields parameter can significantly reduce the size of the response, and subsequently increase response time.
For example, let's say you're making a request to get the last 10 videos you uploaded. The request would look like this:
curl -X GET https://api.vimeo.com/me/videos?page=1&per_page=10
-H 'Accept: application/vnd.vimeo.*+json;version=3.4'
-H 'Authorization: bearer [token]'
The response would return the full and complete video objects for 10 videos, which can be quite large. However if you only need some of the metadata in the response, such as the video's name, description, and its link on vimeo.com, then the same request with the fields param will look like this:
curl -X GET https://api.vimeo.com/me/videos?page=1&per_page=10&fields=uri,name,description,link
-H 'Accept: application/vnd.vimeo.*+json;version=3.4'
-H 'Authorization: bearer [token]'
The fields parameter is documented here: https://developer.vimeo.com/api/common-formats#json-filter
I configured a REST webservice (a Spring Boot webapplication) on WSO2 AM and used the default /* mapping for resources. My webservice takes an assignee (text) and file parameters.
When I perform the calls, I've noticed that request parameters are not forwarded (HTTP Headers are) to the backed services. For example:
curl -i -X POST -H "Content-Type: multipart/form-data" -H "X-PD20-BillingSubscriptionId: e87d4400-b05f-4f40-9c39-06ae4d28cf4d" -H "Authorization: Bearer rrxRV5F6jdkSBcEPXv7I1yFl2x8a" -F "documentFile=#src/test/resources/sample-files/test-fea-1firma.pdf" -F "assignee=bla.bla#gmail.com" http://api.linksmt.it:8280/fea/1.0.0/signRequest
As you can see, It's a form that posts 2 fields, one of them being a file and another a simple text field.
The call is succesfully forwarded to the backed service but without the actual fields values (the headers instead are correctly passed, though their keys are lower-cased, that is "X-PD20-BillingSubscriptionId" is passed as "x-pd20-billingsubscriptionid").
Any hint on why is this happening?
Thanks
Ok, the problem was the same as described in multipart form data file upload using WSO2 API manger ? and I had to uncomment the declarations for
within the $WSO2_AM/repository/conf/axis2/axis2.xml file (and restart the server).