How do I fetch Amazon DynamoDB data using a RESTful API?
Is there a way to get Amazon DynamoDB data using a REST url, and if so what are the required parameters to pass in the url?
We have considered the DynamoDB endpoint as the url and append it with the accesskey and the secretaccesskey, is anything more required to append to the url?
If any one has tried this with DynamoDB RESTful API, can you give me an example of how to get table data?
A sample url would also be good, something showing how to connect to DynamoDB through a RESTful API.
Ideally, a sample url with all the parameters required.
There is an AWS documentation including an example:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/MakingHTTPRequests.html
You can use API gateway which points to a Lambda function to fetch the data from DynamoDB. End point of your API gateway URL will be your rest end point.
That is not how Dynamo works, you will have to build your own RESTful API (i.e. use the AWS SDK for PHP) that hits Dynamo, reformats the data to however you want it then returns it. Quite easy to do :-)
Here is a minimal JavaScript example of performing a GetItem operation on a DynamoDB table via the AWS API:
const DYNAMODB_ENDPOINT = 'https://dynamodb.us-east-1.amazonaws.com';
let aws = new AwsClient({ accessKeyId: AWS_ACCESS_KEY, secretAccessKey: AWS_SECRET_KEY });
async function dynamoDbOp(op, opBody) {
let result = await aws.fetch(DYNAMODB_ENDPOINT, {
method: 'POST',
headers: {
'Content-Type': 'application/x-amz-json-1.0',
'X-Amz-Target': 'DynamoDB_20120810.' + op
},
body: JSON.stringify(opBody)
});
return result.json();
}
let dbResponse = await dynamoDbOp('GetItem', {
TableName: "someTableName",
Key: {
someTableKey: {
S: "someKeyValue"
}
}
});
console.log(dbResponse.json());
Note that AwsClient is a class defined by aws4fetch which takes care of signing AWS requests for you. See the AWS GetItem API doc for more info.
Related
I'm trying to connect my app to firebase but the only response I get is not the response json I need from firebase. I included my call to firebase below. Is the url not correct? The response I'm getting back is not the json object made with firebase that I created.
``
<script>
(async function call () {
console.log("hello")
const endpoint = url
console.log(endpoint)
async function initiation () {
const result = await fetch(endpoint, {mode: "no-cors"})
const data = await result
console.log(data)
}
initiation()
})()
</script>
``
is your database in us-central1?
according to documentation [1] "the form https://<"databaseName">.firebaseio.com (for us-central1 databases) or https://<"databaseName"><"region">.firebasedatabase.app (for databases in all other locations)."
If its in another region you should try with https://<"databaseName"><"region">.firebasedatabase.app
[1]https://firebase.google.com/docs/database/web/start#initialize_the_javascript_sdk
The structure within your code seems odd, if you are implementing the CDN you need to initiate your app with your project credentials, right now you are only accessing a real-time database as a public request and does not provide any additional validators as the database is most likely to have Security Rules enabled.
To request data from the endpoint, you need to also include a .json at the end of the URL https://[PROJECT_ID].firebaseio.com/users/jack/name.json
Source: https://firebase.google.com/docs/reference/rest/database#section-get
We are using LocalStack and having issues interacting with DynamoDB using the AWS js SDK. We are getting an "UnknownError" when trying to use the putItem method of the DynamoDB instance. Here is the error output:
{
“name”: “DynamoDbError”,
“data”: {
“dynamoDbPutObjectParameters”: {
“TableName”: “fooTableName”,
“Item”: {
“hello”: {
“S”: “world”
}
}
}
},
“baseError”: {
“message”: null,
“code”: “UnknownError”,
“time”: “2020-11-23T11:40:26.382Z”,
“statusCode”: 500,
“retryable”: true
}
}
We have set the endpoint in the DynamoDB instance options to be:
http://localhost:4566
4566 is the Edge port we have set for the DynamoDB service in LocalStack.
We get validation errors from the SDK if our put object parameters are incorrect BUT these parameters seem to be okay. They DO work on AWS proper when deployed and the table does update.
We have deployed the DynamoDB tables locally using the AWS CDK and aws-cdk-local and WE CAN query, put and update the table in LocalStack using the AWS CLI successfully.
The issue seems to be using the AWS js SDK to interact with the DynamoDB in LocalStack.
Here is the call to putItem which works on AWS but not with LocalStack via the AWS SDK:
putItem: async (
dynamoDbPutObjectParameters
) => {
try {
const data = await dynamoDbInstance.putItem(dynamoDbPutObjectParameters).promise();
return data;
} catch (error) {
throw new DynamoDbError('Failed to put data to DynamoDB', {
data: { dynamoDbPutObjectParameters },
baseError: error
});
}
},
Does anyone have any ideas what might be going on or how to progress?
Thanks
FWIW the issue that was causing the UnknownError was to do with the putItem command parameter. We were not correctly supplying the sortKey value. The LocalStack implementation of DynamoDB does not provide any detailed error output in the response BUT when we tried a similar call on AWS proper we got detailed error output which lead us to understand what was wrong with the implementation.
Its also worth noting that detailed error information did appear in the output of the LocalStack Docker container, just not in the error response when calling the aws js sdk putItem command.
I have a requirement to call a rest api and store the resulting json in azure storage container. I have tried standalone python coding to extract the data from rest api and able to successfully receive the data from api that has pagination. Now I need to integrate/modify this python coding inside Azure Function and will ultimately store the resulting json data in a azure storage container. I am fairly new to Azure and hence need your guidance on how to tweak this code to suit in Azure function that will in turn push the json to azure container finally.
response = requests.post(base_url,
auth=(client_id, client_secret), data={'grant_type':grant_type,'client_id':client_id,'client_secret':client_secret,'resource':resource})
acc_token_json = response.json()
access_token = json.loads(response.text)
token = access_token['access_token']
#call API to know total pages
API_Key = 'xxxxx'
api_url='https://api.example.com?pageSize=10&page=1&sortBy=orderid&sortDirection=asc'
headers = {
'Authorization': token,
'API-Key': API_Key,
}
r = requests.get(url=api_url, headers=headers).json()
total_record=int(r['pagination']['total'])
total_page=round(total_record/500)+1
#loop through all pages
all_items = []
for page in range(0, total_page):
url = "https://api.example.com?pageSize=500&sortBy=orderid&sortDirection=asc&page="+str(page)
response = requests.get(url=url, headers=headers).json()
response_data=response['data']
all_items.append(response_data)
Your inputs/guidances are very much appreciated.
You can put the logic in the body of the function.(Function is just set the condition of trigger.)
For example, if you are based on HttpTrigger:
import logging
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
'''
#Put the your logic code here.
'''
return func.HttpResponse(
"This is a test.",
status_code=200
)
And you can also use blob output to achieve your requirement, it is easier, have a look of this offcial doc:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-output?tabs=python#example
Let me know if have any problem.
I need to be able to intercept XHR requests on page loaded with Puppeteer and return mock responses in order to organize backendless testing for my web app. What's the best way to do this?
It seems that the way to go is request.respond() indeed, but still, I couldn't find a concrete example in the web on how to use it. The way I did it was like this:
// Intercept API response and pass mock data for Puppeteer
await page.setRequestInterception(true);
page.on('request', request => {
if (request.url() === constants.API) {
request.respond({
content: 'application/json',
headers: {"Access-Control-Allow-Origin": "*"},
body: JSON.stringify(constants.biddersMock)
});
}
else {
request.continue();
}
});
What happens here exactly?
Firstly, all requests are intercepted with page.setRequestInterception()
Then, for each request I look for the one I am interested in, by matching it by URL with if (request.url() === constants.API) where constants.API is just the endpoint I need to match.
If found, I pass my own response with request.respond(), otherwise I just let the request continue with request.continue()
Two more points:
constants.biddersMock above is an array
CORS header is important or access to your mock data will not be allowed
Please comment or refer to resources with better example(s).
Well. In the newest puppeteer,it provide the request.respond() method to handle this situation.
If anyone is interested I ended up creating special app build for my testing needs, which adds Pretender to the page. And I communicate with Pretender server using Puppeteer's evaluate method.
This is not ideal, but I couldn't find a way to achieve what I need with Puppeteer only. There is a way to intercept requests with Puppeteer, but seems to be no way to provide fake response for a given request.
UPDATE:
As X Rene mentioned there is now native support for this in Puppeteer v0.13.0 using request.respond() method. I'm going to rewrite my tests to use it instead of Pretender, since this will simplify many things for me.
UPDATE 2:
There is pptr-mock-server available now to accomplish this. Internally it relies on request interception and request.respond() method. Library is pretty minimal, and may not fit your needs, but it at least provides an example how to implement backendless testing using Puppeteer. Disclaimer: I'm an author of it.
I created a library that uses Puppeteer's page.on('request') and page.on('response') to record and respond with mocked requests.
https://github.com/axiomhq/puppeteer-request-intercepter
npm install puppeteer-request-intercepter
const puppeteer = require('puppeteer');
const { initFixtureRouter } = require('puppeteer-request-intercepter');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// Intercept and respond with mocked data.
const fixtureRouter = await initFixtureRouter(page, { baseUrl: 'https://news.ycombinator.com' });
fixtureRouter.route('GET', '/y18.gif', 'y18.gif', { contentType: 'image/gif' });
await page.goto('https://news.ycombinator.com', { waitUntil: 'networkidle2' });
await page.pdf({ path: 'hn.pdf', format: 'A4' });
await browser.close();
})();
You may want to try out Mockiavelli - request mocking library for Puppeteer. It was build exactly for backendless testing of webapps. It integrates best with jest and jest-puppeteer, but works with any testing library.
What I'm trying to do:
Add events to a google calendar from my site using javascript.
What I can't do:
Find a good tutorial/walk through/example for the google calendar api. All the documentation I've been able to find links back and forth between v1 and v2 api's, or the v3 api doesn't seem to be client based.
For those that are curious, the site I'm developing this for:
http://infohost.nmt.edu/~bbean/banweb/index.php
Google provides a great JS client library that works with all of Google's discovery-based APIs (such as Calendar API v3). I've written a blog post that covers the basics of setting up the JS client and authorizing a user.
Once you have the basic client enabled in your application, you'll need to get familiar with the specifics of Calendar v3 to write your application. I suggest two things:
The APIs Explorer will show you which calls are available in the API.
The Chrome developer tools' Javascript console will automatically suggest method names when you are manipulating gapi.client. For example, begin typing gapi.client.calendar.events. and you should see a set of possible completions (you'll need the insert method).
Here's an example of what inserting an event into JS would look like:
var resource = {
"summary": "Appointment",
"location": "Somewhere",
"start": {
"dateTime": "2011-12-16T10:00:00.000-07:00"
},
"end": {
"dateTime": "2011-12-16T10:25:00.000-07:00"
}
};
var request = gapi.client.calendar.events.insert({
'calendarId': 'primary',
'resource': resource
});
request.execute(function(resp) {
console.log(resp);
});
Hopefully this is enough to get you started.
this should do the trick
//async function to handle data fetching
async function getData () {
//try catch block to handle promises and errors
try {
const calendarId = ''
const myKey = ''
//using await and fetch together as two standard ES6 client side features to extract the data
let apiCall = await fetch('https://www.googleapis.com/calendar/v3/calendars/' + calendarId+ '/events?key=' + myKey)
//response.json() is a method on the Response object that lets you extract a JSON object from the response
//response.json() returns a promise resolved to a JSON object
let apiResponse = await apiCall.json()
console.log(apiResponse)
} catch (error) {
console.log(error)
}
}
getData()