I'm trying to use a simple contains filter with boto3 DynamoDb table resource. My code is as follows:
search = "SomeProductText"
response = table.scan(
FilterExpression='contains(product_text, :pt )',
ExpressionAttributeValues={
':pt': {'S': search}
}
)
print(response['Items'])
while 'LastEvaluatedKey' in response:
response = table.scan(
ExclusiveStartKey=response['LastEvaluatedKey'],
FilterExpression='contains(product_text, :pt )',
ExpressionAttributeValues={
':pt': {'S': search}
}
)
print(response['Items'])
This prints out only empty lists and I'm not getting any results. When I go to the dynamodb console and try this same scan there I get many results where the product_text contains "SomeProductText" as expected. What am I doing wrong here?
As you didn't provide the full code snippet I'll take an educated guess to the issue. The fact you call your client table leads me to believe you're using the Resource client which takes native JSON not DynamoDB-JSON:.
search = "SomeProductText"
response = table.scan(
FilterExpression='contains(product_text, :pt )',
ExpressionAttributeValues={
':pt': search
}
)
print(response['Items'])
while 'LastEvaluatedKey' in response:
response = table.scan(
ExclusiveStartKey=response['LastEvaluatedKey'],
FilterExpression='contains(product_text, :pt )',
ExpressionAttributeValues={
':pt': search
}
)
print(response['Items'])
Related
Hopefully a quick question, I'm trying to connect to the KuCoin API, not super relevant as I think this is more an issue with how I'm using the POST function and how it sends JSON along
Here is my function that is supposed to place an order:
API.Order <- function(pair,buysell,price,size) {
path = "/api/v1/orders"
now = as.integer(Sys.time()) * 1000
json <- list(
clientOid = as.character(now),
side = buysell,
symbol=pair,
type="limit",
price=price,
size=size
)
json=toJSON(json, auto_unbox = TRUE)
str_to_sign = (paste0(as.character(now), 'POST', path, json))
signature = as.character(base64Encode(hmac(api_secret,str_to_sign,"sha256", raw=TRUE)))
passphrase=as.character(base64Encode(hmac(api_secret,api_passphrase,"sha256", raw=TRUE)))
response=content(POST(url=url,
path=path,
body=json,
encode="json",
config = add_headers("KC-API-SIGN"=signature,
"KC-API-TIMESTAMP"=as.character(now),
"KC-API-KEY"=api_key,
"KC-API-PASSPHRASE"=passphrase,
"KC-API-KEY-VERSION"="2")
),
"text",encoding = "UTF-8")
response
data.table(fromJSON(response)$data)
}
API.Order(pair,"sell",1.42,1.0)
And everything works, except I get the following response:
"{\"code\":\"415000\",\"msg\":\"Unsupported Media Type\"}"
Which is puzzling to me. Everything else checks out (the signature and other auth headers), and I set the encode to "json" in the POST.. I also can put it as standard "application/json" and neither works. I've been staring at this for hours now and I can't see what (likely very little) thing I got wrong?
Thanks
I have a table in a cosmosDB where I would like to store some query coming in from a azure data explorer.
In data explorer (using python sdk) I run a query to filter and aggregate my data and I would like to push this query result into a cosmosDB.
I did set all the configurations and connection, but when I run the command upset_item or create_item
I get the following error:
azure.cosmos.exceptions.CosmosHttpResponseError: Status code: 400
{"odata.error":{"code":"InvalidInput","message":{"lang":"en-us","value":"Request url is invalid.\r\nActivityId: d8994113-504e-4198-9976-41316aaafb5f, documentdb-dotnet-sdk/2.11.0 Host/64-bit MicrosoftWindowsNT/6.2.9200.0\nRequestID:d8994113-504e-4198-9976-41316aaafb5f\n"}}}
this is how I configured the azure-cosmos:
url = "url"
key = {"masterkey": "my-key"}
client = CosmosClient(url,key)
database_name = "database"
container_name = "table"
database = client.get_database_client(database_name)
container = database.get_container_client(container_name)
for i in df:
container.upsert_item(i)
df is the result of my query from azure data explorer
the error I believe is due how I am passing the body of the query to the upsert_item
Any advice about this?
UPDATE:
so I tried to follow the sample. I updated my cosmosdb to a sql API.
my current code now looks like this:
url = "url"
key = {"masterkey": "my-key"}
client = CosmosClient(url,key)
database_name = "database"
container_name = "table"
database = client.get_database_client(database_name)
container = database.get_container_client(container_name)
data = json.dumps(test)
data_dict = json.loads(data)
for i in data_dict:
container.create_item(body=i)
I converted the data frame to a json. but when I run the for I get this error:
AttributeError: 'str' object has no attribute 'get'
And I have no idea what am I doing wrong
Thanks for your reply and I think we need more contact on debug.
Here's my code and it worked well.
from azure.cosmos import exceptions, CosmosClient, PartitionKey,ProxyConfiguration
import family
# Initialize the Cosmos client
endpoint = "https://cosmosdb_name.documents.azure.com:443/"
key = 'primary_key_in_keys_blade'
client = CosmosClient(endpoint, key)
database_name = 'AzureSampleFamilyDatabase'
database = client.create_database_if_not_exists(id=database_name)
# </create_database_if_not_exists>
# <create_container_if_not_exists>
container_name = 'FamilyContainer'
container = database.create_container_if_not_exists(
id=container_name,
partition_key=PartitionKey(path="/lastName"),
offer_throughput=400
)
family_items_to_create = [
{
"id":"112233",
"name":"andersen"
},
{
"id":"455646",
"name":"johnson"
},
{
"id":"999999",
"name":"smith"
}
]
# <create_item>
for family_item in family_items_to_create:
container.upsert_item(body=family_item)
Per your error, I think you need to check the data format of data_dict or you can use my sample data for testing to rule out other problems. In my position, I don't know why you'd like to use json.loads() here.
I am trying to use httr to call on an API from IGDB (documentation here). When I use the following query in Postman, I have no problem and get the full request which has 100 entries:
fields rating, game; where game = 114283; limit 100; sort id desc;
Example of entry here:
{
"id": 442667,
"game": 114283,
"rating": 3.0
},
However, when I attempt to make this query in R using httr as follows:
string <- paste0("rating, game; where game = ", ids[1,1], "; limit 100; sort id desc;")
data <- POST("https://api-v3.igdb.com/private/rates/",
add_headers("user-key" = "XXXXXXXXXX"),
query = list(fields = string)
)
fromJSON(rawToChar(data$content))
it returns only a data frame of 23 rows:
id game rating
1 442667 114283 3
...
23 383956 114283 10
Other calls similarly return shortened data frames with varying length depending upon the query.
If anyone has any idea as to why this might be happening, I would love some insight.
Thanks.
I think you're merely passing your query string in wrongly, since it is supposed to be passed into the -d (data) parameter of your http request.
Try the following:
append the initial "fields " into string
pass this to the body param of your POST request
So:
string <- paste0("fields rating, game; where game = ", ids[1,1], "; limit 100; sort id desc;")
data <- POST(
"https://api-v3.igdb.com/private/rates/",
add_headers("user-key" = "XXXXXXXXXX"),
body = string
)
I have written the following spec. Surprisingly it passes even though the result doesn't match
code snippet
val controller = new UserController(mockUserRepository,mockControllerComponents,mockSilhouette)
//val request = FakeRequest[AnyContentAsJson]("POST", "/ws/users/signup").withJsonBody(Json.parse("""{"bad": "field"}"""))//FakeRequest(POST,"/ws/users/signup").withJsonBody(Json.parse("""{"bad":"field"}"""));
val request = FakeRequest("POST","ws/users/signup")
println("sending request",request)
//val result = controller.someMethod()
val result = controller.signupUser(request)
Await.result(result,Duration("10 secs"))
result.map(response => {
println("response from controller:"+response)
response mustBe play.api.mvc.Results.BadRequest
})
console prints
(sending request,POST ws/users/signup)
print in controller. (received request,POST ws/users/signup)
Controller returns 200OK but I am matching it with BadRequest. Still the test passes!
response from controller:Result(200, Map())
I suspect that I am not matching the result correctly. I am doing response mustBe play.api.mvc.Results.BadRequest. I know that response is Results but BadRequest is Status. But I don'w know how else to match and also why the test doesn't fail. I also tried following and can see the the values are different but even then the test passes.
println("response from controller:"+response.header.status+" compared with "+play.api.mvc.Results.BadRequest.header.status)
response.header.status mustBe play.api.mvc.Results.BadRequest.header.status
console print - response from controller:200 compared with 400
Importing members of Helpers object like so
import play.api.test.Helpers._
gives access to various utility methods to extract results from Future[Result] in tests. For example, Helpers.status can be used to extract status code like so:
val result: Future[Result] = controller.signupUser(request)
status(result) mustBe BAD_REQUEST
Another option is to use whenReady from ScalaFutures:
val result: Future[Result] = controller.signupUser(request)
whenReady(result) { _.header.status mustBe BAD_REQUEST }
Another option is to extend AsyncWordSpec and then you can map over the Future like so:
val result: Future[Result] = controller.signupUser(request)
result.map(_.header.status mustBe BAD_REQUEST)
When I issue a linq count against DocumentDb.
The first POST web request results in a 400 Bad request
Request json:
{"query":"SELECT VALUE Count(1) FROM root
WHERE ((((root[\"docType\"] = \"Whatever\")
AND (root[\"field1\"] = false))
AND (root[\"field2\"] = true))
AND (root[\"field3\"] = 1)) "}
Message:
Query that expects full results from aggregate functions is not supported
This is followed by a GET partition key ranges.
Then another POST web request for the count query which works:
Request json:
{"query":"SELECT VALUE [{\"item\": Count(1)}]\r\nFROM root\r\n
WHERE ((((root[\"docType\"] = \"Whatever\")
AND (root[\"field1\"] = false))
AND (root[\"field2\"] = true))
AND (root[\"field3\"] = 1))"}
Am I doing something wrong/silly here or is this expected behaviour for documentDb.
Thanks
Donal
Query that expects full results from aggregate functions is not supported
I do a test and same error "Query that expects full results from aggregate functions is not supported" appears on my side if x-ms-documentdb-query-iscontinuationexpected request header is set to false.
Set x-ms-documentdb-query-iscontinuationexpected request header to True, and request is ok.
Please capture your request and check the actual value of x-ms-documentdb-query-iscontinuationexpected request header.