I have a table in a cosmosDB where I would like to store some query coming in from a azure data explorer.
In data explorer (using python sdk) I run a query to filter and aggregate my data and I would like to push this query result into a cosmosDB.
I did set all the configurations and connection, but when I run the command upset_item or create_item
I get the following error:
azure.cosmos.exceptions.CosmosHttpResponseError: Status code: 400
{"odata.error":{"code":"InvalidInput","message":{"lang":"en-us","value":"Request url is invalid.\r\nActivityId: d8994113-504e-4198-9976-41316aaafb5f, documentdb-dotnet-sdk/2.11.0 Host/64-bit MicrosoftWindowsNT/6.2.9200.0\nRequestID:d8994113-504e-4198-9976-41316aaafb5f\n"}}}
this is how I configured the azure-cosmos:
url = "url"
key = {"masterkey": "my-key"}
client = CosmosClient(url,key)
database_name = "database"
container_name = "table"
database = client.get_database_client(database_name)
container = database.get_container_client(container_name)
for i in df:
container.upsert_item(i)
df is the result of my query from azure data explorer
the error I believe is due how I am passing the body of the query to the upsert_item
Any advice about this?
UPDATE:
so I tried to follow the sample. I updated my cosmosdb to a sql API.
my current code now looks like this:
url = "url"
key = {"masterkey": "my-key"}
client = CosmosClient(url,key)
database_name = "database"
container_name = "table"
database = client.get_database_client(database_name)
container = database.get_container_client(container_name)
data = json.dumps(test)
data_dict = json.loads(data)
for i in data_dict:
container.create_item(body=i)
I converted the data frame to a json. but when I run the for I get this error:
AttributeError: 'str' object has no attribute 'get'
And I have no idea what am I doing wrong
Thanks for your reply and I think we need more contact on debug.
Here's my code and it worked well.
from azure.cosmos import exceptions, CosmosClient, PartitionKey,ProxyConfiguration
import family
# Initialize the Cosmos client
endpoint = "https://cosmosdb_name.documents.azure.com:443/"
key = 'primary_key_in_keys_blade'
client = CosmosClient(endpoint, key)
database_name = 'AzureSampleFamilyDatabase'
database = client.create_database_if_not_exists(id=database_name)
# </create_database_if_not_exists>
# <create_container_if_not_exists>
container_name = 'FamilyContainer'
container = database.create_container_if_not_exists(
id=container_name,
partition_key=PartitionKey(path="/lastName"),
offer_throughput=400
)
family_items_to_create = [
{
"id":"112233",
"name":"andersen"
},
{
"id":"455646",
"name":"johnson"
},
{
"id":"999999",
"name":"smith"
}
]
# <create_item>
for family_item in family_items_to_create:
container.upsert_item(body=family_item)
Per your error, I think you need to check the data format of data_dict or you can use my sample data for testing to rule out other problems. In my position, I don't know why you'd like to use json.loads() here.
Related
i'm currently making a roblox whitelist system and it's almost finished but i need 1 thing more i scripted it and its not work (code below) i didn't found nothing to fix what i have (script and screenshoot of error below), thanks.
local key = 1
local HttpService = game:GetService("HttpService")
local r = HttpService:RequestAsync({
Url = "https://MyWebsiteUrl.com/check.php?key="..key,
Method = "GET"
})
local i = HttpService:JSONDecode(r.Body)
for n, v in pairs(i) do
print(tostring(n)..", "..tostring(v))
end
I assume the website that you are using to validate the key
returns the response in raw if so then
local key = 1
local HttpService = game:GetService("HttpService")
local r = HTTPService:GetAsync("https://MyWebsiteUrl.com/check.php?key="..key)
local response = JSON:Decode(r)
print(response)
I think this is because you tried to concat a string (the url) with a number (the key variable) try to make the key a string
attribute itemJson stored as follow
"itemJson": {
"S": "{\"sold\":\"3\",\"listingTime\":\"20210107211621\",\"listCountry\":\"US\",\"sellerCountry\":\"US\",\"currentPrice\":\"44.86\",\"updateTime\":\"20210302092220\",\"itemLocation\":\"Miami,FL,USA\",\"listType\":\"FixedPrice\",\"categoryName\":\"Machines\",\"itemID\":\"293945109477\",\"sellerID\":\"holiday_for_you\",\"s3Key\":\"US/2021/2/FixedPrice/293945109477.json\",\"visitCount\":\"171\",\"createTime\":\"20210201233158\",\"listingStatus\":\"Completed\",\"endTime\":\"2021-02-28T20:22:57\",\"currencyID\":\"USD\"}"
},
i want to query with filter:contains(itemJson, "sold":"0") with java sdk,i tried those syntax,all fail
expressionValues.put(":v2", AttributeValue.builder().s("\\\"sold\\\":\\\"0\\\"").build());
expressionValues.put(":v2", AttributeValue.builder().s("sold:0"").build());
what is the right way to my filter syntax?
I try #Balu Vyamajala's syntax on the dynamodb web console as follow,did not get the solution yet
contains (itemJson, :subValue) with value of "sold\":\"3\"" seems to be working.
Working example on a Query Api and worked as expected:
QuerySpec querySpec = new QuerySpec()
.withKeyConditionExpression("pk = :v_pk")
.withFilterExpression("contains (itemJson, :subValue)")
.withValueMap(new ValueMap().withString(":v_pk", "6").withString(":subValue", "sold\":\"3\""));
and to test from Aws console we just need to enter "sold":"2"
I am creating a composer from terraform where I want to pass a json as input variable
Terraform code:
software_config{
env_variables{
AIRFLOW_VAR_MYJSON ="{'__comment1__': 'This the global section', 'project_id':'testproject', 'gce_zone':'us-east1-c', 'gce_region':'us-east1','networkname':'vpc1', 'subnetwork':'https://www.googleapis.com/compute/v1/projects/testproject/regions/us-east1/subnetworks/subnet1'}"
}
}
I am trying to read the value of AIRFLOW_VAR_MYJSON in DAG , but it is not working as the value is not recognized as JSON.
I tried converting it and then deserializing it with following code:
JSONList = Variable.get("MYJSON")
jsonvar = json.dumps(JSONList)
setting_var = Variable.set("settings", jsonvar)
dag_config = Variable.get("settings", deserialize_json=True)
but it is not working.
I have also tried using
dag_config =json.loads(jsonvar)
then reading value as
project_id = dag_config["project_id"]
but I get error : "string indices must be integers"
Please suggest a way to resolve this.
NOTE : I know the gcloud command to set variables from json file but that is not working in my case as the project is in VPC and kubernetes clusters are giving timeout or handshake error, so I have ruled out use of this option
Valid JSON can only be " not '. Try switching the quotes.
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array.
software_config{
env_variables{
AIRFLOW_VAR_MYJSON ="{\"__comment1__\": \"This the global section\", \"project_id\":\"testproject\", \"gce_zone\":\"us-east1-c\", \"gce_region\":\"us-east1\",\"networkname\":\"vpc1\", \"subnetwork\":\"https://www.googleapis.com/compute/v1/projects/testproject/regions/us-east1/subnetworks/subnet1\"}"
}
}
Or a little nicer way:
software_config {
env_variables {
AIRFLOW_VAR_MYJSON = jsonencode({
"__comment1__" = "This the global section",
"project_id" = "testproject",
"gce_zone" = "us-east1-c",
"gce_region" = "us-east1",
"networkname" = "vpc1",
"subnetwork" = "https://www.googleapis.com/compute/v1/projects/testproject/regions/us-east1/subnetworks/subnet1",
})
}
}
I am trying to call the stats of a list of players from the call of duty API. This API requires firstly the login in website https://profile.callofduty.com/cod/login. Once logged in, the user can see the stats of a player using the call-of-duty API. For example, the stats of the streamer savyultras90 from Warzone can be seen through the following link: https://my.callofduty.com/api/papi-client/stats/cod/v1/title/mw/platform/psn/gamer/savyultras90/profile/type/wz.
If I log in from the website and try to see the stats of a player and the related json, I am able to do via browser. However, this doesn't seem straightforward in R.
I try to log in using the GET function from httr package as follows
respo <- GET('https://profile.callofduty.com/cod/login', authenticate('USER', 'PWD'))
But when I try to have access to the api and download the JSON file using the function fromJSON from the package jsonlite as follows
data <- fromJSON('https://my.callofduty.com/api/papi-client/stats/cod/v1/title/mw/platform/psn/gamer/savyultras90/profile/type/wz')
I get the error message "Not permitted: not authenticated".
How can I authenticate in one website and stay logged in to call from the API which relies on that authentication?
Seeing I've recently had to develop a PHP API for Warzone, I might be able to guide you in the right direction on how to handle this. But first a few remarks:
You need to authenticate each user individually with the appropriate platform if you want to request that player's data
There is a throttle limit on the amount of API requests
The API of Call of Duty is under strict usage guidelines and should only be used by registered partners. Making usage of the API could result in claims and eventually lawsuits: link
There is no public documentation of the API and the API has changed in the past, breaking several 3rd party tools.
Nevertheless, the process involves several steps as described below:
Register the device making the call
https://profile.callofduty.com/cod/mapp/registerDevice
with a json body in the form of {"deviceId":"INSERT_ID_HERE"}
This will return a response with the authHeader which we will use for as Token in the next calls
Login with Activision credentials
https://profile.callofduty.com/cod/mapp/login
Set the following headers:
Authorization: "INSERT_AUTHHEADER_HERE"
x_cod_device_id: "INSERT_PREVIOUSLY_USED_DEVICEID_HERE"
This in terms will generate a dataset where we will save the following data from:
rtkn, ACT_SSO_COOKIE and atkn.
Make the wanted API call for data
We have all the data now required to make the API call.
For each request we will submit 3 headers:
Authorization: "INSERT_AUTHHEADER_HERE"
x_cod_device_id: "INSERT_PREVIOUSLY_USED_DEVICEID_HERE"
Cookie: ACT_SSO_LOCALE=en_GB;country=GB;API_CSRF_TOKEN=**GENERATE_CSRF_TOKEN**;rtkn=**RTKN_HERE**;ACT_SSO_COOKIE=**ACT_SSO_COOKIE_HERE**;atkn=**ATKN_HERE**
For more reference, you can always look through a Python library or NodeJS library which succesfully implemented the API.
I struggled with this yesterday but finally made some progress. The issue is that you have to obtain an authentication token. The steps can be followed here: https://documenter.getpostman.com/view/7896975/SW7aXSo5#a37a2e5b-84bb-441d-b978-0fd8d42ffd29 but not available in R though.
My code works at first, as long as you don't authenticate again (still trying to figure out why). Basically what I did was to translate the steps in the link and extracted the content in the response from GET:
# Get token ---------------------------------------------------------------
resp <- GET('https://profile.callofduty.com/cod/login')
cookies = c(
'XSRF-TOKEN' = resp$cookies$value[1]
,'new_SiteId' = resp$cookies$value[2]
,'comid' = resp$cookies$value[3]
,'bm_sz' = resp$cookies$value[4]
,'_abck' = resp$cookies$value[5]
# ,'ACT_SSO_COOKIE' = resp$cookies$value[6]
# ,'ACT_SSO_COOKIE_EXPIRY' = resp$cookies$value[7]
# ,'atkn' = resp$cookies$value[8]
# ,'ACT_SSO_REMEMBER_ME' = resp$cookies$value[9]
# ,'ACT_SSO_EVENT' = resp$cookies$value[10]
# ,'pgacct' = resp$cookies$value[11]
# ,'CRM_BLOB' = resp$cookies$value[12]
# ,'tfa_enrollment_seen' = resp$cookies$value[13]
)
headers = c(
)
params = list(
`new_SiteId` = 'cod',
`username` = 'USER',
`password` = 'PWD',
`remember_me` = 'true',
`_csrf` = resp$cookies$value[1]
)
# Authenticate ------------------------------------------------------------
resp_post <- POST('https://profile.callofduty.com/do_login?new_SiteId=cod',
httr::add_headers(.headers=headers),
query = params,
httr::set_cookies(.cookies = cookies))
cookies = c(
'XSRF-TOKEN' = resp_post$cookies$value[1]
,'new_SiteId' = resp_post$cookies$value[2]
,'comid' = resp_post$cookies$value[3]
,'bm_sz' = resp_post$cookies$value[4]
,'_abck' = resp_post$cookies$value[5]
,'ACT_SSO_COOKIE' = resp_post$cookies$value[6]
,'ACT_SSO_COOKIE_EXPIRY' = resp_post$cookies$value[7]
,'atkn' = resp_post$cookies$value[8]
,'ACT_SSO_REMEMBER_ME' = resp_post$cookies$value[9]
,'ACT_SSO_EVENT' = resp_post$cookies$value[10]
,'pgacct' = resp_post$cookies$value[11]
,'CRM_BLOB' = resp_post$cookies$value[12]
,'tfa_enrollment_seen' = resp_post$cookies$value[13]
)
headers = c(
)
params = list(
`new_SiteId` = 'cod',
`username` = 'USER',
`password` = 'PWD',
`remember_me` = 'true',
`_csrf` = resp_post$cookies$value[1]
)
# Get data:
resp_psn <- httr::GET(url = 'https://my.callofduty.com/api/papi-client/stats/cod/v1/title/mw/platform/psn/gamer/savyultras90/profile/type/wz',
httr::add_headers(.headers=headers),
query = params,
httr::set_cookies(.cookies = cookies))
resp_psn_json <- content(resp_psn)
Let me know if you've already managed to resolve this!
I'm having trouble retrieving a filtered list from google app engine datastore (using python for server side). My data entity is defined as the following
class Course_Table(db.Model):
course_name = db.StringProperty(required=True, indexed=True)
....
head_tags_1=db.ListProperty(db.Key)
So the head_tags_1 property is a list of keys (which are the keys to a different entity called Headings_1).
I'm in the Handler below to spin through my Course_Table entity to filter the courses that have a particular Headings_1 key as a member of the head_tags_1 property. However, it doesn't seem like it is retrieving anything when I know there is data there to fulfill the request since it never displays the logs below when I go back to iterate through the results of my query (below). Any ideas of what I'm doing wrong?
def get(self,level_num,h_key):
path = []
if level_num == "1":
q = Course_Table.all().filter("head_tags_1 =", h_key)
for each in q:
logging.info('going through courses with this heading name')
logging.info("course name filtered is %s ", each.course_name)
MANY MANY THANK YOUS
I assume h_key is key of headings_1, since head_tags_1 is a list, I believe what you need is IN operator. https://developers.google.com/appengine/docs/python/datastore/queries
Note: your indentation inside the for loop does not seem correct.
My bad apparently '=' for list is already check membership. Using = to check membership is working for me, can you make sure h_key is really a datastore key class?
Here is my example, the first get produces result, where the 2nd one is not
import webapp2 from google.appengine.ext import db
class Greeting(db.Model):
author = db.StringProperty()
x = db.ListProperty(db.Key)
class C(db.Model): name = db.StringProperty()
class MainPage(webapp2.RequestHandler):
def get(self):
ckey = db.Key.from_path('C', 'abc')
dkey = db.Key.from_path('C', 'def')
ekey = db.Key.from_path('C', 'ghi')
Greeting(author='xxx', x=[ckey, dkey]).put()
x = Greeting.all().filter('x =',ckey).get()
self.response.write(x and x.author or 'None')
x = Greeting.all().filter('x =',ekey).get()
self.response.write(x and x.author or 'None')
app = webapp2.WSGIApplication([('/', MainPage)],
debug=True)