I am working with Google's Datalab service, on a Google managed Computer engine(default), and I would like to call my Google Datastore's API. The documentation points to using the from google.appengine.ext import db library.
But when I execute this in a datalab code block I get ImportError: No module named appengine.ext.
I realize that this likly means that the App Engine SDK is not installed on the Datalab compute engine, My quetion is how can I then access the My Datastore namespace from my Datalab notebook?
It seems that I was better off using the gcloud package. Seeing as I updated the gcloud package before they where able to update documentation this is an example of the code I used:
from gcloud import datastore
from gcloud.datastore.key import Key
from gcloud.datastore.entity import Entity
import datetime
client = datastore.Client('project_id','namespace')
key = client.key('kind_name')
entity = datastore.Entity(key=key)
entity['datetime'] = datetime.datetime.now()
entity['some_other_column'] = 1
query = datastore.Query(client,kind='kind_name')
for result in query.fetch():
print result
Related
we are trying to create an online feature store using cosmosdb following this documentation: https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/publish-features .
But I get an error when I publish the table to cosmosdb: AnalysisException: Catalog 'cosmoscatalog' not found. The issue only happens when using unity-enabled workspaces. I can publish using a non-unity enabled workspace.
P.S. If I create the table using the non-unity enabled workspace, then the unity-enabled workspace can update the cosmosdb. But the unity-enabled worskpace cannot create the cosmos container/database using the fs.publish_table.
I tried the following code:
from databricks.feature_store.online_store_spec import AzureCosmosDBSpec
from databricks.feature_store.client import FeatureStoreClient
fs = FeatureStoreClient()
account_uri = "https://online-feature-store.documents.azure.com:443/"
# Specify the online store.
online_store_spec = AzureCosmosDBSpec(
account_uri=account_uri,
write_secret_prefix="secret/write-cosmos",
read_secret_prefix="secret/read-cosmos",
database_name="online_feature_store_example",
container_name="feature_store_online_wine_features"
)
# Push the feature table to online store.
fs.publish_table("online_feature_store_example.wine_static_features", online_store_spec, mode='merge')
The following code works on workspaces without unity catalog enabled. However, on a unity-catalog enabled workspace, it trhows an error: AnalysisException: Catalog 'cosmoscatalog' not found
You need to create the database and container in CosmosDB with the name you are specifying in AzureCosmosDBSpec.
I am trying to import existing data from firestore to Algolia but can not make it work. I installed the *Firebase Algolia Extension` and tried to follow the documented steps:
running this:
npx firestore-algolia-search
Below are the questions that will be asked:
What is the Region? europe-west3
What is the Project Id? wishlist-88d58
What is the Algolia App Id? 15W2O5H8ZN
What is the Algolia Api Key? { unspecified parameter }
What is the Algolia Index Name? allUsers
What is the Collection Path? allUsers
What are the Fields to extract? { unspecified parameter }
What is the Transform Function? { unspecified parameter }
What is the path to the Google Application Credential File? wishlists_key.json
For the Algolia API Key I added a key:
I did not specify Fields to extracts and Transform Functions.
For path to the Google Application Credential File I created a private key in Firebase and located it on the my desktop as wishlists_key.json, which is where I ran the command above from.
I got a response which also contained the data but said at the beginning there was an error:
{"severity":"WARNING","message":"Warning, FIREBASE_CONFIG and GCLOUD_PROJECT environment variables are missing. Initializing firebase-admin will fail"}
{"location":"europa-west3","algoliaAppId":"15W205H8ZN","algoliaAPIKey":"********","algoliaIndexName":"allUsers","collectionPath":"allUsers","fields":"","transformFunction":"","projectId":"wishlist-88d58","severity":"INFO","message":"Initializing extension with configuration"}
{"severity":"INFO","message":"[ 'Sending rest of the Records to Algolia' ]"}
{"severity":"INFO","message":"[ 'Preparing to send 20 record(s) to Algolia.' ]"}
{"name":"RetryError","message":"Error when performing Algolia index","transporterStackTrace":[{"request":{"data":"{"requests":[{"action":"partialUpdateObject","body":{"signInMethod":"mail","username":"user662572"
...
The command does not finish running but get's stuck in this.
What am I doing wrong here? How do I correctly import data from FireStore to Algolia?
Also, later I will need to import a collection with about 24k documents. Is the documented way also capable of handling these amount of documents?
I was able to make it work through Google Cloud Shell. I had to import my Google Application Credential File to it, ran the command above again.
I still got the same Warning that it will fail, but it worked anyway and all data was correctly imported.
I am looking for a tutorial or document on how to access datastore using cloud functions (python).
However, it seems there is only tutorial for nodejs.
https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/functions/datastore
Can anybody help me out?
Thanks
There are no special setup needed to access datastore from cloud functions in python.
You just need to add google-cloud-datastore into requirements.txt and use datastore client as usual.
requirements.txt
# Function dependencies, for example:
# package>=version
google-cloud-datastore==1.8.0
main.py
from google.cloud import datastore
datastore_client = datastore.Client()
def foo(request):
"""Responds to any HTTP request.
Args:
request (flask.Request): HTTP request object.
Returns:
The response text or any set of values...
"""
query = datastore_client.query(kind=<KindName>)
data = query.fetch()
for e in data:
print(e)
Read more:
Python Client for Google Cloud Datastore
Setting Up Authentication for Server to Server Production Applications
I need to bulk import says 100 records into Cosmos DB.
I found dt.exe, that doesn't help. it throws error when importing csv into cosmos db with table api.
I'm not able to find any reliable way to automate this process.
The command-line Azure Cosmos DB Data Migration tool (dt.exe) can be
used to import your existing Azure Table storage data to a Table API
GA account, or migrate data from a Table API (preview) account into a
Table API GA account. Other sources are not currently supported. The
UI based Data Migration tool (dtui.exe) is not currently supported for
Table API accounts.
According to the above official statement, it seems that other sources(e.g csv file) are not supported to be migrated into Azure Table API account. You could adopt a workaround: Read csv file in the program then import data into Azure Table Storage.
Please refer to the sample python code which I did in this thread.
from azure.cosmosdb.table.tableservice import TableService
from azure.cosmosdb.table.models import Entity
import csv
import sys
import codecs
table_service = TableService(connection_string='***')
reload(sys)
sys.setdefaultencoding('utf-8')
filename = "E:/jay.csv"
with codecs.open(filename, 'rb', encoding="utf-8") as f_input:
csv_reader = csv.reader(f_input)
for row in csv_reader:
task = Entity()
task.PartitionKey = row[0]
task.RowKey = row[1]
task.description = row[2]
task.priority = EntityProperty(EdmType.INT32, row[3])
task.logtime = EntityProperty(EdmType.DATETIME, row[4])
table_service.insert_entity('tasktable', task)
Or you could commit feedback here.
Hope it helps you.
Just for minor update：
If you use python 3.1, there is no need for reload(sys) and sys.setdefaultencoding('utf-8') with 'r' filename = r"E:/jay.csv"
I am working in an Ionic 3 project with ts to integrate Firebase into my app.
The below code I used to integrate firebase with Ionic project
constructor(angFire: AngularFireDatabase){
}
books: FirebaseListObservable<any>;
To send the data from my app to firebase, I used push method and to update entries I used update($key). Now I have all the data's in Firebase backend.
Now, how can I sync the firebase Database with Google Sheets so that each and every entry added to firebase backend has to get updated into sheets. I used a third party ZAPIER for this integration, but it would be nice if I get to learn on how to do this sync on my own.
Upon surfing, there are many tutorials to get the data's from the google sheets into Firebase. But I didn't come across any tutorials for vice versa.
I followed the below tutorial but it doesn't point to spreadsheets.
https://sites.google.com/site/scriptsexamples/new-connectors-to-google-services/firebase
Any help would be greatly appreciated!
I looked into importing Firebase right into Google Scripts either through the JavaScript SDK or or the REST API. Both have requirements/steps that Google Scripts cannot satisfy or that are extremely difficult to satisfy.
There is no foreseeable method of downloading the JavaScript SDK inside a Google Script because almost every method requires a DOM, which you don't have with a Google Sheet.
The REST API requires GoogleCredentials which, at a short glance, appear very difficult to get inside Google Scripts as well
So, the other option is to interact with Firebase in a true server side environment. This would be a lot of code, but here are the steps that I would take:
1) Setup a Pyrebase project so you can interact with your Firebase project via Python.
import pyrebase
config = {
"apiKey": "apiKey",
"authDomain": "projectId.firebaseapp.com",
"databaseURL": "https://databaseName.firebaseio.com",
"storageBucket": "projectId.appspot.com",
"serviceAccount": "path/to/serviceAccountCredentials.json"
}
firebase = pyrebase.initialize_app(config)
...
db = firebase.database()
all_users = db.child("users").get()
2) Setup a Google Scripts/Sheets project as a class that can interact with your Google Sheet
from __future__ import print_function
import httplib2
import os
from apiclient import discovery
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage
try:
import argparse
flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
flags = None
# If modifying these scopes, delete your previously saved credentials
# at ~/.credentials/sheets.googleapis.com-python-quickstart.json
SCOPES = 'https://www.googleapis.com/auth/spreadsheets.readonly'
CLIENT_SECRET_FILE = 'client_secret.json'
APPLICATION_NAME = 'Google Sheets API Python Quickstart'
class GoogleSheets:
...
# The rest of the functions from that link can go here
...
def write(self, sheet, sheet_name, row, col):
"""
Write data to specified google sheet
"""
if sheet == None or sheet == "":
print("Sheet not specified.")
return
day = time.strftime("%m/%d/%Y")
clock = time.strftime("%H:%M:%S")
datetime = day + " - " + clock
values = [[datetime]]
spreadsheetId = sheet
rangeName = sheet_name + "!" + str(row) + ":" + str(col)
body = {
'values': values
}
credentials = self.get_credentials()
http = credentials.authorize(httplib2.Http())
discoveryUrl = ('https://sheets.googleapis.com/$discovery/rest?'
'version=v4')
service = discovery.build('sheets', 'v4', http=http,
discoveryServiceUrl=discoveryUrl)
result = service.spreadsheets().values().update(
spreadsheetId=spreadsheetId, range=rangeName,
valueInputOption="RAW", body=body).execute()
3) Call the Google Sheets somewhere inside your Pyrebase project
from GoogleSheets import GoogleSheets
...
g = GoogleSheets()
g.write(<project-id>, <sheet-name>, <row>, <col>)
...
4) Set up a cron job to run the python script every so often
# every 2 minutes
*/2 * * * * /root/my_projects/file_example.py
You will need some basic server (Heroku, Digital Ocean) to run this.
This is not extensive because there is a lot of code to be written, but you could get the basics done. Makes we want to make a package now.
You can go for Zapier which is a 3rd party service through which you can easily integrate your Firebase and Google spreadsheets and vice versa. It has also got some support for google docs and other features.
https://zapier.com/zapbook/firebase/google-sheets/
Firebase can't be used as a trigger in Zapier, only as an action, so you can't send data from it to Google Sheets.