Other than than re-writing an entity to the datastore, is there a method to get existing entity properties indexed?
A scenario:
I created new queries using a property of previously created entities.
The queries failed with "Cannot query for un-indexed property ", which is correct behaviour as the initial Class definition had (indexed=false)
for the relevant property.
I then set (indexed=true) for the relevant property. The queries using this property than ran without generating errors, the output however was INCORRECT!
I assume this is because existing entities do not automatically get added to the index (although Google documents elude to indexes being automatically generated).
I could get the index updated and queries working correctly only by updating each entity. This is perhaps OK if one is aware of the problem is using small datasets. I am concerned about live data. I need to do more testing with live but the same effect/behaviour could be the same.
It appeared changing index.yaml and restarting the GAE instance had no effect. It also appears gcloud datastore create-indexes index.yaml will not affect this behaviour.
What appears to be a solution
It seems the only solution is to write back the existing entities so an entry is created in the new index. Probably to the effect of:
all_entities_in_kind=theModel.query()
list_of_keys= ndb.put_multi(all_entities_in_kind)
If there's a better way please post.
Simulation of the effect
To illustrate, I ran the code snippet below from the Dev SDK Interactive console. First run of the code below creates test data. The queries will fail (correct behaviour).
Changing bool_p to indexed=true, re-running the modified code means the query will run but the results are incorrect.
Deleting all the data and re-running the code (with bool_p indexed) the queries return the correct result.
Observation
It appears the index on the property is not automatically generated. At least from the Interactive console. Restarting the Instance has no effect. Changing the index.yaml file also seems to make no difference. It appears to be the index on the property needs to be built but so far I have not discovered how. The only solution would be export all the data and re-import with the modified class. This is not so much a problem in development but not for the live datastore.
Code example
from google.appengine.ext import ndb
from datetime import datetime, timedelta
class TModel(ndb.Model):
test_date = ndb.DateProperty(indexed=True)
text = ndb.StringProperty(indexed=True)
bool_p = ndb.BooleanProperty(indexed=False) #First run
#bool_p = ndb.BooleanProperty(indexed=True) #Second run.
query0=TModel.query()
if query0.count() == 0:
print 'Create test data'
TModel(id='a',test_date=datetime.strptime('2017-01-01','%Y-%m-%d').date(),text='One',bool_p=True).put()
TModel(id='b',test_date=datetime.strptime('2017-01-02','%Y-%m-%d').date(),text='Two',bool_p=False).put()
TModel(id='c',test_date=datetime.strptime('2017-01-03','%Y-%m-%d').date(),text='Three',bool_p=True).put()
TModel(id='d',test_date=datetime.strptime('2017-01-01','%Y-%m-%d').date(),text='One').put() #To check behaviour with bool_p undefined
query1 = TModel.query(TModel.bool_p == True)
print query1.count()
query2 = TModel.query(TModel.bool_p == False)
print query2.count()
query3 = TModel.query(TModel.test_date <= datetime.strptime('2017-01-02','%Y-%m-%d').date())
print query3.count()
query4 = TModel.query(TModel.test_date <= datetime.strptime('2017-01-02','%Y-%m-%d').date(), TModel.bool_p == True)
print query4.count()
#Equivalent Queries using GQL
queryG1=ndb.gql('SELECT * FROM TModel WHERE bool_p = True')
print queryG1.count()
queryG2=ndb.gql('SELECT * FROM TModel WHERE bool_p = True')
print queryG2.count()
queryG3 =ndb.gql("SELECT * FROM TModel WHERE test_date <= DATE('2017-01-02')")
print queryG3.count()
queryG4 =ndb.gql("SELECT * FROM TModel WHERE test_date <= DATE('2017-01-02') AND bool_p = True ")
print queryG4.count()
Correct Results/Incorrect Result after changing: (indexed=False) to True
2 0
1 0
3 3
1 0
2 0
2 0
3 3
1 0
It is necessary to rewrite the existing entities when changing a property from unindexed to indexed in order to re-index the properties. From https://cloud.google.com/datastore/docs/concepts/indexes#unindexed_properties:
Note, however, that changing a property from excluded to indexed does not affect any existing entities that may have been created before the change. Queries filtering on the property will not return such existing entities, because the entities weren't written to the query's index when they were created. To make the entities accessible by future queries, you must rewrite them to Cloud Datastore so that they will be entered in the appropriate indexes.
Related
This question already has answers here:
Room database migration if only new table is added
(7 answers)
Closed 2 years ago.
When using Room whenever new tables added to your database you have to create it in your migration. Unfortunately, Room doesn't have such method to create table by just giving class name. Something below is necessary to have
room.createTable(User::class)
The similar method exists for example in OrmLite
TableUtils.createTable(connectionSource, User::class.java)
The necessity comes from complexity of creating tables with just simple SQLite query. Currently, what you can do is within your migrate you write your create SQLite script
db.execSQL("CREATE TABLE IF NOT EXIST `User` (uid INTEGER NON NULL, PRYMARY KEY (`uid`))")
There is no problem in above method, but it gets complex and long SQLite script if your have for example 50 fields. Obviously, you don't write it by yourself, and there are two methods to get the Create Script automatically generated by Room for you so that you just copy past.
After building the app,AppDatabase_Impl will be generated and it will have all necessary table creations. You can get the query from there
You include exportSchema = true in your #Database annotation and it will create versionNumber.json schema of Room database within your schemas folder. You can get the create script from there.
But, both of above methods will require you to run the app without any proper migration (as you don't know correct query) and it will definitely crash. And after that, you have correct query which you can include in your migration method. I think this is not "professional" way of doing it. Plus even after you get the long SQLite query, it's not PR friendly and it's not just good practice to have long SQLite query which is can not be easily debugged.
So, I wanted to make object oriented way of creating tables while migrating. The only approach that I could think of was, obviously, using model data class and generate query according to each field of model. It will supposedly be like this
fun createTable(db: SupportSQLiteDatabase, clazz: KClass<*>) {
val fields = extractColumns(clazz)
val primaryKeys = fields
.filter { it.primaryKey }
.map { it.name }
val createQuery = "CREATE TABLE IF NOT EXISTS `${clazz.simpleName}` (" +
fields.joinToString(", ") { "`${it.name}` ${it.type} ${it.nonNull}" } +
", PRIMARY KEY (" + primaryKeys.joinToString(",") { "`$it`" } +
"))"
db.execSQL(createQuery)
}
fun extractColumns(clazz: KClass<*>): Array<Column>{
val columns = ArrayList<Column>()
for (field in clazz.declaredMemberProperties){
val name = field.findAnnotation<ColumnInfo>()?.name ?: field.name
val type = getSqlType(field.returnType)
val nonNull = if (field.returnType.isMarkedNullable) "" else "NON NULL"
val primaryKey = field.findAnnotation<PrimaryKey>() != null
columns.add(Column(name, type, nonNull, primaryKey))
}
return columns.toTypedArray()
}
But the problem is Room Annotations are all annotated with
#Retention(RetentionPolicy.CLASS) which is can only be accessed during compile time. They are not available during run time. So all of my findAnnotation methods will return null. I was thinking of doing creating during compile time, but couldn't think of how.
So, my question was is there any way to generate CREATE scripts during compile time and if so how to do it?
Apart from my mentioned way of solving, is there any other way of creating tables which doesn't involve first two methods of copy pasting?
And by the way, I am not considering fallbackToDestructiveMigration. I mean, who would want their users to lose all data?
As of current update of Room there is actually the way to create SQL queries using Annotation Processing. With Annotation processing you have to wrote small library that generate Room queries for you when you build it.
Creating Annotation Processing Library is not straightforward and here is related issue.
Room database migration if only new table is added
I am using Python client SDK for Datastore (google-cloud-datastore) version 1.4.0. I am trying to run a key-only query fetch:
query = client.query(kind = 'SomeEntity')
query.keys_only()
Query filter has EQUAL condition on field1 and GREATER_THAN_OR_EQUAL condition on field2. Ordering is done based on field2
For fetch, I am specifying a limit:
query_iter = query.fetch(start_cursor=cursor, limit=100)
page = next(query_iter.pages)
keyList = [entity.key for entity in page]
nextCursor = query_iter.next_page_token
Though there are around 50 entities satisfying this query, each fetch returns around 10-15 results and a cursor. I can use the cursor to get all the results; but this results in additional call overhead
Is this behavior expected?
keys_only query is limited to 1000 entries in a single call. This operation counts as a single entity read.
For another limitations of Datastore, please refer detailed table in the documentation.
However, in the code, you did specify cursor as a starting point for a subsequent retrieval operation. Query can be limited, without cursor:
query = client.query()
query.keys_only()
tasks = list(query.fetch(limit=100))
For detailed instruction how to use limits and cursors, please refer documentation of the Google Gloud Datastore
I have a few objects created on my database and I need to delete some of the repeating attributes related to them.
The query I'm trying to run is:
UPDATE gemp1_product objects REMOVE ingredients[1] WHERE (r_object_id = '08015abd8002cd68')
But all I get is the folloing error message:
Error querying databse.
[DM_QUERY_E_UPDATE_INDEX]error: "UPDATE: Unable to REMOVE tghe attribute ingredients at index 1."
[DM_OBJECT_W_DELETE_ATTR_POSITION_ERROR]warning: "attempt to delete
non-existent attribute 88"
Object 08015abd8002cd68 exists and I can see it on the database. Queries like SELECT and DELETE work fine but I do not want to delete the whole object.
There is no easy way to do this. The reason is that repeating attributes are ordered, to enable multiple repeating attributes to be synchronized for a given object.
Either
set the attribute value to be empty for the given position, and change your code to discard empty attributes, or
use multiple DQL statements to shuffle the order so that the last one becomes empty, or
change your data model, e.g. use a single attribute as a property bag with pre-defined delimiters.
Details (1)
UPDATE gemp1_product OBJECTS SET ingredients[1] = '' WHERE ...
Details (2)
For each index; first find the value of index+1:
SELECT ingredients
FROM gemp1_product
WHERE (i_position*-1)-1 = <index+1>
ENABLE (ROW_BASED)
Use the value in a new query:
UPDATE gemp1_product OBJECTS SET ingredients[1] = '<value_from_above>' WHERE ...
It should also be possible to do this by nesting DQL somehow, but it might not be worth the effort.
Something is either wrong with your query or with your repository. I think you are mistyping your attribute name or using wrong index in your UPDATE query.
If you google for DM_OBJECT_W_DELETE_ATTR_POSITION_ERROR you'll see on this link a bit more detailed explanation:
CAUSE: Program executed a DeleteAttr operation that specified an non-existent attribute position (either a negative number or a number larger than the number of attributes in the object).
From this you could guess that type isn't in consistent state, or that you are trying to remove too big index of your repeating attribute, etc. Did you checked your repository with Consistency checker Job and other similar Jobs?
As of for the removing of repeating property (sttribute) value with DQL query, this is unachievable with single query since you need to specify index position which you don't know at first. But writing a simple script or doing it manually if it's not big amount of values to delete is the way you want to go.
I am new to GAE, please excuse for being naive.
datastore viewer query with where clause returns "No results in Empty namespace.".
For instance:
select * from GaeUser
returns all the entires.
Something like,
select * from GaeUser where firstName = 'somename'
or
select * from GaeUser where dayOfBirth = 5
returns nothing but the message No results in empty namespace.
I am expecting some pointers on how to debug this.
Thanks for reading this!!
Simply you just wrote an incorrect/misspelled query.
Note that GAE datastore is schema-less. Writing a query for a nonexisting entity or for a nonexisting property or specifying a filter condition where you use an incorrect data type will not result in error but rather in an empty result.
Being schema-less also means that 2 entities of the same kind might have the same property with different types. For example you might have a Person entity with an age property of type int and another Person with an age property of type String. Obviously in this case if you write something like
select * from Person where age='5'
will not return the person who has age=5 property having int type.
So just simply double check the names and types of the entity and properties and try again.
Another important note:
Properties are indexed by default. This means when saving an entity, index records for an indexed property will automatically be created and saved, this allows you to find this entity by this indexed property. Properties can be made unindexed. When you save an entity with an unindexed property, index records will not be saved (or if there were any, they will be removed) for this unindexed property and you will not be able to query/find this entity by this unindexed property.
I've got a flex/air app I've been working on, it uses a local sqlite database that is created on the initial application start.
I've added some features to the application and in the process I had to add a new field to one of the database tables. My questions is how to I go about getting the application to create one new field that is located in a table that already exists?
this is a the line that creates the table
stmt.text = "CREATE TABLE IF NOT EXISTS tbl_status ("+"status_id INTEGER PRIMARY KEY AUTOINCREMENT,"+" status_status TEXT)";
And now I'd like to add a status_default field.
thanks!
Thanks - MPelletier
I've add the code you provided and it does add the field, but now the next time I restart my app I get an error - 'status_default' already exists'.
So how can I go about adding some sort of a IF NOT EXISTS statement to the line you provided?
ALTER TABLE tbl_status ADD COLUMN status_default TEXT;
http://www.sqlite.org/lang_altertable.html
That being said, adding columns in SQLite is limited. You cannot add a column anywhere but after the last column in your table.
As for checking if the column already exists, PRAGMA table_info(tbl_status); will return a table listing the various columns of your table.
ADD ON:
I've been using a strategy in database design that allows me to distinguish which modifications are required. For this, you will need a new table (call it DBInfo), with one field (Integer, call it SchemaVersion). Alternately, there is also an internal value in SQLite called user_version, which can be set with a PRAGMA command. Your code can, on program startup, check for schema version number and apply changes accordingly, one version at a time.
Suppose a function named UpdateDBSchema(). This function will check for your database schema version, handle DBInfo not being there, and determine that the database is in version 0. The rest of this function could be just a large switch with different versions, nested in a loop (or other structure available to your platform of choice).
So for this first version, have an UpgradeDBVersion0To1() function, which will create this new table (DBInfo), add your status_default field, and set SchemaVersion to 1. In your code, add a constant that indicates the latest schema version, say LATEST_DB_VERSION, and set it to 1. In that way, your code and your database have a schema version, and you know you need to synch them if they are not equal.
When you need to make another change to your schema, set the LATEST_DB_VERSION constant to 2 and make a new UpgradeDBVersion1To2() function that will perform the required changes.
That way, your program can be ported easily, can connect to and upgrade an old database, etc.
I know this is an old question... however.
I've hit this precise problem in the SQLite implementation in Adobe AIR. I thought it would be possible to use the PRAGMA command to resolve, but since adobe air's implementation does not support the PRAGMA command, we need an alternative.
What I did, that I thought would be worth while sharing here, is this:
var sql:SQLStatement = new SQLStatement();
sql.sqlConnection = pp_db.dbConn;
sql.text = "SELECT NewField FROM TheTable";
sql.addEventListener(SQLEvent.RESULT, function(evt:SQLEvent):void {
});
sql.addEventListener(SQLErrorEvent.ERROR, function(err:SQLErrorEvent):void {
var sql:SQLStatement = new SQLStatement();
sql.sqlConnection = pp_db.dbConn;
sql.text = "ALTER TABLE TheTable ADD COLUMN NewField NUMERIC;";
sql.execute();
sql.addEventListener(SQLEvent.RESULT, function (evt:SQLEvent):void {
});
});
sql.execute();
Hope it helps someone.
I solved a similar problem using the answer from this question:
ALTER TABLE ADD COLUMN IF NOT EXISTS in SQLite
Use built in user_version parameter to keep track of your updates. You set it using:
PRAGMA user_version = 1
and you retrieve it using
PRAGMA user_version
So basically retrieve user_version (it's 0 by default), check if it's 0. If yes, perform your updates and set it to 1. If you have more updates in the future, check if it's 1, perform updates and set it to 0. And so on...
In some cases I execute the command and get the exception for "duplicate column". Just a quick solution, not the perfect.