I am using DataNucleus with HBase. I had a table user. It contained 4 rows. Now I added a new column to the table. Now everytime I access any old user object which does not have this column DataNucleus throws an exception as it is trying to map the column with the property in the POJO. Is there no other way than updating the old 'user' objects with dummy data? My object mapping looks something like this:
#Persistent(columns={#Column(name="next_mail_timestamp", insertValue="#NULL", defaultValue = "#NULL", allowsNull = "true")}, name="nextMailTimestamp", cacheable="false", nullValue=NullValue.DEFAULT)
private long nextMailTimestamp;
As you can see I have tried using insertValue, defaultValue , allowsNull, nullValue. But nothing seems to work.
The stacktrace looks like this:
at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:479)
at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:453)
at org.datanucleus.store.hbase.fieldmanager.FetchFieldManager.fetchLongField(FetchFieldManager.java:269)
at org.datanucleus.state.AbstractStateManager.replacingLongField(AbstractStateManager.java:2133)
at com.kuliza.sitepulse.data.User.jdoReplaceField(User.java)
at com.kuliza.sitepulse.data.User.jdoReplaceFields(User.java)
at org.datanucleus.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:1989)
at org.datanucleus.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2009)
at org.datanucleus.store.hbase.query.HBaseQueryUtils$2.fetchFields(HBaseQueryUtils.java:226)
at org.datanucleus.state.JDOStateManagerImpl.loadFieldValues(JDOStateManagerImpl.java:803)
at org.datanucleus.state.JDOStateManagerImpl.initialiseForHollow(JDOStateManagerImpl.java:210)
at org.datanucleus.state.ObjectProviderFactory.newForHollowPopulated(ObjectProviderFactory.java:88)
at org.datanucleus.ObjectManagerImpl.findObject(ObjectManagerImpl.java:2794)
at org.datanucleus.store.hbase.query.HBaseQueryUtils.getObjectUsingApplicationIdForResult(HBaseQueryUtils.java:221)
at org.datanucleus.store.hbase.query.HBaseQueryUtils.getObjectsOfType(HBaseQueryUtils.java:168)
at org.datanucleus.store.hbase.query.HBaseQueryUtils.getObjectsOfCandidateType(HBaseQueryUtils.java:80)
at org.datanucleus.store.hbase.query.JDOQLQuery.performExecute(JDOQLQuery.java:271)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1766)
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1655)
at org.datanucleus.store.query.Query.execute(Query.java:1628)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221)
at com.kuliza.sitepulse.service.DataService.getUserWithCredentials(DataService.java:111)
at com.kuliza.sitepulse.service.AuthenticationService.getUserWithCredentials(AuthenticationService.java:46)
at com.kuliza.sitepulse.controller.AuthenticationController.signIn(AuthenticationController.java:69)
and my method is (in DataService.java:111)(which throws the exception)
#Override
public User getUserWithCredentials(String userName, String password){
PersistenceManager pm = pmf.getPersistenceManager();
Query q = pm.newQuery("SELECT FROM " + User.class.getName() + " WHERE userName == \""+userName+"\"" +" && password == " +
" \""+password + "\"");
List<User> c = (List<User>)q.execute();
pm.close();
if(c.size() > 0)
return c.get(0);
else
return null;
}
I have actually added two new columns (mailIntervalInMilliseconds, nextMailTimestamp) which are both long and in the stacktrace I see its trying to convert the db column to Long (AFAIK)
Released versions of DataNucleus HBase plugin don't currently support all modes of schema evolution. In particular the addition of fields/properties when they are of primitive types. However DataNucleus SVN does have support for this, so you could use that.
Related
During development structures and requirements change. Key and index settings need to be changed, that might break incremental table update. So my solution so far is to delete the table and recreate it from the cloudformation stack.
But how to solve this problem with a production deployment? Is it possible to automate dynamodb deployment as follows?
Create new table
Migrate data from old table to new table
Delete old table
Yes, it is perfectly possible to automate such a deployment structure. As long as you have code to create a table, it should be fairly straightforward to get all of the data from an old table, change the data, and then upload it all to a new table without any drops in up-time. If you write what language you would like to do such a thing in I can help a bit more.
I've done this before and I've added below a small generified code-sample on how you could do this in Java.
Java method for creating a table given the class of the object type stored in dynamo:
/**
* Creates a single table with its appropriate configuration (CreateTableRequest)
*/
public void createTable(Class tableClass) {
DynamoDBMapper mapper = createMapper(); // you'll need your own function to do this.
ProvisionedThroughput pt = new ProvisionedThroughput(1L, 1L);
CreateTableRequest ctr = mapper.generateCreateTableRequest(tableClass);
ctr.withProvisionedThroughput(new ProvisionedThroughput(1L, 1L));
// Provision throughput and configure projection for secondary indices.
if (ctr.getGlobalSecondaryIndexes() != null) {
for (GlobalSecondaryIndex idx : ctr.getGlobalSecondaryIndexes()) {
if (idx != null) {
idx.withProvisionedThroughput(pt).withProjection(new Projection().withProjectionType("ALL"));
}
}
}
TableUtils.createTableIfNotExists(client, ctr);
}
Java method to delete table:
private static void deleteTable(String tableName) {
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable(tableName);
try {
System.out.println("Issuing DeleteTable request for " + tableName);
table.delete();
System.out.println("Waiting for " + tableName + " to be deleted...this may take a while...");
table.waitForDelete();
}
catch (Exception e) {
System.err.println("DeleteTable request failed for " + tableName);
System.err.println(e.getMessage());
}
}
I would scan the whole table and plop all of the content into a List and then map through that list, converting the objects into your new type, and then create a new table of that type but with a different name, push all of your new objects, and then delete the old table after switching any references you might have of the old table to the new one. Unfortunately this does mean that everything consuming your tables are going to need to be able to switch between your two staging tables.
I'm newbie in SQLite.
I would like to query my SQLite database to get multiple rows.
When I add a new item in my local database I call this method Add:
public bool Add<T>(string key, T value)
{
return this.Insert(new SQliteCacheTable(key, this.GetBytes(value))) == 1;
}
_simpleCache.Add("favorite_1", data1);
_simpleCache.Add("favorite_2", data2);
_simpleCache.Add("favorite_3", data2);
Then,
I would like to retrieve from local database all entries where key starts with "favorite_"
to returns all objects in the database which are "favorite" objects.
I'm experienced in Linq, and I would like to do something like this:
IEnumerable<Element> = repository.Find((element) => element.Key.StartWith("favorite_"))
In the SQLiteConnection class there is a method like this:
SQLite.Net.SQLiteConnection.Find<T>(System.Linq.Expressions.Expression<System.Func<T,bool>>)
But I would like the same with in returns a collection IEnumerable<T>.
Can you help me please?
Thank you.
Jool
You have to build your query on the table itself, not the connection:
Assuming:
SQLiteConnection repository;
Then the code would look like:
var favorites = repository.Table<SQliteCacheTable>().Where(item => item.StartsWith("favorite_"));
The favorites variable is of type TableQuery<SQliteCacheTable> though, so it does not yet contain your data. The execution of the actual SQL query will be deferred until you try to access the results (by enumerating with foreach or converting to a list with ToList, for example).
To actually observe what's going on on the database, you can turn on tracing in sqlite-net, by setting repository.Trace = true on your SQLiteConnection object.
Finally, it's worth mentioning that you can also use the C# query syntax on TableQuery<T> objects, if your comfortable with it. So your query could become:
var favorites = from item in repository.Table<SQliteCacheTable>()
where item.StartsWith("favorite_")
select item;
I'm writing some code updating database with a SQL statement that has some placeholders . But it doesn't seem to update these placeholders.
I got the following error:
Cannot update '#columnName'; field not updateable
Here is the method:
public void updateDoctorTableField(string columnName, string newValue, string vendorNumber) {
sqlStatement = "update Doctor set #columnName = #newValue where `VENDOR #` = #vendorNumber;";
try {
_command = new OleDbCommand(sqlStatement, _connection);
_command.Parameters.Add("#columnName", OleDbType.WChar).Value = columnName;
_command.Parameters.Add("#newValue", OleDbType.WChar).Value = newValue;
_command.Parameters.Add("#vendorNumber", OleDbType.WChar).Value = vendorNumber;
_command.ExecuteNonQuery();
} catch (Exception ex) {
processExeption(ex);
} finally {
_connection.Close();
}
}
Not all parts of the query are parameterisable.
You can't parametrise the name of the column. This needs to be specified explicitly in your query text.
If this is sent via user input you need to take care against SQL Injection. In fact in any event it would be best to check it against a whitelist of known valid column names.
The reason the language does not allow for parameters for things like table names, column names and such is exactly the same reason why your C# program does not allow for substitution of variables in the code. Basically your question can be rephrased like this in a C# program:
class MyClass
{
int x;
float y;
string z;
void DoSomething(string variableName)
{
this.#variable = ...
}
}
MyCLass my = new MyClass();
my.DoSomething("x"); // expect this to manuipulate my.x
my.DoSomething("y"); // expect this to manuipulate my.y
my.DoSomething("z"); // expect this to manuipulate my.z
This obviously won't compile, because the compiler cannot generate the code. Same for T-SQL: the compiler cannot generate the code to locate the column "#columnName" in your case. And just as in C# you would use reflection to do this kind of tricks, in T-SQL you would use dynamic SQL to achieve the same.
You can (and should) use the QUOTENAME function when building your dynamic SQL to guard against SQL injection.
I am using ASP.NET MVC2 in Visual Studio 2008. I believe the SQL Server is 2005.
I have two tables: EquipmentInventory and EquipmentRequested
EquipmentInventory has a primary key
of sCode
EquipmentRequested has a
foreign key called sCode based upon
sCode in EquipmentInventory.
I am trying the following code (lots of non-relevent code removed):
try
{
EChODatabaseConnection myDB = new EChODatabaseConnection();
//this section of code works fine. The data shows up in the database as expected
foreach (var equip in oldData.RequestList)
{
if (equip.iCount > 0)
{
dbEquipmentInventory dumbEquip = new dbEquipmentInventory();
dumbEquip.sCode = equip.sCodePrefix + newRequest.iRequestID + oldData.sRequestor;
myDB.AddTodbEquipmentInventorySet(dumbEquip);
}
}
myDB.SaveChanges(); //save this out immediately so we can add in new requests
//this code runs fine
foreach (var equip in oldData.RequestList)
{
if (equip.iCount > 0)
{
dbEquipmentRequested reqEquip = new dbEquipmentRequested();
reqEquip.sCode = equip.sCodePrefix + newRequest.iRequestID + oldData.sRequestor;
myDB.AddTodbEquipmentRequestedSet(reqEquip);
}
}
//but when I try to save the above result, I get an error
myDB.SaveChanges();
oldData is passed into the function. newRequest is the result of adding to a "non-related" table. newRequest.iRequestID does have a value.
In looking at the reqEquip is the watch window, I do notice that EquipInventory is null.
The error message I receive is:
"Entities in 'EChODatabaseConnection.dbEquipmentRequestedSet' participate in the 'FK_EquipmentRequested_EquipmentInventory_sCode' relationship. 0 related 'EquipmentInventory' were found. 1 'EquipmentInventory' is expected."
Obviously I'm doing something wrong but thus far, I can not seem to find where I am having a problem.
Anyone have some hints on how to properly insert a record into a table that has a foreign key reference?
UPDATE:
I am using the Data Entity Framework.
UPDATE:
Thanks to Rob's answer, I was able to figure out my error.
As Rob mentioned, I needed to set my reference for the foreign key.
My coding result looks like:
foreach (var equip in oldData.RequestList)
{
if (equip.iCount > 0)
{
dbEquipmentInventory dumbEquip = new dbEquipmentInventory();
dumbEquip.sCode = equip.sCodePrefix + newRequest.iRequestID + oldData.sRequestor;
myDB.AddTodbEquipmentInventorySet(dumbEquip);
//add in our actual request items
dbEquipmentRequested reqEquip = new dbEquipmentRequested();
reqEquip.EquipmentInventory = dumbEquip;
myDB.AddTodbEquipmentRequestedSet(reqEquip);
}
}
myDB.SaveChanges();
Does anyone see a better method for doing this?
What are you using as an ORM? I believe that regardless of which one you're using, you could use the foreign key handling of most ORMs to handle this for you. For example, you make a new dumbEquip, don't do the immediate save. Do your dbEquipmentRequested reqEquip = new dbEquipmentRequested(); and add the data to it and then say dumbEquip.dbEquipmentRequested.Add(reqEquip). Then save the record and the ORM should save the records in the correct order required for the FK and even enter the FK ID into the reqEquip record.
OK, this thing just puzzles me.
I have a table, say Users, with columns UserID, Name, etc. Have an object mapped to it using CTP5. So now I want to test it, and do the following:
List<User> users = new List();
// Some init code here, making say 3 users.
using (UsersDbContext)
{
// insert users
}
So far so good, works fine.
Now I want to see if the records match, so I select the users back using the following code.
using (UsersDbContext dbc = UsersDbContext.GetDbContext())
{
List<Users> usersRead = dbc.Users.Where(x => x.ID >= users[0].ID && x.ID <= users[users.Count - 1].ID).ToList();
}
This throws and exception:
System.NotSupportedException: LINQ to
Entities does not recognize the method
'User get_Item(Int32)' method, and
this method cannot be translated into
a store expression.
EF has difficulties seeing that I'm just asking to return an int in Users[0].ID ?
If I replace a call to users[0].ID with a straight int - works fine.
I get what it's trying to do, but I thought it should be pretty easy to check if the method belongs to .NET or Sql Server ?
You are trying to access an indexer in an EF expression, which doesn't translate to an SQL query. You'll have to move the parameters outside the query like this:
int first = users[0].ID;
int last = users[users.Count - 1].ID;
List<Users> usersRead = dbc.Users.Where(x => x.ID >= first && x.ID <= last).ToList();