I created a mixed index on Titan 1.0 with Dynamo DB backend and Elasticsearch and I'm trying to remove it using the following code
public static void removeIndex(TitanGraph graph, String indexStr) throws ExecutionException, InterruptedException {
TitanManagement m = graph.openManagement();
TitanGraphIndex nameIndex = m.getGraphIndex(indexStr);
Preconditions.checkState(nameIndex!=null, "index "+ indexStr +" doesn't exist");
TitanManagement.IndexJobFuture futureDisable = m.updateIndex(nameIndex, SchemaAction.DISABLE_INDEX);
m.commit();
graph.tx().commit();
futureDisable.get();
// Block until the SchemaStatus transitions from to DISABLED
ManagementSystem.awaitGraphIndexStatus(graph, indexStr)
.status(SchemaStatus.DISABLED).call();
// Delete the index using TitanManagement
m = graph.openManagement();
nameIndex = m.getGraphIndex(indexStr);
TitanManagement.IndexJobFuture futureRemove =
m.updateIndex(nameIndex, SchemaAction.REMOVE_INDEX);
m.commit();
graph.tx().commit();
Preconditions.checkState(futureRemove!=null,
"Couldn't remove index/es because seems like indexes were not disabled."); // fails here.
futureRemove.get();
m = graph.openManagement();
nameIndex = m.getGraphIndex(indexStr);
Preconditions.checkArgument(nameIndex==null);
}
The key doesn't get removed. I get this warning indicating that the index never gets disabled.
---
INFO com.thinkaurelius.titan.graphdb.database.management.GraphIndexStatusWatcher - Some key(s) on index verticesIndex do not currently have status DISABLED: position=INSTALLED
INFO com.thinkaurelius.titan.graphdb.database.management.GraphIndexStatusWatcher - Timed out (PT1M) while waiting for index verticesIndex to converge on status DISABLED
[WARNING]
The code fails on the precondition test.
What am I doing wrong?
It turns out I was using position to define a property key which is a reserved keyword in DynamoDB.
The only way was to:
Delete all DynamoDB tables of Titan from the web console,
change 'position' to 'fieldPosition',
and start my code to create tables and indexes from scratch.
Related
I'm getting the error code SQLITE_BUSY when trying to write to a table after selecting from it. The select statement and result is properly closed prior to my insert.
If I'm removing the select part the insert works fine. And this is what I'm not getting. According to the documentation SQLITE_BUSY should mean that a different process or connection (which is definetly not the case here) is blocking the database.
There's no SQLite manager running. Also jdbcConn is the only connection to the database I have. No parallel running threads aswell.
Here's my code:
try {
if(!jdbcConn.isClosed()) {
ArrayList<String> variablesToAdd = new ArrayList<String>();
String sql = "SELECT * FROM VARIABLES WHERE Name = ?";
try (PreparedStatement stmt = jdbcConn.prepareStatement(sql)) {
for(InVariable variable : this.variables.values()) {
stmt.setString(1, variable.getName());
try(ResultSet rs = stmt.executeQuery()) {
if(!rs.next()) {
variablesToAdd.add(variable.getName());
}
}
}
}
if(variablesToAdd.size() > 0) {
String sqlInsert = "INSERT INTO VARIABLES(Name, Var_Value) VALUES(?, '')";
try(PreparedStatement stmtInsert = jdbcConn.prepareStatement(sqlInsert)) {
for(String name : variablesToAdd) {
stmtInsert.setString(1, name);
int affectedRows = stmtInsert.executeUpdate();
if(affectedRows == 0) {
LogManager.getLogger().error("Error while trying to add missing database variable '" + name + "'.");
}
}
}
jdbcConn.commit();
}
}
}
catch(Exception e) {
LogManager.getLogger().error("Error creating potentially missing database variables.", e);
}
This crashes on int affectedRows = stmtInsert.executeUpdate();. Now if I remove the first block (and manually add a value to the variablesToAdd list) the value inserts fine into the database.
Am I missing something? Am I not closing the ResultSet and PreparedStatement properly? Maybe I'm blind to my mistake from looking at it for too long.
Edit: Also executing the select in a separate thread does the trick. But that can't be the solution. Am I trying to insert into the database too fast after closing previous statements?
Edit2: I came across a busy_timeout, which promised to make updates/queries wait for a specified amount of time before returning with SQLITE_BUSY. I tried setting the busy timeout like so:
if(jdbcConn.prepareStatement("PRAGMA busy_timeout = 30000").execute()) {
jdbcConn.commit();
}
The executeUpdate() function still immedeiately returns with SQLITE_BUSY.
I'm dumb.
I was so thrown off by the fact that removing the select statement worked (still not sure why that worked, probably bad timing) that I missed a different thread using the same file.
Made both threads use the same java.sql.Connection and everything works fine now.
Thank you for pushing me in the right direction #GordThompson. Wasn't aware of the jdbc:sqlite::memory: option which led to me finding the issue.
Here is the configuration options i am using.
storage.backend=cassandra
storage.hostname=192.168.56.121
storage.cassandra.keyspace=graphs
cache.db-cache = false
cache.db-cache-clean-wait = 20
index.search.backend=elasticsearch
index.search.hostname=192.168.56.122
index.search.elasticsearch.client-only=true
index.search.index-name=graphs
TitanGraph graph = GraphFactory.getInstance().getGraph();
TitanManagement mgmt = null;
try {
mgmt = graph.openManagement();
PropertyKey name = mgmt.getPropertyKey(Schema.NAME);
if (name == null) {
name = mgmt.makePropertyKey(Schema.NAME).dataType(String.class).make();
}
TitanGraphIndex graphIndex = mgmt.getGraphIndex("byName");
if (graphIndex == null) {
IndexBuilder builder = mgmt.buildIndex("byName", Vertex.class).addKey(name);
builder.buildCompositeIndex();
}
PropertyKey id = mgmt.getPropertyKey(Schema.ID);
if (id == null) {
id = mgmt.makePropertyKey(Schema.ID).dataType(Long.class).make();
}
PropertyKey sourceType = mgmt.getPropertyKey(Schema.SOURCE_TYPE);
if (sourceType == null) {
sourceType = mgmt.makePropertyKey(Schema.SOURCE_TYPE).dataType(String.class).make();
}
TitanGraphIndex uniqueIndex = mgmt.getGraphIndex("uniqueIndex");
if (uniqueIndex == null) {
IndexBuilder builder = mgmt.buildIndex("uniqueIndex", Vertex.class).addKey(id).addKey(sourceType);
builder.unique().buildCompositeIndex();
}
// Edges
EdgeLabel deps = mgmt.getEdgeLabel("deps");
if (deps == null) {
deps = mgmt.makeEdgeLabel("deps").multiplicity(Multiplicity.SIMPLE).make();
}
RelationTypeIndex depsIndex = mgmt.getRelationIndex(deps, "depsIndex");
if(depsIndex == null) {
depsIndex = mgmt.buildEdgeIndex(deps, "depsIndex", Direction.BOTH, Order.decr);
}
mgmt.commit();
// Re index the existing data
if (reIndexData) {
mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("uniqueIndex"), SchemaAction.REINDEX).get();
mgmt.updateIndex(mgmt.getGraphIndex("byName"), SchemaAction.REINDEX).get();
deps = mgmt.getEdgeLabel("deps");
mgmt.updateIndex(mgmt.getRelationIndex(deps,"depsIndex"), SchemaAction.REINDEX).get();
mgmt.commit();
}
} catch (Throwable e) {
log.error(e.getMessage(), e);
if (mgmt != null) {
mgmt.rollback();
}
}
I have created lots of documents and every thing is working fine. But when i observed the number document available in the elastic search is 0.
I am wondered whether titan db really using the elastic search or not.
Any idea what i am missing here ? And why documents are not getting created in elastic search.
And i also tried the belown configuration as well but no luck
storage.backend=cassandra
storage.hostname=192.168.56.121
storage.cassandra.keyspace=graphs
cache.db-cache = false
cache.db-cache-clean-wait = 20
index.graphs.backend=elasticsearch
index.graphs.hostname=192.168.56.122
index.graphs.elasticsearch.client-only=true
index.graphs.index-name=graphs
Titan uses storage backend (cassandra/hbase) for Composite Index and index backend (Solr/Elastic Search) for Mixed Index
Mixed indexes retrieve vertices or edges by any combination of previously added property keys. Mixed indexes provide more flexibility than composite indexes and support additional condition predicates beyond equality. On the other hand, mixed indexes are slower for most equality queries than composite indexes.
Unlike composite indexes, mixed indexes require the configuration of an indexing backend and use that indexing backend to execute lookup operations. Titan can support multiple indexing backends in a single installation. Each indexing backend must be uniquely identified by name in the Titan configuration which is called the indexing backend name.
In you schema you are creating only composite index. That's why there is not data in ElasticSearch.
Here is a example how to create a mixed index :
IndexBuilder builder = mgmt.buildIndex('byName', Vertex.class).addKey(name);
builder.buildMixedIndex("search");
mgmt.commit();
Read More
Source : http://s3.thinkaurelius.com/docs/titan/1.0.0/indexes.html
I researched this and all I can find is a suggestion to turn on .Trace = true like this:
db1 = DependencyService.Get<ISQLite>().GetConnection();
db1.Trace = true;
I also tried this:
db2.Trace = true;
var categories = db2.Query<Category>("SELECT * FROM Category ORDER BY Name").ToList();
Debug.WriteLine("xxxx");
Well I did this and then restarted the application. When I view the Application output I just see information on threads started and the xxxx but don't see any SQL trace information.
Can anyone give me advice on this. Thanks
You need to set Trace and Tracer (action) properties on your SQLiteConnection to print queries to output:
db.Tracer = new Action<string>(q => Debug.WriteLine(q));
db.Trace = true;
Look in the Application Output window for lines that begin Executing
Example Output after setting Trace to true:
Executing: create table if not exists "Valuation"(
"Id" integer primary key autoincrement not null ,
"StockId" integer ,
"Time" datetime ,
"Price" float )
Executing Query: pragma table_info("Valuation")
Executing: create index if not exists "Valuation_StockId" on "Valuation"("StockId")
Executing: insert into "Stock"("Symbol") values (?)
Executing Query: select * from "Stock" where ("Symbol" like (? || '%'))
0: A
Ref: https://github.com/praeclarum/sqlite-net/blob/38a5ae07c886d6f62cecd8fdeb8910d9b5a77546/src/SQLite.cs
The SQLite PCL uses Debug.WriteLine which means that the logs are only included in Debug builds of the PCL.
Remove your nuget reference to the sqlite.net PCL (leave the native reference), and instead add SQLite.cs as a class to your project, and execute a debug build, with the Trace flag set, and you'll see the tracing.
I didn't have to do anything special other than include the SQLite.cs file in my Xamarin iOS project for this to work:
using (var conn = new SQLite.SQLiteConnection("mydb.sqlite") { Trace = true }) {
var rows = conn.Table<PodcastMetadata>().Where(row => row.DurationMinutes < 10).Select(row => new { row.Title });
foreach (var row in rows) {
Debug.WriteLine(row);
}
}
Output:
Executing Query: select * from "PodcastMetadata" where ("DurationMinutes" < ?)
0: 10
Data to be inserted has just two TEXT columns whose individual length don't even exceed 256.
I initially used executeSimpleSQL since I didn't need to get any results.
It worked for simulataneous inserts of upto 20K smoothly i.e. in the bakground no lag or freezing observed.
However, with 0.1 million I could see horrible freezing during insertion.
So, I tried these two,
Insert in chunks of 500 records - This didn't work well since even for 20K records it showed visible freezing. I didn't even try with 0.1million.
So, I decided to go async and used executeAsync alongwith Bind etc. This also shows visible freezing for just 20K records. This was the whole array being inserted and not in chunks.
var dirs = Cc["#mozilla.org/file/directory_service;1"].
getService(Ci.nsIProperties);
var dbFile = dirs.get("ProfD", Ci.nsIFile);
var dbService = Cc["#mozilla.org/storage/service;1"].
getService(Ci.mozIStorageService);
dbFile.append('mydatabase.sqlite');
var connectDB = dbService.openDatabase(dbFile);
let insertStatement = connectDB.createStatement('INSERT INTO my_table
(my_col_a,my_col_b) VALUES
(:myColumnA,:myColumnB)');
var arraybind = insertStatement.newBindingParamsArray();
for (let i = 0; i < my_data_array.length; i++) {
let params = arraybind.newBindingParams();
// Individual elements of array have csv
my_data_arrayTC = my_data_array[i].split(',');
params.bindByName("myColumnA", my_data_arrayTC[0]);
params.bindByName("myColumnA", my_data_arrayTC[1]);
arraybind.addParams(params);
}
insertStatement.bindParameters(arraybind);
insertStatement.executeAsync({
handleResult: function(aResult) {
console.log('Results are out');
},
handleError: function(aError) {
console.log("Error: " + aError.message);
},
handleCompletion: function(aReason) {
if (aReason != Components.interfaces.mozIStorageStatementCallback.REASON_FINISHED)
console.log("Query canceled or aborted!");
console.log('We are done inserting');
}
});
connectDB.asyncClose(function() {
console.log('[INFO][Write Database] Async - plus domain data');
});
Also, I seem to get the async callbacks after a long time. Usually, executeSimpleSQL is way faster than this.If I use SQLite Manager Tool extension to open the DB immediately this is what I get ( as expected )
SQLiteManager: Error in opening file mydatabase.sqlite - either the file is encrypted or corrupt
Exception Name: NS_ERROR_STORAGE_BUSY
Exception Message: Component returned failure code: 0x80630001 (NS_ERROR_STORAGE_BUSY) [mozIStorageService.openUnsharedDatabase]
My primary objective was to dump data as big as 0.1 million + and then later on perform reads when needed.
Trying to get working ScalikeJDBC and SQLite. Have a simple code based on provided examples:
import scalikejdbc._, SQLInterpolation._
object Test extends App {
Class.forName("org.sqlite.JDBC")
ConnectionPool.singleton("jdbc:sqlite:test.db", null, null)
implicit val session = AutoSession
println(sql"""SELECT * FROM kv WHERE key == 'seq' LIMIT 1""".map(identity).single().apply()))
}
It fails with exception:
Exception in thread "main" java.sql.SQLException: Cannot change read-only flag after establishing a connection. Use SQLiteConfig#setReadOnly and QLiteConfig.createConnection().
at org.sqlite.SQLiteConnection.setReadOnly(SQLiteConnection.java:447)
at org.apache.commons.dbcp.DelegatingConnection.setReadOnly(DelegatingConnection.java:377)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setReadOnly(PoolingDataSource.java:338)
at scalikejdbc.DBConnection$class.readOnlySession(DB.scala:138)
at scalikejdbc.DB.readOnlySession(DB.scala:498)
...
I've tried both scalikejdbc 1.7 and 2.0, error remains. As sqlite driver I use "org.xerial" % "sqlite-jdbc" % "3.7.+".
What can I do to fix the error?
The following will create two separate connections, one for read-only operations and the other for writes.
ConnectionPool.add("mydb", s"jdbc:sqlite:${db.getAbsolutePath}", "", "")
ConnectionPool.add(
"mydb_ro", {
val conf = new SQLiteConfig()
conf.setReadOnly(true)
val source = new SQLiteDataSource(conf)
source.setUrl(s"jdbc:sqlite:${db.getAbsolutePath}")
new DataSourceConnectionPool(source)
}
)
I found that the reason is that you're using "org.xerial" % "sqlite-jdbc" % "3.7.15-M1". This version looks still unstable.
Use "3.7.2" as same as #kawty.
Building on #Synesso's answer, I expanded slightly to be able to get config value from config files and to set connection settings:
import scalikejdbc._
import scalikejdbc.config.TypesafeConfigReader
case class SqlLiteDataSourceConnectionPool(source: DataSource,
override val settings: ConnectionPoolSettings)
extends DataSourceConnectionPool(source)
// read settings for 'default' database
val cpSettings = TypesafeConfigReader.readConnectionPoolSettings()
val JDBCSettings(url, user, password, driver) = TypesafeConfigReader.readJDBCSettings()
// use those to create two connection pools
ConnectionPool.add("db", url, user, password, cpSettings)
ConnectionPool.add(
"db_ro", {
val conf = new SQLiteConfig()
conf.setReadOnly(true)
val source = new SQLiteDataSource(conf)
source.setUrl(url)
SqlLiteDataSourceConnectionPool(source, cpSettings)
}
)
// example using 'NamedDB'
val name: Option[String] = NamedDB("db_ro") readOnly { implicit session =>
sql"select name from users where id = $id".map(rs => rs.string("name")).single.apply()
}
This worked for me with org.xerial/sqlite-jdbc 3.28.0:
String path = ...
SQLiteConfig config = new SQLiteConfig();
config.setReadOnly(true);
return DriverManager.getConnection("jdbc:sqlite:" + path, config.toProperties());
Interestingly, I wrote a different solution on the issue on the xerial repo:
PoolProperties props = new PoolProperties();
props.setDriverClassName("org.sqlite.JDBC");
props.setUrl("jdbc:sqlite:...");
Properties extraProps = new Properties();
extraProps.setProperty("open_mode", SQLiteOpenMode.READONLY.flag + "");
props.setDbProperties(extraProps);
// This line can be left in or removed; it no longer causes a problem
// as long as the open_mode code is present.
props.setDefaultReadOnly(true);
return new DataSource(props);
I don't recall why I needed the second, and was then able to simplify it back to the first one. But if the first doesn't work, you might try the second. It uses a SQLite-specific open_mode flag that then makes it safe (but unnecessary) to use the setDefaultReadOnly call.