I have created a slice of structs that has 3 properties
type Person struct {
age int
gender string
name string
}
How can I pull the item from the slice which matches my criteria?
For example I would like to do
var persons []Person = mySliceOfPersons
person := getFrom(persons).Where(age ==10).Where(gender == "male")
The purpose here is to keep the data in memory, and not be restricted by IO. (I'm expecting thousands of calls per second). I am new to Go and I am not sure where to find a package that does this. The data comes from Json and not a Database so I don't think I can use the sql package.
This solution IS a database, but you can embed it into your application for use rather than relying on an outside db: https://github.com/HouzuoGuo/tiedot
Another possibility is an approach like this one, which uses the sql package against local flat files, which could potentially be adapted to run against map?: https://github.com/dinedal/textql
Related
I'm new to Android, and rather new to SQL in general.
I have a data model where I have a Text that consists of TextMetadata as well as a long string, which is the text content itself. So
Text {
metadata: {
author: string,
title: string
// more associated metadata
},
textContent: long string, or potentially array of lines or paragraphs
}
I'd like to load a list of the metadata for all texts on the App's landing page, without incurring the cost of reading all the long strings (or having operations be slowed down because the table has a column with a long string?).
What is the proper pattern here? Should I use two tables, and related them? Or can I use one table/one #Entity, with embedded metadata, and do some fancy stuff in the DAO to just list/sort/operate on the embedded metadata?
Most of my background is with NoSQL databases, so I could be thinking about this entirely wrong. Advice on the general best practices here would be helpful, but I guess I have two core questions:
Does having a long/very long string/TEXT column cause performance considerations when operating on that specific table/row?
Is there a clean way using Kotlin annotations to express embedded metadata that would make it easy to fetch in the DAO, without having use a long SELECT for each individual column?
This is a good question that is also relevant to other environments.
The Core Issue: How to store large data without effecting your database?
As a rule of thumb you should avoid storing information in your database that is not queryable. Large strings, images, or event metadata which you will never query - does not belong in your db. I was surprised when I realized how many design patterns there are regarding to mongo db (which are relevant to other noSQL databases as well)
So, we know that this data should NOT be stored in the DB. But, because the alternative (file system) is WAY worse than that (unless you would like to implement your own secured file-system-based store) - we should at least try to minimize its footprint.
Our Strategy: save large data chunks in a different table without defining it as an entity (there is no need to wrap it as entity anyway)
How Are We Going To Do That?
Well, thankfully, android room has a direct access to sqLite and it can be used directly (read the docs). This is the place to remind us that android room is built on-top of sqLite - which is (in my own opinion) a fascinating database. I enjoy working with it very much and it's just getting better as the time goes by (personal opinion). Advantages? we are still using android APIs while storing large data in a performant, unified and secure way. yay
Steps we are going to perform:
Initiate a class which will manage a new database - for storing large data only
Define a command that will create our table - constructed of 2 columns
key (primary key) - the id of the item
value - the item itself
In original db for the Text entity - define a column that will hold the id (key) of the large text stored
Whenever you save an item to your large items table - get the id and store it in your entity
You can of course use only 1 table for this.. but.. I know that sqLite requires a certain amount of understanding and it is NOT as easy as android room so.. it's your choice whenever to use 1 or 2 tables in your solution
Below is a code that demonstrates the main principal of my proposal
object LargeDataContract {
// all tables for handling large data will be defined here
object TextEntry : BaseColumns {
const val TABLE_NAME = "text_entry"
const val COLUMN_NAME_KEY = "key"
const val COLUMN_NAME_VALUE = "value"
}
}
// in the future - append this "create statement" whenever you add more tables to your database
private const val SQL_CREATE_ENTRIES =
"CREATE TABLE ${TextEntry.TABLE_NAME} (" +
"${TextEntry.COLUMN_NAME_KEY} INTEGER PRIMARY KEY," +
"${TextEntry.COLUMN_NAME_VALUE} TEXT)"
// create a helper that will assist you to initiate your database properly
class LargeDataDbHelper(context: Context) : SQLiteOpenHelper(context, DATABASE_NAME, null, DATABASE_VERSION) {
override fun onCreate(db: SQLiteDatabase) {
db.execSQL(SQL_CREATE_ENTRIES)
}
companion object {
// If you change the database schema, you must increment the database version. Also - please read `sqLite` documentation to better understand versioning ,upgrade and downgrade operations
const val DATABASE_VERSION = 1
const val DATABASE_NAME = "LargeData.db"
}
}
// create an instance and connect to your database
val dbHelper = LargeDataDbHelper(context)
// write an item to your database
val db = dbHelper.writableDatabase
val values = ContentValues().apply {
put(TextEntry.COLUMN_NAME_VALUE, "some long value goes here")
}
val key = db?.insert(TextEntry.TABLE_NAME, null, values)
// now take the key variable and store it in you entity. this is the only reference you should need
Bottom Line: This approach will assist you to gain as much performance as possible while using android APIs. Sure thing, not the most "intuitive" solution, but - this is how we gain performance and making great apps as well as educating ourselves and upgrading our knowledge and skillset. Cheers
I know how to use IFNULL to get default values for non-existent rows or null values, but for creating queries that are compatible with older schema versions, it would be nice to be able to do this:
Schema v1: CREATE TABLE Employee (Name TEXT, Phone TEXT)
Schema v2: CREATE TABLE Employee (Name TEXT, Phone TEXT, Address TEXT)
Theoretical backward compatible query:
SELECT Name, Phone, IFNULL(Address, '') FROM Employee
Obviously this doesn't work for a file created with schema v1. Is there some way to do this though?
There are 2 alternative workflows, but both are rather annoying. Either 1) update the old db by adding missing columns (which would start with null values); or 2) build the query code dynamically based on schema version.
Create a temporary view that references a particular schema, substituting default values (or even transforming other data) for individual columns which differ between the base schemas.
Sqlite views can even be made modifiable by defining appropriate triggers.
This still requires programming some conditional logic upon connection, but it would allow more uniform queries and interaction with different versions of the schema.
The suggested syntax would perhaps be convenient in some limited cases, but this approach is much more useful since it can be expanded beyond simple "if column exists" Boolean operations and instead could be used to perform dynamic transformation of one schema into another, perhaps joining tables and providing more advanced logic for updates of differing schema, etc.
Pseudo code mixed with view definitions to demonstrate:
db <- Open database connection
db_schema <- determine schema version
If db_schema == 1 Then
db.execute( "CREATE VIEW temp.EmployeeX AS
SELECT Name, Phone, '' AS Address
FROM main.Employee;" )
Else If db_schema == 2 Then
db.execute( "CREATE VIEW temp.EmployeeX AS
SELECT Name, Phone, Address
FROM main.Employee;" )
End If
#Later in code
data <- db.getdata("SELECT Name, Address
FROM EmployeeX")
If you're really averse to conditional statements for the schema this may still be annoying, but it would at least reduce/eliminate conditional statements throughout the code--ideally occurring as part of the connection logic at one location in the code.
You might further notice that this pattern is really what object-oriented programming is supposed to solve. There's no mention of the language in the question, but a well-designed object model could be created in a similar fashion so that all database access is done through a unified interface. The implementation details for different schemas are internal to different objects that derive (i.e. implement interfaces and/or inherit from base class) from a basic set of interfaces. Consider the language you're using to see if the problem could be solved this way.
I'd like to use flyway for a DB update with the situation that an DB already exists with productive data in it. The problem I'm looking at now (and I did not find a nice solution yet), is the following:
There is an existing DB table with numeric IDs, e.g.
create table objects ( obj_id number, ...)
There is a sequence "obj_seq" to allocate new obj_ids
During my DB migration I need to introduce a few new objects, hence I need new
object IDs. However I do not know at development time, what ID
numbers these will be
There is a DB trigger which later references these IDs. To improve performance I'd like to avoid determine the actual IDs every time the trigger runs but rather put the IDs directly into the trigger
Example (very simplified) of what I have in mind:
insert into objects (obj_id, ...) values (obj_seq.nextval, ...)
select obj_seq.currval from dual
-> store this in variable "newID"
create trigger on some_other_table
when new.id = newID
...
Now, is it possible to dynamically determine/use such variables? I have seen the flyway placeholders but my understanding is that I cannot set them dynamically as in the example above.
I could use a Java-based migration script and do whatever string magic I like - so, that would be a way of doing it, but maybe there is a more elegant way using SQL?
Many thx!!
tge
If the table you are updating contains only reference data, get rid of the sequence and assign the IDs manually.
If it contains a mix of reference and user data, you need to select the id based on values in other columns.
I need to apply two successive aggregate functions to a dataset (the sum of a series of averages), something that is easily and routinely done with common table expressions in SQL Server or another DBMS that supports CTEs. Unfortunately, I am currently stuck with SQLite which does not support CTEs. Is there an alternative or workaround for achieving the same result in SQLite without performing two queries and rolling up the results in code?
To add a few more details, I don't think it could be easily done with views because the first set of aggregate values need to be retrieved based on a WHERE clause with several parameters. E.g.,
SELECT avg(elapsedTime)
FROM statisticsTable
WHERE connectionId in ([lots of values]) AND
updateTime > [startTime] AND
updateTime < [endTime]
GROUP BY connectionId
And then I need the sum of those averages.
Now that we are in THE FUTURE, let me note here that SQLite now does support Common Table Expressions, as of version 3.8.3 of 2014-02-03.
http://www.sqlite.org/lang_with.html
Would this work?
SELECT SUM(t.time) as sum_of_series_of_averages
FROM
(
SELECT avg(elapsedTime) as time
FROM statisticsTable
WHERE connectionId in ([lots of values]) AND
updateTime > [startTime] AND
updateTime < [endTime]
GROUP BY connectionId
) as t
By converting your averages into an inline view, you can SUM() the averages.
Is this what you are looking for?
As you've mentioned, SQLite doesn't support CTEs, window functions, or any of the like.
You can, however, write your own user functions that you can call inside SQLite by registering them to the database with the SQLite API using sqlite_create_function(). You register them with the database, and then you can use them in your own application code. You can make an aggregate function that would perform the sum of a series of averages based on the individual column values. For each value, a step-type callback function is called that allows you to perform some calculation on the data, and a pointer for holding state data is also available.
In your SQL, then, you could register a custom function called sum_of_series_of_averages and have:
SELECT sum_of_series_of_averages(columnA,columnB)
FROM table
WHERE ...
For some good examples on how those work, you should check out the SQLite source code, and also check out this tutorial (search for Defining SQLite User Functions).
I currently have a function that gets some data from the database and puts it into a dataset. The return type on my function is GenericCollection (Of CustomerDetails)
If I do this:
Dim dataset As DataSet = Read(strSQL.ToString) 'Gets Data from DB
What's the most efficient way to map the dataset results to an collection of objects. More importantly, since I'm using GenericCollection, is there a way to do this in which I can call a function from the ObjectType class (CustomerDetails) that would have a means to converting that specific object.
Or is there a way in which I can use a function that would handle all types?
Is there a way to do something like:
Return returnedResults.TransformDataSet(dataset)
In which returnedResults is an object collection Of CustomerDetails, or would it simply be easier to have TransformDataSet return an object collection Of CustomerDetails by itself?
Thanks for any help.
Do you plan to generate the dataset, create your collection, then throw away the dataset? If so, I would suggest dispensing with the dataset completely, and use a data reader (SqlDataReader if you are using sql server). You can iterate through the reader and create your collection as you go.
The dataset is a heavy, xml-based gadget that is great if you need to keep it around and use it for other stuff, but if you are just going to use it as a temporary intermediate data store between the db and you collection, then I would lose it.