I have millions of rows in a sqlite3 database. The column 'points' of a single row contains the following example:
{1399808086,1366221142,1374614902,1374608759,1375598069,1375270116,1935207612,1914502332,1913478333,1930188205,1934563311,1942881023,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988 ... up to nearly 1'000 numbers, ending with }
What is the best way to search for rows where the column 'points' includes
a) all three example search values
1366221142,1374614902,1374608759 (Position two, three and four in the above content)
b) as much as possible (3 or 2 or 1) of the above 3 example search values
I tried it with Indexes, but a search with LIKE and '%1366221142%' takes "forever".
Actually I try it with FTS5, but the import into a new created virtual table seems to take several days.
Do you know any other possibility?
If you can use Python then this approach is 'pretty fast'.
import sqlite3
conn = sqlite3.connect(':memory:')
points = '{1399808086,1366221142,1374614902,1374608759,1375598069,1375270116,1935207612,1914502332,1913478333,1930188205,1934563311,1942881023,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860,1929778348,1900414380,1883651988,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,-778051210,-740437658,-749943514,-754136794,-770946570,1376606678,1380850053,1380854148,1381902468,1381971092,308228244,324940180,324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860}'
conn.execute('CREATE TABLE something (recno, points)')
for r in range(10000):
conn.execute('INSERT INTO something (recno, points) values (?,?)', (r, points))
seeking = ['1366221142', '1374614902', '1374608759']
first = True
for row in conn.execute('SELECT recno, points FROM something'):
pointsList = points[1:-1].split(',')
counts = { _:pointsList.count(_) for _ in seeking }
if first:
print (row)
print (counts)
first = False
The (abridged) output is:
(0, '{1399808086,1366221142,1374614902,1374608759,1375598069,1375270116,1935207612,1914502332,1913478333,1930188205,1934563311,1942881023,1373508175,-778100129,-788765075,-788763091,-790856156,-790835404,-791756027,-795938489,-779165370,... 324948372,327078869,292409717,275550503,275554606,275547438,812554558,812489022,1894554398,1895733774,1895741966,1912515343,1943993629,1935471709,1918694493,1914490972,1913409788,1913475260,1913458860}')
{'1374614902': 1, '1374608759': 1, '1366221142': 1}
Note that the code arranges to put 10,000 copies of your string, extended to 1,000 numbers, into the database and then processes them. Of course, the database is in memory, which is a factor to consider.
You could just try it.