I'm trying to sample multiple records from big table randomly. Currently, a way I can go is to make random numbers in python and use them in sqlite3. Though I know random() of sqlite gives a random number, I don't know how to get (DISTINCT) multiple random numbers in sqlite. If I can make a table filled with random numbers, it would be OK.
If you know how many rows you want, you can use something like:
SELECT DISTINCT col1, col2, etc FROM yourtable ORDER BY random() LIMIT 10;
Related
I'm inserting data into table from another table using below query in Teradata and I want to run this statement until table reaches 20GB. So I want to run below statement in a loop to achieve that. However I written one but it's giving query invalid error when I'm trying to execute. Could you please help me as I'm new to Teradata. Thanks.
insert into schema1.xyx select * from schema2.abc;
if/loop/etc. are only allowed in Stored Procedures.
Looping a billion times will be quite inefficient (and will result in much more than 20GB). Better check the current size of table abc from dbc.TableSizeV, calculate how many loops you need and then cross join.
insert into schema1.xyx
select t.*
from schema2.abc AS t
cross join
( -- calculated number of loops
select top 100000 *
-- any table with a large number of rows
from sys_calendar_calendar
);
But much easier is using sampling.
Calculate the number of rows needed and then
insert into schema1.xyx
select *
from schema2.abc
sample with replacement 100000000;
I have a database that has two integer columns, and I'm trying to find a way to select the top 'x' amount of rows with the highest sums of these two columns. I'm trying to eliminate the need of creating a third column that stores the sum of the two, unless there's a way to to automatically update this column every time one of the other two are altered. I'm using SQLite by the way, as I know there are some slight differences here and there between SQL/SQLite syntax.
Any help is appreciated.
Something like
SELECT a, b
FROM yourtable
ORDER BY a + b DESC
LIMIT :x
should do it.
I am trying to find a way to format any large number 1000 or more with commas in SQlite. So, 1000 would be 1,000. 1000000 would be 1,000,000, and so on.
I can easily handle individual cases by changing the numbers to text and then using SUBSTR, but I need a catch all solution that can be done in SQLite.
Any help is appreciated.
You can do it with the function printf():
SELECT printf("%,d", col)
FROM tablename
Replace tablename and col with your table's and the column's name.
See the demo.
SELECT col1 FROM tbl ORDER BY RAND() LIMIT 10;
This can work fine for small tables. However, for big table, it will have a serious performance problem as in order to generate the list of random rows, MySQL need to assign random number to each row and then sort them.
Even if you want only 10 random rows from a set of 100k rows, MySQL need to sort all the 100k rows and then, extract only 10 of them.
My solution for this problem, is to use RAND in the WHERE clause and not in the ORDER BY clause. First, you need to calculate the fragment of your desired result set rows number from the total rows in your table. Second, use this fragment in the WHERE clause and ask only for RAND numbers that smallest (or equal) from this fragment.
SELECT col1 FROM tbl WHERE RAND()<=0.0005;
In order to get exactly 100 row in the result set, we can increase the fragment number a bit and limit the query:
For example:
I am trying to populate everyrow in a column with random ranging from 0 to row count.
So far I have this
UPDATE table
SET column = ABS (RANDOM() % (SELECT COUNT(id) FROM table))
This does the job but produces duplicate values, which turned out to be bad. I added a Unique constraint but that just causes it to crash.
Is there a way to update a column with random unique values from certain range?
Thanks!
If you want to later read the records in a random order, you can just do the ordering at that time:
SELECT * FROM MyTable ORDER BY random()
(This will not work if you need the same order in multiple queries.)
Otherwise, you can use a temporary table to store the random mapping between the rowids of your table and the numbers 1..N.
(Those numbers are automatically generated by the rowids of the temporary table.)
CREATE TEMP TABLE MyOrder AS
SELECT rowid AS original_rowid
FROM MyTable
ORDER BY random();
UPDATE MyTable
SET MyColumn = (SELECT rowid
FROM MyOrder
WHERE original_rowid = MyTable.rowid) - 1;
DROP TABLE MyOrder;
What you seem to be seeking is not simply a set of random numbers, but rather a random permutation of the numbers 1..N. This is harder to do. If you look in Knuth (The Art of Computer Programming), or in Bentley (Programming Pearls or More Programming Pearls), one suggested way is to create an array with the values 1..N, and then for each position, swap the current value with a randomly selected other value from the array. (I'd need to dig out the books to check whether it is any arbitrary position in the array, or only with a value following it in the array.) In your context, then you apply this permutation to the rows in the table under some ordering, so row 1 under the ordering gets the value in the array at position 1 (using 1-based indexing), etc.
In the 1st Edition of Programming Pearls, Column 11 Searching, Bentley says:
Knuth's Algorithm P in Section 3.4.2 shuffles the array X[1..N].
for I := 1 to N do
Swap(X[I], X[RandInt(I,N)])
where the RandInt(n,m) function returns a random integer in the range [n..m] (inclusive). That's nothing if not succinct.
The alternative is to have your code thrashing around when there is one value left to update, waiting until the random number generator picks the one value that hasn't been used yet. As a hit and miss process, that can take a while, especially if the number of rows in total is large.
Actually translating that into SQLite is a separate exercise. How big is your table? Is there a convenient unique key on it (other than the one you're randomizing)?
Given that you have a primary key, you can easily generate an array of structures such that each primary key is allocated a number in the range 1..N. You then use Algorithm P to permute the numbers. Then you can update the table from the primary keys with the appropriate randomized number. You might be able to do it all with a second (temporary) table in SQL, especially if SQLite supports UPDATE statements with a join between two tables. But it is probably nearly as simple to use the array to drive singleton updates. You'd probably not want a unique constraint on the random number column while this update is in progress.