I have an array where each element contains array
$arryItem = [
'surname' => 'user_surname',
'name' => 'user_name',
'sex' => 'user_sex',
'ratio' => 'player_ratio'
];
Also I have 3 tables. Users, Players and users_roles
users
id | name | surname | sex
players
id | user_id | ratio
users_roles
user_id | role_id
On each iteration I have to
Create user and get inserted ID
Create player and insert user_id as inserted user ID
Create new entry into users_roles, where user_id will inserted user ID
The problem is that there will 100k+ iterations with 3 inserts and other logic such as srtolower, generate random email and password. I think that my script will fail with timeout or memory limit.
Are there ways to do this fast? DB - PostgreSQL
Related
I am trying to retrieve a list of items based on loggedInUserId. Below is a rough schema I have
{
personId: 1,
accountManagerId: 2,
accountExecId: 3,
successManagerId: 5
}
I want to retrieve all the records if loggedInUserId is any of above mentioned fields. What will be the best way to organize the data?
Thanks for your help
Typically in DynamoDB you denormalize the data, unlike the normalized schema you show. That would look like the following:
pk | sk | data
1 | personId | other info
1 | accountManagerId | other info
2 | personId | other info
4 | successManagerId | other info
Now if you have a loggedInUserId as 1 and you do a Query API call to your table you get all the items that user relates to.
SELECT * FROM mytable WHERE pk= 1
1 | personId | other info
1 | accountManagerId | other info
We you have two options.
One would look something like
PK SK PersonId AccountManagerId AccountExecId SuccessManagerId
<partition value> <sort value> 1 2 3 5
The other would be to just store the whole object in a json string
PK SK UserData
<partition value> <sort value> <user data>
I prefer the second one, but both are valid
Edit:
For querying, if you want to search by the loggedInUserId you can set it as a partition key
PK=LOGGED_IN_USER_ID#1
And a simple Java example for query
var dynamoClient = DynamoDbEnhancedClient.builder().dynamoDbClient(dynamoDbClient).build();
var table = dynamoClient.table("Users", TableSchema.fromClass(Users.class));
table.getItem(Key.builder().partitionValue("1").build());
Its easier to use string partition and sort keys
I am having some issues designed a lookup table in dynamodb, without running into issues with GSI.
I have an object U, this object has 4 (that is of interest) attributes. Id, A, B and C. I have another object T that has 2 (that is of interest) attributes Id and a list of U.Id's.
I need to create a DynamoDB table where i can do quick looks up such as.
Given T.Id, give me all U objects that has A, or Given T.Id, give me all connected U objects that has B. Lastly given U.Id, give me the T.Id which has the U.Id in it's list.
I where thinking something like
| T.Id | Sort Key | U.Id | U.A | U.B | U.C |
| T1 | U1 | U1 | abc | rew | bgt |
| T1 | U2 | U2 | bgf | red | bcs |
| T2 | U3 | U3 | abc | rew | bgt |
There T.Id is the Primary Key, Sort Key is the U.Id.
Then a GSI on U.ID, U.A, U.B and U.C.
This approach should work for now, but i am limited to 5 GSI, and i know that later on more attributes on U would be added.
T could have up towards 2000 U.Id's in its list.
How should i do this DynamoDB design to have the quickest looks ups and not run into problems with a limit on GSI?
The use case like get all U.Id given T.Id, U.A and U.B would be nice to support but is not a requirements, i can ask the users of this table to do a intersection of multiple calls.
The idea of my solution is to spend space complexity to reduce time complexity.
Create a table, the hash key should be the U.id or T.id, which have prefix 'U.' or 'T.', then let's call the range key as Lookup. The Lookup column should be a compositing key, if you want to set a lookup for T.Id to U.Id, the Lookup key's prefix is 'id.', if the lookup is for T.Id to A, the lookup key's prefix is 'A.' and suffix is U.Id, same to B and C.
The attributes A,B,C should only store in the record whose id and lookup are both U.Id's value
To allow U.Id to refer back to T.Id, you should create a GSI for Lookup column.
|Id |Lookup |Attributes|
+----+-----------+----------+
|U.Id|U.Id |A,B,C |
|T.Id|id.value | |
|T.Id|A.value:Id | |
|T.Id|B.value:Id | |
|T.Id|C.value:Id | |
Given T.Id (value is 'tid'), give me all U objects that has A (value is 'a')
SELECT 'Lookup' FROM table WHERE Id = T.tid & Lookup.beginsWith('A.a');
Then we can get all U's id by split the lookup's value with ':'. And then use batch get to get all U's value
same idea to B and C
Lastly given U.Id (value is 'uid'), give me the T.Id which has the U.Id in it's list.
SELECT 'Id' FROM table WHERE Lookup = id.uid;
Then you have the T.Id
Hope my answer can help you, if you have any problems just let me know.
Notice: the SQL I wrote is just pseudo code, to provide you an idea only.
What you have described is a one-to-many relationship. Have a read up on how to model one-to-many relationships in NOSQL stores.
To make this less abstract I will assume u=user and t=team. One team can have many users. One user can only be in one team.
The way to model this is to have two tables - a user table and a team table. Note that if Teams have no attributes then just the user table will suffice.
User table:
Partiton Key: UserId
Attribute: A
Attribute: B
Attribute: C
Attribute: TeamId
User table GSI:
Partition Key: Team Id
Team table:
Partition Key: TeamId
Attribute: X
Given T.Id, give me all U objects that has A
Query the User table using the GSI, with partition key = TeamId, use a filterexpression on A
Given T.Id, give me all connected U objects that has B
Query the User table using the GSI, with partition key = TeamId, use a filterexpression on B
Lastly given U.Id, give me the T.Id which has the U.Id in it's list.
Use GetItem on the User table using the Primary Partition Key
The solution described by #Yu-Huang is a graph-node implementation. You can read about it in the context of DynamoDB at https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html
I do not recommend this implementation. It is suitable for many-to-many relationships and will add a great deal of complexity where it is not required.
This seems to be the hardest thing to search for - I can find a lot of "how do I find empties in two tables", or how to do so for non-sqlite, but...
Three tables, item (id, name), user (id, name) and item_user (item_id, user_id) - the last table connects the first two
Three users, Bob, Jane, Danny
Two items, hammer, nail
How do I find the users who haven't made an order for an item?
So, if...
bob has ordered a hammer and a nail
so has Jane
Danny has only ordered a hammer
...then I want to return one row:
user.name item.name
--------- ---------
Danny nail
Can I do a search to show this? In sqlite?
If I'm understanding your question correctly, you want to find every item that each user has not ordered? That can be accomplished with the following query:
SELECT
user.name AS user_name,
item.name AS item_name
-- Get all users and items.
FROM user
CROSS JOIN item
-- Exclude items that users have ordered.
WHERE NOT EXISTS (
SELECT *
FROM item_user
WHERE item_user.item_id = item.id
AND item_user.user_id = user.id
);
This yields:
user_name | item_name
----------+----------
'Danny' | 'nail'
Query used to construct tables:
CREATE TABLE item (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL
);
INSERT INTO item (name)
VALUES ('hammer'), ('nail');
CREATE TABLE user (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL
);
INSERT INTO user (name)
VALUES ('Bob'), ('Jane'), ('Danny');
CREATE TABLE item_user (
item_id INTEGER NOT NULL,
user_id INTEGER NOT NULL,
FOREIGN KEY (item_id) REFERENCES item (id),
FOREIGN KEY (user_id) REFERENCES user (id)
);
INSERT INTO item_user (user_id, item_id)
VALUES
(1, 1), -- Bob, hammer
(1, 2), -- Bob, nail
(2, 1), -- Jane, hammer
(2, 2), -- Jane, nail
(3, 1); -- Danny, hammer
I have a user table with user_id as primary key
user_id | name | email | deleted
1 | John | john#gmail.com | NULL
Field user_id is used like reference in some other tables.
Now, when I want to delete the user, I would like to delete the row if is not used in some other tables or update (mark users.deleted=1) if used
In this moment I do it from 2 sequences, but I would like to do it from one.
I've try with CASE but no chance... said "Near CASE syntax error"
CASE
WHEN (SELECT count(resp_user_id) AS counter FROM equip WHERE user_id = 1) > 0
THEN (UPDATE users SET deleted = 1 WHERE user_id=1)
ELSE (DELETE FROM users WHERE user_id=1)
END
What did I write wrong?
SQLite is an embedded database, i.e., it is designed to be used together with a 'real' programming language.
There is no such logic in SQLite itself. You have to write the logic in your program, and use multiple SQL statements:
cursor = db.execute("SELECT resp_user_id AS counter FROM equip WHERE user_id = ?", [id])
if not cursor.empty:
db.execute("UPDATE users SET deleted = 1 WHERE user_id = ?", [id])
else:
db.execute("DELETE FROM users WHERE user_id = ?", [id])
To make the accesses atomic, wrap them into a transaction.
Not in one sequence, but in a secured way the command can be:
begin transaction;
delete from users where user_id=1 and ((select count(resp_user_id) from equip where resp_user_id = 1) = 0);
update users set deleted = 1 where user_id=1 and ((select count(resp_user_id) from equip where resp_user_id = 1) > 0);
end transaction;
I'm in visual studio, looking to create a composite index on 2 columns for several tables. There are 2 columns: UserID is in all tables and acts as the foreign key; then, each table has its own key to refer to the parts of the object, such as phone, address... Like this:
TablePhones:
PhoneID | UserID | PhonePrefix | PhoneNumber | PhoneExtention
TableAddresses:
AddressID | UserID | AddressStreet1 | AddressStreet2 | AddressCity...
Note that users can have more than 1 address and more than 1 phone number.
I'm using linq to sql and the where clauses queries to get the objects look like this:
read queries:
where x.UserID == TheUserID
update/delete queries:
where x.UserID == TheUserID && x.PhoneID = ThePhoneID
At the moment, the primary keys are on PhoneID and AddressID and I'm looking to create composite indexes on PhoneID/UserID and AddressID/UserID. Is the order of the columns in the database fine as it is or should I move UserID in first position for all tables.
Thanks for suggestions.
Order of columns in table doesn't matter; at least for SQLServer. The important thing is in which order fields are listed in an index. Queries with conditions on leading column[s] will very benefit from the index.
If your primary key is clustered, you can create index on only userID, no need for composite key. Anyway, it will have a reference to clustered key.