When a table is being used as a hash key, does it have anything to do with the hex id you get when you print out the table?
For example,
obj = {}
print(obj)
You might get something like table: 153CF5A0. Is this value used in the hashing process of this table?
I noticed that if you create two identical tables that aren't references of each other, they hash to different things:
obj1 = {}
obj2 = {}
map = {}
map[obj1] = 'obj1'
map[obj2] = 'obj2'
print(map[obj1]) -- obj1
print(map[obj2]) -- obj2
Yes, the memory address of a table is used as a key for hashing when the table is used to index another table. See the source code (tables are handled in the default case).
Related
Im trying to write a recursive query for a use on a old and poorly designed database - and so the queries get quite complex.
Here is the (relevant) table relationships
Because people asked - here is the creation code for these tables:
CREATE TABLE CircuitLayout(
CircuitLayoutID int,
PRIMARY KEY (CircuitLayoutID)
);
CREATE TABLE LitCircuit (
LitCircuitID int,
CircuitLayoutID int,
PRIMARY KEY (LitCircuitID)
FOREIGN KEY (CircuitLayoutID) REFERENCES CircuitLayout(CircuitLayoutID)
);
CREATE TABLE CircuitLayoutItem(
CircuitLayoutItemID int,
CircuitLayoutID int,
TableName varchar(255),
TablePK int,
PRIMARY KEY (CircuitLayoutItemID)
FOREIGN KEY (CircuitLayoutID) REFERENCES CircuitLayout(CircuitLayoutID)
);
TableName refers to another table in the database and thus TablePK is a primary key from the specified table
One of the valid options for TableName is LitCircuit
I'm trying to write a query that will select a circuit and any circuit it is related to
I am having trouble understanding the syntax for recursive ctes
my non-functional attempt is this:
WITH RECURSIVE carries AS (
SELECT LitCircuit.LitCircuitID AS recurseList FROM LitCircuit
JOIN CircuitLayoutItem ON LitCircuit.CircuitLayoutID = CircuitLayoutItem.CircuitLayoutID
WHERE CircuitLayoutItem.TableName = "LitCircuit" AND CircuitLayoutItem.TablePK IN (00340)
UNION
SELECT LitCircuit.LitCircuitID AS CircuitIDs FROM LitCircuit
JOIN CircuitLayout ON LitCircuit.CircuitLayoutID = CircuitLayoutItem.CircuitLayoutID
WHERE CircuitLayoutItem.TableName = "LitCircuit" AND CircuitLayoutItem.TablePK IN (SELECT recurseList FROM carries)
)
SELECT * FROM carries;
the "00340" is a dummy number for testing, and it would get replaced with an actual list in usage
What i'm attempting to do is get a list of LitCircuitIDs based on one or many LitCircuitIDs - that's the anchor member, and that works fine.
What I want to do is take this result and feed it back into itself.
I lack an understanding of how to access data from the anchor member:
I don't know if it is a table with the columns from the select in the anchor or if it is simply a list of resulting values
I dont understand if or where I need to include "carries" in the FROM part of a query
If I were to write this function in python I would do it like this:
def get_circuits(circuit_list):
result_list = []
for layout_item_key, layout_item in CircuitLayoutItem.items():
if layout_item['TableName'] == "LitCircuit" and layout_item['TablePK'] in circuit_list:
layout = layout_item['CircuitLayoutID']
for circuit_key, circuit in LitCircuit.items():
if circuit["CircuitLayoutID"] == layout:
result_list.append(circuit_key)
result_list.extend(get_circuits(result_list))
return result_list
How do I express this in SQL?
danblack's comment made me realize something I was missing:
Here is what I was trying to do:
WITH RECURSIVE carries AS (
SELECT LitCircuit.LitCircuitID FROM LitCircuit
JOIN CircuitLayoutItem ON LitCircuit.CircuitLayoutID = CircuitLayoutItem.CircuitLayoutID
WHERE CircuitLayoutItem.TableName = 'LitCircuit' AND CircuitLayoutItem.TablePK IN (00340)
UNION ALL
SELECT LitCircuit.LitCircuitID FROM carries
JOIN CircuitLayoutItem ON carries.LitCircuitID = CircuitLayoutItem.TablePK
JOIN LitCircuit ON CircuitLayoutItem.CircuitLayoutID = LitCircuit.CircuitLayoutID
WHERE CircuitLayoutItem.TableName = 'LitCircuit'
)
SELECT DISTINCT LitCircuitID FROM carries;
I did not think of the CTE as a table to query against - rather just a result set, so I did not realize you have to SELECT from it - or in general treat it like a table.
How to translate this snippet of executable pseudo code into ABAP?
phone_numbers = {
'hans': '++498912345',
'peter': '++492169837',
'alice': '++6720915',
}
# access
print (phone_numbers['hans'])
# add
phone_numbers['bernd']='++3912345'
# update
phone_numbers['bernd']='++123456'
if 'alice' in phone_numbers:
print('Yes, alice is known')
# all entries
for name, number in phone_numbers.items():
print(name, number)
Modern ABAP is possible up to 752, less chars, more upvotes :-)
P.S. BTW, up to now no one has added abap to pleac (Programming Language Examples Alike Cookbook)
Well, how about the following solution?
REPORT ZZZ.
TYPES: BEGIN OF t_phone_number,
name TYPE char40,
number TYPE char40,
END OF t_phone_number.
DATA: gt_phone_number TYPE HASHED TABLE OF t_phone_number WITH UNIQUE KEY name.
START-OF-SELECTION.
gt_phone_number = VALUE #(
( name = 'hans' number = '++498912345' )
( name = 'peter' number = '++492169837' )
( name = 'alice' number = '++6720915' )
).
* access
WRITE / gt_phone_number[ name = 'hans' ]-number.
* add
gt_phone_number = VALUE #( BASE gt_phone_number ( name = 'bernd' number = '++3912345' ) ).
* update
MODIFY TABLE gt_phone_number FROM VALUE #( name = 'bernd' number = '++123456' ).
IF line_exists( gt_phone_number[ name = 'alice' ] ).
WRITE / 'Yes, Alice is known.'.
ENDIF.
* all entries
LOOP AT gt_phone_number ASSIGNING FIELD-SYMBOL(<g_phone_number>).
WRITE: /, <g_phone_number>-name, <g_phone_number>-number.
ENDLOOP.
#Jagger's answer is great, but #guettli asked for shorter syntax. So just for completeness, there is of course always the possibility to wrap this in a class:
CLASS dictionary DEFINITION.
PUBLIC SECTION.
TYPES:
BEGIN OF row_type,
key TYPE string,
data TYPE string,
END OF row_type.
TYPES hashed_map_type TYPE HASHED TABLE OF row_type WITH UNIQUE KEY key.
METHODS put
IMPORTING
key TYPE string
data TYPE string.
METHODS get
IMPORTING
key TYPE string
RETURNING
VALUE(result) TYPE string.
METHODS get_all
RETURNING
VALUE(result) TYPE hashed_map_type.
METHODS contains
IMPORTING
key TYPE string
RETURNING
VALUE(result) TYPE abap_bool.
PRIVATE SECTION.
DATA map TYPE hashed_map_type.
ENDCLASS.
CLASS dictionary IMPLEMENTATION.
METHOD put.
READ TABLE map REFERENCE INTO DATA(row) WITH TABLE KEY key = key.
IF sy-subrc = 0.
row->*-data = data.
ELSE.
INSERT VALUE #( key = key
data = data )
INTO TABLE map.
ENDIF.
ENDMETHOD.
METHOD get.
result = map[ key = key ]-data.
ENDMETHOD.
METHOD get_all.
INSERT LINES OF map INTO TABLE result.
ENDMETHOD.
METHOD contains.
result = xsdbool( line_exists( map[ key = key ] ) ).
ENDMETHOD.
ENDCLASS.
Leading to:
DATA(phone_numbers) = NEW dictionary( ).
phone_numbers->put( key = 'hans' data = '++498912345' ).
phone_numbers->put( key = 'peter' data = '++492169837' ).
phone_numbers->put( key = 'alice' data = '++6720915' ).
" access
WRITE phone_numbers->get( 'hans' ).
" add
phone_numbers->put( key = 'bernd' data = '++3912345' ).
" update
phone_numbers->put( key = 'bernd' data = '++123456' ).
IF phone_numbers->contains( 'alice' ).
WRITE 'Yes, alice is known'.
ENDIF.
" all entries
LOOP AT phone_numbers->get_all( ) INTO DATA(row).
WRITE: / row-key, row-data.
ENDLOOP.
People rarely do this in ABAP because internal tables are so versatile and powerful. From my personal point of view, I'd like to see people build more custom data structures. Implementation details like HASHED or SORTED, see discussion in #Jagger's answer, are hidden away in a natural way when doing this.
Our team is currently exploring the ways to encrypt PII data on the field level within BigQuery and we found out the following way to encrypt/decrypt using Crypto-JS:
#standardSQL
CREATE TEMPORARY FUNCTION encrypt(_text STRING) RETURNS STRING LANGUAGE js AS
"""
let key = CryptoJS.enc.Utf8.parse("<key>");
let options = { iv: CryptoJS.enc.Utf8.parse("<iv>"), mode: CryptoJS.mode.CBC };
let _encrypt = CryptoJS.AES.encrypt(_text, key, options);
return _encrypt;
""";
CREATE TEMPORARY FUNCTION decrypt(_text STRING) RETURNS STRING LANGUAGE js AS
"""
let key = CryptoJS.enc.Utf8.parse("<key>");
let options = { iv: CryptoJS.enc.Utf8.parse("<iv>"), mode: CryptoJS.mode.CBC };
let _decrypt = CryptoJS.AES.decrypt(_text, key, options).toString(CryptoJS.enc.Utf8);
return _decrypt;
""" OPTIONS (library="gs://path/to/Crypto-JS/crypto-js.js");
-- query to encrypt fields
SELECT
<fields>, encrypt(<pii-fields>)
FROM
`<project>.<dataset>.<table>`
-- query to decrypt fields
SELECT
<fields>, decrypt(<pii-fields>)
FROM
`<project>.<dataset>.<table>`
I am trying to benchmark the performance of AES CBC encryption & decryption using Crypto JS library in the big query before deploying it into our production. We found out the rate of data to encrypt & decrypt is growing exponential per records with increasing number of data compared to the usual query. However with the increasing number of data to process, the progress of processing per record & record processing time is improving.
As there are no available documentation regarding this, could someone from the community help provide better ways, optimize query, best practices to use field level encryption & decryption within the big query?
BigQuery now supports encryption functions. From the documentation, here is a self-contained example that creates some keysets and uses them to encrypt data. In practice, you would want to store the keysets in a real table so that you can later use them to decrypt the ciphertext.
WITH CustomerKeysets AS (
SELECT 1 AS customer_id, KEYS.NEW_KEYSET('AEAD_AES_GCM_256') AS keyset UNION ALL
SELECT 2, KEYS.NEW_KEYSET('AEAD_AES_GCM_256') UNION ALL
SELECT 3, KEYS.NEW_KEYSET('AEAD_AES_GCM_256')
), PlaintextCustomerData AS (
SELECT 1 AS customer_id, 'elephant' AS favorite_animal UNION ALL
SELECT 2, 'walrus' UNION ALL
SELECT 3, 'leopard'
)
SELECT
pcd.customer_id,
AEAD.ENCRYPT(
(SELECT keyset
FROM CustomerKeysets AS ck
WHERE ck.customer_id = pcd.customer_id),
pcd.favorite_animal,
CAST(pcd.customer_id AS STRING)
) AS encrypted_animal
FROM PlaintextCustomerData AS pcd;
Edit: if you want to decrypt using AES-CBC with PKCS padding (it's not clear what kind of padding you are using in your example) you can use the KEYS.ADD_KEY_FROM_RAW_BYTES function to create a keyset, then call AEAD.DECRYPT_STRING or AEAD.DECRYPT_BYTES. For example:
SELECT
AEAD.DECRYPT_STRING(
KEYS.ADD_KEY_FROM_RAW_BYTES(b'', 'AES_CBC_PKCS', b'1234567890123456'),
FROM_HEX('deed2a88e73dccaa30a9e6e296f62be27db30db16f76d3f42c85d31db3f46376'),
'')
This returns abcdef. The IV is expected to be the first 16 bytes of the ciphertext.
I'm familiar with MySQL and am starting to use Amazon DynamoDB for a new project.
Assume I have a MySQL table like this:
CREATE TABLE foo (
id CHAR(64) NOT NULL,
scheduledDelivery DATETIME NOT NULL,
-- ...other columns...
PRIMARY KEY(id),
INDEX schedIndex (scheduledDelivery)
);
Note the secondary Index schedIndex which is supposed to speed-up the following query (which is executed periodically):
SELECT *
FROM foo
WHERE scheduledDelivery <= NOW()
ORDER BY scheduledDelivery ASC
LIMIT 100;
That is: Take the 100 oldest items that are due to be delivered.
With DynamoDB I can use the id column as primary partition key.
However, I don't understand how I can avoid full-table scans in DynamoDB. When adding a secondary index I must always specify a "partition key". However, (in MySQL words) I see these problems:
the scheduledDelivery column is not unique, so it can't be used as a partition key itself AFAIK
adding id as unique partition key and using scheduledDelivery as "sort key" sounds like a (id, scheduledDelivery) secondary index to me, which makes that index pratically useless
I understand that MySQL and DynamoDB require different approaches, so what would be a appropriate solution in this case?
It's not possible to avoid a full table scan with this kind of query.
However, you may be able to disguise it as a Query operation, which would allow you to sort the results (not possible with a Scan).
You must first create a GSI. Let's name it scheduled_delivery-index.
We will specify our index's partition key to be an attribute named fixed_val, and our sort key to be scheduled_delivery.
fixed_val will contain any value you want, but it must always be that value, and you must know it from the client side. For the sake of this example, let's say that fixed_val will always be 1.
GSI keys do not have to be unique, so don't worry if there are two duplicated scheduled_delivery values.
You would query the table like this:
var now = Date.now();
//...
{
TableName: "foo",
IndexName: "scheduled_delivery-index",
ExpressionAttributeNames: {
"#f": "fixed_value",
"#d": "scheduled_delivery"
},
ExpressionAttributeValues: {
":f": 1,
":d": now
},
KeyConditionExpression: "#f = :f and #d <= :d",
ScanIndexForward: true
}
In SQLite I can run the following query to get a list of columns in a table:
PRAGMA table_info(myTable)
This gives me the columns but no information about what the primary keys may be. Additionally, I can run the following two queries for finding indexes and foreign keys:
PRAGMA index_list(myTable)
PRAGMA foreign_key_list(myTable)
But I cannot seem to figure out how to view the primary keys. Does anyone know how I can go about doing this?
Note: I also know that I can do:
select * from sqlite_master where type = 'table' and name ='myTable';
And it will give the the create table statement which shows the primary keys. But I am looking for a way to do this without parsing the create statement.
The table_info DOES give you a column named pk (last one) indicating if it is a primary key (if so the index of it in the key) or not (zero).
To clarify, from the documentation:
The "pk" column in the result set is zero for columns that are not
part of the primary key, and is the index of the column in the primary
key for columns that are part of the primary key.
Hopefully this helps someone:
After some research and pain the command that worked for me to find the primary key column name was:
SELECT l.name FROM pragma_table_info("Table_Name") as l WHERE l.pk = 1;
For the ones trying to retrieve a pk name in android, and while using the ROOM library.
#Oogway101's answer was throwing an error: "no such column [your_table_name] ... etc.. etc...
my way of query submition was:
String pkSearch = "SELECT l.name FROM pragma_table_info(" + tableName + ") as l WHERE l.pk = 1;";
database.query(new SimpleSQLiteQuery(pkSearch)
I tried using the (") quotations and still error.
String pkSearch = "SELECT l.name FROM pragma_table_info(\"" + tableName + "\") as l WHERE l.pk = 1;";
So my solution was this:
String pragmaInfo = "PRAGMA table_info(" + tableName + ");";
Cursor c = database.query(new SimpleSQLiteQuery(pragmaInfo));
String id = null;
c.moveToFirst();
do {
if (c.getInt(5) == 1) {
id = c.getString(1);
}
} while (c.moveToNext() && id == null);
Log.println(Log.ASSERT, TAG, "AbstractDao: pk is: " + id);
The explanation is that:
A) PRAGMA table_info returns a cursor with various indices, the response is atleast of length 6... didnt check more...
B) index 1 has the column name.
C) index 5 has the "pk" value, either 0 if it is not a primary key, or 1 if its a pk.
You can define more than one pk so this will not bring an accurate result if your table has more than one (IMHO more than one is bad design and balloons the complexity of the database beyond human comprehension).
So how will this fit into the #Dao? (you may ask...)
When making the Dao "abstract" you have access to a default constructor which has the database in it:
from the docummentation:
An abstract #Dao class can optionally have a constructor that takes a Database as its only parameter.
this is the constructor that will grant you access to the query.
There is a catch though...
You may use the Dao during a database creation with the .addCallback() method:
instance = Room.databaseBuilder(context.getApplicationContext(),
AppDatabase2.class, "database")
.addCallback(
//You may use the Daos here.
)
.build();
If you run a query in the constructor of the Dao, the database will enter a feedback loop of infinite instantiation.
This means that the query MUST be used LAZILY (just at the moment the user needs something), and because the value will never change, it can be stored. and never re-queried.