knowing if a Kusto table has overridden cahcing/retention policy - azure-data-explorer

When we fire the following command against a specific database , we get all the policy info for all the tables in that database:-
.show tables details
But how to determine if a certain policy (e.g. Caching or Retention) that is shown there is because it is inherited from database policy or it is specifically overridden for that table (which is supported too) ? Is there a command to get that information as well , some sort of per table per policy level flag which will tell us if it was overridden.

to see a policy defined at database level (or null, if not defined at database-level), run: .show database DATABASE_NAME policy POLICY_KIND.
for example: .show database MyDatabase policy retention
to see a policy defined at table level (or null, if not defined at table-level), run: .show table TABLE_NAME policy POLICY_KIND.
for example: .show table MyTable policy retention.
if this returns null as the Policy - the database-level policy, if set, is in effect.
as you mentioned correctly, .show tables details shows you the effective policy on the table, taking into account both database-level and table-level policies, if those are set (not null).

Related

Choosing Primary key for DynamoDB

A bit of context: I am trying to build an inventory to list my AWS resources in various accounts and I am planning to use DynamoDB to store the data. These will be the columns for my table: ResourceARN, ResourceName, ResourceType, StandardTag, IsDeleted, LastUpdateTime and ResourceCreationDate ( this field is available only for a few resource types like Ec2).
Question: I want to query my DDB table using account ID, resource type and tag name. I am stumped on choosing the primary key for the table. Since primary key should be unique and has to have 1:many relationship. Hence, I cannot use a combination of resourceType and account Id. Nor can I use resourceArn as my primary key since it is 1:1 relationship. Also, using the resourceARN as the sort key does not make sense to me. I understand that I can use a simple scan operation, but that is very costly and will take time if I add more data in my DDB.
I would appreciate any suggestions or guidance over the same.
Short answer
Partition key: Account ID
Sort key: <resource type>/<resource ID>
Rationale
It's a common pattern for a sort key to be a string concatenating multiple attributes. Since sort keys can be queried by prefix, you can leverage this in your queries:
Get all account resources: query all sort keys on the Account ID partition key
Get all EC2 instances of an account: query with partition key = <your account ID> and sort key begins_with('ec2-instance').
You may notice that ARNs follow such a hierarchy as well (what's probably not a coincidence). This would be effectively using a subset of the ARN as the sort key.
Some notes:
DynamoDB is about attributes as much as about columns. You don't need to include ResourceCreationDate in the records which don't have it, and doing so will save you space (see next point).
Attribute names count as storage for every record, which impacts cost and also throughput. It's common to use shorthand for names for this reason (rct instead of ResourceCreationTime for example).
You can use LSIs (Local Secondary Indexes) to order by creation and update times if you need this.

Creation of Flyway "schema_version" fails for dashDB

I'm using Flyway to manage db migration on IBM dashDB. This database organizes by default table content 'by column', which in particular makes the creation of the "schema_version" table fail.
To get it to work, the table creation SQL statement should only include the "ORGANIZE BY ROW" directive:
CREATE TABLE (...)
(...)
) ORGANIZE BY ROW
What would be the best approach to handle this issue ? I'm looking for a solution that does not impact the default table organization.
Thanks for helping,
Cheers.
dashDB will perform best when all tables are column-based. When you start to mix row and column based tables, many operations are then performed in "compensation" which basically means they won't take full advantage of the columnar engine.
There are currently some compatibility reasons why a columnar table cannot be created and thus a row based table must be used, but the original DDL nor error are stated so I can't tell in this case. If you can provide the full CREATE TABLE statement and the resulting error (if you have it), I can possibly provide an alternative solution that would allow you to still use all column-based tables.
If you only want to change a particular table from column organized to row organized then a "ORGANIZE BY ROW" on the table definition would be the recommended way to approach this. (This seems to be what you're doing)
Changing the default table org will change how tables are created when you don't put an "ORGANIZE BY " in your table ddl.
If you have admin privileges on your dashDB instance you can change the default table org via 'Run SQL' in the dashDB console or using a dashDB client. (for exampl: clp/clpplus)
Set default table organization to ROW:
call ADMIN_CMD('UPDATE DB CFG USING DFT_TABLE_ORG ROW');
Set default table organization to COLUMN: (default dashDB configuration)
call ADMIN_CMD('UPDATE DB CFG USING DFT_TABLE_ORG COLUMN');
Analytics will perform much better with Column organized tables so it's recommended to have the majority of your tables as column organized.

How to restrict DB user to view table data if he has SELECT permission

An Oracle DB user has SELECT permission on all the tables of a DB schema. Can i restrict the user to view the table data.The user should be able to select the table but should not be able to see the data.
This specific requirement is required for user who reviews DB design and generates ALTER script for DB using Oracle Data Modeler 3.3 where he can just see the table design and can compare it with ERD
Can i achieve it using FGAC or RLS?
You can achieve this by granting only references
GRANT references ON schema_a.table TO erd_user;
the erd_user can then use
DESC schema_a.table
to get a definition but not select any data.
This might be preferred to giving the SELECT CATALOGUE where they can see a lot more information that you might like.

Creating multiple schemas for a user - Oracle 11g

Can we create multiple schemas for a particular user? I am currently logged in as X/Y user and when I tried creating a schema using create schema authorization sample_schema, I got the error the schema name is missing or is incorrect in an authorization clause of a create schema statement. I do know that a default schema X would have been created.
CREATE SCHEMA in Oracle does - contrary to its name - not create a new schema.
It is merely a shorthand to create several tables in a single statement.
Quote from the manual:
Use the CREATE SCHEMA statement to create multiple tables and views and perform multiple grants in your own schema in a single transaction
and further down the explanation on what the "schema" name parameter is:
The schema name must be the same as your Oracle Database username.
Well you could create a user named sample_schema (From the above example) and give user X/Y permission to use sample_schema tablespace.

How to implement a cross-database foreign key constraint?

Let's say I have two schemas: HR and Orders.
[HR].Employees [Orders].Entries
-------------- ----------------
Id_Employee ----> Employee
Fullname Id_Entry
Birthday Description
Amount
As you can see, what I'd want is to be able to establish a cross-database foreign key, but when I try this using a database link, I get:
-- From [Orders]
ALTER TABLE Entries
ADD CONSTRAINT FK_Entries_Employees FOREIGN KEY (Employee)
REFERENCES Employees#HR;
COMMIT;
ORA-02021: DDL operations are not allowed on a remote database
Is there a way around this? It's a legacy database, so I can't change the existing schema.
For the NHibernate crowd: I would then use this relation to map the NHibernate's domain objects.
One option would be to create a materialized view of Employees on [Orders] and then use that as the parent for the foreign key.
Of course, that has some drawbacks. In particular,
-- you won't be able to do a complete refresh of the materialized view without disabling the foreign key, so it'll have to fast refresh.
-- keys entered into EMPLOYEES won't be available to ENTRIES until the materialized view refresh. If that's critical, you may want to set it to refresh on commit.
Other alternatives are to handle the key enforcement yourself through a trigger or through a post cleanup process. Or convince the DBA's that these schemas can reside on the same database instance.
As far as I know constraints and referential integrity are only supported within one single database.
If you need to cross the boundaries of the database, you'd have to be creative. Maybe write some triggers checking for data in the other database or enforce these constraints on the application level. But then you may encounter the problem with transaction scope limited to one single database.

Resources