I am trying to fetch all the data from my dynamodb table but unable to get as all the methods for Query / Scan operates with input parameter. so i tried getting all the rows which having primary key greater than 0.
var table = Table.LoadTable(client,Utilities.Utility.EmployeeTable);
ScanFilter filter = new ScanFilter();
filter.AddCondition("iemp_id", ScanOperator.GreaterThan, 0);
ScanOperationConfig config = new ScanOperationConfig()
{
Filter = filter,
// Optional parameters.
Select = SelectValues.SpecificAttributes,
AttributesToGet = new List<string> { "iemp_id", "demp_salary", "semp_name" }
//ConsistentRead = true
};
Search search = table.Scan(config);`
Here i am getting search.Matches = 0 where it should return data from my table.
You have only two options
1.Query : You need to supply Partition Key(mandatory) and optionally Range key.
2.Scan: Full scan of the table with out Partition key/Range Key.
In your case you will have to do full scan of the table.
DynamoDBQScanExpression scanExpression = new DynamoDBScanExpression();
scanExpression .withFilterExpression(filterexpression)
.withExpressionAttributeValues(expression values);
Related
I'm trying to query all the rows which are created the last one week. I have created an index for created key in AWS console. In my query, I added the Key Condition with ComparisonOperator.GT for the created key. But When I run the query it throws an error like Query key condition not supported. If I give the condition as ComparisonOperator.EQ, it will return a single row. But not working for the ComparisonOperator.GT.
Code :
Condition rangeKeyCondition = new Condition();
rangeKeyCondition.withComparisonOperator(ComparisonOperator.GT).withAttributeValueList(new AttributeValue().withS("11:26 23/10/2018 "));
Map<String, Condition> keyConditions = new HashMap<String, Condition>();
keyConditions.put("created", rangeKeyCondition);
QueryRequest queryRequest = new QueryRequest();
queryRequest.withTableName(getTableName(TABLE_NAME));
queryRequest.withIndexName("created-index");
queryRequest.withKeyConditions(keyConditions);
QueryResult result = EventStoreInitializer.getAmazonDynamoDBClient().query(queryRequest);
I have read your case so for your desired output you should not create 'created' as Index bz it becomes key for the table and in keyCondition you can't use GT and LT operator.
you need to use FilterExpression and there you can use GT and LT for the 'created' field.
I'm using sqlite3.exe to execute queries against my DB, using the following code.
public static string QueryDB(string query)
{
string output = System.String.Empty;
string error = System.String.Empty;
System.Diagnostics.ProcessStartInfo startInfo = new System.Diagnostics.ProcessStartInfo();
startInfo.FileName = "C:\\sqlite\\sqlite3.exe";
startInfo.Arguments = "test.db " + query;
startInfo.UseShellExecute = false;
startInfo.CreateNoWindow = true;
startInfo.RedirectStandardError = true;
startInfo.RedirectStandardOutput = true;
startInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
try
{
using(System.Diagnostics.Process sqlite3 = System.Diagnostics.Process.Start(startInfo))
{
output = sqlite3.StandardOutput.ReadToEnd();
error = sqlite3.StandardError.ReadToEnd();
sqlite3.WaitForExit();
}
}
catch (System.Exception ex)
{
System.Console.WriteLine(ex.ToString());
return null;
}
return output;
}
I'm inserting data into a table, and I'd like it to return the id of the inserted data. Is there a way to get SQLite to do this?
For example, my query might look like this "INSERT INTO mytable (some_values) VALUES ('some value');". After this query is run, I'd like output to contain the rowid of the inserted data. Is there any way to do this (a command line switch, etc.)?
A possible workaround, is to run two commands against the DB. First insert the data, then get the last inserted row id. In which case, the query would look like this "\"INSERT INTO mytable (some_values) VALUES ('some value'); SELECT last_insert_rowid();\""
You should not use max(id) or similar function in DB.
In this specific case it can work, under the condition that you use ONE connection and ONE thread to write data to DB.
In case of multiple connections you can get wrong answer.
From version SQLite 3.35.0 it supports returning close in the insert statement (SQLite Returning Close)
create table test (
id integer not null primary key autoincrement,
val text
);
insert into table test(val) values (val) returning id;
Would you consider this:
select max(id) from your_table_name;
or embedded function last_insert_rowid()?
I would like to encrypt an existing database column with always encrypted. My project is a ASP.NET project using code first and database is SQL Server. The database has already data. I created a migration to achieve my goal.
First I tried to alter the column type, using the following.
ALTER TABLE [dbo].[TestDecrypted] ALTER COLUMN [FloatCol] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
I got the following error.
Operand type clash: float is incompatible with float encrypted with (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt')
Then I decided to created another column and migrate the data.
ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = [FloatCol]
And I got the same error.
After I looked at this, I noticed that it is possible to insert data like the following
DECLARE #floatCol FLOAT = 1.1
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #floatCol
But if I try to obtain the value from my existing column, it fails.
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
The error follows.
Encryption scheme mismatch for columns/variables '#floatCol'. The encryption scheme for the columns/variables is (encryption_type = 'PLAINTEXT') and the expression near line '4' expects it to be (encryption_type = 'RANDOMIZED', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'CEK_Auto1', column_encryption_key_database_name = 'TestEncrypt').
Does anyone knows how can I achieve my goal?
Update 1
#Nikhil-Vithlani-Microsoft did some interesting suggestions.
Always Encrypted Wizard in SSMS - I would like to achieve my goal with code first migrations, so this idea does not fit.
SqlBulkCopy - It does not work inside migrations, because the new column will only exist after all 'Up' method is run. Therefore we cannot insert data into this column in this way inside this method.
Anyway, his suggestions drove me to another attempt: obtain the decrypted values and update the encrypted column with them.
var values = new Dictionary<Guid, double>();
var connectionString = ConfigurationManager.ConnectionStrings["MainDb"].ConnectionString;
using (var sourceConnection = new SqlConnection(connectionString))
{
var myCommand = new SqlCommand("SELECT * FROM dbo.TestDecrypted", sourceConnection);
sourceConnection.Open();
using (var reader = myCommand.ExecuteReader())
{
while (reader.Read())
{
values.Add((Guid)reader["Id"], (double)reader["FloatCol"]);
}
}
}
Sql("ALTER TABLE [dbo].[TestDecrypted] ADD [FloatCol2] [float] ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL");
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
In fact, I did not try to create another column and to migrate the data, as mentioned in an example above. I tried it only on SSMS.
And now I got a different error.
Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
I tried to do it without encrypting the new column, and it worked properly.
Any idea why this error occurs?
You will have to do the always encrypted related migration outside of entity framework. This blog should help
https://blogs.msdn.microsoft.com/sqlsecurity/2015/08/27/using-always-encrypted-with-entity-framework-6/
If you want to encrypt an existing column, you can use Always Encrypted Wizard in SSMS, or use this article that explains how to migrate existing data.
Also, please note that doing bulk inserts through a C# (.NET 4.6.1+ client) app is supported.
You can do this in c# using SqlBulkCopy specifically using SqlBulkCopy.WriteToServer(IDataReader) Method.
Create a new table (encryptedTable) with the same schema as that of your plaintext table (unencryptedTable) but with the encryption turned on for the desired columns.
Do select * from unencryptedTable to load the data in a SqlDataReader then use SqlBulkCopy to load it to the encryptedTable using SqlBulkCopy.WriteToServer(IDataReader) Method
For example,
Plaintext Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) NOT NULL)
Encrypted Table
CREATE TABLE [dbo].[Patients](
[PatientId] [int] IDENTITY(1,1),
[SSN] [char](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (ENCRYPTION_TYPE = DETERMINISTIC,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256',
COLUMN_ENCRYPTION_KEY = CEK1) NOT NULL)
As for why your method does not work,
when you use parameterization for always encrypted, the right hand side (RHS) of the declare statement needs to be a literal. Because the driver will identify the literal and encrypt it for you. So, the following will not work, since RHS is a sql expression and cannot be encrypted by the driver
DECLARE #floatCol FLOAT = (SELECT TOP 1 FloatCol FROM TestDecrypted)
UPDATE [dbo].[TestDecrypted] SET FloatCol2 = #floatCol
Update:
The following code will not work because parameterization for Always Encrypted only applies to SSMS
foreach (var valuePair in values)
{
// The error occurs here
Sql($#"DECLARE #value FLOAT = {valuePair.Value}
UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #value WHERE Id = '{valuePair.Key}'");
}
However, if you rewrite your code as follows, that should work
foreach (var valuePair in values)
{
SqlCommand cmd = _sqlconn.CreateCommand();
cmd.CommandText = #"UPDATE [dbo].[TestDecrypted] SET [FloatCol2] = #FloatVar WHERE Id = '{valuePair.Key}'");";
SqlParameter paramFloat = cmd.CreateParameter();
paramFloat.ParameterName = #"#FloatVar";
paramFloat.DbType = SqlDbType.Float;
paramFloat.Direction = ParameterDirection.Input;
paramFloat.Value = floatValue;
cmd.Parameters.Add(paramFloat);
cmd.ExecuteNonQuery();
}
Hope that helps, if you have additional question, please leave them in the comments.
I am trying to work out the best method for querying all the values down a column. Should I set it as a GSI (it is currently). is it possible with a query or would I need to to do a scan?
Thanks for your help
You have to do a scan if you are not filtering by key. Here is the sample code to scan the index and get all the values.
List<String> categoryList = new ArrayList<>();
DynamoDB dynamoDB = new DynamoDB(dynamoDBClient);
Table table = dynamoDB.getTable("Music");
Index index = table.getIndex("Secondary Index Name");
ItemCollection<ScanOutcome> items = null;
ScanSpec scanSpec = new ScanSpec().withSelect(Select.SPECIFIC_ATTRIBUTES).withAttributesToGet("Category");
items = index.scan(scanSpec);
Iterator<Item> pageIterator = items.iterator();
while (pageIterator.hasNext() ) {
categoryList.add(pageIterator.next().getString("Category"));
}
I am using linq to Entity to retrieve data from to different tables by joining them, but I also want to group them by the field problemDesc in order to get rid of unnecessary duplicate entries for the same problem.
here is the code:
using (AssistantEntities context = new AssistantEntities())
{
var problems = context.tblProblems;
var customers = context.tblCustomers;
var query =
from problem in problems
join customer in customers
on problem.CustID equals customer.custID
where problem.IsActive == true
orderby customer.isMonthlyService == true descending
select new
{
problemID = problem.problemID,
ProblemCreateDate = problem.ProblemCreateDate,
CustID = problem.CustID,
name = customer.name,
isMonthlyService = customer.isMonthlyService,
StationName = problem.StationName,
problemDesc = problem.problemDesc,
LogMeIn = problem.LogMeIn
};
return query.ToList();
}
I am doing query.toList() in order to use that list in a gridview as a dataSource.
and if it possible, also add a field that count the duplicate problems.
You have plenty of examples in the following link.
LINQ - Grouping Operators