Consider a sample application where we need to store Customer name as primary key and list of his transactions as value in Berkeley db. Do berkeley support storing list of values against a primary key? If so, how to approach designing the entity?
The most common approach would be to save the records as a data storage type, such as XML (or JSON, if you prefer that sort of thing). Here's an example in C# that I came up with that stores the records as JSON objects in the database:
static void Main(string[] args)
{
var db = BTreeDatabase.Open("customers.db", new BTreeDatabaseConfig {Creation = CreatePolicy.ALWAYS});
for (int i = 0; i < 10000; i++)
{
var customer = new Customer
{
customerName = nameGenerator.GenerateRandomFirstAndLastName()
};
var list = new List<Transaction>();
for (int j = 0; j < 10; j++)
{
list.Add(new Transaction
{txAmount = random.NextDouble(), txDate = DateTime.Now, txId = randomString()});
}
customer.transactions = list;
db.Put(new DatabaseEntry(Encoding.Default.GetBytes(customer.customerName)),
new DatabaseEntry(Encoding.Default.GetBytes(JsonConvert.SerializeObject(customer))));
}
WriteLine("Completed!");
ReadLine();
}
Granted, this isn't the most efficient way, since I store the customer name twice, but it should be a good starting point.
Related
I am looking to retrieve the items from my Cosmos Db collection and I have a list of Ids which I want to retrieve. all code are running in an Azure Function and below is the code where I get all Ids in List filteredResult where I get list of all Ids. I am looking the best way to complete this code to retrieve all items from my Cosmos Db collection considering 30-40 Ids at a time.
public static void Run([ServiceBusTrigger("testSB", "SubscriberName", Connection = "AzureServiceBusString")] string mySbMsg,
[CosmosDB(
databaseName: "DBName",
collectionName: "CollectionName",
ConnectionStringSetting = "CosmosDBConnection")] DocumentClient client,
ILogger log)
{
try {
log.LogInformation($"C# ServiceBus topic trigger function processed message: {mySbMsg}");
var jsonSerializerSettings = new JsonSerializerSettings();
jsonSerializerSettings.MissingMemberHandling = MissingMemberHandling.Ignore;
List<MyItem> lists = JsonConvert.DeserializeObject<List<MyItem>>(mySbMsg, jsonSerializerSettings);
List<string> filteredResult = (from s in lists
where s.DocType == "TEST"
select s.Id).ToList();
}
This issue has recently been covered under Microsoft Q&A forum. Posting the answer here again for broader reach.
This code can be used after you created the 'filteredResult' (just replace the 'input' list with 'filteredResult')
List<string> input = new List<string>();
input.Add("1");
input.Add("2");
input.Add("3");
var option = new FeedOptions { EnableCrossPartitionQuery = true };
IQueryable<Family> queryable = client.CreateDocumentQuery<Family>(UriFactory.CreateDocumentCollectionUri("families", "items").ToString(), "SELECT * FROM books where books.id IN " + "('" + string.Join( "','", input) + "')",option);
List<Family> posts = queryable.ToList();
Console.WriteLine("Read count = {0}", posts.Count);
Also notice I created a model class for document properties as below:
public class Family
{
public int id;
public string city;
}
is there a way to remove primary key from the datatable Or is there any way to remove the constraints of "PK" first and then remove the column itself?
Thanks!
UPDATED:
dtTable.Columns.Add(new System.Data.DataColumn("PRIMARY_KEY", typeof(System.Int32)));
dtTable.PrimaryKey = new DataColumn[1] { dtTable.Columns["PRIMARY_KEY"] }; // throws an error
dtTable.Columns["PRIMARY_KEY"].AutoIncrement = true;
You can remove primay key using
DataTable.PrimaryKey = null;
you can delete data table column using
DataTable.Columns.Remove("column name here");
I found that sometimes after removing PrimaryKey from a DataTable:
MyDataTable.PrimaryKey = null;
the Unique setting remains true on the member columns of the deleted PrimaryKey.
My solution:
public static void KillPrimaryKey(DataTable LocDataTable)
{
int LocPriKeyCount = LocDataTable.PrimaryKey.Length;
string[] PrevPriColumns = new string[LocPriKeyCount];
// 1. Store ColumnNames in a string Array
for (int ii = 0; ii < LocPriKeyCount; ii++) PrevPriColumns[ii] = LocDataTable.PrimaryKey[ii].ColumnName;
// 2. Clear PrimaryKey
LocDataTable.PrimaryKey = null;
// 3. Clear Unique settings
for (int ii = 0; ii < LocPriKeyCount; ii++) LocDataTable.Columns[PrevPriColumns[ii]].Unique = false;
}
I can't seem to get this right.
I want to do a scan of a table and only return records where a particular field does not exist.
I've tried the following two things:
HashMap<String, Condition> scanFilter = new HashMap();
Condition scanFilterCondition = new Condition().withComparisonOperator(ComparisonOperator.NULL.toString());
scanFilter.put("field", scanFilterCondition);
ScanRequest scan = new ScanRequest()
.withTableName("table name")
.withScanFilter(scanFilter)
etc
and
ScanRequest scan = new ScanRequest()
.withTableName("table")
.withFilterExpression("attribute_not_exists(attributeName)")
.withLimit(100)
etc
However they return no records (and most records are missing this field). Note, that if I remove the filter the scan does return and process all records as expected so the basic query is correct.
How do I do this?
EDIT added full method in case it helps
// Get information on the table so that we can set the read capacity for the operation.
List<String> tables = client.listTables().getTableNames();
String tableName = tables.stream().filter(table -> table.equals(configuration.getTableName())).findFirst().get();
if(Strings.isNullOrEmpty(tableName))
return 0;
TableDescription table = client.describeTable(tableName).getTable();
//Set the rate limit to a third of the provisioned read capacity.
int rateLimit = (int) (table.getProvisionedThroughput().getReadCapacityUnits() / 3);
RateLimiter rateLimiter = RateLimiter.create(rateLimit);
// Track how much throughput we consume on each page
int permitsToConsume = 1;
// Initialize the pagination token
Map<String, AttributeValue> exclusiveStartKey = null;
int count = 1;
int writtenCount = 0;
do {
// Let the rate limiter wait until our desired throughput "recharges"
rateLimiter.acquire(permitsToConsume);
//We only want to process records that don't have the field key set.
HashMap<String, Condition> scanFilter = new HashMap<>();
Condition scanFilterCondition = new Condition().withComparisonOperator(ComparisonOperator.NULL.toString());
scanFilter.put("field", scanFilterCondition);
ScanRequest scan = new ScanRequest()
.withTableName(configuration.getNotificationsTableName())
.withScanFilter(scanFilter)
.withLimit(100)
.withReturnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
.withExclusiveStartKey(exclusiveStartKey);
ScanResult result = client.scan(scan);
exclusiveStartKey = result.getLastEvaluatedKey();
// Account for the rest of the throughput we consumed,
// now that we know how much that scan request cost
double consumedCapacity = result.getConsumedCapacity().getCapacityUnits();
permitsToConsume = (int)(consumedCapacity - 1.0);
if(permitsToConsume <= 0) {
permitsToConsume = 1;
}
// Process results here
} while (exclusiveStartKey != null);
The NULL condition seems to be fine. You need to do recursive search using Scan. The Dynamodb scan doesn't scan the whole database in one go. It scans the data based on the amount of consumed provisioned throughput.
Sample code to perform scan in loop based on LastEvaluatedKey:-
ScanResult result = null;
do {
HashMap<String, Condition> scanFilter = new HashMap<>();
Condition scanFilterCondition = new Condition().withComparisonOperator(ComparisonOperator.NULL);
scanFilter.put("title", scanFilterCondition);
ScanRequest scanRequest = new ScanRequest().withTableName(tableName).withScanFilter(scanFilter);
if (result != null) {
scanRequest.setExclusiveStartKey(result.getLastEvaluatedKey());
}
result = dynamoDBClient.scan(scanRequest);
LOGGER.info("Number of records ==============>" + result.getItems().size());
for (Map<String, AttributeValue> item : result.getItems()) {
LOGGER.info("Movies ==================>" + item.get("title"));
}
} while (result.getLastEvaluatedKey() != null);
NULL : The attribute does not exist. NULL is supported for all data
types, including lists and maps. Note This operator tests for the
nonexistence of an attribute, not its data type. If the data type of
attribute "a" is null, and you evaluate it using NULL, the result is a
Boolean false. This is because the attribute "a" exists; its data type
is not relevant to the NULL comparison operator.
LastEvaluatedKey The primary key of the item where the operation
stopped, inclusive of the previous result set. Use this value to start
a new operation, excluding this value in the new request.
If LastEvaluatedKey is empty, then the "last page" of results has been
processed and there is no more data to be retrieved.
If LastEvaluatedKey is not empty, it does not necessarily mean that
there is more data in the result set. The only way to know when you
have reached the end of the result set is when LastEvaluatedKey is
empty.
var expr = new Expression();
expr.ExpressionStatement = "contains(#Name, :Name) and attribute_not_exists(#FullName) or #FullName = :FullName";
expr.ExpressionAttributeNames["#FullName"] = "FullName";
expr.ExpressionAttributeValues[":FullName"] = "sumit singh";
expr.ExpressionAttributeNames["#Name"] = "Name";
expr.ExpressionAttributeValues[":Name"] = "sumit singh";
ScanOperationConfig config = new ScanOperationConfig()
{
Limit = 2,
PaginationToken = "{}",
//Filter = filter,
FilterExpression = expr,
//AttributesToGet = attributesToGet,
//Select = SelectValues.SpecificAttributes,
TotalSegments = 1
};
var item = _tableContext.FromScanTableAsync(config);
do
{
documentList.AddRange(await item.GetNextSetAsync());
} while (!item.IsDone);
I have in DB table Called Students We make some thing like lottery to know which students are going to be the winner in some thing like game in a school
public static List<Students> GetAll()
{
List<Students> list = new List<Students>();
var query = from c in new HsInternetDBDataContext().Students
orderby c.DateCreated ascending
select c;
foreach (DAL.Student dr in query.ToList())
{
l.Add(new Quotes(dr));
}
return list;
}
what i want to do is to retrieve random student from this table
the same question are posted in another blog and i quote the answer when i post it :
public static Quotes GetRandomStudents()
{
HsInternetDBDataContext db = new HsInternetDBDataContext();
var query = from c in db.Students select c;
int count = query.Count();
if (query.Count() > 0)
{
Random r = new Random();
return new Students(query.ToList()[r.Next(0, count)]);
}
else
{
return new Students();
}
}
Hope that helps because it works with another one has the same situation
I'm doing a little bit of planning of an application that uses Mondrian OLAP engine with Olap4j and should present/display data to user. I understand all the back-end stuff, but I'm not sure how should I display the data in the view layer.
For example olap4j has a formatter that prints the SELECT nicely into the console.
How is the data that I get from olap4j displayed in view layer ? I just went through the olap4j API, and there doesn't seem to be anything for getting the result in a form that can be somehow further processed and displayed. Is this process part of the Pentaho solution ? So that otherwise it is really not easy to present data just from Mondrian OLAP engine and olap4j ?
EDIT: I'm used to traditionally get some data from a database into my DTO and display it in view layer. But how do I create DTOs for such a complicated result set ?
You can create your own view layer it's just a little bit tricky.
OlapStatement.executeOlapQuery() returns a CellSet, you will have to work with that. Also read the specifications, it's a good source of information.
Here is an example, that creates List<List<MyCell>> (not the best representation but it's easy to undarstand how it works). This creates a table similar to http://www.olap4j.org/api/index.html?org/olap4j/Position.html (without the "Gender" and "Product" labels).
private final static int COLUMNS = 0; //see Cellset javadoc
private final static int ROWS= 1; //see Cellset javadoc
/**
* Outer list: rows, inner list: elements in a row
*/
private List<List<MyCell>> getListFromCellSet(CellSet cellSet) {
List<List<MyCell>> toReturn= new ArrayList<List<MyCell>>();
//Column header
//See http://www.olap4j.org/api/index.html?org/olap4j/Position.html on how Position works, it helps a lot
//Every position will be a column in the header
for (Position pos : cellSet.getAxes().get(COLUMNS).getPositions()) {
for (int i = 0; i < pos.getMembers().size(); i++) {
if (toReturn.size() <= i) {
toReturn.add(i, new ArrayList<MyCell>());
}
Member m = pos.getMembers().get(i);
MyCell myCell = new MyCell(m); //use m.getCaption() for display
toReturn.get(i).add(myCell );
}
}
//Put empty elements to the beginning of the list, so there will be place for the rows header
if (cellSet.getAxes().get(ROWS).getPositions().size() > 0) {
for (int count=0; count < cellSet.getAxes().get(1).getPositions().get(0).getMembers().size(); count++) {
for (int i = 0; i < toReturn.size(); i++) {
toReturn.get(i).add(0, new MyCell());
}
}
}
//Content + row header
for(int i = 0; i < cellSet.getAxes().get(ROWS).getPositionCount(); i++) {
List<MyCell> row = new ArrayList<MyCell>();
//Header
for (org.olap4j.metadata.Member m : cellSet.getAxes().get(ROWS).getPositions().get(i).getMembers()) {
row.add(new MyCell(m));
}
//Content
for (int j = 0; j < cellSet.getAxes().get(COLUMNS).getPositionCount(); j++) {
ArrayList<Integer> list = new ArrayList<Integer>();
list.add(j); //coordinte
list.add(i); //coordinte
row.add(new MyCell(cellSet.getCell(list))); //use cell.getFormattedValue() for display
}
toReturn.add(row);
}
return toReturn;
}
Create the MyCell class with these constructors:
public class MyCell {
...
public MyCell(){...}
public MyCell(Member m){...}
public MyCell(Cell c){...}
}
Don't forget to display the filters, use Cellset.getFilterAxis() for that.
You can also check the Rectangular formatter on SourceForge, but it's a bit longer.