I have following Persistable classes with 1-to-N relationship.
#PersistenceCapable
public class Pet {
#Persistent(primaryKey = "true", valueStrategy = IdGeneratorStrategy.IDENTITY)
Long id;
#Persistent
String name;
#Element(column = "PET_ID")
List<Photo> photos;
// getters and setters
and
#PersistenceCapable
public class Photo {
#Persistent(primaryKey = "true", valueStrategy = IdGeneratorStrategy.IDENTITY)
Long id;
#Persistent
String desc;
#Persistent(serialized="true")
Object image;
// getters and setters
// hash and equal using field id
Field List photos uses FK to establish 1-N relationship between Pet (1) and Photo (N). Field Object image in Photo is a serialized to hold the image object.
For datastore operations I use PetDao, which has following methods
public final static PersistenceManagerFactory pmf = JDOHelper
.getPersistenceManagerFactory("datastore");
public void storePet(Pet pet) {
// get PM and current tx
try {
tx.begin();
pm.makePersistent(pet);
tx.commit();
} catch (Exception e) {
// rollback and close pm
}
}
public void storePhoto(Long petId, Photo photo) {
// get PM and current tx
try {
tx.begin();
Pet pet = pm.getObjectById(Pet.class,petId);
pet.addPhoto(photo);
tx.commit();
} catch (Exception e) {
// rollback and close pm
}
}
I create and persist objects as
Pet pet = new Pet();
pet.setName("Nicky");
Photo photo = new Photo();
photo.setDesc("Photo 1");
photo.setImage(new Image("image 1"));
pet.addPhoto(photo);
.... add photo 2 and photo 3
PetDao petDao = new PetDao();
petDao.storePet(pet);
// i have one more photo so add it directly
photo = new Photo();
photo.setDesc("Photo 4");
photo.setImage(new Image ("image 4"));
petDao.storePhoto((long)0, photo);
Everything persists as required and datastore ends up with 1 pet in PET table and 4 photo in PHOTO table.
But when I analyze the DataNucleus log for petDao.storePhoto((long)0, photo) code, I see that DataNucleus retrieves all the image objects from datastore.
Native [DEBUG] INSERT INTO PHOTO ("DESC",IMAGE,PET_ID,PHOTOS_INTEGER_IDX) VALUES (<'Photo 4'>,<UNPRINTABLE>,<0>,<3>)
Persist [DEBUG] Execution Time = 70 ms (number of rows = 1) on PreparedStatement "org.datanucleus.store.rdbms.ParamLoggingPreparedStatement#190a0d6"
Persist [DEBUG] Object "in.m.pet.Photo#10deb5f" was inserted in the datastore and was given strategy value of "3"
Native [DEBUG] SELECT A0.IMAGE FROM PHOTO A0 WHERE A0.ID = <1>
Retrieve [DEBUG] Execution Time = 1 ms
Native [DEBUG] SELECT A0.IMAGE FROM PHOTO A0 WHERE A0.ID = <0>
Retrieve [DEBUG] Execution Time = 0 ms
Native [DEBUG] SELECT A0.IMAGE FROM PHOTO A0 WHERE A0.ID = <2>
Retrieve [DEBUG] Execution Time = 0 ms
After adding the "Photo 4" using INSERT INTO PHOTO... statement, DataNucleus retrieves the earlier three image Objects by firing 3 SELECT IMAGE FROM PHOTO statements. These retrieves may be quite large as number of image objects increases, resulting in unnecessary load on datastore impacting the performance.
Same thing happens if I select the pet using pm.getObjectById() and detach the Pet object and add photo to detached object and then attach it back to object graph with pm.makePersistent(pet). FetchGroup is as follows
#PersistenceCapable(detachable="true")
#FetchGroup(name="detachPhotos", members={#Persistent(name="photos")})
public class Pet {
....
}
and detach pet with fetchgroup
public Pet getPet(Long id){
PersistenceManager pm = pmf.getPersistenceManager();
pm.getFetchPlan().addGroup("detachPhotos");
Pet pet = pm.getObjectById(Pet.class, id);
return pm.detachCopy(pet);
}
My question is how to avoid these unnecessary retrials of Object image from datastore.
One more observation: if i call petDao.storePhoto((long)0, photo) from another app or use a separate instance of PMF in PetDao.storePhoto method, then DataNucleus will not fire the SELECT to retrieve image objects.
If a 1-N relationship can get large, you may want to consider mapping it relationally, i.e. instead of mapping Pet.photos, map Photo.pet. This will prevent you from navigating from Pet to Photo in a OO fashion without a query, but will prevent the SQL statements you are concerned about.
You storePhoto would then look like below and the 1-N would not be fetched.
public void storePhoto(Photo photo) {
// get PM and current tx
try {
tx.begin();
pm.makePersistent(photo); // assuming pet was already set
tx.commit();
} catch (Exception e) {
// rollback and close pm
}
}
Got answer in DataNucleus Performance Tuning, which recommends to set datanucleus.persistenceByReachabilityAtCommit=false if reach-ability is not needed by the app. Setting this as false, resolves the image retrieval issue without any other side effect with Pet/Photo.
To quote from DN doc
DataNucleus verifies if newly persisted objects are memory reachable
on commit, if they are not, they are removed from the database. This
process mirrors the garbage collection, where objects not referenced
are garbage collected or removed from memory. Reachability is
expensive because it traverses the whole object tree and may require
reloading data from database. If reachability is not needed by your
application, you should disable it.
Check whether reach-ability is required by your app before setting it to false.
Related
I'm trying to work with DynamoDb streams, I am using the example code shown in this article. I've modified it to work in a basic Spring Boot app (initializr), utilizing an existing DynamoDb table which has streams enabled. Everything appears to work, however; I'm not seeing any new updates.
This particular database has a bulk update once per day at a specific time, it may get some minor changes now and then during the day. I'm trying to monitor these minor updates. When I run the application I can see the records from the bulk update, however if my application is running and I use the AWS Console to modify, create or delete a record I don't seem to get any output.
I'm using:
Spring Boot:2.3.9.RELEASE
amazon-kinesis-client:1.14.2
Java 11
Running on Mac Catalina (though that shouldn't matter)
In my test application I did the following:
package com.test.dynamodb_streams_test_kcl.service;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreams;
import com.amazonaws.services.dynamodbv2.model.*;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.util.List;
#Slf4j
#Service
#RequiredArgsConstructor
public class LowLevelKclProcessor {
private static final String dynamoDbTableName = "global-items";
private final AmazonDynamoDB dynamoDB;
private final AmazonDynamoDBStreams dynamoDBStreams;
private final ZonedDateTime startTime = ZonedDateTime.now();
#PostConstruct
public void initialize() {
log.info("Describing table={}", dynamoDbTableName);
DescribeTableResult itemTableDescription = dynamoDB.describeTable(dynamoDbTableName);
log.info("Got description");
String itemTableStreamArn = itemTableDescription.getTable().getLatestStreamArn();
log.info("Got stream arn ({}) for table={} tableArn={}", itemTableStreamArn,
itemTableDescription.getTable().getTableName(), itemTableDescription.getTable().getTableArn());
// Get all the shard IDs from the stream. Note that DescribeStream returns
// the shard IDs one page at a time.
String lastEvaluatedShardId = null;
do {
DescribeStreamResult describeStreamResult = dynamoDBStreams.describeStream(
new DescribeStreamRequest()
.withStreamArn(itemTableStreamArn)
.withExclusiveStartShardId(lastEvaluatedShardId));
List<Shard> shards = describeStreamResult.getStreamDescription().getShards();
// Process each shard on this page
for (Shard shard : shards) {
String shardId = shard.getShardId();
System.out.println("Shard: " + shard);
// Get an iterator for the current shard
GetShardIteratorRequest getShardIteratorRequest = new GetShardIteratorRequest()
.withStreamArn(itemTableStreamArn)
.withShardId(shardId)
.withShardIteratorType(ShardIteratorType.LATEST);
GetShardIteratorResult getShardIteratorResult =
dynamoDBStreams.getShardIterator(getShardIteratorRequest);
String currentShardIter = getShardIteratorResult.getShardIterator();
// Shard iterator is not null until the Shard is sealed (marked as READ_ONLY).
// To prevent running the loop until the Shard is sealed, which will be on average
// 4 hours, we process only the items that were written into DynamoDB and then exit.
int processedRecordCount = 0;
while (currentShardIter != null && processedRecordCount < 100) {
System.out.println(" Shard iterator: " + currentShardIter.substring(380));
// Use the shard iterator to read the stream records
GetRecordsResult getRecordsResult = dynamoDBStreams.getRecords(new GetRecordsRequest()
.withShardIterator(currentShardIter));
List<Record> records = getRecordsResult.getRecords();
for (Record record : records) {
// I set a breakpoint on the line below, but it was never hit after the bulk update info
if (startTime.isBefore(ZonedDateTime.ofInstant(record.getDynamodb()
.getApproximateCreationDateTime().toInstant(), ZoneId.systemDefault()))) {
System.out.println(" " + record.getDynamodb());
}
}
processedRecordCount += records.size();
currentShardIter = getRecordsResult.getNextShardIterator();
}
}
// If LastEvaluatedShardId is set, then there is
// at least one more page of shard IDs to retrieve
lastEvaluatedShardId = describeStreamResult.getStreamDescription().getLastEvaluatedShardId();
} while (lastEvaluatedShardId != null);
}
}
Note that your test is based on the low-level API, not on the Kenisis client library. So it's normal to have some tricky technical details to deal with.
Your test application has some similarities with the example given in the doc, but it has issues:
When I run the application I can see the records from the bulk update
ShardIteratorType.LATEST will not look for old records that happened before running the test (It starts reading just after the most recent stream records in the shard)
So, I will assume that the iterator type was different (ex: TRIM_HORIZON) and changed later to LATEST during your tests.
The main issue comes from the fact that your application will sequentially poll shards, and it will bloque in the first shard until it finds 100 new records in this shard (due to LATEST iterator type).
So, you may not see the new minor changes while the test is running if they belong to a different shard.
Solutions:
1- Poll shards in parallel using threads.
2- Filter returned shards using the sequence number of the last logged record, and try to guess the shard that may contain minor changes.
3- Dangerous & I'm not sure if it works :)
In a test table, and if your data model allows this: close the current stream, and enable a new one, then make sure that all your writes belong to one partition. In the majority of cases, table partitions have a one-to-one relationship with active shards. Theoretically, you have only one active shard to deal with.
I just started using Microsoft Exchange Web Services for the first time. Want I want to be able to do is the following:
Create Meeting
Update Meeting
Cancel/Delete Meeting
These meetings are created in an ASP.NET MVC application and saved into a SQL Server database. I simply wish to integrate this with the on site Exchange Server. So far, I'm able to created my meeting with the following code:
public static Task<string> CreateMeetingAsync(string from, List<string> to, string subject, string body, string location, DateTime begin, DateTime end)
{
var tcs = new TaskCompletionSource<string>();
try
{
ExchangeService service = new ExchangeService(ExchangeVersion.Exchange2013);
service.Credentials = CredentialCache.DefaultNetworkCredentials;
//service.UseDefaultCredentials = true;
// I suspect the Service URL needs to be set from the user email address because this is then used to set the organiser
// of the appointment constructed below. The Organizer is a read-only field that cannot be manually set. (security measure)
service.AutodiscoverUrl(from);
//service.Url = new Uri(WebConfigurationManager.AppSettings["ExchangeServer"]);
Appointment meeting = new Appointment(service);
meeting.Subject = subject;
meeting.Body = "<span style=\"font-family:'Century Gothic'\" >" + body + "</span><br/><br/><br/>";
meeting.Body.BodyType = BodyType.HTML;
meeting.Start = begin;
meeting.End = end;
meeting.Location = location;
meeting.ReminderMinutesBeforeStart = 60;
foreach (string attendee in to)
{
meeting.RequiredAttendees.Add(attendee);
}
meeting.Save(SendInvitationsMode.SendToAllAndSaveCopy);
tcs.TrySetResult(meeting.Id.UniqueId);
}
catch (Exception ex)
{
tcs.TrySetException(ex);
}
return tcs.Task;
}
This successfully creates my meeting, places it into the user's calendar in outlook and sends a meeting request to all attendees. I noticed the following exception when attempting to call meeting.Save(SendInvitationsMode.SendToAllAndSaveCopy); twice:
This operation can't be performed because this service object already has an ID. To update this service object, use the Update() method instead.
I thought: Great, it saves the item in exchange with a unique id. I'll save this ID in my application's database and use it later to edit/cancel the meeting. That is why I return the id: tcs.TrySetResult(meeting.Id.UniqueId);
This is saved nicely into my application's database:
Now, I am attempting to do the next part where I update the meeting, but I cannot find a way to search for the item based on the unique identifier that I'm saving. An example I found on code.msdn uses the service.FindItems() method with a query that searches the subject:
string querystring = "Subject:Lunch";
FindItemsResults<Item> results = service.FindItems(WellKnownFolderName.Calendar, querystring, view);
I don't like this. There could be a chance that the user created a meeting outside of my application that coincidentally has the same subject, and here come's my application and cancel's it. I tried to determine wether it's possible to use the unique id in the query string, but this does not seem possible.
I did notice on the above query string page that the last property you can search on is (property is not specified) that searches in "all word phase properties.". I tried thus simply putting the id into the query, but this returns no results:
FindItemsResults<Item> results = service.FindItems(WellKnownFolderName.Calendar, "AAMkADJhZDQzZWFmLWYxYTktNGI1Yi1iZTA5LWVmOTE3MmJiMGIxZgBGAAAAAAAqhOzrXRdvRoA6yPu9S/XnBwDXvBDBMebkTqUWix3HxZrnAAAA2LKLAAB5iS34oLxkSJIUht/+h9J1AAFlEVLAAAA=", view);
Use the Appointment.Bind static function, providing a service object and the ItemId saved in your database. Be aware with meeting workflow (invite, accept, reject) can re-create a meeting on the same calendar with a new ItemId. But if you are just looking at the meeting you make on your own calendar, you should be OK.
When creating a user, entries are required in multiple tables. I am trying to create a transaction that creates a new entry into one table and then pass the new entityid into the parent table and so on. The error I am getting is
The transaction manager has disabled its support for remote/network
transactions. (Exception from HRESULT: 0x8004D024)
I believe this is caused by creating multiple connections within a single TransactionScope, but I am unsure on what the best/most efficient way of doing this is.
[OperationBehavior(TransactionScopeRequired = true)]
public int CreateUser(CreateUserData createData)
{
// Create a new family group and get the ID
var familyGroupId = createData.FamilyGroupId ?? CreateFamilyGroup();
// Create the APUser and get the Id
var apUserId = CreateAPUser(createData.UserId, familyGroupId);
// Create the institution user and get the Id
var institutionUserId = CreateInsUser(apUserId, createData.AlternateId, createData.InstitutionId);
// Create the investigator group user and return the Id
return AddUserToGroup(createData.InvestigatorGroupId, institutionUserId);
}
This is an example of one of the function calls, all the other ones follow the same format
public int CreateFamilyGroup(string familyGroupName)
{
var familyRepo = _FamilyRepo ?? new FamilyGroupRepository();
var familyGroup = new FamilyGroup() {CreationDate = DateTime.Now};
return familyRepo.AddFamilyGroup(familyGroup);
}
And the repository call for this is as follows
public int AddFamilyGroup(FamilyGroup familyGroup)
{
using (var context = new GameDbContext())
{
var newGroup = context.FamilyGroups.Add(familyGroup);
context.SaveChanges();
return newGroup.FamilyGroupId;
}
}
I believe this is caused by creating multiple connections within a single TransactionScope
Yes, that is the problem. It does not really matter how you avoid that as long you avoid it. A common thing to do is to have one connection and one EF context per WCF request. You need to find a way to pass that EF context along.
The method AddFamilyGroup illustrates a common anti-pattern with EF: You are using EF as a CRUD facility. It's supposed to me more like a live object graph connected to the database. The entire WCF request should share the same EF context. If you move in that direction the problem goes away.
I am creating a java fx application for openfire chat client.
i am using smack 4.1 rc1 to connect to the server.
i am able to to connect to server send presence information to others and send messages to other users as well.
however i am not able to iterate through the roster.
when i get roster object and debug it its shows a hash map of 3 roster entries that means the roster is getting loaded in roster object. however when i use roster.getentries method to store it into the Collection of roster entries it shows 0 object. even the roster.getentriescount() method returns 0 though i can see the roster user names in the debug view
try {
config = XMPPTCPConnectionConfiguration.builder()
.setUsernameAndPassword(mUserName+ "#" + Domain, mPassword)
.setServiceName(HostName)
.setHost(HostName)
.setPort(PortName)
.setResource(Resource)
.setSecurityMode(ConnectionConfiguration.SecurityMode.disabled)
.build();
mXmppConnection = new XMPPTCPConnection(config);
mXmppConnection.connect();
mXmppConnection.login();
// Presence presence=new Presence();
Presence presence ;
if(mPresence) presence = new Presence(Presence.Type.available);
else presence = new Presence(Presence.Type.unavailable);
presence.setStatus("On Smack");
XMPPConnection conn=(XMPPConnection) mXmppConnection;
Chat chat = ChatManager.getInstanceFor(mXmppConnection).createChat
("monika#ipaddress");
chat.sendMessage("Howdy from smack!");
// Send the packet (assume we have a XMPPConnection instance called "con").
mXmppConnection.sendPacket(presence);
System.out.println("Connected successfully");
Roster roster = Roster.getInstanceFor(conn);
Collection<RosterEntry> entries = roster.getEntries();
int i=0;
for (RosterEntry entry : entries) {
System.out.println(entry);
i++;
}
System.out.println("Rosters Count - "+ i+ roster.getEntryCount());
has any one encountered the same problem before?
You may have to check if roster is loaded before calling getEntries.
Roster roster = Roster.getInstanceFor(connection);
if (!roster.isLoaded())
roster.reloadAndWait();
Collection <RosterEntry> entries = roster.getEntries();
thanks for Deepak Azad
Here full code
public void getRoaster(final Callback<Collection<RosterEntry>> callback) {
final Roster roster = Roster.getInstanceFor(connection);
if (!roster.isLoaded())
try {
roster.reloadAndWait();
} catch (SmackException.NotLoggedInException | SmackException.NotConnectedException | InterruptedException e) {
e.printStackTrace();
}
Collection<RosterEntry> entries = roster.getEntries();
for (RosterEntry entry : entries) {
android.util.Log.d(AppConstant.PUBLIC_TAG, entry.getName());
}
}
I have just solved this problem. I'm using OpenFire as XMPP server. I checked the field "Subscription" in the users in roster and it was "None". After change it by "Both", it worked, and entries are being fetched.
Hope it helps!
Using Async-based Webservice and Async framework in WinRT (Win8) to get a large recordsets(1000 to 5000) from a remote Ms SQL Server.
I want to know :
1) Which is the best approach to handle to insert large recordsets into SQLite?
2) Using RollBack transaction will start all over again if there is connection error. The below method will insert whatever and I can update the data later if the records are not complete. Is this a good approach?
3) Any better way to enhance my below solution?
This foreach statement to handle
each reords in returned result which returned from Async-Based WebService:
foreach (WebServiceList _List in IList)
{
InsertNewItems(_List.No, _List.Description, _List.Unit_Price, _List.Base_Unit_of_Measure);
}
private void InsertNewItems(string ItemNo, string ItemName, decimal ItemPrice, string ItemBUoM)
{
var existingItem = (db2.Table().Where(c => c.No == ItemNo)).SingleOrDefault();
if (existingItem != null)
{
existingItem.No = ItemNo;
existingItem.Description = ItemName;
existingItem.Unit_Price = ItemPrice;
existingItem.BaseUnitofMeasure = ItemBUoM;
int success = db2.Update(existingItem);
}
else
{
int success = db2.Insert(new Item()
{
No = ItemNo,
Description = ItemName,
Unit_Price = ItemPrice,
Base_Unit_of_Measure = ItemBUoM
});
}
}
You should use RunInTransaction from sqlite-net. The documentation for it says,
Executes action within a (possibly nested) transaction by wrapping it
in a SAVEPOINT. If an exception occurs the whole transaction is rolled
back, not just the current savepoint. The exception is rethrown.
using (var db = new SQLiteConnection(DbPath))
{
db.RunInTransaction(() =>
{
db.InsertOrReplace(MyObj);
});
}
Wiki article for Transactions at GitHub
The most important performance aspect for bulk inserts is to use a single transaction. If you want to handle aborts, I suggest that you feed the data in sufficiently large parts and restart from that point on next time. An SQL transaction either finishes completely or rolls back completely, so unless the input data changes between two runs, there should be no need to do an insert-or-update.
See, for example, here for a discussion of SQLite bulk insert performance using different methods.