I want to back up rocksdb while not blocking my writing calls, the status I expected is rocksdb should back up the data at that moment and ignoring the writing calls while backing up, is there a proper way to accomplish this?
I tested this in java using rocksdb-jni:
RocksDB.loadLibrary();
RocksDB rocksDb = RocksDB.open("C:/rocksdb/data");
BackupEngine engine = BackupEngine.open(rocksDb.getEnv(), new BackupEngineOptions("C:/rocksdb/backup"));
for(int i = 0;i < 100000;i++){
byte[] k = ByteBuffer.allocate(4).putInt(0, i).array();
rocksDb.put(k,k);
}
ExecutorService executorService = Executors.newFixedThreadPool(2);
CountDownLatch countDownLatch = new CountDownLatch(1);
executorService.execute(() -> {
try {
Thread.currentThread().setPriority(Thread.MAX_PRIORITY);
countDownLatch.await();
engine.createNewBackup(rocksDb, false);
} catch (RocksDBException | InterruptedException e) {
throw new RuntimeException(e);
}
});
executorService.execute(() -> {
try {
Thread.currentThread().setPriority(Thread.NORM_PRIORITY);
countDownLatch.await();
//Thread.sleep(1);
for(int i = 100000;i < 200000;i++){
byte[] k = ByteBuffer.allocate(4).putInt(0, i).array();
rocksDb.put(k,k);
}
} catch (InterruptedException | RocksDBException e) {
throw new RuntimeException(e);
}
});
countDownLatch.countDown();
the result I expected from restoring is exactly 100000 records, but I always got more records
Yes - both backup and checkpoint engines can achieve this i.e. take a live backup while not blocking writes
https://github.com/facebook/rocksdb/wiki/How-to-backup-RocksDB
Related
I am creating a Javafx chat app which also allows for file transfer. My issue is I open a FileOutputStream for the received file within the below method. I can see my listener.statusTransferring() updating the UI only if I enable Platform.runLater. I think I now need to enable the same on the fos.write(b, 0, tmpTransferred) within the while loop but don't know how to do this. I have tried unsuccessfully wrapping the whole method within Platform runlater. Note: If I don't use platform runlater I don't get any errors however the UI does not update until the file transfer is complete eg listener.statusCompleted() is called;. The error I get now as a result of the fos being in Platform runlater. is below.. Line 185 is fos.write(b, 0, tmpTransferred); The other listener calls appear to work fine. Just not listener.statusTransferring(); or listener.transferUpdate(); which utilise the fos. Any help will be greatly appreciated. Also for your own sanity I am a self taught google programmer. Yep the worst kind I am sure. Thanks in advance.
Exception in thread "JavaFX Application Thread" java.lang.NullPointerException
at net.thebowdens.net.FileReceiver.transfer(FileReceiver.java:185)
at net.thebowdens.net.DefaultMessageResponder.fileSend(DefaultMessageResponder.java:543)
public boolean transfer() {
listener.statusConnecting();
received = false;
cancel = false;
try {
if (sSock != null) {
sock = sSock.accept();
listener.statusTransferring();
Platform.runLater(() ->{
try {
fos = new FileOutputStream(file);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
});
is = sock.getInputStream();
final byte[] b = new byte[1024];
transferred = 0;
percent = 0;
int tmpTransferred = 0;
int tmpPercent = 0;
int transCounter = 0;
bCounter.prepare();
while (!cancel && (tmpTransferred = is.read(b)) != -1) {
fos.write(b, 0, tmpTransferred);
transferred += tmpTransferred;
percent = (int) ((transferred * 100) / size);
bCounter.addBytes(tmpTransferred);
transCounter++;
if (percent > tmpPercent || transCounter >= 250) {
transCounter = 0;
tmpPercent = percent;
listener.transferUpdate();
}
}
if (!cancel && transferred == size) {
received = true;
listener.statusCompleted();
}
else {
listener.statusFailed();
}
}
}
catch (final IOException e) {
LOG.log(Level.SEVERE, e.toString());
listener.statusFailed();
}
finally {
stopReceiver();
cleanupConnections();
}
return received;
}
Keep in mind that you should use Platform.runLater only for updating the UI, everything else should be outside it otherwhise the UI will become unresponsive.
I suggest you to to refactor your code according to this.
Well after much discussion over the correct language and other issues I solved my problem of the UI updating. I had two issues. My choice selector and Filechooser methods were not on the Javafx application thread (hope this is the right terminology) so I had to do the following:
private ExecutorService executorService = Executors.newCachedThreadPool();
executorService.execute(new Runnable() {
#Override
public void run() {
Platform.runLater(() -> {
try {
receiveRequest(tmpUser, fileRes, user, fileName, size, fileHash);
} catch (IOException | ServerException | CommandException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
});
}
});
I then had to do the same within the Filechooser and file accept method for the transfer class UI to update
executorService.execute(new Runnable() {
#Override public void run() {
if (fileRes.transfer()) {
ui.showSystemMessage("Successfully received " + fileName +
" from " + user + ", and saved as " + fileRes.getFile().getName());
}
else {
ui.showSystemMessage("Failed to receive " + fileName + " from " + user);
fileRes.cancel();
}
}
});
}
I am using kcl api version 2 ,and dont want to use Dynamodb Client for storing the records .
private static final Logger LOG = LoggerFactory.getLogger(DisplayConsumerApplication.class);
public static void main(String... args) {
KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder().credentialsProvider(ProfileCredentialsProvider.create())
.region(Region.of("US-EAST-1")).build();
//DynamoDbAsyncClient dynamoClient =
// DynamoDbAsyncClient.builder().credentialsProvider(ProfileCredentialsProvider.
// create()) .region(Region.of("US-EAST-1")).build();
CloudWatchAsyncClient cloudWatchClient = CloudWatchAsyncClient.builder().credentialsProvider(ProfileCredentialsProvider.create())
.region(Region.of("US-EAST-1")).build();
ConfigsBuilder configsBuilder = new ConfigsBuilder("Sample","Sample", kinesisClient,null,
cloudWatchClient, UUID.randomUUID().toString(), new DisplayConsumerFactory());
Scheduler scheduler = new Scheduler(configsBuilder.checkpointConfig(), configsBuilder.coordinatorConfig(),
configsBuilder.leaseManagementConfig(), configsBuilder.lifecycleConfig(), configsBuilder.metricsConfig(),
configsBuilder.processorConfig(), configsBuilder.retrievalConfig());
Thread schedulerThread = new Thread(scheduler);
schedulerThread.setDaemon(true);
schedulerThread.start();
System.out.println("Press enter to shutdown");
BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
try {
reader.readLine();
} catch (IOException ioex) {
LOG.error("Caught exception while waiting for confirm. Shutting down", ioex);
}
Future<Boolean> gracefulShutdownFuture = scheduler.startGracefulShutdown();
LOG.info("Waiting up to 20 seconds for shutdown to complete.");
try {
gracefulShutdownFuture.get(20, TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOG.info("Interrupted while waiting for graceful shutdown. Continuing.");
} catch (ExecutionException e) {
LOG.error("Exception while executing graceful shutdown.", e);
} catch (TimeoutException e) {
LOG.error("Timeout while waiting for shutdown. Scheduler may not have exited.");
}
LOG.info("Completed, shutting down now.");
}
}
As you can see I commented initialize of DynamodbClient ,but in that method it is manadatory to pass the object of Dynamoclient .So I passed as null ,but getting null pointer exception ,Could you please share your idea how I can use Scheduler without dynamodb client ?
I'm using Dynamo DB streams + Lamdba as trigger to invoke kinesis which puts my Dynamo DB data to Redshift.
Could someone suggest a way to load Dynamo DB data to Redshift in different regions using dynamo streams.
I had written one program that can move the data from Dynamo db to Redshift but that is not working without streams, You can take a look of code and see whether this helps your case or you get any Idea corresponding to this.
1.Creating the Connection with Redshift.
2.Create the Prepeared Statement for insertion into Redshift.
3.Fetching the data from Dynamo in batches using pagination.
4.Inserting the batch data into Resdhift in batches.
public void createConnectionWithRedshift() {
final String DB_URL = "jdbc:redshift://ao.cepuhmobd.us-west-2.redshift.amazonaws.com:5439/events";
// final String DB_URL = args[0];
// Database credentials
final String USER = "abc";
final String PASS = "abc";
Connection conn = null;
try {
// STEP 3: Open a connection
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL, USER, PASS);
// createNewTable(conn);
// STEP 4: Execute a query
preparedStatement = conn.prepareStatement("insert into Events " + "(Vin,timestamp,eventtype,source,data)" + "VALUES (?,?,?,?,?)");
} catch (SQLException se) {
se.printStackTrace();
}
}// end main
public void replicateDynamoToRedshidt(int pages, int batchSize, int scanSize)
throws TableNeverTransitionedToStateException, InterruptedException {
createConnectionWithRedshift();//Redshift Connection
for (int i = 0; i < pages; i = i + 1) {
List<EventLogEntity> results = findAll(new PageRequest(i, batchSize));//Fetching the data from Dynamo in batches
List<HeadUnitData> headUnitDataList = headUnitEvents(results);
for (int j = 0; j < headUnitDataList.size(); j++) {
HeadUnitData headUnitData = headUnitDataList.get(j);
insertData(headUnitData.getVin(), headUnitData.getType(), headUnitData.getSource(), headUnitData.getData());//Inserting the data into Redshidt in batches
}
try {
preparedStatement.executeBatch();
System.out.println("Inserted in Database : " + results.size());
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Can someone advise how to create new threads in a loop. Data are retrieved from the database and I want to make each row was processed in a new thread. Everything happens in the controller JavaFX. You probably need to use the service but I can not find an example of this approach. Thanks in advance
Are you sure you want to create a new thread for every row? If you have a million rows, you will have a million threads.
If so, it should be as simple as this:
ResultSet rs = ...;
while (rs.next()) {
new Thread(() -> {
doSomething();
Platform.runLater(() -> doSomethingThatUpdatesUI());
}).start();
}
Update
An example using javafx.concurrent.Task (only one thread created):
Task task = new Task<Void>() {
#Override
public Void call() {
ResultSet rs = ...;
int max = getSizeOfResultSet(rs);
int count = 0;
while (rs.next()) {
if (isCancelled()) {
break;
}
updateProgress(count++, max);
}
return null;
}
};
yourProgressBarIfYouHaveOne.progressProperty().bind(task.progressProperty());
new Thread(task).start();
task.setOnSucceeded(event -> {
System.out.println("OK, all done!");
releaseUserInterface();
});
Can anybody help telling me what is wrong with my code? I am trying to connect to SQLite database, and executing some queries. when trying to create and open the database, create and insert the table, no exception returned. but when I try to execute delete statement,
DatabaseIOException: File system error (12)
always returned. I don't know the cause of the exception exactly. would you tell me what usually cause this kind of exception? I don't even know when I need to close the database and when I don't need to. this solution also makes me confused.
here is my code:
public class DatabaseManager {
Logger log = new Logger();
Database db;
public DatabaseManager() {
createDatabase();
}
private void createDatabase() {
// Determine if an SDCard is present
boolean sdCardPresent = false;
String root = null;
Enumeration enum = FileSystemRegistry.listRoots();
while (enum.hasMoreElements()) {
root = (String) enum.nextElement();
if(root.equalsIgnoreCase("sdcard/")) {
sdCardPresent = true;
}
}
if(!sdCardPresent) {
alert("This application requires an SD card to be present. Exiting application...");
}
else {
try {
URI uri = URI.create("/SDCard/databases/MyAdvanceUI/myadvanceui.db");
db = DatabaseFactory.openOrCreate(uri);
db.close();
//alert("Database OK!");
} catch (Exception e) {
// TODO Auto-generated catch block
//alert("Exception in createDatabase(): " + e);
}
}
}
private void alert(final String message) {
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run() {
Dialog.inform(message);
System.exit(0);
}
});
}
private void createTableTask() {
try {
URI uri = URI.create("/SDCard/databases/MyAdvanceUI/myadvanceui.db");
db = DatabaseFactory.open(uri);
Statement st = db.createStatement("CREATE TABLE IF NOT EXISTS t_task (id INTEGER PRIMARY KEY AUTOINCREMENT, "
+ "client TEXT, task TEXT)");
st.prepare();
st.execute();
st.close();
db.close();
//alert("Table Task created!");
} catch (Exception e) {
// TODO: handle exception
//alert("Exception in createTableTask(): " + e);
}
}
private void insertTableTask() {
String[] clients = { "Budi Setiawan", "Dian Kusuma", "Joko Ahmad", "Titi Haryanto", "Wahyu" };
String[] tasks = {
"Penawaran terhadap instalasi server",
"Follow up untuk keperluan produk terbaru",
"Pendekatan untuk membina relasi",
"Penawaran jasa maintenance",
"Penawaran terhadap instalasi database"
};
try {
URI uri = URI.create("/SDCard/databases/MyAdvanceUI/myadvanceui.db");
db = DatabaseFactory.open(uri);
for(int i = 0; i < clients.length; i++) {
Statement st = db.createStatement("INSERT INTO t_task (client, task) VALUES (?, ?)");
st.prepare();
st.bind(1, clients[i]);
st.bind(2, tasks[i]);
st.execute();
st.close();
}
db.close();
} catch (Exception e) {
// TODO: handle exception
//alert("Exception in insertTableTask(): " + e);
}
}
public void loadInitialData() {
createTableTask();
insertTableTask();
}
public Cursor getTasks() {
// TODO Auto-generated method stub
Cursor results = null;
try {
URI uri = URI.create("/SDCard/databases/MyAdvanceUI/myadvanceui.db");
db = DatabaseFactory.open(uri);
Statement st = db.createStatement("SELECT client, task FROM t_task");
st.prepare();
results = st.getCursor();
return results;
} catch (Exception e) {
// TODO: handle exception
//alert("Exception: " + e);
}
return results;
}
public void delete(String string) {
// TODO Auto-generated method stub
try {
URI uri = URI.create("/SDCard/databases/MyAdvanceUI/myadvanceui.db");
db = DatabaseFactory.open(uri);
Statement st = db.createStatement("DELETE FROM t_task WHERE client=?");
st.prepare();
st.bind(1, string);
st.execute();
} catch (Exception e) {
// TODO: handle exception
alert("Exception: " + e);
}
}
}
thank you for your help.
I don't see that you close the statement and close the database after select and delete actions. Most probably you can't open database because it wasn't closed correctly.
Big warning SD card isn't available when user mounted devices to PC as external drive. Some devices are going without SD card preinstalled. DB operations are really slow on 5 OS devices. Your alert method code wan't close db what could be issue to open it after on the next application start.
Warning As #pankar mentioned in comment you should add finally {} where you will close resources for sure. In your current implementation if you get exception in execution you will never close database.
Big improvements You don't need to create and prepare statement every loop. Just do it before for. Do bind and execute every loop. And close statement after for.
Improvements You could keep one opened db during application run cycle. It will save you some line of code and time for opening closing.
Notation It's bad practice to have parameter named like 'string'. I would rename it to something more meaningful.