There is a lot of documentation and mentions online for removing a node from the cluster that you simply need to do:
SET GLOBAL wsrep_desync = ON;
SET GLOBAL wsrep_on = OFF;
However this doesn't seem to remove the node at all from the cluster, changes made on other nodes still propagate to this one and changes on the node that should be removed still propagate to the rest of the cluster.
I found the reason as I was posting this question but may be useful for someone else. Until the node that has been desynced receives some operation that causes read lock on the db it stays synced. Slightly confusing behaviour but fine once understood.
Related
The azure cosmos db documentation mentions following in the index transformation section :
When you move to None indexing mode, the index is dropped immediately. Moving to None is useful when you want to cancel an in-progress transformation and start fresh with a different indexing policy.
I have a partitioned collection with custom indexing policy with indexing mode consistent. Since there is no transformation progress available for a partitioned collection, so before starting any new indexing policy update, I was thinking of using None mode to cancel any in progress transformation and start the new one(consistent mode). But will this cause the entire index to get recreated, even if, just a new path was added?
If answer is yes, what is the best way of checking if no transformation is in progress before starting a new one?
Welcome to StackOverflow, Jatin! I am from the CosmosDB engineering team.
We do have transformation progress now for a partitioned collections - this was released recently. This can be fetched by setting the header 'x-ms-documentdb-populatequotainfo' to True. This header, in addition to providing quota and usage information for a partitioned collection, is also used to piggyback the index transformation progress information for partitioned collections.
However, you do not need to wait for the progress to complete before changing your indexing policy. Even though this would mean that it could take longer for our background re-indexing to finish all previous index updates and catch up to your latest indexing policy, there is an opportunity for the transformations to be online, meaning that queries could continue to work based on the changes between the indexing policy updates. For example, if the only change was adding paths throughout the transformations, queries on paths that are already indexed will continue to work throughout the reindexing process, assuming that the indexing mode was consistent through all the updates.
However, if you want your new index to be effective potentially sooner, then your suggestion of transitioning to None and then to Consistent could work. Please note that this is equivalent to an offline index rebuild, meaning the existing index will be deleted and the new one will be constructed from scratch. Queries will not return consistent results on any path up until the reindexing process completes.
Welcome to Stack Overflow!
Simply change the indexing policy from Custom to Consistent. Skip going to None and go directly to Consistent indexing policy:
All index transformations are performed online. The items indexed per the old policy are efficiently transformed per the new policy without affecting the write availability or the throughput provisioned to the container. The consistency of read and write operations that are performed by using the REST API, SDKs, or from stored procedures and triggers is not affected during index transformation.
Index transformations
While reindexing is in progress, your query may not return all matching results, if the queries happen to use the index that is being modified, and queries will not return any errors/failures. While reindexing is in progress, queries are eventually consistent regardless of the indexing mode configuration.
If you feel that None is safe step before going to Consistent, you are free to do so but it is not absolutely necessary.
I am using riak version 1.4.10 and it is in a ring with two hosts. I am unable to get rid of keys left over from previous operations using simple delete operations on keys. When I list the keys for a bucket, it shows me the old keys, however if I try to retrieve the data associated with a key, no data is found. When I try to delete the key, it still persists. What could be the cause of this? Is there a way to wipe the keys in the bucket so it starts from a clean slate? I don't care about any of the data in riak, but I would rather not have to reinstall everything again.
You are probably seeing the tombstones of the old data. Since Riak is an eventually consistent data store, it needs to keep track of deletes as if they were ordinary writes, at least for a little while.
If data is present on one node, but not another, how do you tell if it is a PUT that hasnt' propagated yet, or a DELETE?
Riak solves this by using a tombstone. Whenever you delete something, instead of just wiping the data immediately, Riak replaces the existing value with a special value that it knows means deleted. This special value contains a vclock that is descended from the previous value, and metadata indicating deleted. So when it comes time to decide the above question, Riak simply compares the vclock of the value with that of the tombstone. Whichever descends from the other must be the correct one.
To solve the problem of an ever growing data size that contains mostly tombstones, tombstones are reaped after a time. The time is set using the delete_mode setting. After the DELETE is processed, and the tombstone has been written to the primary vnodes, the delete process issues a GET request for the key. Whenever the GET process encounters a tombstone, and all of the primary vnodes responded with the same tombstone, it schedules the tombstone to be reaped according to the delete_mode setting.
So if you want to actually get rid of the tombstones, check your delete_mode setting to make sure it is not set to 'keep', and issue a get for each one to make sure it is really gone.
Or if you are just wiping the data store to restart your tests, stop Riak, delete all the files under the data_root for the backend you are using, and restart.
we're having some weird things happening with a cleanup cronjob and riak:
the objects we store (postboxes) have a 2i for modification date (which is a unix timestamp).
there's a cronjob running freqently deleting all postboxes that have not been modified within 180 days. however we've found evidence that postboxes that some (very little) postboxes that were modified in the last three days were deleted by this cronjob.
After reviewing and debugging several times over every line of code, I am confident, that this is not a problem of the cronjob.
I also traced back all delete calls to that bucket - and no one else is deleting objects there.
Of course I also checked with Riak to read the postboxes with r=ALL: they're definitely gone. (and they are stored with w=QUORUM)
I also checked the logs: updating the post boxes did succeed (there were no errors reported back from the write operations)
This leaves me with two possible causes for this:
riak loses data (which I am not willing to believe that easily)
the secondary indexes are corrupt and queries to them return wrong keys
So my questions are:
Can 2is actually break?
Is it possible to verify that?
Am I missing something completely different?
Cheers,
Matthias
Secondary index queries in Riak are coverage queries, which means that they will only use one of the stored replicas, and not perform a quorum read.
As you are writing with w=QUORUM, it is possible that one (or more) of the replicas may not get updated if you have n_val set to 3 or higher while the operation still is deemed successful. If this is the one selected for the coverage query, you could end up deleting based on the old value. In order to avoid this, you will need to perform updates with w=ALL.
Under the condition of "allow_versions" set to "FALSE" or "TRUE", for both cases, how Swift response for the scenario that a file is under overwriting while delete request come in simultaneously(with the order of overwriting first then deletion)?
Please share your thoughts.
Many thanks!
The timestamp assigned to the request coming in to the proxy is what will ultimately decide which "last" write wins.
If you have a long running upload and issue the delete during, the tombstone will have a later timestamp and will eventually take precedence even if the upload finishes after.
When using the container versioning feature, overwriting in a versioned container will cause the object data to be COPY'd off the current tip before the PUT data is sent to the storage node with the assigned timestamp. For deletes in a versioned container the "previous version" is discovered at the time the overwriting request is made and subject to eventual consistency in the container listing, but is only deleted once it has been copied into the current location for the object.
More information about object versioning is available here:
http://docs.openstack.org/developer/swift/overview_object_versioning.html
Well, a quick summary comes, though still a very high level view but hope it helps understanding how it works under the hood.
The diagram(below link) sets two simultaneous scenarios(A and B) against the enable/disable of the Swift object versioning feature. The outcome for each scenario is shown in the diagram.
Download the diagram.
Please share your thoughts if any.
I have several graphs. The breadth and depth of each graph can vary and will undergo changes and alterations during runtime. See example graph.
There is a root node to get a hold on the whole graph (i.e. tree). A node can have several children and each child serves a special purpose. Furthermore a node can access all its direct children in order to retrieve certain informations. On the other hand a child node may not be aware of its own parent node, nor other siblings. Nothing spectacular so far.
Storing each graph and updating it with an object database (in this case DB4O) looks pretty straightforward. I could have used a relational database to accomplish data persistence (including database triggers, etc.) but I wanted to realize it with an object database instead.
There is one peculiar thing with my graphs. See another example graph.
To properly perform calculations some nodes require informations from other nodes. These other nodes may be siblings, children/grandchildren or related in some other kind. In this case a specific node knows the other relevant nodes as well (and thus can get the required informations directly from them). For the sake of simplicity the first image didn't show all potential connections.
If one node has a change of state (e.g. triggered by an internal timer or triggered by some other node) it will inform other nodes (interested obsevers, see also observer pattern) about the change. Each informed node will then take appropriate actions to update its own state (and in turn inform other observers as needed). A root node will not know about every change that occurs, since only the involved nodes will know that something has changed. If such a chain of events is triggered by the root node then of course it's not much of an issue.
The aim is to assure data persistence with an object database. Data in memory should be in sync with data stored within the database. What adds to the complexity is the fact that the graphs don't consist of simple (and stupid) data nodes, but that lots of functionality is integrated in each node (i.e. events that trigger state changes throughout a graph).
I have several rough ideas on how to cope with the presented issue (e.g. (1) stronger separation of data and functionality or (2) stronger integration of the database or (3) set an arbitrary time interval to update data and accept that data may be out of synch for a period of time). I'm looking for some more input and options concerning such a key issue (which will definitely leave significant footprints on a concrete implementation).
(edited)
There is another aspect I forgot to mention. A graph should not reside all the time in memory. Graphs that are not needed will be only present in the database and thus put in a state of suspension. This is another issue which needs consideration. While in suspension the update mechanisms will probably be put to sleep as well and this is not intended.
In the case of db4o check out "transparent activation" to automatically load objects on demand as you traverse the graph (this way the graph doesn't have to be all in memory) and check out "transparent persistence" to allow each node to persist itself after a state change.
http://www.gamlor.info/wordpress/2009/12/db4o-transparent-persistence/
Moreover you can use db4o "callbacks" to trigger custom behavior during db4o operations.
HTH
German
What's the exact question? Here a few comments:
As #German already mentioned: For complex object graphs you probably want to use transparent persistence.
Also as #German mentione: Callback can help you to do additional stuff when objects are read/written etc on the database.
To the Observer-Pattern. Are you on .NET or Java? Usually you don't want to store the observers in the database, since the observers are usually some parts of your business-logic, GUI etc. On .NET events are automatically not stored. On Java make sure that you mark the field holding the observer-references as transient.
In case you actually want to store observers, for example because they are just other elements in your object-graph. On .NET, you cannot store delegates / closures. So you need to introduce a interface for calling the observer. On Java: Often we use anonymous inner classes as listener: While db4o can store those, I would NOT recommend that. Because a anonymous inner class gets generated name which can change. Then db4o will not find that class later if you've changed your code.
Thats it. Ask more detailed questions if you want to know more.