I have an enormous directed graph I need to traverse in search for the shortest path to a specific node from a given starting point. The graph in question does not exist explicitly; the child nodes are determined algorithmically from the parent nodes.
(To give an illustration: imagine a graph of chess positions. Each node is a chess position and its children are all the legal moves from that position.)
So I have a queue for open nodes, and every time I process the next node in the queue I enqueue all of its children. But since the graph can have cycles I also need to maintain a hashset of all visited nodes so I can check if I have visited one before.
This works okay, but since this graph is so large, I run into memory problems. All of the nodes in the queue are also stored in the hashset, which tends to be around 50% of the total number or visited nodes in practice in my case.
Is there some magical way to get rid of this redundancy while keeping the speed of the hashset? (Obviously, I could get rid of the redundancy by NOT hashing and just doing a linear search, but that is out of the question.)
I solved it by writing a class that stores the keys in a list and stores the indices of the keys in a hashtable. The next node "in the queue" is always the the next node in the list until you find what you're looking for or you've traversed the entire graph.
class IndexMap<T>
{
private List<T> values;
private LinkedList<int>[] buckets;
public int Count { get; private set; } = 0;
public IndexMap(int capacity)
{
values = new List<T>(capacity);
buckets = new LinkedList<int>[NextPowerOfTwo(capacity)];
for (int i = 0; i < buckets.Length; ++i)
buckets[i] = new LinkedList<int>();
}
public void Add(T item) //assumes item is not yet in map
{
if (Count == buckets.Length)
ReHash();
int bucketIndex = item.GetHashCode() & (buckets.Length - 1);
buckets[bucketIndex].AddFirst(Count++);
values.Add(item);
}
public bool Contains(T item)
{
int bucketIndex = item.GetHashCode() & (buckets.Length - 1);
foreach(int i in buckets[bucketIndex])
{
if (values[i].Equals(item))
return true;
}
return false;
}
public T this[int index]
{
get => values[index];
}
private void ReHash()
{
LinkedList<int>[] newBuckets = new LinkedList<int>[2 * buckets.Length];
for (int i = 0; i < newBuckets.Length; ++i)
newBuckets[i] = new LinkedList<int>();
for (int i = 0; i < buckets.Length; ++i)
{
foreach (int index in buckets[i])
{
int bucketIndex = values[index].GetHashCode() & (newBuckets.Length - 1);
newBuckets[bucketIndex].AddFirst(index);
}
buckets[i] = null;
}
buckets = newBuckets;
}
private int NextPowerOfTwo(int n)
{
if ((n & n-1) == 0)
return n;
int output = 0;
while (n > output)
{
output <<= 1;
}
return output;
}
}
The old method of maintaining both an array of the open nodes and a hashtable of the visited nodes needed n*(1+a)*size(T) space, where a is the ratio of nodes_in_the_queue over total_nodes_found and size(T) is the size of a node.
This method needs n*(size(T) + size(int)). If your nodes are significantly larger than an int, this can save a lot.
We all know you can't do the following because of ConcurrentModificationException:
for (Object i : l) {
if (condition(i)) {
l.remove(i);
}
}
But this apparently works sometimes, but not always. Here's some specific code:
public static void main(String[] args) {
Collection<Integer> l = new ArrayList<>();
for (int i = 0; i < 10; ++i) {
l.add(4);
l.add(5);
l.add(6);
}
for (int i : l) {
if (i == 5) {
l.remove(i);
}
}
System.out.println(l);
}
This, of course, results in:
Exception in thread "main" java.util.ConcurrentModificationException
Even though multiple threads aren't doing it. Anyway.
What's the best solution to this problem? How can I remove an item from the collection in a loop without throwing this exception?
I'm also using an arbitrary Collection here, not necessarily an ArrayList, so you can't rely on get.
Iterator.remove() is safe, you can use it like this:
List<String> list = new ArrayList<>();
// This is a clever way to create the iterator and call iterator.hasNext() like
// you would do in a while-loop. It would be the same as doing:
// Iterator<String> iterator = list.iterator();
// while (iterator.hasNext()) {
for (Iterator<String> iterator = list.iterator(); iterator.hasNext();) {
String string = iterator.next();
if (string.isEmpty()) {
// Remove the current element from the iterator and the list.
iterator.remove();
}
}
Note that Iterator.remove() is the only safe way to modify a collection during iteration; the behavior is unspecified if the underlying collection is modified in any other way while the iteration is in progress.
Source: docs.oracle > The Collection Interface
And similarly, if you have a ListIterator and want to add items, you can use ListIterator#add, for the same reason you can use Iterator#remove — it's designed to allow it.
In your case you tried to remove from a list, but the same restriction applies if trying to put into a Map while iterating its content.
This works:
Iterator<Integer> iter = l.iterator();
while (iter.hasNext()) {
if (iter.next() == 5) {
iter.remove();
}
}
I assumed that since a foreach loop is syntactic sugar for iterating, using an iterator wouldn't help... but it gives you this .remove() functionality.
With Java 8 you can use the new removeIf method. Applied to your example:
Collection<Integer> coll = new ArrayList<>();
//populate
coll.removeIf(i -> i == 5);
Since the question has been already answered i.e. the best way is to use the remove method of the iterator object, I would go into the specifics of the place where the error "java.util.ConcurrentModificationException" is thrown.
Every collection class has a private class which implements the Iterator interface and provides methods like next(), remove() and hasNext().
The code for next looks something like this...
public E next() {
checkForComodification();
try {
E next = get(cursor);
lastRet = cursor++;
return next;
} catch(IndexOutOfBoundsException e) {
checkForComodification();
throw new NoSuchElementException();
}
}
Here the method checkForComodification is implemented as
final void checkForComodification() {
if (modCount != expectedModCount)
throw new ConcurrentModificationException();
}
So, as you can see, if you explicitly try to remove an element from the collection. It results in modCount getting different from expectedModCount, resulting in the exception ConcurrentModificationException.
You can either use the iterator directly like you mentioned, or else keep a second collection and add each item you want to remove to the new collection, then removeAll at the end. This allows you to keep using the type-safety of the for-each loop at the cost of increased memory use and cpu time (shouldn't be a huge problem unless you have really, really big lists or a really old computer)
public static void main(String[] args)
{
Collection<Integer> l = new ArrayList<Integer>();
Collection<Integer> itemsToRemove = new ArrayList<>();
for (int i=0; i < 10; i++) {
l.add(Integer.of(4));
l.add(Integer.of(5));
l.add(Integer.of(6));
}
for (Integer i : l)
{
if (i.intValue() == 5) {
itemsToRemove.add(i);
}
}
l.removeAll(itemsToRemove);
System.out.println(l);
}
In such cases a common trick is (was?) to go backwards:
for(int i = l.size() - 1; i >= 0; i --) {
if (l.get(i) == 5) {
l.remove(i);
}
}
That said, I'm more than happy that you have better ways in Java 8, e.g. removeIf or filter on streams.
Same answer as Claudius with a for loop:
for (Iterator<Object> it = objects.iterator(); it.hasNext();) {
Object object = it.next();
if (test) {
it.remove();
}
}
With Eclipse Collections, the method removeIf defined on MutableCollection will work:
MutableList<Integer> list = Lists.mutable.of(1, 2, 3, 4, 5);
list.removeIf(Predicates.lessThan(3));
Assert.assertEquals(Lists.mutable.of(3, 4, 5), list);
With Java 8 Lambda syntax this can be written as follows:
MutableList<Integer> list = Lists.mutable.of(1, 2, 3, 4, 5);
list.removeIf(Predicates.cast(integer -> integer < 3));
Assert.assertEquals(Lists.mutable.of(3, 4, 5), list);
The call to Predicates.cast() is necessary here because a default removeIf method was added on the java.util.Collection interface in Java 8.
Note: I am a committer for Eclipse Collections.
Make a copy of existing list and iterate over new copy.
for (String str : new ArrayList<String>(listOfStr))
{
listOfStr.remove(/* object reference or index */);
}
People are asserting one can't remove from a Collection being iterated by a foreach loop. I just wanted to point out that is technically incorrect and describe exactly (I know the OP's question is so advanced as to obviate knowing this) the code behind that assumption:
for (TouchableObj obj : untouchedSet) { // <--- This is where ConcurrentModificationException strikes
if (obj.isTouched()) {
untouchedSet.remove(obj);
touchedSt.add(obj);
break; // this is key to avoiding returning to the foreach
}
}
It isn't that you can't remove from the iterated Colletion rather that you can't then continue iteration once you do. Hence the break in the code above.
Apologies if this answer is a somewhat specialist use-case and more suited to the original thread I arrived here from, that one is marked as a duplicate (despite this thread appearing more nuanced) of this and locked.
With a traditional for loop
ArrayList<String> myArray = new ArrayList<>();
for (int i = 0; i < myArray.size(); ) {
String text = myArray.get(i);
if (someCondition(text))
myArray.remove(i);
else
i++;
}
ConcurrentHashMap or ConcurrentLinkedQueue or ConcurrentSkipListMap may be another option, because they will never throw any ConcurrentModificationException, even if you remove or add item.
Another way is to use a copy of your arrayList just for iteration:
List<Object> l = ...
List<Object> iterationList = ImmutableList.copyOf(l);
for (Object curr : iterationList) {
if (condition(curr)) {
l.remove(curr);
}
}
A ListIterator allows you to add or remove items in the list. Suppose you have a list of Car objects:
List<Car> cars = ArrayList<>();
// add cars here...
for (ListIterator<Car> carIterator = cars.listIterator(); carIterator.hasNext(); )
{
if (<some-condition>)
{
carIterator().remove()
}
else if (<some-other-condition>)
{
carIterator().add(aNewCar);
}
}
Now, You can remove with the following code
l.removeIf(current -> current == 5);
I know this question is too old to be about Java 8, but for those using Java 8 you can easily use removeIf():
Collection<Integer> l = new ArrayList<Integer>();
for (int i=0; i < 10; ++i) {
l.add(new Integer(4));
l.add(new Integer(5));
l.add(new Integer(6));
}
l.removeIf(i -> i.intValue() == 5);
Java Concurrent Modification Exception
Single thread
Iterator<String> iterator = list.iterator();
while (iterator.hasNext()) {
String value = iter.next()
if (value == "A") {
list.remove(it.next()); //throws ConcurrentModificationException
}
}
Solution: iterator remove() method
Iterator<String> iterator = list.iterator();
while (iterator.hasNext()) {
String value = iter.next()
if (value == "A") {
it.remove()
}
}
Multi thread
copy/convert and iterate over another one collection. For small collections
synchronize[About]
thread safe collection[About]
I have a suggestion for the problem above. No need of secondary list or any extra time. Please find an example which would do the same stuff but in a different way.
//"list" is ArrayList<Object>
//"state" is some boolean variable, which when set to true, Object will be removed from the list
int index = 0;
while(index < list.size()) {
Object r = list.get(index);
if( state ) {
list.remove(index);
index = 0;
continue;
}
index += 1;
}
This would avoid the Concurrency Exception.
for (Integer i : l)
{
if (i.intValue() == 5){
itemsToRemove.add(i);
break;
}
}
The catch is the after removing the element from the list if you skip the internal iterator.next() call. it still works! Though I dont propose to write code like this it helps to understand the concept behind it :-)
Cheers!
Example of thread safe collection modification:
public class Example {
private final List<String> queue = Collections.synchronizedList(new ArrayList<String>());
public void removeFromQueue() {
synchronized (queue) {
Iterator<String> iterator = queue.iterator();
String string = iterator.next();
if (string.isEmpty()) {
iterator.remove();
}
}
}
}
I know this question assumes just a Collection, and not more specifically any List. But for those reading this question who are indeed working with a List reference, you can avoid ConcurrentModificationException with a while-loop (while modifying within it) instead if you want to avoid Iterator (either if you want to avoid it in general, or avoid it specifically to achieve a looping order different from start-to-end stopping at each element [which I believe is the only order Iterator itself can do]):
*Update: See comments below that clarify the analogous is also achievable with the traditional-for-loop.
final List<Integer> list = new ArrayList<>();
for(int i = 0; i < 10; ++i){
list.add(i);
}
int i = 1;
while(i < list.size()){
if(list.get(i) % 2 == 0){
list.remove(i++);
} else {
i += 2;
}
}
No ConcurrentModificationException from that code.
There we see looping not start at the beginning, and not stop at every element (which I believe Iterator itself can't do).
FWIW we also see get being called on list, which could not be done if its reference was just Collection (instead of the more specific List-type of Collection) - List interface includes get, but Collection interface does not. If not for that difference, then the list reference could instead be a Collection [and therefore technically this Answer would then be a direct Answer, instead of a tangential Answer].
FWIWW same code still works after modified to start at beginning at stop at every element (just like Iterator order):
final List<Integer> list = new ArrayList<>();
for(int i = 0; i < 10; ++i){
list.add(i);
}
int i = 0;
while(i < list.size()){
if(list.get(i) % 2 == 0){
list.remove(i);
} else {
++i;
}
}
One solution could be to rotate the list and remove the first element to avoid the ConcurrentModificationException or IndexOutOfBoundsException
int n = list.size();
for(int j=0;j<n;j++){
//you can also put a condition before remove
list.remove(0);
Collections.rotate(list, 1);
}
Collections.rotate(list, -1);
Try this one (removes all elements in the list that equal i):
for (Object i : l) {
if (condition(i)) {
l = (l.stream().filter((a) -> a != i)).collect(Collectors.toList());
}
}
You can use a while loop.
Iterator<Map.Entry<String, String>> iterator = map.entrySet().iterator();
while(iterator.hasNext()){
Map.Entry<String, String> entry = iterator.next();
if(entry.getKey().equals("test")) {
iterator.remove();
}
}
I ended up with this ConcurrentModificationException, while iterating the list using stream().map() method. However the for(:) did not throw the exception while iterating and modifying the the list.
Here is code snippet , if its of help to anyone:
here I'm iterating on a ArrayList<BuildEntity> , and modifying it using the list.remove(obj)
for(BuildEntity build : uniqueBuildEntities){
if(build!=null){
if(isBuildCrashedWithErrors(build)){
log.info("The following build crashed with errors , will not be persisted -> \n{}"
,build.getBuildUrl());
uniqueBuildEntities.remove(build);
if (uniqueBuildEntities.isEmpty()) return EMPTY_LIST;
}
}
}
if(uniqueBuildEntities.size()>0) {
dbEntries.addAll(uniqueBuildEntities);
}
If using HashMap, in newer versions of Java (8+) you can select each of 3 options:
public class UserProfileEntity {
private String Code;
private String mobileNumber;
private LocalDateTime inputDT;
// getters and setters here
}
HashMap<String, UserProfileEntity> upMap = new HashMap<>();
// remove by value
upMap.values().removeIf(value -> !value.getCode().contains("0005"));
// remove by key
upMap.keySet().removeIf(key -> key.contentEquals("testUser"));
// remove by entry / key + value
upMap.entrySet().removeIf(entry -> (entry.getKey().endsWith("admin") || entry.getValue().getInputDT().isBefore(LocalDateTime.now().minusMinutes(3)));
The best way (recommended) is use of java.util.concurrent package. By
using this package you can easily avoid this exception. Refer
Modified Code:
public static void main(String[] args) {
Collection<Integer> l = new CopyOnWriteArrayList<Integer>();
for (int i=0; i < 10; ++i) {
l.add(new Integer(4));
l.add(new Integer(5));
l.add(new Integer(6));
}
for (Integer i : l) {
if (i.intValue() == 5) {
l.remove(i);
}
}
System.out.println(l);
}
Iterators are not always helpful when another thread also modifies the collection. I had tried many ways but then realized traversing the collection manually is much safer (backward for removal):
for (i in myList.size-1 downTo 0) {
myList.getOrNull(i)?.also {
if (it == 5)
myList.remove(it)
}
}
In case ArrayList:remove(int index)- if(index is last element's position) it avoids without System.arraycopy() and takes not time for this.
arraycopy time increases if(index decreases), by the way elements of list also decreases!
the best effective remove way is- removing its elements in descending order:
while(list.size()>0)list.remove(list.size()-1);//takes O(1)
while(list.size()>0)list.remove(0);//takes O(factorial(n))
//region prepare data
ArrayList<Integer> ints = new ArrayList<Integer>();
ArrayList<Integer> toRemove = new ArrayList<Integer>();
Random rdm = new Random();
long millis;
for (int i = 0; i < 100000; i++) {
Integer integer = rdm.nextInt();
ints.add(integer);
}
ArrayList<Integer> intsForIndex = new ArrayList<Integer>(ints);
ArrayList<Integer> intsDescIndex = new ArrayList<Integer>(ints);
ArrayList<Integer> intsIterator = new ArrayList<Integer>(ints);
//endregion
// region for index
millis = System.currentTimeMillis();
for (int i = 0; i < intsForIndex.size(); i++)
if (intsForIndex.get(i) % 2 == 0) intsForIndex.remove(i--);
System.out.println(System.currentTimeMillis() - millis);
// endregion
// region for index desc
millis = System.currentTimeMillis();
for (int i = intsDescIndex.size() - 1; i >= 0; i--)
if (intsDescIndex.get(i) % 2 == 0) intsDescIndex.remove(i);
System.out.println(System.currentTimeMillis() - millis);
//endregion
// region iterator
millis = System.currentTimeMillis();
for (Iterator<Integer> iterator = intsIterator.iterator(); iterator.hasNext(); )
if (iterator.next() % 2 == 0) iterator.remove();
System.out.println(System.currentTimeMillis() - millis);
//endregion
for index loop: 1090 msec
for desc index: 519 msec---the best
for iterator: 1043 msec
you can also use Recursion
Recursion in java is a process in which a method calls itself continuously. A method in java that calls itself is called recursive method.
As far as I know, DP is either you start with bigger problem and recursively come down, and keep saving the value each time for future use or you do it iteratively and keep saving values bottom up. But what if I am doing it bottom up but recursively going up?
Say for example the following question, Longest Common Subsequence
Here's my solution
public class LongestCommonSubseq {
/**
* #param args
*/
public static List<Character> list = new ArrayList<Character>();
public static int[][] M = new int[7][7];
public static void main(String[] args) {
String s1 = "ABCDGH";
String s2 = "AEDFHR";
for(int i=0;i<=6;i++)
for(int j=0;j<=6;j++)
M[i][j] = -1;
int max = getMax(s1,s2,0,0);
System.out.println(max);
Collections.sort(list);
for(int i = 0;i < max;i++)
System.out.println(list.get(i));
}
public static int getMax(String s1, String s2,int i ,int j){
if(i >= s1.length() || j>= s2.length()){
M[i][j] = 0;
return M[i][j];
}
if(M[i][j] != -1)
return M[i][j];
if(s1.charAt(i) == s2.charAt(j)){
M[i][j] = 1 + getMax(s1,s2,i+1,j+1);
list.add(s1.charAt(i));
}
else
M[i][j] = max(getMax(s1,s2,i+1,j) , getMax(s1, s2, i, j+1));
return M[i][j];
}
public static int max(int a,int b){
return a > b ? a : b;
}
}
So you see,I am going from M[0][0] in the other direction but I am not doing it iteratively.
But I guess it should be fine. Just needed to confirm.
Thanks
The direction does not matter. What is more important is that you go from more general(complex) problem to simpler ones. What you have done is dynamic programming.
For dynamic programming it doesn't matter if you follow the bottom-up or top-down-paradigm. The basic thesis (like you have correctly mentioned) of dynamic programming is known as Bellman's Principle of Optimality which is the following:
Principle of Optimality: An optimal policy has the property that
whatever the initial state and initial decision are, the remaining
decisions must constitute an optimal policy with regard to the state
resulting from the first decision.
Resource: Wikipedia (http://en.wikipedia.org/wiki/Bellman_equation#Bellman.27s_Principle_of_Optimality)
An great approach to cut of some of these optimal sub-solutions from the recursive-call-tree is to use Caching (like in your code).
As the title says, I have to trim a binary tree based on a given min and max value. Each node stores a value, and a left/right node. I may define private helper methods to solve this problem, but otherwise I may not call any other methods of the class nor create any data structures such as arrays, lists, etc.
An example would look like this:
overallRoot
_____[50]____________________
/ \
__________[38] _______________[90]
/ \ /
_[14] [42] [54]_____
/ \ \
[8] [20] [72]
\ / \
[26] [61] [83]
trim(52, 65);
should return:
overallRoot
[54]
\
[61]
My attempted solution has three methods:
public void trim(int min, int max) {
rootFinder(overallRoot, min, max);
}
First recursive method finds the new root perfectly.
private void rootFinder(IntTreeNode node, int min, int max) {
if (node == null)
return;
if (overallRoot.data < min) {
node = overallRoot = node.right;
rootFinder(node, min, max);
}
else if (overallRoot.data > max) {
node = overallRoot = node.left;
rootFinder(node, min, max);
}
else
cutter(overallRoot, min, max);
}
This second method should eliminate any further nodes not within the min/max, but it doesn't work as I would hope.
private void cutter(IntTreeNode node, int min, int max) {
if (node == null)
return;
if (node.data <= min) {
node.left = null;
}
if (node.data >= max) {
node.right = null;
}
if (node.data < min) {
node = node.right;
}
if (node.data > max) {
node = node.left;
}
cutter(node.left, min, max);
cutter(node.right, min, max);
}
This returns:
overallRoot
[54]_____
\
[72]
/
[61]
Any help is appreciated. Feel free to ask for further explanation as needed.
This assumes that a node x has the following values:
left (pointer to the left child)
right (pointer to the right child)
parent (pointer to parent)
You might want to make a method called CutBranch, which should simply remove a node an all it's subtrees from your tree. Let T be your tree, and let T.root be a pointer to it's root. It could then work like this:
CutBranch(x,T) {
y = T.root;
while (y.left != x && y.right != x) {
if (y < x) y = y.right;
else y = y.left;
}
if (y < x) y.right = Nil;
else y.left = Nil;
}
This assumes that your tree doesn't include nodes with equal values of course, but it takes O(lg n) time. It doesn't do any garbage collection however.
now you can iterate through the nodes, and every time you reach a node smaller than your lower bound, you can call CutBranch on it's left child, and then delete the node itself. If the node is larger than your upper bound, then you can CutBranch it's right child and delete it.
Great question, thumbs up, although I think if you consider a different approach it gets easier. Like, for every node, first "TRIM" the children, and then "TRIM" itself.
The following method assumes the tree is a BST as per the example in your question.
public Node trim(Node root, int min, int max){
if(root==null)
return root;
root.rightChild = trim (root.rightChild, min, max);
root.leftChild = trim (root.leftChild,min,max);
if(root.key>max || root.key<min){
if(root.rightChild!=null)
return root.rightChild;
return root.leftChild;
}
return root;
}
Although, if you want it to work for any Binary Tree, weather BST or not. Just make the following changes to the if statement above.
if(root.key>max || root.key<min){
if(root.rightChild==null)
return root.leftChild;
else if(root.leftChild==null)
return root.rightChild;
else{
//randomly select one of the children to be parent and add the other child to the first free space in its sub tree
//This is based on personal preferences
Node temp = root.leftChild;
while(temp.leftChild!=null || temp.rightChild!=null){
temp = temp.leftChild;
}
if(temp.leftChild==null)
temp.leftChild=root.rightChild;
else
temp.rightChild=root.rightChild;
return root.leftChild;
}
}
The method should be called like
tree.root = trim(tree.root, min, max);
When working with a tree I find it easiest to have recursive methods which take a Node and return a Node, the idea being that I can then call the method to "update" nodes beneath me by calling the method on them.
In this case, for instance, you could have Node minBound(Node) which returns the subtree of this node which is above the lower bound. If the current Node is in the bound then apply recursively to each child and return yourself. If the current Node is not in the bound then return the updated child Node in the correct direction. If the current Node is null then just return null.
An equivalent method must be written for maxBound.
Then you can just do minBound(maxBound(root)) to get the new root for the tree.
(You could combine minBound and maxBound into the one method, but for ease of explanation I decided to split them out.)
EDIT: Since it's been so long, I thought I'd actually put a code sample up to show what I mean.
public void trim(int min, int max) {
overallRoot = minBound(maxBound(overallRoot,max),min);
}
private IntTreeNode minBound(IntTreeNode node, int min) {
if (node == null) // base case of our recursion
return null;
if (node.value < min) // we're too small, but our larger children might be in
return minBound(node.right, min);
// if we make it to here then we're in bounds, so update our left child
// (our right child is bigger than us, so doesn't need to be processed)
node.left = minBound(node.left, min);
return node;
}
private IntTreeNode maxBound(IntTreeNode node, int max) {
if (node == null) // base case of our recursion
return null;
if (node.value > max) // we're too big, but our smaller children might be in
return maxBound(node.left, max);
// if we make it to here then we're in bounds, so update our right child
// (our left child is smaller than us, so doesn't need to be processed)
node.right = maxBound(node.right, max);
return node;
}