(conj coll x), for performance reasons, will add x to the end of some collections and to the beginning of other collections.
Is there a way to check whether the collection will be naturally add-at-end or add-at-beginning? I.e.
(conj-at-end? x) ; false for lists and true for vectors.
It is fairly easy to write my own, by checking if something is a vector or list, etc, but this is error prone in case there is a new collection added or something changes.
conj is a part of IPersistentCollection interface. This interface defines only three methods:
conj - adds new element;
empty - returns an empty collection of the same type;
equiv - performs equality checks.
So, there is nothing like conj-at-end?.
I don't think it could be implemented, either. For example, how would you define it for internally unsorted collections like hash-maps and hash-sets?
(conj #{1 2 3 4 5} 6)
; => #{1 4 6 3 2 5}
Does conj add to the beginning or the end of a collection?
The question only makes sense for a sequential collection, for which you can just try it:
Define
(defn conj-at-end? [coll]
(and (sequential? coll)
(= (conj (empty coll) 1 2) [1 2])))
Then
(conj-at-end? []) ;=> true
(conj-at-end? ()) ;=> false
and
(conj-at-end? #{}) ;=> false
We also need the corresponding
(defn conj-at-beginning? [coll]
(and (sequential? coll)
(= (conj (empty coll) 1 2) [2 1])))
Then
(conj-at-beginning? []) ;=> false
(conj-at-beginning? ()) ;=> true
and
(conj-at-beginning? #{}) ;=> false
Note that a set is neither conj-at-beginning? nor conj-at-end?.
This rather pat solution would be upset if, for example, clojure.lang.PersistentArrayMaps, produced by array-map,
were stable under conj, which they are not, and
elected to comply with the empty clojure.lang.Sequential interface,
tested for by sequential?, which they do not.
Related
I am trying to understand the below program to find the Fibonacci series using recursion in Clojure.
(defn fib
[x]
(loop [i '(1 0)]
(println i)
(if (= x (count i))
(reverse i)
(recur
(conj i (apply + (take 2 i))))))) // This line is not clear
For ex for a call fib(4) I get the below output,
(1 0)
(1 1 0)
(2 1 1 0)
(0 1 1 2)
Which as per my inference the conj seems to add the value of (apply + (take 2 i)) to the start of the i. But that is not the behaviour of conj. Can someone help me understand how exactly this works?
That is the behavior of conj, for lists. conj doesn't always add to the end:
(conj '(1) 2) ; '(2 1)
(conj [1] 2) ; [1 2]
The placement of the added element depends on the type of the collection. Since adding to the end of a list is expensive, conj adds to to front instead. It's the same operation (adding to a list), but optimized for the collection being used.
Per Clojure documentation:
The 'addition' may happen at different 'places' depending on the concrete type.
Appending to list happens to beginning of list, appending to vector happens to the end...
See more examples at https://clojuredocs.org/clojure.core/conj
I know this is a recurring question (here, here, and more), and I know that the problem is related to creating lazy sequencies, but I can't see why it fails.
The problem: I had written a (not very nice) quicksort algorithm to sort strings that uses loop/recur. But applied to 10000 elements, I get a StackOverflowError:
(defn qsort [list]
(loop [[current & todo :as all] [list] sorted []]
(cond
(nil? current) sorted
(or (nil? (seq current)) (= (count current) 1)) (recur todo (concat sorted current))
:else (let [[pivot & rest] current
pred #(> (compare pivot %) 0)
lt (filter pred rest)
gte (remove pred rest)
work (list* lt [pivot] gte todo)]
(recur work sorted)))))
I used in this way:
(defn tlfnum [] (str/join (repeatedly 10 #(rand-int 10))))
(defn tlfbook [n] (repeatedly n #(tlfnum)))
(time (count (qsort (tlfbook 10000))))
And this is part of the stack trace:
[clojure.lang.LazySeq seq "LazySeq.java" 49]
[clojure.lang.RT seq "RT.java" 521]
[clojure.core$seq__4357 invokeStatic "core.clj" 137]
[clojure.core$concat$fn__4446 invoke "core.clj" 706]
[clojure.lang.LazySeq sval "LazySeq.java" 40]
[clojure.lang.LazySeq seq "LazySeq.java" 49]
[clojure.lang.RT seq "RT.java" 521]
[clojure.core$seq__4357 invokeStatic "core.clj" 137]]}
As far as I know, loop/recur performs tail call optimization, so no stack is used (is, in fact, an iterative process written using recursive syntax).
Reading other answers, and because of the stack trace, I see there's a problem with concat and adding a doall before concat solves the stack overflow problem. But... why?
Here's part of the code for the two-arity version of concat.
(defn concat [x y]
(lazy-seq
(let [s (seq x)]
,,,))
)
Notice that it uses two other functions, lazy-seq, and seq. lazy-seq is a bit like a lambda, it wraps some code without executing it yet. The code inside the lazy-seq block has to result in some kind of sequence value. When you call any sequence operation on the lazy-seq, then it will first evaluate the code ("realize" the lazy seq), and then perform the operation on the result.
(def lz (lazy-seq
(println "Realizing!")
'(1 2 3)))
(first lz)
;; prints "realizing"
;; => 1
Now try this:
(defn lazy-conj [xs x]
(lazy-seq
(println "Realizing" x)
(conj (seq xs) x)))
Notice that it's similar to concat, it calls seq on its first argument, and returns a lazy-seq
(def up-to-hundred
(reduce lazy-conj () (range 100)))
(first up-to-hundred)
;; prints "Realizing 99"
;; prints "Realizing 98"
;; prints "Realizing 97"
;; ...
;; => 99
Even though you asked for only the first element, it still ended up realizing the whole sequence. That's because realizing the outer "layer" results in calling seq on the next "layer", which realizes another lazy-seq, which again calls seq, etc. So it's a chain reaction that realizes everything, and each step consumes a stack frame.
(def up-to-ten-thousand
(reduce lazy-conj () (range 10000)))
(first up-to-ten-thousand)
;;=> java.lang.StackOverflowError
You get the same problem when stacking concat calls. That's why for instance (reduce concat ,,,) is always a smell, instead you can use (apply concat ,,,) or (into () cat ,,,).
Other lazy operators like filter and map can exhibit the exact same problem. If you really have a lot of transformation steps over a sequence consider using transducers instead.
;; without transducers: many intermediate lazy seqs and deep call stacks
(->> my-seq
(map foo)
(filter bar)
(map baz)
,,,)
;; with transducers: seq processed in a single pass
(sequence (comp
(map foo)
(filter bar)
(map baz))
my-seq)
Arne had a good answer (and, in fact, I'd never noticed cat before!). If you want a simpler solution, you can use the glue function from the Tupelo library:
Gluing Together Like Collections
The concat function can sometimes have rather surprising results:
(concat {:a 1} {:b 2} {:c 3} )
;=> ( [:a 1] [:b 2] [:c 3] )
In this example, the user probably meant to merge the 3 maps into one. Instead, the three maps were mysteriously converted into length-2 vectors, which were then nested inside another sequence.
The conj function can also surprise the user:
(conj [1 2] [3 4] )
;=> [1 2 [3 4] ]
Here the user probably wanted to get [1 2 3 4] back, but instead got a nested vector by mistake.
Instead of having to wonder if the items to be combined will be merged, nested, or converted into another data type, we provide the glue function to always combine like collections together into a result collection of the same type:
; Glue together like collections:
(is (= (glue [ 1 2] '(3 4) [ 5 6] ) [ 1 2 3 4 5 6 ] )) ; all sequential (vectors & lists)
(is (= (glue {:a 1} {:b 2} {:c 3} ) {:a 1 :c 3 :b 2} )) ; all maps
(is (= (glue #{1 2} #{3 4} #{6 5} ) #{ 1 2 6 5 3 4 } )) ; all sets
(is (= (glue "I" " like " \a " nap!" ) "I like a nap!" )) ; all text (strings & chars)
; If you want to convert to a sorted set or map, just put an empty one first:
(is (= (glue (sorted-map) {:a 1} {:b 2} {:c 3}) {:a 1 :b 2 :c 3} ))
(is (= (glue (sorted-set) #{1 2} #{3 4} #{6 5}) #{ 1 2 3 4 5 6 } ))
An Exception will be thrown if the collections to be 'glued' are not all of the same type. The allowable input types are:
all sequential: any mix of lists & vectors (vector result)
all maps (sorted or not)
all sets (sorted or not)
all text: any mix of strings & characters (string result)
I put glue into your code instead of concat and still got a StackOverflowError. So, I also replaced the lazy filter and remove with eager versions keep-if and drop-if to get this result:
(defn qsort [list]
(loop [[current & todo :as all] [list] sorted []]
(cond
(nil? current) sorted
(or (nil? (seq current)) (= (count current) 1))
(recur todo (glue sorted current))
:else (let [[pivot & rest] current
pred #(> (compare pivot %) 0)
lt (keep-if pred rest)
gte (drop-if pred rest)
work (list* lt [pivot] gte todo)]
(recur work sorted)))))
(defn tlfnum [] (str/join (repeatedly 10 #(rand-int 10))))
(defn tlfbook [n] (repeatedly n #(tlfnum)))
(def result
(time (count (qsort (tlfbook 10000)))))
-------------------------------------
Clojure 1.8.0 Java 1.8.0_111
-------------------------------------
"Elapsed time: 1377.321118 msecs"
result => 10000
> (conj [0] 1 2 3)
[0 1 2 3]
> (conj {:a "ei"} {:b "bi"})
{:b "bi", :a "ei"}
>
See, when it acts on vector, it puts 1,2,3 at end of it.
Whereas it put :b "bi" in front of :a map k-v pair
Why is this?
thanks
As with many hashed maps implementations, Clojure's hashed maps do not sort their entries, not retain the order in which they were inserted. This allows for better performance.
Note that conj does not have general ordering semantics either (it has ordering semantics for some concrete types, such as vectors).
You don't have to go as far as maps to get inconsistent behaviour from conj:
(conj [1] 2) ; [1 2]
(conj (list 1) 2) ; (2 1)
Hash maps have no defined order. But, for any map,
the seq of entries will always be the same
the vals and keys will be in consistent order.
Thus, for map m,
(= (keys m) (map key m))
(= (vals m) (map val m))
(= m (zipmap (keys m) (vals m)))
Currently, this sequence seems to be independent of insertion order. I tested this on sets by randomly shuffling random integers.
New to Clojure so I've been going through 4Clojure's questions to get familiar with the core library before jumping on a project and have ran into this question:
// Write a function which reverses a sequence.
(= (__ [1 2 3 4 5]) [5 4 3 2 1])
This is just one of the test-cases but here is what I had come up with:
(fn [x my-seq]
(if (empty? my-seq)
x
(do
(into x (take-last 1 my-seq))
(recur x (into [] (drop-last my-seq)))))) []
I was getting an empty vector [ ] after executing this code in a repl so
I modified the code like so:
(fn [x my-seq]
(if (empty? my-seq)
x
(recur (into x (take-last 1 my-seq)) (into [] (drop-last my-seq))))) []
Question is why was my previous code not working? To me it seems logically equivalent, the modified code just seems cleaner where it avoids the do form. Again I'm new to Clojure so i'm not entirely familiar with the do and recur forms.
(into x (take-last 1 my-seq)) is the problem.
Note that you cannot change x. (into x ...) creates a new vector and returns it.
However the return values of every statement inside (do ...) except for the last one are dropped. You are recurring with the original - empty - x.
Say that I have a function:
(defn get-token [char]
(defn char->number? []
(re-matches #"\d" (str char)))
(defn whitespace? []
(or
(= \space char)
(= \newline char)))
(defn valid-ident-char? []
(re-matches #"[a-zA-Z_$]" (str char)))
(cond
(whitespace?)
nil
(= \' char)
:quote
(= \) char)
:rparen
(= \( char)
:lparen
(char->number?)
:integer
(valid-ident-char?)
:identifier
:else
(throw (Exception. "invalid character"))))
When I run this function on, for instance, the string "(test 1 2)", I get a list of symbols for each character:
'(:lparen :identifier :identifier :identifier nil :integer nil :integer :rparen)
Seeing that this is not entirely what I want, I am trying to write a function that takes a collection and "condenses" the collection to combine adjacent elements that are equal.
A final example might do this:
(defn combine-adjacent [coll]
implementation...)
(->>
"(test 1 2)"
(map get-token)
(combine-adjacent)
(remove nil?))
; => (:lparen :identifier :integer :integer :rparen)
What is the idiomatic Clojure way to achieve this?
Clojure 1.7 will introduce a new function called dedupe to accomplish exactly this:
(dedupe [0 1 1 2 2 3 1 2 3])
;= (0 1 2 3 1 2 3)
If you're prepared to use 1.7.0-alpha2, you could use it today.
The implementation relies on transducers (dedupe produces a transducer when called with no arguments; the unary overload is defined simply as (sequence (dedupe) coll)), so it wouldn't be straightforward to backport.
Not sure how idiomatic it is but one way to do it would be to use partition-by to group elements of the incoming sequence into lists containing subsequences of the same element and then use map to get the first element from each of those lists.
So, in code
(defn combine-adjacent [input]
(->> input (partition-by identity) (map first)))
or
(defn combine-adjacent [input]
(->> (partition-by identity input) (map first))
should work.
There are a couple of tricks for comparing items to the one next door:
first, we can compare it to its tail:
(defn combine-adjacent
[s]
(mapcat #(when (not= % %2) [%]) (rest s) s))
alternatively, we can take the sequence by twos, and drop repeats
(defn combine-adjacent
[s]
(mapcat (fn [[a b]] (if (not= a b) [a]) ())
(partition 2 1[0 1 2 2 2 3 2 3])))
both of these take advantage of the helpful property of concat when combined with map that you can return zero or more elements for the result sequence for each input. The empty list for the false case in the second version is not needed, but may help with clarity.