(st-json:write-json-to-string
(st-json:read-json "{\"a\":0.1}"))
Output is not valid JSON:
"{\"a\":1.e-1}"
Desired output:
"{\"a\":0.1}"
I don't see any options or other arguments for write-json-to-string at https://marijnhaverbeke.nl/st-json/
I would still be ok if the number came out a little modified like .1 or 0.10
Update: There is a system variable for float format: "The printer uses *read-default-float-format* to guide the choice of exponent markers when printing floating-point numbers."
According to this example ...
(let ((*read-default-float-format* 'double-float))
(read-from-string "(1.0 1.0e0 1.0s0 1.0f0 1.0d0 1.0L0)"))
==> (1.0d0 1.0d0 1.0 1.0 1.0d0 1.0d0)
The implementation I'm using (SBCL 2.0.11) has float formats F and D. http://www.lispworks.com/documentation/lw50/CLHS/Body/v_rd_def.htm
I'm not sure how "The printer uses *read-default-float-format* to guide the choice of exponent markers when printing floating-point numbers." ... Is there something wrong with my test below?
(progn
(setf *read-default-float-format* 'long-float)
(princ(st-json:write-json-to-string (st-json:read-json "{\"a\":0.1}")))
(setf *read-default-float-format* 'single-float)
(princ(st-json:write-json-to-string (st-json:read-json "{\"a\":0.1}")))
(setf *read-default-float-format* 'double-float)
(princ(st-json:write-json-to-string (st-json:read-json "{\"a\":0.1}")))
(setf *read-default-float-format* 'short-float)
(princ(st-json:write-json-to-string (st-json:read-json "{\"a\":0.1}"))))
{"a":1.e-1}{"a":1.e-1}{"a":1.e-1}{"a":1.e-1}
==> "{\"a\":1.e-1}"
Thanks in advance for any help!
It got nothing to do with *read-default-float-format*.
Using M-. in SLIME, I could see that write-json-to-string calls write-json which in turn calls write-json-element.
write-json-element is a generic function with different methods for different type. method for real is:
(defmethod write-json-element ((element real) stream)
(format stream "~,,,,,,'eE" element))
That format string is standard Common Lisp, resulting in the behaviour you are seeing.
CL-USER> (format t "~,,,,,,'eE" 0.1)
1.e-1
NIL
CL-USER>
I have no idea why author did that, but if you are stuck with st-json, Just redefine the method as (which will issue a re-definition warning, but ignore that):
(defmethod st-json:write-json-element ((element real) stream)
(format stream "~F" element))
However, if you could switch to another library, take a look at jzon, it is in latest Quicklisp.
Related
I want some reader macros to print as as shortened expression that the macro understands. Lets say I want to extend the #' macro to take #'~[rest-of-symbol] and turn that into (complement #'rest-of-symbol).
What controls how that is printed? On SBCL, for instance, '(function +) prints as #'+. How do i make '(complement #'listp) print as #~listp?
My first thought was
(defmethod print-object :around ((obj cons) stream)
;; if #'~fn-name / (complement (function fn-name))
;; => fn-name otherwise NIL
(let ((fn-name
(ignore-errors
(destructuring-bind (complement (function fn-name))
obj
(when (and (eq complement 'complement)
(eq function 'function))
fn-name)))))
(if fn-name
(format stream "#'~~~S" fn-name)
(call-next-method))))
This works insofar as (print-object '(complement #'evenp) *standard-output*) prints it the way I want, but the REPL doesn't. Also (print-object '#'+ *standard-output*) prints it as (function +) so the REPL isn't using print-object. With defining the print-object method for user defined classes the REPL always picks up on the new definition.
This is my first post and I'm sorry I can't get the code to format properly. If someone can put a link on how to do that I would appreciate it.
Evaluation time
You are mixing code with data in your example:
(function +)
Is a special form that evaluates to a function object, which admits a shorter syntax:
#'+
But when you are writing:
'(function +)
or
'(complement fn)
Then in both cases you are writing quoted, literal lists, which evaluates to themselves (namely a list starting with symbol function or complement, followed respectively by symbol + and fn).
However, you want the code to be evaluated at runtime to actual function objects; if you type this in the REPL:
(complement #'alpha-char-p)
The result is a value that is printed as follows:
#<FUNCTION (LAMBDA (&REST SB-IMPL::ARGUMENTS) :IN COMPLEMENT) {101AAC8D9B}>
You have an actual function object that you can funcall. In other words, by the time you reach print-object, you no longer have access to source code, you are manipulating data at runtime which happens to be functions. So you cannot use destructuring-bind to get the complement symbol that was present in the source code.
What you need to do instead is to attach metadata to your function. There is a way to do that in Common Lisp by defining a new type of function, thanks to the Meta-Object Protocol.
Funcallable objects
I'm relying on Closer-MOP for all the symbols prefixed with c2cl: below. I define a new class of functions, annotated-fn, which is a function with addditional data:
(defclass annotated-fn (c2cl:funcallable-standard-object)
((data :initform :data :initarg :data :reader annotated-fn-data))
(:metaclass c2cl:funcallable-standard-class))
Notice that this class is a funcallable-standard-object (like the usual functions), and its metaclass is funcallable-standard-class. Such an object has an additional implicit slot that is a function to call.
More precisely, you have to call c2cl:set-funcallable-instance-function to set a function associated with the object, and when later you use funcall or apply with the object, then the wrapped function is called instead. So you can transparently use this class of functions wherever you usually use a function. It just has additional slots (here data).
For example, here is how I instantiate it, with a function to wrap additional data:
(defun annotate-fn (function data)
(let ((object (make-instance 'annotated-fn :data data)))
(prog1 object
(c2cl:set-funcallable-instance-function object function))))
Let's try it:
(describe
(annotate-fn (constantly 3)
'(:category :constantly)))
#<ANNOTATED-FN {1006275C7B}>
[funcallable-instance]
Lambda-list: UNKNOWN
Derived type: FUNCTION
Documentation:
T
Source file: SYS:SRC;CODE;FUNUTILS.LISP
Slots with :INSTANCE allocation:
DATA = (:CATEGORY :CONSTANTLY)
You can also use this object like any other function.
Now, your reader macros can expand into calls to annotate-fn, and add any kind of additional metadata you need to the function.
Reader macro
For our example, imagine you define a reader macros for constant functions:
(set-macro-character #\[ 'read-constantly t)
(set-macro-character #\] (get-macro-character #\) nil))
(defun read-constantly (stream char)
(declare (ignore char))
(let* ((list (read-delimited-list #\] stream t))
(value (if (rest list) list (first list)))
(var (gensym)))
`(let ((,var ,value))
(annotate-fn (constantly ,var)
(list :category :constantly
:constant ,var)))))
Using this syntax:
> [(+ 8 5)]
=> #<ANNOTATED-FN ...>
By the way, the syntax I defined also allows the following:
> [+ 8 5]
Pretty-printing
Let's define a generic function that prints an annotated function given its :category field:
(defgeneric print-for-category (category data object stream))
(defmethod print-object ((o annotated-fn) s)
(let* ((data (annotated-fn-data o))
(category (getf data :category)))
(print-for-category category data o s)))
Then, we can specialize it for :constantly, and here we assume also that the data associated with the function contains a :constant field:
(defmethod print-for-category ((_ (eql :constantly)) data o s)
(format s "[~s]" (getf data :constant)))
For example:
(let ((value (+ 8 6)))
(annotate-fn (constantly value)
`(:constant ,value
:category :constantly)))
This above is printed as:
[14]
Which would be the same as your hypothetical reader macro.
To do this you need to understand the pretty printer. I have understood it in the past but no longer do completely. It dispatches on type and the trick for things like this is that you can specify very specific types for trees of conses, although doing so is verbose.
Here is an example which is almost certainly not completely correct, but does achieve what you want in this case:
(defparameter *ppd* (copy-pprint-dispatch))
(defun pprint-complement-function (s form)
;; This is the thing that the pretty printer will call. It can
;; assume that the form it wants to print is already correct.
(destructuring-bind (complement (function name)) form
(declare (ignore complement function))
(format s "#'~~~W" name)))
;;; Now set this in the table with a suitable hairy type specification
;;;
(set-pprint-dispatch '(cons (eql complement)
(cons (cons (eql function)
(cons t null))
null))
'pprint-complement-function
0
*ppd*)
And now
> (let ((*print-pprint-dispatch* *ppd*))
(pprint '(complement (function foo)))
(pprint '((complement (function foo)) (function foo))))
#'~foo
(#'~foo #'foo)
You can make the awful nested cons type specifier easier by defining this (which, perhaps, should be the compound type specifier for list except you can't do that):
(deftype list-of-types (&rest types)
(labels ((lot (tt)
(if (null tt)
'null
`(cons ,(first tt) ,(lot (rest tt))))))
(lot types)))
And then
(set-pprint-dispatch '(list-of-types (eql complement)
(list-of-types (eql function)
*))
'pprint-complement-function
0
*ppd*)
is perhaps easier to read.
I have a struct with :name and :value that I'd like to use as arguments to a macro. But I'm not sure how to tell lisp that.
I can write out the call like
(sxql:yield (sxql:set= :name "a" :value 1))
"SET name = ?, value = ?"
("a" 1)
But I'd like to use an already existing structure
(defstruct my-struct name value)
(setq x (make-my-struct :name "a" :value 1))
; #S(MY-STRUCT :NAME "a" :VALUE 1)
using answers from Common LISP: convert (unknown) struct object to plist?
I've made
(defun struct-plist (x)
"make struct X into a property list. ugly kludge"
(let* ((slots (sb-mop:class-slots (class-of x)))
(names (mapcar 'sb-mop:slot-definition-name slots)))
(alexandria:flatten
(mapcar (lambda (n) (list (intern (string n) "KEYWORD")
(slot-value x n)))
names))))
(setq p (struct-plist x)) ; (:NAME "a" :VALUE 1)
My naive attempts are
(sxql:set= p) ; error in FORMAT: No more argument SET ~{~A = ~A~^, ~}
(funcall 'sxql:set= p) ; SXQL:SET= is a macro, not a function.
(macroexpand (sxql:set= p)) ; error in FORMAT ...
I imagine this is an easy/fundamental lisp programming question. But I'm not sure how to ask it (or search for answers). I'm also hoping there is an better struct<->plist story than what I've stumbled across so far.
EDIT: In case this is really an xy-problem. I've used flydata:defmodel to create the struct and I want to insert to a database using the same model.
This is definitely an xy problem: unfortunately I don't understand y (flydata?) well enough to answer the y part.
Here's why what you are trying to do can't work however. Consider this code in a file being compiled:
(defstruct mine name value)
...
(sxql:set= <anything derived from mine>)
Compiling this file must satisfy two constraints:
It does not fully create the structure type mine (see defstruct);
It must macroexpand sxql:set=.
What these constraints mean is that sxql:set= can't know about the structure at the time it is expanded. So any trick which relies on information about the structure must make that information available at compile time.
As I said, I don't understand the y part well enough to understand what you are trying to do, but a hacky approach to this is:
write a wrapper for defstruct which stashes information at compile time (strictly: at macro-expansion time);
write a wrapper for sxql:set= which uses that information to expand into something which makes sense.
Here is a mindless wrapper for defstruct. Note that this is mindless: it can only understand the most simple defstruct forms, and even then it may be wrong. It exists only as an example.
(eval-when (:compile-toplevel :load-toplevel :execute)
(defvar *structure-information* '()))
(defmacro define-mindless-structure (name &body slots)
(assert (and (symbolp name)
(every #'symbolp slots))
(name slots)
"I am too mindless")
(let ((found (or (assoc name *structure-information*)
(car (push (list name) *structure-information*)))))
(setf (cdr found) (mapcar (lambda (slot)
(list slot (intern (symbol-name slot)
(find-package "KEYWORD"))
(intern (concatenate 'string
(symbol-name name)
"-"
(symbol-name slot)))))
slots)))
`(defstruct ,name ,#slots))
So now
(define-mindless-structure mine
name value)
Will expand into (defstruct mine name value) and, at macroexpansion time will stash some information about this structure in *structure-information*.
Now I stop really understanding what you need to do because I don't know what sxql:set= is meant to do, but it might be something like this:
(defmacro mindless-set= ((s o))
(let ((info (assoc s *structure-information*))
(ov (make-symbol "O")))
(unless info
(error "no information for ~A" s))
`(let ((,ov ,o))
(sxql:set= ,#(loop for (slot initarg accessor) in (cdr info)
;; the compiler will whine about slot annoyingly
collect initarg
collect `(,accessor ,ov))))))
So with this macro, assuming a suitable define-mindless-structure for mine form has been seen by the time the macro is expanded, then
(mindless-set= (mine it))
Will expand into
(let ((#:o it))
(set= :name (mine-name #:o) :value (mine-value #:o)))
But, as I said, I am not sure what the expansion you actually want is.
Finally, before even thinking about using anything like the above, it would be worth looking around to see if there are portability libraries which provide compile/macroexpansion-time functionality like this: there very well may be such, as I don't keep up with things.
A simple reflection warning example:
lein repl
user=> (set! *warn-on-reflection* true)
true
user=> (eval '(fn [x] (.length x)))
Reflection warning, NO_SOURCE_PATH:1:16 - reference to field length can't be resolved.
#object[user$eval2009$fn__2010 0x487ba4b8 "user$eval2009$fn__2010#487ba4b8"]
I want to make this into a function. But where do reflection warnings go?
//clojure/compile.java 63
RT.errPrintWriter()
.format("Reflection warning, %s:%d:%d - reference to field %s can't be resolved.\n",
SOURCE_PATH.deref(), line, column, fieldName);
//clojure/RT.java 269
public static PrintWriter errPrintWriter(){
Writer w = (Writer) ERR.deref();
//clojure/RT.java 188
final static public Var ERR =
Var.intern(CLOJURE_NS, Symbol.intern("*err*"),
new PrintWriter(new OutputStreamWriter(System.err), true)).setDynamic();
Ok so they go to System.err. Lets capture it's output:
(def pipe-in (PipedInputStream.))
(def pipe-out (PipedOutputStream. pipe-in))
(System/setErr (PrintStream. pipe-out))
(defn reflection-check [fn-code]
(binding [*warn-on-reflection* true]
(let [x (eval fn-code)
;_ (.println (System/err) "foo") ; This correctly makes us return "foo".
n (.available pipe-in)
^bytes b (make-array Byte/TYPE n)
_ (.read pipe-in b)
s (apply str (mapv char b))]
s)))
However, calling it gives no warning, and no flushing seems to be useful:
(println "Reflection check:" (reflection-check '(fn [x] (.length x)))) ; no warning.
How can I extract the reflection warning?
You have correctly discovered how *err* is initialized, but since vars are rebindable this is no guarantee about its current value. The REPL often rebinds it to something else, e.g. a socket. If you want to redirect it yourself, you should simply rebind *err* to a Writer of your choosing.
Really I'm not sure your approach would work even if *err* were never rebound. The Clojure runtime has captured a pointer to the original value of System.err, and then you ask the Java runtime to use a new value for System.err. Clojure certainly won't know about this new value. Does the JRE maintain an extra level of indirection to allow it to do these swaps behind the scenes even for people who have already captured System.err? Maybe, but if so it's not documented.
I ran into a similar problem a while back and created some helper functions modelled on with-out-str. Here is a solution to your problem:
(ns tst.demo.core
(:use tupelo.core tupelo.test) )
(defn reflection-check
[fn-code]
(let [err-str (with-err-str
(binding [*warn-on-reflection* true]
(eval fn-code)))]
(spyx err-str)))
(dotest
(reflection-check (quote (fn [x] (.length x)))))
with result:
-------------------------------
Clojure 1.10.1 Java 14
-------------------------------
err-str => "Reflection warning, /tmp/form-init3884945788481466752.clj:12:36
- reference to field length can't be resolved.\n"
Note that binding and let forms can be in either order and still work.
Here is the source code:
(defmacro with-err-str
"Evaluates exprs in a context in which *err* is bound to a fresh
StringWriter. Returns the string created by any nested printing
calls."
[& body]
`(let [s# (new java.io.StringWriter)]
(binding [*err* s#]
~#body
(str s#))))
If you need to capture the Java System.err stream, it is different:
(defmacro with-system-err-str
"Evaluates exprs in a context in which JVM System/err is bound to a fresh
PrintStream. Returns the string created by any nested printing calls."
[& body]
`(let [baos# (ByteArrayOutputStream.)
ps# (PrintStream. baos#)]
(System/setErr ps#)
~#body
(System/setErr System/err)
(.close ps#)
(.toString baos#)))
See the docs here.
There are 5 variants (plus clojure.core/with-out-str):
with-err-str
with-system-out-str
with-system-err-str
discarding-system-out
discarding-system-err
Source code is here.
I want to retrieve the string, generated by write for further processing without doing any actual output, but write seems always to also output into the REPL
CL-USER>(let ((err-string (write (make-instance 'error) :stream nil)))
(do-awesome-stuff-with-string err-string))
<ERROR> ;;this is the printing I want to get rid of
"awesome-result"
Why does write still outputs into the REPL, and how do I get rid of that?
You can use with-output-to-string for this. Here's an example:
(flet ((do-awesome-stuff-with-string (string)
(concatenate 'string string " is awesome!")))
(let ((err-string (with-output-to-string (s)
(write (make-instance 'error) :stream s))))
(do-awesome-stuff-with-string err-string)))
;; => "#<ERROR {25813951}> is awesome!"
Here's here's the HyperSpec entry on with-output-to-string.
The reason (write (make-instance 'error) :stream nil) doesn't work is that the :stream argument to write is a stream designator and in that context nil is shorthand for *standard-output*. (The fact that format instead takes nil to mean that it should return a string is a common point of confusion).
Keep in mind that portably errors are made with MAKE-CONDITION. The standard does not say that errors are CLOS classes, so MAKE-INSTANCE might not work in some implementations.
There are two simple ways to get a string:
a) a textual description:
CL-USER 15 > (princ-to-string (make-condition 'error))
"The condition #<ERROR 4020311270> occurred"
b) the error object printed:
CL-USER 16 > (prin1-to-string (make-condition 'error))
"#<ERROR 402031158B>"
To put it another, way, "Okay, so code is data..."
That thread addresses how to read from a source file, but I'm wondering how to get the s-expression of an already-loaded function into a data structure that I can read and manipulate.
In other words, if I say,
(defn example [a b] (+ a b))
can't I get that list at runtime? Isn't this the whole point of "code as data"?
This is really a general Lisp question, but I'm looking for an answer in Clojure.
You can use the clojure.repl/source macro to get the source of a symbol:
user> (source max)
(defn max
"Returns the greatest of the nums."
{:added "1.0"
:inline-arities >1?
:inline (nary-inline 'max)}
([x] x)
([x y] (. clojure.lang.Numbers (max x y)))
([x y & more]
(reduce1 max (max x y) more)))
nil
But this is only part of the answer. AFAICT source looks up the source filename and line number that define the given symbol, and then prints the source code from the file. Therefore, source will not work on symbols that you do not have the source for, i.e. AOT-compiled clojure code.
Coming back to your original question, you can think of source as reading the meta data associated with the given symbol and simply printing that. I.e. it's cheating. It's not in any way returning "code as data" to you, where with code I mean a compiled clojure function.
In my mind "code as data" refers to the feature of lisps where source code is effectively a lisp data structure, and therefore it can be read by the lisp reader. That is, I can create a data structure that is valid lisp code, and eval that.
For example:
user=> (eval '(+ 1 1))
2
Here '(+ 1 1) is a literal list which gets read by the clojure reader and then evaluated as clojure code.
Update: Yehonathan Sharvit was asking in one of the comments if it's possible to modify the code for a function. The following snippet reads in the source for a function, modifies the resulting data structure, and finally evaluates the data structure resulting in a new function, my-nth, being defined:
(eval
(let [src (read-string (str (source-fn 'clojure.core/nth) "\n"))]
`(~(first src) my-nth ~#(nnext src))))
The syntax-quote line replaces nth with my-nth in the defn form.
You can get the source in recent versions of clojure with the source function.
user=> (source nth)
(defn nth
"Returns the value at the index. get returns nil if index out of
bounds, nth throws an exception unless not-found is supplied. nth
also works for strings, Java arrays, regex Matchers and Lists, and,
in O(n) time, for sequences."
{:inline (fn [c i & nf] `(. clojure.lang.RT (nth ~c ~i ~#nf)))
:inline-arities #{2 3}
:added "1.0"}
([coll index] (. clojure.lang.RT (nth coll index)))
([coll index not-found] (. clojure.lang.RT (nth coll index not-found))))
nil
to get the string as a value you can wrap this in with-out-str:
user=> (with-out-str (source nth))
"(defn nth\n \"Returns the value at the index. get returns nil if index out of\n bounds, nth throws an exception unless not-found is supplied. nth\n also works for strings, Java arrays, regex Matchers and Lists, and,\n in O(n) time, for sequences.\"\n {:inline (fn [c i & nf] `(. clojure.lang.RT (nth ~c ~i ~#nf)))\n :inline-arities #{2 3}\n :added \"1.0\"}\n ([coll index] (. clojure.lang.RT (nth coll index)))\n ([coll index not-found] (. clojure.lang.RT (nth coll index not-found))))\n"
user=>
That was my message; nice to meet you ;-) BTW, the references given in that thread for answers were excellent reading; so if you're interested, you might want to take the time to read them. Back to your question though source seems to work for code that was loaded through a file, but it doesn't work in all cases. I think, specifically, it doesn't work for functions defined in the repl.
user=> (def foo (fn [] (+ 2 2)))
#'user/foo
user=> (source foo)
Source not found
nil
user=> (defn foo2 [] (+ 2 2))
#'user/foo2
user=> (source foo2)
Source not found
nil
Digging a little bit...
user=> (source source)
(defmacro source
"Prints the source code for the given symbol, if it can find it.
This requires that the symbol resolve to a Var defined in a
namespace for which the .clj is in the classpath.
Example: (source filter)"
[n]
`(println (or (source-fn '~n) (str "Source not found"))))
nil
user=> (source clojure.repl/source-fn)
(defn source-fn
"Returns a string of the source code for the given symbol, if it can
find it. This requires that the symbol resolve to a Var defined in
a namespace for which the .clj is in the classpath. Returns nil if
it can't find the source. For most REPL usage, 'source' is more
convenient.
Example: (source-fn 'filter)"
[x]
(when-let [v (resolve x)]
(when-let [filepath (:file (meta v))]
(when-let [strm (.getResourceAsStream (RT/baseLoader) filepath)]
(with-open [rdr (LineNumberReader. (InputStreamReader. strm))]
(dotimes [_ (dec (:line (meta v)))] (.readLine rdr))
(let [text (StringBuilder.)
pbr (proxy [PushbackReader] [rdr]
(read [] (let [i (proxy-super read)]
(.append text (char i))
i)))]
(read (PushbackReader. pbr))
(str text)))))))
nil
So yeah, it looks like it tries to load the source file off the classpath to try to spit it out for you. One thing I've learned when working with Clojure is that 9 times out of 10 it is useful to look at the source.