Lisp cl-dbi - how to embed control characters in sqlite queries - sqlite

How do you embed control characters (ie " or ' or :) in cl-dbi queries for sqlite3?
currently using a make-string-output/get-output-stream-string to build a variable that contains quoted data (json). Now, I want to be able to store the data in a sqlite3 db, but I'm obviously building the string wrong, because I get an error
DB Error: unrecognized token: ":" (Code: ERROR)
how do I escape characters in cl-dbi to pass them through to sqlite3?
EDIT - here's a brief passage of the JSON data that I'm trying to store (as text) in the sqlite3 db:
{
"type": "artificial",
Update: GAH! Took me a day to find an errant : in the prepared query string. :/

As far as I can make out, what you are trying to do is generate, in Lisp, some string which contains valid SQL, which itself contains an SQL literal string.
First of all this is an instance of an antipattern I call 'language in a string'. I keep thinking I have a big diatribe about this to point people at but it seems I haven't. Suffice it to say it's kind of the antithesis of what Lisp people have tried to achieve for more than 60 years and it's why we have SQL injection attacks and a lot of the other crap that afflicts us. But that battle is long lost and all we can do is try to avoid actually drowning in the sea of mud and rotting bits of people that now litters the battlefield.
So, to do this you need to be able to do two things.
You need to be able to generate an SQL literal string from a sequence of characters (or from a string). This means you need to know the syntax of literal SQL strings and in particular what characters are legal in them and how you express characters which are not.
You need to be able to interpolate this string into a CL string.
The second of these is trivial: this is what format's ~A directive does. Or if you want to get fancy you could use cl-interpol.
For the first, I don't know the syntax of SQL literal strings, but I will give an example which assumes the following simple rules:
literal strings are delimited by " characters;
the character \ escapes the following character to remove it of any special significance;
all other characters are allowed (this is almost certainly wrong).
Well, there are lots of ways of doing this, all of which involve walking along the sequence of characters looking for the ones that need to be escaped. Here is something reasonably horrible and quick which I wrote. It needs a macro called nlet which is Scheme's 'named let' construct, and it assumes TRO in the implementation (if your implementation does not do this, get one that does).
(defmacro nlet (name bindings &body forms)
"named let"
(multiple-value-bind (vars vals) (values (mapcar (lambda (b)
(etypecase b
(symbol b)
(cons (first b))))
bindings)
(mapcar (lambda (b)
(etypecase b
(symbol 'nil)
(cons
(unless (null (cddr b))
(error "bad binding ~A" b))
(second b))))
bindings))
`(labels ((,name ,vars ,#forms))
(,name ,#vals))))
(defun ->sql-string (seq)
;; turn SEQ (a sequence of characters) into a string representing an
;; SQL literal string (perhaps)
(nlet loupe ((tail (coerce seq 'list))
(accum '()))
(if (null tail)
(coerce (cons #\" (nreverse (cons #\" accum))) 'string)
(destructuring-bind (first . rest) tail
(loupe rest
(case first
((#\\ #\")
(append (list first #\\) accum))
(otherwise
(cons first accum))))))))
So now:
> (->sql-string "foo")
"\"foo\""
> (->sql-string '(#\f #\\ #\o #\" #\o))
"\"f\\\\o\\\"o\""
This is made ugly by the Lisp printer, but (see above) we can see what the strings actually are:
> (format t "~&select x from y where x.y = ~A;~%"
(->sql-string '(#\f #\\ #\o #\" #\o)))
select x from y where x.y = "f\\o\"o";
nil
And you can see that the SQL literal string obeys the rules I set out above.
Before using anything like this check what the rules are, because if you get them wrong you are possibly open to SQL injection attacks.

Related

Usefulness/point of function "symbol-name"?

On first look it seems somewhat silly to have a function which returns the name of a symbol, which must be called using the name of that same symbol. i.e. it should already be obvious in the calling context what the function will return. On the other hand there is identity which is sometimes useful (I forget where just now) but I supposed (perhaps wrongly) that symbol-function is there for a reason other than simply to act as a kind of identity.
However, the hyperspec offers a possible hint:
(symbol-name 'temp) => "TEMP"
(symbol-name :start) => "START"
(symbol-name (gensym)) => "G1234" ;for example
I note that :start means get the name of the symbol start from the keyword package, where the keyword package is denoted simply by :.
(:keyword being its longer form, unnecessary to use). Thus, in this case symbol-name plays the role of simply removing the package prefix.
The other thing it might do is, given an implementation is case insensitive, it would get the actual name by removing case in the symbol name supplied.
Is that roughly it or is there any importance to this function I am missing?
One thing I was confused by about symbols (cleared up now) is that symbol-plist does not tell you everything about the symbol (say, whether it holds the value of a special variable or function). Rather, plist is a mainly legacy feature now largely replaced by hashtables. So, a call to symbol-plist is going to return NIL even if one has set a special variable on the symbol.
One final question on that, Paul Graham says in Chapter 8 (p133), that "You can use symbols as data objects and as names for things without understanding how the two are related". Would it be correct say that if we rarely now use plists, that, today, we generally don't use symbols "as data objects" at all, instead, just as names for things (allbeit with the duality in CL of course, i.e. symbol-function and symbol-value simultaneously).
Symbols are objects. You can create them and pass them around programmatically. One of the properties of these objects is their name, which is a string. Symbol-name is the reader (not accessor) for that. That's all.
Symbols are also used in the representation of code. In this case, they are created by the reader and used by the compiler. This is not their only use, however, and the spec makes no such assumptions either. I hope that this addresses the core of your question.
Here is a function that, given a symbol, returns a symbol interned in the same package but with a reversed name:
(defun reverse-symbol (symbol)
(intern (make-symbol (reverse (symbol-name symbol)))
(symbol-package symbol)))
Here is a function that, given a string designator, returns the designated string:
(defun designated-string (string-designator)
(ctypecase string-designator
(string string-designator)
(symbol (symbol-name string-designator))))
You can also do all kinds of shenanigans in macros and compiler macros by inspecting the names of the symbols in the given form and applying some conventions (not that you should…).
Let's assume you want to write some protocol for transmitting bits of structure between two different systems over some serial channel
(defgeneric transmit-object (o stream))
What is the method for symbols going to look like?
(defmethod transmit-object ((o symbol) stream)
... (symbol-name o) ...)
In particular you certainly do not know the name of the symbol at the point where you need to reason about it, and you could not write such a protocol without using symbol-name (or some absolutely appalling hack like printing the symbol to a string and using that).
Regarding symbol-plist, the implementation is completely at liberty to keep all sorts of information on the property list of a symbol. I am not even sure that the implementation is not allowed to do (a cleverer version of):
(defun symbol-value (s)
(let* ((d (load-time-value (cons nil nil)))
(v (get s secret-value-indicator d)))
(when (eq v d)
(error ...))
v))
It may be that this is not allowed, but I am not sure. Certainly it was once fairly common for function definitions to be kept in this way.
It's not so silly when you process symbol names. When you for example build preprocessor - you read some data, convert it to list structures and then some code is processing those list and trigger some additional logic when symbol-name is lets say defun-my-ubermacro. It's exactly what's in Graham quote - you treat symbols as data.
And (in my opinion) it's not correct to say, that when you don't use plists, you generally don't use symbols as data. Plists are only on of many places where it's useful feature. A lot of what macros do is processing symbols as data / names.
On first look it seems somewhat silly to have a function which returns the name of a symbol, which must be called using the name of that same symbol.
That's wrong. symbol-name is called with a symbol and not a name. It returns the name as a string.
I note that :start means get the name of the symbol start from the keyword package, where the keyword package is denoted simply by :. (:keyword being its longer form, unnecessary to use). Thus, in this case symbol-name plays the role of simply removing the package prefix.
No, symbol-name returns the name of the symbol as a string. The keyword symbol is an object of type symbol.
A symbol is a data type and has several cells:
the name, a string
possibly a function
possibly a value
a property list
optionally the home package it is interned in
don't use symbols "as data objects" at all, instead, just as names for things
No, symbols as data objects have many purposes. For example Lisp source code uses many symbols. But it can also appear in all sorts of data.
CL-USER 6 > 'foo
FOO
CL-USER 7 > (type-of 'foo)
SYMBOL
CL-USER 8 > (symbol-name 'foo)
"FOO"
CL-USER 9 > (type-of (symbol-name 'foo))
SIMPLE-BASE-STRING
CL-USER 10 > (mapcar #'symbol-name '(a b c defun foo bar))
("A" "B" "C" "DEFUN" "FOO" "BAR")
CL-USER 11 > (mapcar #'type-of *)
(SIMPLE-BASE-STRING SIMPLE-BASE-STRING SIMPLE-BASE-STRING SIMPLE-BASE-STRING SIMPLE-BASE-STRING SIMPLE-BASE-STRING)
Since you haven't accepted an answer, here is my take.
For most day-to-day programming symbols, and therefore SYMBOL-NAME,
aren't very useful. Mostly they are used for their
unique-ness. However they shine when you are modifying the compiler
with macros.
Here are 3 examples where SYMBOL-NAME is used:
Firstly, LOOP is the CL generic looping construct, it works via
placeholder symbols in a way that some claim is un-lispy, but it
remains very handy, especially if you are stepping things in parallel.
The following 3 forms are equivalent (at run-time):
(loop for i in list collect i)
(loop :for i :in list :collect i)
(loop #:for i #:in list #:collect i)
I, personally, prefer the third form because it makes it really
obvious where the LOOP magic is happening, and also avoids interning
symbols in any package (which is usually harmless, but not
always). That the third works at all requires the existence of
SYMBOL-NAME
Secondly, I don't make much use of CLOS but classes are undeniably
useful. DEFCLASS tends to be too verbose for what I want to do though,
so I often employ a macro that uses implied symbols. So:
(defbean peer ()
(id)
(next
beats))
Becomes:
(defclass peer nil
((id :initarg :id :reader peer-id)
(next :initarg :next :accessor peer-next)
(beats :initarg :beats :accessor peer-beats)))
Saving much typing. Notice how the slot name is converted to a keyword
for the initargs, and how reader and accessor names are created.
Some don't like this sort of macro, and I can see how it might be
problematic if you have a large team with everyone doing this all over
the shop (though M-. and C-c ret are always available), but for
small teams I think this is one of the best reasons to use lisp,
customising it to how you actually want to use it.
Thirdly, my sqlite helper library uses SYMBOL-NAME to generate SQL
statements (along with some more implied symbols) for even more
code-saving:
(defsqlite-table artists
(id :auto-key)
(artist-name :t)
(sort-artist-name :t :n))
Becomes something pretty huge:
(progn
(defparameter +create-artists-sql+
"CREATE TABLE artists (
id INTEGER PRIMARY KEY NOT NULL,
artist_name TEXT NOT NULL,
sort_artist_name TEXT
)")
(defun create-artists-table (pjs-sqlite::db pjs-sqlite::recursive log)
###)
(eval-when (:load-toplevel)
###)
(defun insert-artists
(pjs-sqlite::db artist-name sort-artist-name &key (id nil id-p))
###)
(defun update-artists
(pjs-sqlite::db id
&key (artist-name nil artist-name-p)
(sort-artist-name nil sort-artist-name-p))
###)
(defun delete-artists (pjs-sqlite::db id)
(with-sqlite-statements (pjs-sqlite::db
(pjs-sqlite::stmt
"DELETE FROM artists WHERE id = ?"))
###)))
(I stripped out a lot of code that would be distracting, but it should
be clear how I mapped symbols to sql identifiers)
I hope this gives you some ideas on what sort of things SYMBOL-NAME
can be used for.

How to define globally a user input as variable

Is there a way, in common lisp, to receive a user input, say "foo", and defvar a global variable *foo*?
For example (which does NOT work):
(defun global-name (s)
"Takes s and changes it to *s*"
(concatenate 'string "*" s "*"))
(defun add-global-var (var)
"defvars a global variable and adds it to *global-list*"
(let ((var-name (global-name var)))
(defvar var-name var)
(push var-name *global-list*)))
; Used like this:
(add-global-var "myvar")
In this case, the var-name is a string, and will not work with defvar.
Déjà vu... I asked these kinds of questions 20+ years ago ;-)
Your question
Yes, you can do that (but no, you do not want to!)
(defun add-global-var (var-name &optional (package *package*))
(let ((var (intern var-name package)))
(proclaim `(special ,var))
(push var *global-list*)))
Please see
proclaim
intern
*package*
Alternatively, you can use a macro as the other answer suggests - in
fact, symbol creation at macroexpansion time (which is part of
compilation) is a very common thing,
cf. gensym.
Your problem
There is little reason to do this though.
Global variables created at run time were not available at compile time
and are, therefore, pretty useless.
Why do you want to do this?
If you want to map strings to values, you are much better off using an
equal hash table.
If you want to integrate with read,
you should call it while binding
*package*
to your internal temp package and then use
symbol-value
to store and retrieve values.
You will use intern to
map "variable names" to the symbols.
This is most likely a XY problem since it's very unusual to need to make a variable with a name made up in runtime. It's very common in compile time, but not runtime. #coredump has already covered compile time macros if that is what you are after.
Here is how you do it though:
(defun add-global-var (var)
"defvars a global variable and adds it to *global-list*"
(let ((var-name (intern (string-upcase (global-name var)))))
(set var-name var)
(push var-name *global-list*)))
set is deprecated, but I doubt it will ever be removed. Implementations might not be able to run as fast though since this is like messing with internals.
Since the names are not from source you you have no good use for the bidnings. because of this I would rather use a hash:
(defvar *bindings* (make-hash-table :test #'eq))
(defun add-binding (var)
(let ((var-name (intern (string-upcase (global-name var)))))
(setf (gethash var-name *bindings*) var)
*bindings*))
A reason to do this is as a part of your own little interpreter symbol table or something. You don't need a list of them since you can get all the keys from the hash as well as get the bound values.
Yes, with a macro:
(defvar *global-list* nil)
I changed global-name so that it also accepts symbols, to avoid thinking about whether the string should be upcased or not. With a symbol, the case is given by readtable-case (you can use uninterned symbols if you want to avoid polluting packages).
(defun global-name (name)
(check-type name (or string symbol))
(intern
(concatenate 'string "*" (string name) "*")))
I named the macro defvar*:
(defmacro defvar* (name)
`(push
(defvar ,(global-name name) ',name)
*global-list*))
Tests:
CL-USER> (defvar* #:foo)
(*FOO*)
CL-USER> (defvar* #:bar)
(*BAR* *FOO*)
Note:
You can also add an optional package argument like in #sds's answer, that's better.

common lisp: read list into list

It might seem simple, but I can't get it to work. I simply need to read a file where the contents are just one big list
(a b c d)
. . . as is . . . into a list in my program. I have
(let ((ardplst nil))
...
(with-open-file (in ardpfile :direction :input :if-does-not-exist nil)
(when in
(read-sequence ardplst in))
(format t "~a" ardplst))
But it's not working. I get NIL. What am I doing wrong?
What does read-sequence do? It reads some elements from the stream, typically characters (but it depends on the element-type of the stream) and destructively insert them into the input sequence. So, you would collect characters #\(, then #\a, then #\Space, then #\b, etc. However, reading stops as soon as you reach the end of your sequence: with your empty list, that means immediately (you are supposed to pass a buffer, e.g. a vector). In you case, read-sequence returns 0.
The reason you get nil is because your last expression is format, which in the above code outputs to the standard output (because of t) and returns nil. You could have used print, which returns the printed object.
I don't understand why you are explicitely using :if-does-not-exist nil. Are you sure you want to silently skip the task if the file cannot be opened? What if the list you read is empty? You should probably let an error be signaled in case the file is not found.
I would use read while disabling read-time evaluation:
(with-open-file (in my-file)
(let* ((*read-eval* nil)
(list (read in)))
(prog1 list
(check-type list list))))
Note that the default :direction is :input. In my opinion it does not hurt to omit this argument here, though sometimes it can be more readable to write it explicitely.

Error with define in Racket

I just discovered Racket a few days ago, and I'm trying to get more comfortable with it by writing a little script that generates images to represent source code using #lang slideshow.
I know that when programming in a functional paradigm it's good practice to create almost all your variables with let, but I find that it introduces too many levels of nesting and that Racket's let has an overcomplicated API which requires superfluous parentheses. I'm sure this is to remove ambiguity when using let in more powerful ways, but for my purposes it's just an annoyance. Consequently, I'm creating all my variables with define, and writing blocks with begin if I need to (such as in the body of an if statement).
The problem is that I've repeatedly been getting what seem to be very mysterious errors. I'm sure I'm just making some silly beginner's mistake, being new to the language, but I really can't seem to find the source of the complaint.
Here's the offending code:
(define sub-code (foldr ht-append (rectangle 0 0) (map internal-style (rest code))))
although what we're defining sub-code to seems pretty irrelevant. If I replace it with
(define sub-code '())
I receive the same error. DrRacket is saying that define is being used in an expression context. I understand what this error would normally mean - IE that it would raise when you write code like (print (define x 10)), but I can't see what would trigger it here.
If it helps, this define is at the beginning of a begin block, inside an if statement
(if (list? code)
(begin
(define sub-code '())
; a few more define statements and finally an expression ))
The specific error message DrRacket is printing is
define: not allowed in an expression context in: (define sub-code (quote ()))
I thought maybe define isn't allowed in begin blocks, but I checked the docs and one of the examples for begin is
(begin
(define x 10)
x)
So I don't really know what to do. Thanks in advance!
Definitions are allowed in a 'body' context, like in lambda and let among others. The consequent and alternate clauses of if are not body contexts; they are expression contexts and therefore definitions are not allowed.
begin is special - begin in a body context allows definitions, but begin in an expression contexts forbids definitions. Your case falls in to the later.
For example:
(define (foo . args) #| body context #|)
(define foo (lambda args #| body context |#))
(define (foo . args)
(let (...)
#| body context |#))
Syntactic keywords that requires expressions: if, cond, case, and, or, when, unless, do, begin. Check out the formal syntax in any Scheme report (r{4,5,6,7}rs); look for <body>, <sequence>, <command>, and <expression>.
Also, if you need a body context in an expression, just wrap a let syntactic form, as such:
(if test
(let ()
(define foo 'foo)
(list foo foo))
alternate)
As GoZoner explained, you can't use define in an expression context.
What could you do instead?
Use let:
(if (list? code)
(let ([x '()])
x)
...
Or it would work with an "empty" let and define:
(if (list? code)
(let ()
(define x '())
x)
...
But that's a bit silly.
Or use cond and define:
(cond [(list? code)
(define x '())
x]
...
This last way -- using cond and define -- is closest to what the current Racket style guide recommends.
Here's more details, from the Racket docs.
The different contexts are required because macros must expand differently, depending on which language forms are allowed.
As others have said, definitions are not allowed in expression contexts ("expr ..." in the docs), but are ok in other contexts.
In other doc entries, "body ..." indicates an internal-definition context (guide, reference), for example in lambda bodies, and "form ..." indicates all non-expression contexts, like in the docs for begin.
Or you could wrap the expressions in (begin)
e.g.(begin
(define x 10)
(define y 100)
(define z 1000))

Defining a variable with a name that is kept in an other variable

I need to define a variable with a name contained in another variable.
The following code does not work and it uses eval, therefore not very good style.
(defvar symbol "zap")
(eval `(defvar ,symbol 100))
(print zap)
It's usually better to use a hash table for that purpose.
First of all -- are you sure you need to do this?
Now, with that out of the way --
(defvar *symbol-name* "zap")
(let ((symbol (intern *symbol-name*)))
(proclaim `(special ,symbol))
(setf (symbol-value symbol) 100))
(print |zap|)
Note that |zap| will normally need to be quoted with pipes, because intern is case-sensitive, while the default readtable upcases symbols by default. So if you don't quote, as in (print zap), that is interpreted as (PRINT ZAP), with the wrong case for zap. Alternatively, you can upcase the interned symbol, with (intern (string-upcase *symbol-name*)).
Update after seeing what you're actually trying to do.
You definitely don't want to be defining global variables for that. A data structure that maps keys (A1...H4, etc) to numbers is what you want. In this case, your keys can be symbols or strings. And the data structure could be a hashtable, a plist or alist (among other choices). For example, a string-keyed hashtable would look like:
(defvar *table* (make-hash-table :test #'equalp))
(setf (gethash "A1" *table*) 42)
(print (gethash "A1" *table*)) ==> prints 42
If you absolutely want to do this, with global variables, I suspect that SET and SYMBOL-VALUE (and using symbols instead of strings) might do the trick. It definitely falls in the "hm, yes, you can do it that way, but I am not convinced it's the best way" territory, though.
However, you are most probably better off either using a hash-table or (skipping storage completely, if there's no need to mutate what A1 means further down the line) a function that parses the letter and digit apart and calculates the relevant value.
use SET:
CL-USER 13 > (defvar *some-symbol* '*this-is-some-symbol*)
*SOME-SYMBOL*
CL-USER 14 > (set *some-symbol* 42)
42
CL-USER 15 > *this-is-some-symbol*
42
or if it is a string:
(setf (symbol-value (find-symbol some-symbol-name)) 42)

Resources