I'm trying to use drakma-async in my small project. But I just can't understand what's happening. (I use emacs + slime + ccl). I need to get data with http(s) and parse it in a callback. I assume I can get wrong data that cannot be parsed, so I want to make a retry. But when I tried to make some tests I just can't understand what's happening...
(defun my-callback (data)
(prin1 data)
(restart-case
(error "Some error parsing data...")
(just-continue () (prin1 "Continue..."))))
(defun simple-test ()
(let ((future (asf:make-future)))
(as:delay #'(lambda () (asf:finish future "Some data")) :time 2)
(prin1 (asf:future-finished-p future))
(asf:attach future #'my-callback)))
(defun drakma-test ()
(asf:alet ((response (das:http-request "http://www.google.com")))
;(prin1 (asf:future-finished-p response))
(asf:attach response #'my-callback)))
(defun drakma-test-let ()
(let ((response (das:http-request "http://www.google.com")))
;(prin1 (asf:future-finished-p response))
(asf:attach response #'my-callback)))
(defun run-test (test)
(as:start-event-loop test))
1) So I will that's what I have with my simple example (that's what I've planned)
? (run-test #'simple-test)
NIL"Some data" ;I get debugger here with simple-error and choose my restart
Invoking restart: #<RESTART JUST-CONTINUE #x7F0578EC20AD>
"Continue..."
1
2) Here what I get in second test:
? (run-test #'drakma-test)
"<A LOT OF HTML>
"
1
Where are my debugger and my restart?
3) Uncomment the ;(prin1 (asf:future...)) line in drakma-test
? (run-test #'drakma-test)
1
No finished/unfinished bool, No Data is not printed, I don't get a restart, I just get 1 as result.
4) I assume if i write (let ((reponse (das:http-request "http://www.google.com"))) ... )
instad of (asf:alet ...) the response will contain not future object, but will block until the request will be finished and the response will contain the data.
? (run-test #'drakma-test-let)
1
5) Uncomment the ;(prin1 (asf:future...)) line in drakma-test-let
? (run-test #'drakma-test-let)
NIL ;future is not finished
1
Data is not printed, just that is not finished and the result of run-test.
I've run tests for cl-async and they all passed except the ipv6 test. So I just don't know where to start to understand whats happening... Why I get no debugger and restart in 2nd test? Why nothing happens in 3rd test (it's the same as 2nd, but with prin1). Why nothing happens in 5th and 5th tests?
P.S. Don't have enough reputation to create drakma-async or cl-async tags for this libraries. I know that drakma-async is built over drakma so I put this tag.
Thanks for m-n's comment that made the situation clearer and explained shortly the situation.
I made some examples and want to show what happens in each case:
Example:
(defun my-callback (&rest data)
(format t "Echo from callback: ~A~%" data)
(restart-case
(error "Some error parsing data...")
(just-continue () (prin1 "Continue..."))))
(defun my-errback (e)
(format t "Echo from errback: ~A~%" e))
(defun make-example-future ()
(let ((future (asf:make-future))) ;creating future
(as:delay #'(lambda () ;finishing future in 2 seconds
(asf:future-handler-case ;wrapping asf:finish
(asf:finish future
"Result data")
(t (e) (asf:signal-error future e)))) ;signal future an error
:time 2)
future))
(defun simple-test-2 ()
(let ((future (make-example-future)))
(format t "Is future?: ~A~%Finished?: ~A~%"
(asf:futurep future) (asf:future-finished-p future))
(asf:alet ((result future))
(asf:attach-errback future #'my-errback)
(format t "Finished? ~A~%" (asf:future-finished-p future))
(asf:future-finished-p result)
(asf:attach result #'my-callback))))
And here is what's happening:
? (as:start-event-loop #'simple-test-2)
Is future?: T
Finished?: NIL
;<here we have a 2 sec pause>
Finished? T
Echo from errback: There is no applicable method for the generic function:
#<STANDARD-GENERIC-FUNCTION CL-ASYNC-FUTURE:FUTURE-FINISHED-P #x302001B67A8F>
when called with arguments:
("Result data")
A) asf:alet wait for result and bind the result value to the variable.
So I was wrong thinking that asf:alet bind a future.
B) In make-example-future we wrap asf:finish with asf:future-handler-case
and use asf:signal-error to send error to future.
That means that error is handled and the errback will be called.
Even if the callback is attached later in the code.
Moreover, the error with (asf:future-finished-p result)
was handled with future-handler-case because it was wrapped in asf:alet (At least I think so).
C) Comment the (asf:future-finished-p result) and the result is
Is future?: T
Finished?: NIL
Finished? T
Echo from callback: (Result data) ;here is my data
Echo from errback: Some error parsing data... ;;here is my error
1
In drakma-async there is similar future-handler-case wrapper that wraps asf:finish.
So this explains the #2 test result. I got the data and asf:alet returned me the string. The error from callback was passed to errback, which I didn't have.
Moreover. In drakma-test using only asf:alet I just can't attach errback because I don't have access to future. I need to call http-request in let, not in alet.
Also this explains the result of the #3 test: I got error in (future-finished-p) which was sent to errback.
If we look at the result of #4 and #5 test with new my-callback: It can be seen, that
cl-async try to call my callback with all values the drakma returned. There are 7 of them (the values that drakma:http-request return).
So I tried to attach wrong number of arguments callback and my #4 and #5 tests were signalling an error that was simply handled by that future-hander-case and send it to errback.
Result:
Anyway, it seems impossible to use restarts with drakma-async without removing that future-handler-case because it send error to errback, but lose all restarts.
Hope this post helps if somebody fill face up with my question.
Related
I am using SBCL, Emacs, and Slime. In addition, I am using the library Dexador.
Dexador documentation provides an example on how to handle failed HTTP requests.
From the official documentation, it says:
;; Handles 400 bad request
(handler-case (dex:get "http://lisp.org")
(dex:http-request-bad-request ()
;; Runs when 400 bad request returned
)
(dex:http-request-failed (e)
;; For other 4xx or 5xx
(format *error-output* "The server returned ~D" (dex:response-status e))))
Thus, I tried the following. It must be highlighted that this is part of a major system, so I am simplifying it as:
;;Good source to test errors: https://httpstat.us/
(defun my-get (final-url)
(let* ((status-code)
(response)
(response-and-status (multiple-value-bind (response status-code)
(handler-case (dex:get final-url)
(dex:http-request-bad-request ()
(progn
(setf status-code
"The server returned a failed request of 400 (bad request) status.")
(setf response nil)))
(dex:http-request-failed (e)
(progn
(setf status-code
(format nil
"The server returned a failed request of ~a status."
(dex:response-status e)))
(setf response nil))))
(list response status-code))))
(list response-and-status response status-code)))
My code output is close to what I want. But I do not understand it is output.
When the HTTP request is successful, this is the output:
CL-USER> (my-get "http://www.paulgraham.com")
(("<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">
<html><script type=\"text/javascript\">
<!--
... big HTML omitted...
</script>
</html>"
200)
NIL NIL)
I was expecting (or wished) the output to be something like: '(("big html" 200) "big html" 200).
But, things happen to be even weirder when the HTTP request fails. For instance:
CL-USER> (my-get "https://httpstat.us/400")
((NIL NIL) NIL
"The server returned a failed request of 400 (bad request) status.")
I was expecting: '((NIL "The server returned a failed request of 400 (bad request) status.") NIL "The server returned a failed request of 400 (bad request) status.")
Or:
CL-USER> (my-get "https://httpstat.us/425")
((NIL NIL) NIL "The server returned a failed request of 425 status.")
Again, I was expecting: ((NIL "The server returned a failed request of 425 status.") NIL "The server returned a failed request of 425 status.")
I am afraid there is a variable overshadowing problem happening - not sure, though.
How can I create a function so that I can safely store in variables the response and the status-code independent of being a failed or successful request?
If the request is successful, I have ("html" 200). If it fails, it would be: (nil 400) or other number (nil 425) - depending on the error message.
Your problem is that you failed to understand how multiple-value-bind, handler-case, and let* work. (Probably also setf and progn.)
TLDR
Quick fix:
(defun my-get (final-url)
(let* ((status-code)
(response)
(response-and-status
(multiple-value-bind (bresponse bstatus-code)
(handler-case (dex:get final-url)
(dex:http-request-bad-request ()
(values nil
"The server returned a failed request of 400 (bad request) status."))
(dex:http-request-failed (e)
(values nil
(format nil "The server returned a failed request of ~a status." (dex:response-status e)))))
(list (setf response bresponse)
(setf status-code bstatus-code)))))
(list response-and-status response status-code)))
Output:
CL-USER> (my-get "http://www.paulgraham.com")
(("big html" 200) "big html" 200)
CL-USER> (my-get "https://httpstat.us/400")
((NIL "The server returned a failed request of 400 (bad request) status.") NIL "The server returned a failed request of 400 (bad request) status.")
CL-USER> (my-get "https://httpstat.us/425")
((NIL "The server returned a failed request of 425 status.") NIL "The server returned a failed request of 425 status.")
So, Why Do You Get Your Current Result?
Preliminary
For multiple-value-bind, it binds multiple values returning from a values form into the corresponding variables.
CL-USER> (multiple-value-bind (a b)
nil
(list a b))
(NIL NIL)
CL-USER> (multiple-value-bind (a b)
(values 1 2)
(list a b))
(1 2)
CL-USER> (multiple-value-bind (a b)
(values 1 2 3 4 5)
(list a b))
(1 2)
For handler-case, it returns the value of the expression form when there is no error. When there is an error, it will execute the corresponding error-handling code.
CL-USER> (handler-case (values 1 2 3)
(type-error () 'blah1)
(error () 'blah2))
1
2
3
CL-USER> (handler-case (signal 'type-error)
(type-error () 'blah1)
(error () 'blah2))
BLAH1
CL-USER> (handler-case (signal 'error)
(type-error () 'blah1)
(error () 'blah2))
BLAH2
For let* form, variables are initialised to nil if init-forms are not provided. Same goes to the let form. The parentheses around these variables without init-forms are unneeded. Example:
CL-USER> (let* (a b (c 3))
(list a b c))
(NIL NIL 3)
Combining These Knowledges
When dex:get success (i.e. no error), it returns the values of dex:get, which is (values body status response-headers uri stream). With multiple-value-bind, your response-and-status is bound to the value of (list response status-code), which is ("big html" 200).
Since the code in both dex:http-request-bad-request and dex:http-request-failed will only be executed when dex:get failed, both response and status-code have the initial value nil. That's why you get (("big html" 200) nil nil) on success.
When dex:get failed, both response and status-code are setf to new values. Since (setf response nil) mutates the value of response and returns nil (the new value set), and since progn returns the value of last form, your progn returns nil for both error handling cases. That's why your response-and-status is bound to (nil nil) when failed.
This is an addendum to zacque's answer, which describes the problem and your confusions about multiple-value-bind & other things pretty well.
As mentioned in that answer, dex:get returns five values: body, status, response-headers, uri, stream. But it also signals one of various conditions on failure. So one obvious thing to do is simply have a function which returns the same five values (there's no reason to package some of them into a list), but handles the errors, relying on the handler to return suitable values. Such a function is terribly simple, and has no assignment.
(defun my-get (final-url)
(handler-case (dex:get final-url)
(dex:http-request-failed (e)
;; These are probably not the right set of values, but if the
;; first one is NIL we're basically OK.
(values nil
(dex:response-status e)
e
(format nil
"The server returned a failed request of ~a status."
(dex:response-status e))
nil))))
If you really want to package the values up into some structure you can do that, but generally it's easy to just handle multiple values. For instance a user of this function can now just ignore all but the first value rather than unpacking all sorts of grot, so:
(defun my-get-user (url)
(or (my-get url)
(error "oops")))
I'm using IOLIB with this code to resolve a hostname:
(sockets:address-to-string (sockets:lookup-hostname name))
I works, but the functions does not take any timeout paramenter, and i cannot figure out how to set these using socket options.
Unfortunately this is not easy to find (in particular, this is not documented), but following the chain of calls from lookup-hostname (M-. in Emacs), you can see your code eventually calls dns-query:
(defun dns-query (name &key (type :a) (search *dns-search-domain*)
(nameservers *dns-nameservers*) decode
(repeat *dns-repeat*) (timeout *dns-timeout*))
...)
The timeout argument defaults to a special variable iolib/sockets::*dns-timeout*, which is globally bound to 10. You then only need to bind it around your code to set a different timeout:
(let ((iolib/sockets::*dns-timeout* 1))
...)
The variable is not exported, but dns-query is, maybe it is better to call that function directly.
CL-USER> (iolib:dns-query "http://example.com" :timeout 0.0001)
NIL
CL-USER> (iolib:dns-query "http://example.com" :timeout 1)
#<DNS RESPONSE Id: 61273, Question: #(#<"http://example.com." A IN>) Flags: :OP/S :RD :RA :NAME-ERROR, Sections: QD(1) AN(0) NS(1) AD(0)>
I've tried to replace SBCL with Clozure CL when working in IPv6 only network, but encountered an error like that:
MIGRATIONS> (ignore-errors (ccl:make-socket :remote-host "ya.ru" :remote-port 443))
NIL
#<CCL:NO-APPLICABLE-METHOD-EXISTS #x302005215E5D>
MIGRATIONS> (ignore-errors (ccl:make-socket :remote-host "ya.ru" :remote-port 443 :address-family :internet))
NIL
#<CCL:NO-APPLICABLE-METHOD-EXISTS #x3020052549AD>
MIGRATIONS> (ignore-errors (ccl:make-socket :remote-host "ya.ru" :remote-port 443 :address-family :internet6))
#<BASIC-TCP-STREAM ISO-8859-1 (SOCKET/16) #x3020051D4A9D>
The problem is that many libraries when using CCL:MAKE-TCP-SOCKET don't specify address-family or specify an :internet.
Is there is a way to patch ccl:make-socket at runtime to override this setting?
Advise a function
Several implementations of Common Lisp allow advising (-> patching) of normal functions. Advising is a non-standard feature and different implementations provide it in slightly different ways. A related mechanism is standardized for CLOS generic functions with :before, :after and :around methods.
The purpose is to add one or more patches to a function, after it has been defined and without altering the original source code.
Typically this requires that the function call to this function is not inlined.
The macro ADVISE in Clozure Common Lisp
Patching functions in Clozure CL can be done with the macro ADVISE. See the documentation for advising.
Let's say we have a function FOOBAR:
? (defun foobar (a b &key c (d :foobar)) (list a b c d))
FOOBAR
FOOBAR gets called inside TEST:
? (defun test (a) (foobar a 20 :c 30))
TEST
? (test 10)
(10 20 30 :FOOBAR)
We now want to patch FOOBAR such that named arg :D gets called with a different value.
We change the arglist to insert the new named argument after the two required args:
? (advise foobar (let ((arglist (list* (first arglist)
(second arglist)
:d :ipv6
(cddr arglist))))
(:do-it)) ; calling the original function
:when :around ; advise around it
:name :ipv6) ; the name of this advise
#<Compiled-function (CCL::ADVISED 'FOOBAR) (Non-Global) #x3020010D1CCF>
Now we can call our TEST function and it will call the advised function FOOBAR.
? (test 10)
(10 20 30 :IPV6)
Advise for CCL:MAKE-SOCKET
You could write a similar advise for CCL:MAKE-SOCKET.
Untested:
(advise ccl:make-socket (let ((arglist (list* :address-family
:internet6
arglist)))
(:do-it))
:when :around
:name :internet6)
This can be done!
First make a copy of the original make-socket
(IN-PACKAGE :ccl)
(DEFPARAMETER original-make-socket #'make-socket)
Then redefine make-socket. Note: You will have to provide the full spec for all keyword parameters. As it is, I've used only the ones from your question for demonstration.
(defun make-socket (&key (remote-host "defau.lt")
(remote-port 443)
(address-family :internet6))
(declare (ignore address-family))
(format t "Calling new make-socket with address-family as internet6!")
(funcall original-make-socket
:remote-host remote-host
:remote-port remote-port
:address-family :internet6))
This will signal a continuable error.
Type :go at the repl to continue.
This will successfully patch make-socket.
Now any calls to make-socket will be to the new definition. Try:
(IN-PACKAGE :cl-user)
(ccl:make-socket :remote-host "ya.ru" :remote-port 443 :address-family :IRRELEVANT)
Another way to do it, would be to override the global variable *warn-if-redefine-kernel* before redefining make-socket.
(setf *warn-if-redefine-kernel* nil)
This will avoid the continuable error signal, and straight patch the kernel function.
I am using clj-kafka, and I am trying to make a core.async interface to it in the REPL.
I am getting some messages, but my structure feels wrong : I either cannot stop receiving messages, or have to launch the go routine again to receive more messages.
Here is my attempt :
(defn consume [topic]
(let [consume-chan (chan)]
(with-resource [c (consumer config)]
shutdown
(go (doseq [m (messages c "test")]
(>! chan message) ;; should I check the return value?
)))
consume-chan)) ;; is it the right place to return a channel ?
(def consume-chan (consume "test"))
;;(close! consume-chan)
(go (>! consume-chan "hi")) ;; manual test, but I have some messages in Kafka already
(def cons-ch (go
(with-resource [c (consumer config)]
shutdown
(doseq [m (messages c "test")]
(>! consume-chan m))))) ;; should I check something here ?
;;(close! cons-ch)
(def go-ch
(go-loop []
(if-let [km (<! consume-chan)]
(do (println "Got a value in this loop:" km)
(recur))
(do (println "Stop recurring - channel closed")))))
;;(close! go-ch)
How do I consume a lazy-sequence of messages with a core.async interface ?
Here's what I would do:
>! and <! return nil if the channel is closed, so make sure that the loop exits when this happens - that way you can easily end the loop from the outside by closing the channel.
Use a try/catch to check for exceptions inside the go block, and make any exception the return value so that they don't get lost.
Check for exceptions on read values, to catch anything from inside the channel.
The go blocks return a channel, and the return value of the code inside the block (like the exceptions from above) will be put on the channel. Check these channels for exceptions, possibly to rethrow.
You can now write to a channel like this:
(defn write-seq-to-channel
[channel
values-seq]
(a/go
(try
(loop [values values-seq]
(when (seq values)
(when (a/>! channel (first values))
(recur (rest values)))))
(catch Throwable e
e))))
and you read like this:
(defn read-from-channel-and-print
[channel]
(a/go
(try
(loop []
(let [value (a/<! channel)]
(when value
(when (instance? Throwable value)
(throw value))
(println "Value read:" value)
(recur))))
(catch Throwable e
e))))
You will now have two channels, so use something like alts! or alts!! to check for your loops exiting. Close the channel when you are done.
While using Clojure proxies, fns passed to proxy should override existing methods or are they called in conjunction with super.method()?
In the following code, RequestHandler.get() is invoked along with the proxy get [].
;see: http://github.com/paulosuzart/JTornado
(ns org.ctornadoweb)
(import '(org.jtornadoweb Web$RequestHandler))
(import '(org.jtornadoweb HttpServer Web$Application))
(let [myHandler (proxy [Web$RequestHandler] []
(get []
(.write "Hi CLJ"))
(post []
(.write "POST")))]
(.listen
(HttpServer.
(.add (Web$Application.) "/" (class myHandler))
false nil false) 8089))
The same happens to the compiled/inheritance version:
; Starts a JTornado HTTP Server and a sample RequestHandler.
; Bit verbose due to compilation directives. Recommendation is to generate
; a set of macros to hide this.
(ns org.ctornadoweb
; Compiled and implements a static main method to start the server
(:import (org.jtornadoweb HttpServer)
(org.jtornadoweb.Web$Application)
(org.jtornadoweb.Web$RequestHandler))
(:gen-class :prefix "server-"))
(gen-class
:name org.ctornadoweb.MyHandler
:extends org.jtornadoweb.Web$RequestHandler
:prefix "do")
(defn do-get [this]
"Handles the HTTP GET method"
(.write "hello clojure"))
(defn do-post [this]
"Handles the HTTP POST method"
(.write (.getArgument "name" "default" false)))
(defn server-main []
"main method"
(.listen
(org.jtornadoweb.HttpServer.
(.add (org.jtornadoweb.Web$Application.) "/" org.ctornadoweb.MyHandler)
false nil false) 8089))
;use (compile 'org.ctornadoweb)
The trace shows the proxy get being invoked and then the super.get, what throws (by default) an exception.
HTTP 405: Method Not Allowed
at org.jtornadoweb.Web$RequestHandler.get(Web.java:72)
at org.ctornadoweb.proxy$org.jtornadoweb.Web$RequestHandler$0.get(Unknown Source)
I tried to find some words about the actual behavior of Clojure Proxies. Can someone give this help?
No, the super method will not be called automatically, though you can explicitly call it with proxy-super.
The following test case shows things working as they should:
user=> (def foo
(proxy [java.util.ArrayList] []
(clear [] (println "not clearing"))))
#'user/foo
user=> (.size foo)
0
user=> (.add foo "hi")
true
user=> (.add foo "bye")
true
user=> (.size foo)
2
user=> (.clear foo)
not clearing
nil
user=> (.size foo)
2
If super.clear() were getting called, the size would show as 0.