Does a Clojure proxy always invoke super.method()? - functional-programming

While using Clojure proxies, fns passed to proxy should override existing methods or are they called in conjunction with super.method()?
In the following code, RequestHandler.get() is invoked along with the proxy get [].
;see: http://github.com/paulosuzart/JTornado
(ns org.ctornadoweb)
(import '(org.jtornadoweb Web$RequestHandler))
(import '(org.jtornadoweb HttpServer Web$Application))
(let [myHandler (proxy [Web$RequestHandler] []
(get []
(.write "Hi CLJ"))
(post []
(.write "POST")))]
(.listen
(HttpServer.
(.add (Web$Application.) "/" (class myHandler))
false nil false) 8089))
The same happens to the compiled/inheritance version:
; Starts a JTornado HTTP Server and a sample RequestHandler.
; Bit verbose due to compilation directives. Recommendation is to generate
; a set of macros to hide this.
(ns org.ctornadoweb
; Compiled and implements a static main method to start the server
(:import (org.jtornadoweb HttpServer)
(org.jtornadoweb.Web$Application)
(org.jtornadoweb.Web$RequestHandler))
(:gen-class :prefix "server-"))
(gen-class
:name org.ctornadoweb.MyHandler
:extends org.jtornadoweb.Web$RequestHandler
:prefix "do")
(defn do-get [this]
"Handles the HTTP GET method"
(.write "hello clojure"))
(defn do-post [this]
"Handles the HTTP POST method"
(.write (.getArgument "name" "default" false)))
(defn server-main []
"main method"
(.listen
(org.jtornadoweb.HttpServer.
(.add (org.jtornadoweb.Web$Application.) "/" org.ctornadoweb.MyHandler)
false nil false) 8089))
;use (compile 'org.ctornadoweb)
The trace shows the proxy get being invoked and then the super.get, what throws (by default) an exception.
HTTP 405: Method Not Allowed
at org.jtornadoweb.Web$RequestHandler.get(Web.java:72)
at org.ctornadoweb.proxy$org.jtornadoweb.Web$RequestHandler$0.get(Unknown Source)
I tried to find some words about the actual behavior of Clojure Proxies. Can someone give this help?

No, the super method will not be called automatically, though you can explicitly call it with proxy-super.
The following test case shows things working as they should:
user=> (def foo
(proxy [java.util.ArrayList] []
(clear [] (println "not clearing"))))
#'user/foo
user=> (.size foo)
0
user=> (.add foo "hi")
true
user=> (.add foo "bye")
true
user=> (.size foo)
2
user=> (.clear foo)
not clearing
nil
user=> (.size foo)
2
If super.clear() were getting called, the size would show as 0.

Related

Async Compojure-Api Middleware not Working

My middleware is throwing an error only for async requests, not sure why:
project.clj
(defproject asyncy "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:dependencies [[org.clojure/clojure "1.9.0"]
[metosin/compojure-api "1.1.11"]
[org.clojure/core.async "0.3.442"]]
:ring {:handler asyncy.handler/app}
:uberjar-name "server.jar"
:profiles {:dev {:dependencies [[javax.servlet/javax.servlet-api "3.1.0"]]
:plugins [[lein-ring "0.12.0"]]}})
handler.clj
To make this even easier to talk about and diagnose, i've made a bare minimum version of the problem using the lein new compojure-api template
(ns asyncy.handler
(:require [compojure.api.sweet :refer :all]
[ring.util.http-response :refer :all]
[schema.core :as s]
[clojure.core.async :as async]))
(s/defschema Pizza
{:name s/Str
(s/optional-key :description) s/Str
:size (s/enum :L :M :S)
:origin {:country (s/enum :FI :PO)
:city s/Str}})
(def app
(api
{:async? true
:swagger
{:ui "/"
:spec "/swagger.json"
:data {:info {:title "Asyncy"
:description "Compojure Api example"}
:tags [{:name "api", :description "some apis"}]}}}
(context "/api" []
:tags ["api"]
:middleware [(fn [handler]
;; this sync-style handler always handles both sync and async requests
(fn ([request]
(clojure.pprint/pprint request)
(if (some-> request :params :x (= "1"))
(bad-request {:error true :message "one is the loneliest number that you'll ever do"})
;; this throws an error on async requests
(handler request)))
;; this async 3-arity handler is never called
([request b c]
(clojure.pprint/pprint ["never called, doesnt matter." request b c]))
))]
(GET "/plus" []
:return {:result Long}
:query-params [x :- Long, y :- Long]
:summary "adds two numbers together"
(ok {:result (+ x y)}))
(GET "/plus-async" []
:return {:result Long}
:query-params [x :- Long, y :- Long]
:summary "adds two numbers together"
(fn [request respond raise]
(respond (ok {:result (+ x y)}))))
)))
(handler request) in the middleware is what throws the error, for async requests only:
ERROR Wrong number of args (1) passed to: handler/fn--26289/fn--26301/fn--26303
clojure.lang.ArityException: Wrong number of args (1) passed to: handler/fn--26289/fn--26301/fn--26303
at clojure.lang.AFn.throwArity(AFn.java:429)
at clojure.lang.AFn.invoke(AFn.java:32)
at compojure.response$eval1960$fn__1961.invoke(response.clj:47)
at compojure.response$eval1882$fn__1883$G__1873__1890.invoke(response.clj:7)
at compojure.core$wrap_response$fn__3839.invoke(core.clj:158)
at compojure.core$pre_init$fn__3938.invoke(core.clj:328)
at compojure.api.coerce$body_coercer_middleware$fn__14642.invoke(coerce.clj:51)
at compojure.core$pre_init$fn__3940$fn__3943.invoke(core.clj:335)
at compojure.core$wrap_route_middleware$fn__3823.invoke(core.clj:127)
at compojure.core$wrap_route_info$fn__3828.invoke(core.clj:137)
at compojure.core$wrap_route_matches$fn__3832.invoke(core.clj:146)
at compojure.core$wrap_routes$fn__3950.invoke(core.clj:348)
at compojure.api.routes.Route.invoke(routes.clj:74)
at compojure.core$routing$fn__3847.invoke(core.clj:185)
at clojure.core$some.invokeStatic(core.clj:2693)
at clojure.core$some.invoke(core.clj:2684)
at compojure.core$routing.invokeStatic(core.clj:185)
at compojure.core$routing.doInvoke(core.clj:182)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$apply.invoke(core.clj:652)
at compojure.core$routes$fn__3851.invoke(core.clj:192)
at asyncy.handler$fn__26289$fn__26290$fn__26291.invoke(handler.clj:32)
at compojure.core$routing$fn__3847.invoke(core.clj:185)
at clojure.core$some.invokeStatic(core.clj:2693)
at clojure.core$some.invoke(core.clj:2684)
at compojure.core$routing.invokeStatic(core.clj:185)
at compojure.core$routing.doInvoke(core.clj:182)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$apply.invoke(core.clj:652)
at compojure.core$routes$fn__3851.invoke(core.clj:192)
at compojure.core$make_context$handler__3919.invoke(core.clj:285)
at compojure.core$make_context$fn__3921.invoke(core.clj:293)
at compojure.api.routes.Route.invoke(routes.clj:74)
at compojure.api.core$handle$fn__14853.invoke(core.clj:8)
at clojure.core$some.invokeStatic(core.clj:2693)
at clojure.core$some.invoke(core.clj:2684)
at compojure.api.core$handle.invokeStatic(core.clj:8)
at compojure.api.core$handle.invoke(core.clj:7)
at clojure.core$partial$fn__5561.invoke(core.clj:2616)
at compojure.api.routes.Route.invoke(routes.clj:74)
at ring.swagger.middleware$wrap_swagger_data$fn__14015.invoke(middleware.clj:35)
at ring.middleware.http_response$wrap_http_response$fn__8034.invoke(http_response.clj:19)
at ring.swagger.middleware$wrap_swagger_data$fn__14015.invoke(middleware.clj:35)
at compojure.api.middleware$wrap_options$fn__14077.invoke(middleware.clj:74)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at compojure.api.middleware$wrap_exceptions$fn__14067.invoke(middleware.clj:43)
at ring.middleware.format_response$wrap_format_response$fn__7930.invoke(format_response.clj:194)
at ring.middleware.keyword_params$wrap_keyword_params$fn__8076.invoke(keyword_params.clj:36)
at ring.middleware.nested_params$wrap_nested_params$fn__8134.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__4079.invoke(params.clj:67)
at compojure.api.middleware$wrap_options$fn__14077.invoke(middleware.clj:74)
at compojure.api.routes.Route.invoke(routes.clj:74)
at clojure.lang.Var.invoke(Var.java:381)
at ring.middleware.reload$wrap_reload$fn__1829.invoke(reload.clj:39)
at ring.middleware.stacktrace$wrap_stacktrace_log$fn__1211.invoke(stacktrace.clj:26)
at ring.middleware.stacktrace$wrap_stacktrace_web$fn__1277.invoke(stacktrace.clj:96)
at ring.adapter.jetty$proxy_handler$fn__487.invoke(jetty.clj:25)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
How does one do middleware with async compojure-api? Feel like i'm just totally off on how to implement this.

Clojure Hystrix exceptions because of too many http requests

I need to build a system that needs to make a a ton of external http requests and I have to use Netflix's Hystrix to create fallbacks and re-route exceptions. Right now, I've got a very simple setup:
(hystrix/defcommand fetch-request
{:hystrix/group-key "c0"
:hystrix/command-key "URLFetch"
:hystrix/fallback-fn (fn [url]
{:status 419
:headers {}
:body "failed"})}
[url]
#(http/get url))
(defn test3 []
(let [n 4000
m (range 0 n)
p (partition 300 m)]
(doseq [t p]
(thread
(doseq [x t]
(let [res (fetch-request "http://localhost:3000/comments")]
(match (:status res)
200 (prn x)
:else (prn nil)))
)))
))
When I execute test3, I keep getting nil. If I decrease the value of n, I get a status of 200 (which is what I need). Furthermore, if I just use http/get directly on test3 function instead of fetch-request command, it works without any problem (even when the value of n is above 7000).
Note: The reason I'm using partitions+threads is to parallelize the http requests. If you know a better way to execute large volume of http requests in clojure as fast as possible, that would be really awesome.
Update:
I played w/ a variety of configurations. Most of them didn't yield a different outcome. Most of the reqs are still not getting executed so the fallback is immediately triggered. I tried disabling the circuitBreaker (which I don't want to, that's why I'm using hystrix) to see if it did anything -- and it did. 80% of the requests passed through.
(hystrix/defcommand fetch-request
{:hystrix/group-key "ct0"
:hystrix/command-key "URLFetch"
:hystrix/init-fn
(fn [_ ^com.netflix.hystrix.HystrixCommand$Setter setter]
(.andCommandPropertiesDefaults
setter
^com.netflix.hystrix.HystrixCommandProperties$Setter
(.withCircuitBreakerEnabled
(HystrixCommandProperties/Setter)
false)
; ^com.netflix.hystrix.HystrixCommandProperties$Setter
; (.withExecutionTimeoutInMilliseconds
; (HystrixCommandProperties/Setter)
; 1000))
; (.andCommandPropertiesDefaults
; setter
; (.withExecutionIsolationStrategy
; (HystrixCommandProperties/Setter)
; com.netflix.hystrix.HystrixCommandProperties$ExecutionIsolationStrategy/THREAD)
))
:hystrix/fallback-fn
(fn fetch-req-fallback [url]
{:status 419
:headers {}
:body "failed"})}
[url]
#(http/get url))
Update 2:
Removing the thread block fixes the problem. However, I do need to execute these requests across multiple thread so it doesn't necessarily solve my issue.
Fixed it. I had to modify the threadpool properties.
(hystrix/defcommand fetch-request
{:hystrix/group-key "ct0"
:hystrix/command-key "URLFetch"
:hystrix/thread-pool-key "URLThread"
:hystrix/init-fn
(fn [_ ^com.netflix.hystrix.HystrixCommand$Setter setter]
(.andThreadPoolPropertiesDefaults
setter
(doto (HystrixThreadPoolProperties/Setter)
(.withMaxQueueSize 10)
(.withQueueSizeRejectionThreshold 10)))
(.andCommandPropertiesDefaults
setter
(doto (HystrixCommandProperties/Setter)
(.withExecutionIsolationStrategy
com.netflix.hystrix.HystrixCommandProperties$ExecutionIsolationStrategy/THREAD)
(.withExecutionTimeoutInMilliseconds
1500))))
:hystrix/fallback-fn
(fn fetch-req-fallback [url]
{:status 419
:headers {}
:body "failed"})}
[url]
#(http/get url))

How can I make an http call in clojure/ring?

My web client (written in cljs) connects to backend (written in clj) which needs to make a few third party API calls. It has to be done on the server and then the result should be transformed in a specific way and sent back to the client.
Here's my handler for one of the urls
(defn get-orders [req]
(let [{:keys [sig uri]} (api-signature :get-orders)]
(client/get uri
{:async? true}
(fn [response] {:body "something"})
(fn [exception] {:body "error"}))))
Instead of returning {:body "something"}, it is returning the following error:
No implementation of method: :render of protocol: #'compojure.response/Renderable found for class: org.apache.http.impl.nio.client.FutureWrapper
What am I doing wrong?
When you specify {:async? true}, clj-http.client/get will return a future which is a FutureWrapper in the error message you got.
So if you don't need async, don't use it. This is an example of a synchronous ring handler which calls a third-party url and returns the response that got back.
(defn handler [request]
(response {:result (client/get "http://example.com")}))
If you really need async, use async version of ring handler.
(defn handler [request respond raise]
(client/get "http://example.com"
{:async? true}
(fn [response] (respond {:body "something"}))
(fn [exception] (raise {:body "error"}))))
Don't forget to config webserver adapter to use the async handler. For example, for Jetty, set :async? flag to true like so
(jetty/run-jetty app {:port 4000 :async? true :join? false})
If you want to concurrently call to multiple third-party urls and return once to web client, use promise to help
(defn handler [request]
(let [result1 (promise)
result2 (promise)]
(client/get "http://example.com/"
{:async? true}
(fn [response] (deliver result1 {:success true :body "something"}))
(fn [exception] (deliver result1 {:success false :body "error"})))
(client/get "http://example.com/"
{:async? true}
(fn [response] (deliver result2 {:success true :body "something"}))
(fn [exception] (deliver result2 {:success false :body "error"})))
(cond
(and (:success #result1)
(:success #result2))
(response {:result1 (:body #result1)
:result2 (:body #result2)})
(not (:success #result1))
(throw (ex-info "fail1" (:body #result1)))
(not (:success #result2))
(throw (ex-info "fail2" (:body #result2))))))

async clojure http-kit client with callback writing to database sqlite3

I am stucked with http-kit async behavior when it comes to write to an sqlite table.
The i/o to the database depends on whether I send the code to a boot repl or run it as a boot script. The i/o proceed only in the repl case. What am I missing? Here 's my code:
#!/usr/bin/env boot
(defn deps [new-deps]
(boot.core/merge-env! :dependencies new-deps))
(deps '[
[http-kit "2.2.0"]
[org.clojure/core.async "0.2.395"]
[org.clojure/java.jdbc "0.7.0-alpha1"]
[org.xerial/sqlite-jdbc "3.16.1"]
[org.slf4j/slf4j-nop "1.7.22"]
])
(require
'[org.httpkit.client :as http]
'[clojure.java.jdbc :as jdbc]
)
(def db-spec
{:classname "org.sqlite.JDBC"
:subprotocol "sqlite"
:subname "sqlite.db"})
;(jdbc/db-do-commands
;db-spec
;(jdbc/create-table-ddl "test" [[:msg :text]]))
(def ins! (partial jdbc/insert! db-spec "test"))
(http/get "http://locahost" {} (fn [_] (ins! {:msg "repl"})))
(defn -main []
(println (System/getProperty "user.dir"))
(http/get "http://locahost" {} (fn [_] (ins! {:msg "exec"}))))
Thanks
The issue with the script not working from command line is that http-kit asynchronous callbacks are handled by daemon threads and the only non-daemon thread is the main thread running your script.
When your -main function ends after submitting HTTP request to http-kit for async processing, the main thread terminates and causes that the JVM shutdowns before daemon threads processing your async callback get a chance to run.
You can check that by adding a sleep expression in the end of -main function to see that your callback is executed:
(defn -main []
(println (System/getProperty "user.dir"))
(http/get "http://locahost" {} (fn [_] (ins! {:msg "exec"})))
(Thread/sleep 60000))
The best way to make sure that the -main function will wait for the result to be processed is to keep the promise returned by http/get call. The promise eventually will contain the result produced by your callback function:
(let [result-promise (http/get "https://www.google.com" {} (fn [_] "Result"))]
#result-promise)
#result-promise is a reader macro/shortcut for (deref result-promise).
The full form might be better when you would like to not block indefinitely - just call deref with timeout-ms and timeout-value arguments:
(deref result-promise 5000 "Didn't get response in 5 seconds. Giving up")

Clojure - core.async interface for apache kafka

I am using clj-kafka, and I am trying to make a core.async interface to it in the REPL.
I am getting some messages, but my structure feels wrong : I either cannot stop receiving messages, or have to launch the go routine again to receive more messages.
Here is my attempt :
(defn consume [topic]
(let [consume-chan (chan)]
(with-resource [c (consumer config)]
shutdown
(go (doseq [m (messages c "test")]
(>! chan message) ;; should I check the return value?
)))
consume-chan)) ;; is it the right place to return a channel ?
(def consume-chan (consume "test"))
;;(close! consume-chan)
(go (>! consume-chan "hi")) ;; manual test, but I have some messages in Kafka already
(def cons-ch (go
(with-resource [c (consumer config)]
shutdown
(doseq [m (messages c "test")]
(>! consume-chan m))))) ;; should I check something here ?
;;(close! cons-ch)
(def go-ch
(go-loop []
(if-let [km (<! consume-chan)]
(do (println "Got a value in this loop:" km)
(recur))
(do (println "Stop recurring - channel closed")))))
;;(close! go-ch)
How do I consume a lazy-sequence of messages with a core.async interface ?
Here's what I would do:
>! and <! return nil if the channel is closed, so make sure that the loop exits when this happens - that way you can easily end the loop from the outside by closing the channel.
Use a try/catch to check for exceptions inside the go block, and make any exception the return value so that they don't get lost.
Check for exceptions on read values, to catch anything from inside the channel.
The go blocks return a channel, and the return value of the code inside the block (like the exceptions from above) will be put on the channel. Check these channels for exceptions, possibly to rethrow.
You can now write to a channel like this:
(defn write-seq-to-channel
[channel
values-seq]
(a/go
(try
(loop [values values-seq]
(when (seq values)
(when (a/>! channel (first values))
(recur (rest values)))))
(catch Throwable e
e))))
and you read like this:
(defn read-from-channel-and-print
[channel]
(a/go
(try
(loop []
(let [value (a/<! channel)]
(when value
(when (instance? Throwable value)
(throw value))
(println "Value read:" value)
(recur))))
(catch Throwable e
e))))
You will now have two channels, so use something like alts! or alts!! to check for your loops exiting. Close the channel when you are done.

Resources