My web client (written in cljs) connects to backend (written in clj) which needs to make a few third party API calls. It has to be done on the server and then the result should be transformed in a specific way and sent back to the client.
Here's my handler for one of the urls
(defn get-orders [req]
(let [{:keys [sig uri]} (api-signature :get-orders)]
(client/get uri
{:async? true}
(fn [response] {:body "something"})
(fn [exception] {:body "error"}))))
Instead of returning {:body "something"}, it is returning the following error:
No implementation of method: :render of protocol: #'compojure.response/Renderable found for class: org.apache.http.impl.nio.client.FutureWrapper
What am I doing wrong?
When you specify {:async? true}, clj-http.client/get will return a future which is a FutureWrapper in the error message you got.
So if you don't need async, don't use it. This is an example of a synchronous ring handler which calls a third-party url and returns the response that got back.
(defn handler [request]
(response {:result (client/get "http://example.com")}))
If you really need async, use async version of ring handler.
(defn handler [request respond raise]
(client/get "http://example.com"
{:async? true}
(fn [response] (respond {:body "something"}))
(fn [exception] (raise {:body "error"}))))
Don't forget to config webserver adapter to use the async handler. For example, for Jetty, set :async? flag to true like so
(jetty/run-jetty app {:port 4000 :async? true :join? false})
If you want to concurrently call to multiple third-party urls and return once to web client, use promise to help
(defn handler [request]
(let [result1 (promise)
result2 (promise)]
(client/get "http://example.com/"
{:async? true}
(fn [response] (deliver result1 {:success true :body "something"}))
(fn [exception] (deliver result1 {:success false :body "error"})))
(client/get "http://example.com/"
{:async? true}
(fn [response] (deliver result2 {:success true :body "something"}))
(fn [exception] (deliver result2 {:success false :body "error"})))
(cond
(and (:success #result1)
(:success #result2))
(response {:result1 (:body #result1)
:result2 (:body #result2)})
(not (:success #result1))
(throw (ex-info "fail1" (:body #result1)))
(not (:success #result2))
(throw (ex-info "fail2" (:body #result2))))))
Related
I have the following simple server in Clojure using Compojure (which is some flavor of the ring pattern). Everything was working fine in development, and now that I'm in prod I can't get CORS to work for the life of me - I have a wrap-preflight function which seems to work fine, but I keep getting CORS errors in terminal and neither the post or get requests for my comment system work. I am totally lost and very frustrated, I've asked around and no one else seems to know.
Here is the main core.clj code - If anyone has any ideas please let me know. You can see the errors live at thedailyblech.com (not an advert, but maybe it will help debug).
Thank you!
(ns clojure-play.core
(:use org.httpkit.server
[compojure.core :refer :all]
[compojure.route :as route]
[clojure.data.json :as json]
[clojure.tools.logging :only [info]]
[clojure-play.routes :as routes]
[ring.middleware.json :only [wrap-json-body]]
[ring.middleware.cors :refer [wrap-cors]])
(:require [monger.core :as mg]
[monger.collection :as mc]
[clojure.edn :as edn]
[clojure.java.io :as io]
[compojure.handler :as handler])
(:import [org.bson.types ObjectId]
[com.mongodb DB WriteConcern])
(:gen-class))
(println "in the beginning was the command line...")
(defonce channels (atom #{}))
(defn connect! [channel]
(info "channel open")
(swap! channels conj channel))
(defn notify-clients [msg]
(doseq [channel #channels]
(send! channel msg)))
(defn disconnect! [channel status]
(info "channel closed:" status)
(swap! channels #(remove #{channel} %)))
(defn ws-handler [request]
(with-channel request channel
(connect! channel)
(on-close channel (partial disconnect! channel))
(on-receive channel #(notify-clients %))))
(defn my-routes [db]
(routes
(GET "/foo" [] "Hello Foo")
(GET "/bar" [] "Hello Bar")
(GET "/json_example/:name" [] routes/json_example)
(GET "/json_example" [] routes/json_example)
(POST "/email" [] routes/post_email)
(POST "/write_comment" [] (fn [req] (routes/write_comment req db)))
(POST "/update_comment" [] (fn [req] (routes/update_comment req db)))
(GET "/read_comments/:path" [path] (fn [req] (routes/read_comments req db path)))
(GET "/read_comments/:path1/:path2" [path1 path2] (fn [req] (routes/read_comments req db (str path1 "/" path2))))
(GET "/ws" [] ws-handler)))
(defn connectDB []
(defonce connection
(let
[uri "mongodb://somemlabthingy"
{:keys [conn db]} (mg/connect-via-uri uri)]
{:conn conn
:db db}))
{:db (:db connection)
:conn (:conn connection)})
(def cors-headers
"Generic CORS headers"
{"Access-Control-Allow-Origin" "*"
"Access-Control-Allow-Headers" "*"
"Access-Control-Allow-Methods" "GET POST OPTIONS DELETE PUT"})
(defn preflight?
"Returns true if the request is a preflight request"
[request]
(= (request :request-method) :options))
(defn -main
"this is main"
[& args]
(println "hello there main")
(def db (get (connectDB) :db))
(println (read-string (slurp (io/resource "environment/config.edn"))))
(defn wrap-preflight [handler]
(fn [request]
(do
(println "inside wrap-preflight")
(println "value of request")
(println request)
(println "value of handler")
(println handler)
(if (preflight? request)
{:status 200
:headers cors-headers
:body "preflight complete"}
(handler request)))))
(run-server
(wrap-preflight
(wrap-cors
(wrap-json-body
(my-routes db)
{:keywords? true :bigdecimals? true})
:access-control-allow-origin [#"http://www.thedailyblech.com"]
:access-control-allow-methods [:get :put :post :delete :options]
:access-control-allow-headers ["Origin" "X-Requested-With"
"Content-Type" "Accept"]))
{:port 4000}))
The CORS middleware handles the preflight stuff automatically -- you do not need separate middleware for it, nor do you need to produce your own headers etc.
You have it wrapping the routes which is correct -- so CORS-checking happens first, then routing. You should remove your custom preflight middleware and it should work at that point.
We use wrap-cors at work and the only complication we hit was in allowing enough headers (some inserted by production infrastructure, like load balancers). We ended up with this:
:access-control-allow-headers #{"accept"
"accept-encoding"
"accept-language"
"authorization"
"content-type"
"origin"}
For what it's worth, here's what we have for methods:
:access-control-allow-methods [:delete :get
:patch :post :put]
(you do not need :options in there)
After digging around for hours, I found this to be super helpful on an issues post on the ring-cors github, for which documentation was severely lacking.
Using the linked gist, I was able to get past the CORS issues:
; Copied from linked gist
(def cors-headers
"Generic CORS headers"
{"Access-Control-Allow-Origin" "*"
"Access-Control-Allow-Headers" "*"
"Access-Control-Allow-Methods" "GET"})
(defn preflight?
"Returns true if the request is a preflight request"
[request]
(= (request :request-method) :options))
(defn all-cors
"Allow requests from all origins - also check preflight"
[handler]
(fn [request]
(if (preflight? request)
{:status 200
:headers cors-headers
:body "preflight complete"}
(let [response (handler request)]
(update-in response [:headers]
merge cors-headers )))))
; my -main
(defn -main
"Main entry point"
[& args]
(let [port (Integer/parseInt (or (System/getenv "PORT") "8081"))]
(server/run-server
(all-cors
(wrap-defaults #'app-routes site-defaults))
{:port port})
(println "Running on" port)))
This finally allowed me to see the headers properly set in Chrome dev tools and also got rid of the warning on my React front-end.
Access-Control-Allow-Headers: *
Access-Control-Allow-Methods: GET
Access-Control-Allow-Origin: *
Might be worth trying and adding an explicit
(OPTIONS "/*" req handle-preflight)
route to your Compojure routes - in my case that's what made it work.
(RING-MIDDLEWARE-CORS/wrap-cors
:access-control-allow-credentials "true"
:access-control-allow-origin [#".*"]
:access-control-allow-headers #{"accept"
"accept-encoding"
"accept-language"
"authorization"
"content-type"
"origin"}
:access-control-allow-methods [:get :put :post :delete :options])
My middleware is throwing an error only for async requests, not sure why:
project.clj
(defproject asyncy "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:dependencies [[org.clojure/clojure "1.9.0"]
[metosin/compojure-api "1.1.11"]
[org.clojure/core.async "0.3.442"]]
:ring {:handler asyncy.handler/app}
:uberjar-name "server.jar"
:profiles {:dev {:dependencies [[javax.servlet/javax.servlet-api "3.1.0"]]
:plugins [[lein-ring "0.12.0"]]}})
handler.clj
To make this even easier to talk about and diagnose, i've made a bare minimum version of the problem using the lein new compojure-api template
(ns asyncy.handler
(:require [compojure.api.sweet :refer :all]
[ring.util.http-response :refer :all]
[schema.core :as s]
[clojure.core.async :as async]))
(s/defschema Pizza
{:name s/Str
(s/optional-key :description) s/Str
:size (s/enum :L :M :S)
:origin {:country (s/enum :FI :PO)
:city s/Str}})
(def app
(api
{:async? true
:swagger
{:ui "/"
:spec "/swagger.json"
:data {:info {:title "Asyncy"
:description "Compojure Api example"}
:tags [{:name "api", :description "some apis"}]}}}
(context "/api" []
:tags ["api"]
:middleware [(fn [handler]
;; this sync-style handler always handles both sync and async requests
(fn ([request]
(clojure.pprint/pprint request)
(if (some-> request :params :x (= "1"))
(bad-request {:error true :message "one is the loneliest number that you'll ever do"})
;; this throws an error on async requests
(handler request)))
;; this async 3-arity handler is never called
([request b c]
(clojure.pprint/pprint ["never called, doesnt matter." request b c]))
))]
(GET "/plus" []
:return {:result Long}
:query-params [x :- Long, y :- Long]
:summary "adds two numbers together"
(ok {:result (+ x y)}))
(GET "/plus-async" []
:return {:result Long}
:query-params [x :- Long, y :- Long]
:summary "adds two numbers together"
(fn [request respond raise]
(respond (ok {:result (+ x y)}))))
)))
(handler request) in the middleware is what throws the error, for async requests only:
ERROR Wrong number of args (1) passed to: handler/fn--26289/fn--26301/fn--26303
clojure.lang.ArityException: Wrong number of args (1) passed to: handler/fn--26289/fn--26301/fn--26303
at clojure.lang.AFn.throwArity(AFn.java:429)
at clojure.lang.AFn.invoke(AFn.java:32)
at compojure.response$eval1960$fn__1961.invoke(response.clj:47)
at compojure.response$eval1882$fn__1883$G__1873__1890.invoke(response.clj:7)
at compojure.core$wrap_response$fn__3839.invoke(core.clj:158)
at compojure.core$pre_init$fn__3938.invoke(core.clj:328)
at compojure.api.coerce$body_coercer_middleware$fn__14642.invoke(coerce.clj:51)
at compojure.core$pre_init$fn__3940$fn__3943.invoke(core.clj:335)
at compojure.core$wrap_route_middleware$fn__3823.invoke(core.clj:127)
at compojure.core$wrap_route_info$fn__3828.invoke(core.clj:137)
at compojure.core$wrap_route_matches$fn__3832.invoke(core.clj:146)
at compojure.core$wrap_routes$fn__3950.invoke(core.clj:348)
at compojure.api.routes.Route.invoke(routes.clj:74)
at compojure.core$routing$fn__3847.invoke(core.clj:185)
at clojure.core$some.invokeStatic(core.clj:2693)
at clojure.core$some.invoke(core.clj:2684)
at compojure.core$routing.invokeStatic(core.clj:185)
at compojure.core$routing.doInvoke(core.clj:182)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$apply.invoke(core.clj:652)
at compojure.core$routes$fn__3851.invoke(core.clj:192)
at asyncy.handler$fn__26289$fn__26290$fn__26291.invoke(handler.clj:32)
at compojure.core$routing$fn__3847.invoke(core.clj:185)
at clojure.core$some.invokeStatic(core.clj:2693)
at clojure.core$some.invoke(core.clj:2684)
at compojure.core$routing.invokeStatic(core.clj:185)
at compojure.core$routing.doInvoke(core.clj:182)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invokeStatic(core.clj:659)
at clojure.core$apply.invoke(core.clj:652)
at compojure.core$routes$fn__3851.invoke(core.clj:192)
at compojure.core$make_context$handler__3919.invoke(core.clj:285)
at compojure.core$make_context$fn__3921.invoke(core.clj:293)
at compojure.api.routes.Route.invoke(routes.clj:74)
at compojure.api.core$handle$fn__14853.invoke(core.clj:8)
at clojure.core$some.invokeStatic(core.clj:2693)
at clojure.core$some.invoke(core.clj:2684)
at compojure.api.core$handle.invokeStatic(core.clj:8)
at compojure.api.core$handle.invoke(core.clj:7)
at clojure.core$partial$fn__5561.invoke(core.clj:2616)
at compojure.api.routes.Route.invoke(routes.clj:74)
at ring.swagger.middleware$wrap_swagger_data$fn__14015.invoke(middleware.clj:35)
at ring.middleware.http_response$wrap_http_response$fn__8034.invoke(http_response.clj:19)
at ring.swagger.middleware$wrap_swagger_data$fn__14015.invoke(middleware.clj:35)
at compojure.api.middleware$wrap_options$fn__14077.invoke(middleware.clj:74)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at ring.middleware.format_params$wrap_format_params$fn__7105.invoke(format_params.clj:119)
at compojure.api.middleware$wrap_exceptions$fn__14067.invoke(middleware.clj:43)
at ring.middleware.format_response$wrap_format_response$fn__7930.invoke(format_response.clj:194)
at ring.middleware.keyword_params$wrap_keyword_params$fn__8076.invoke(keyword_params.clj:36)
at ring.middleware.nested_params$wrap_nested_params$fn__8134.invoke(nested_params.clj:89)
at ring.middleware.params$wrap_params$fn__4079.invoke(params.clj:67)
at compojure.api.middleware$wrap_options$fn__14077.invoke(middleware.clj:74)
at compojure.api.routes.Route.invoke(routes.clj:74)
at clojure.lang.Var.invoke(Var.java:381)
at ring.middleware.reload$wrap_reload$fn__1829.invoke(reload.clj:39)
at ring.middleware.stacktrace$wrap_stacktrace_log$fn__1211.invoke(stacktrace.clj:26)
at ring.middleware.stacktrace$wrap_stacktrace_web$fn__1277.invoke(stacktrace.clj:96)
at ring.adapter.jetty$proxy_handler$fn__487.invoke(jetty.clj:25)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
How does one do middleware with async compojure-api? Feel like i'm just totally off on how to implement this.
I am stucked with http-kit async behavior when it comes to write to an sqlite table.
The i/o to the database depends on whether I send the code to a boot repl or run it as a boot script. The i/o proceed only in the repl case. What am I missing? Here 's my code:
#!/usr/bin/env boot
(defn deps [new-deps]
(boot.core/merge-env! :dependencies new-deps))
(deps '[
[http-kit "2.2.0"]
[org.clojure/core.async "0.2.395"]
[org.clojure/java.jdbc "0.7.0-alpha1"]
[org.xerial/sqlite-jdbc "3.16.1"]
[org.slf4j/slf4j-nop "1.7.22"]
])
(require
'[org.httpkit.client :as http]
'[clojure.java.jdbc :as jdbc]
)
(def db-spec
{:classname "org.sqlite.JDBC"
:subprotocol "sqlite"
:subname "sqlite.db"})
;(jdbc/db-do-commands
;db-spec
;(jdbc/create-table-ddl "test" [[:msg :text]]))
(def ins! (partial jdbc/insert! db-spec "test"))
(http/get "http://locahost" {} (fn [_] (ins! {:msg "repl"})))
(defn -main []
(println (System/getProperty "user.dir"))
(http/get "http://locahost" {} (fn [_] (ins! {:msg "exec"}))))
Thanks
The issue with the script not working from command line is that http-kit asynchronous callbacks are handled by daemon threads and the only non-daemon thread is the main thread running your script.
When your -main function ends after submitting HTTP request to http-kit for async processing, the main thread terminates and causes that the JVM shutdowns before daemon threads processing your async callback get a chance to run.
You can check that by adding a sleep expression in the end of -main function to see that your callback is executed:
(defn -main []
(println (System/getProperty "user.dir"))
(http/get "http://locahost" {} (fn [_] (ins! {:msg "exec"})))
(Thread/sleep 60000))
The best way to make sure that the -main function will wait for the result to be processed is to keep the promise returned by http/get call. The promise eventually will contain the result produced by your callback function:
(let [result-promise (http/get "https://www.google.com" {} (fn [_] "Result"))]
#result-promise)
#result-promise is a reader macro/shortcut for (deref result-promise).
The full form might be better when you would like to not block indefinitely - just call deref with timeout-ms and timeout-value arguments:
(deref result-promise 5000 "Didn't get response in 5 seconds. Giving up")
I have some long running process that returns a core.async channel with the result on it when the process has finished.
Now I'd like to return that result using long-polling with HTTP-kit.
Unfortunately I'm a bit confused what the right way of doing so is.
Currently I have a handler (hooked up to a route) that initiates the processing call and sends the result when done:
(defn handler
[request]
(let [c (process request)] ;; long running process that returns a channel
(http/with-channel request channel
(http/send! channel {:status 200
:body (<!! (go (<! c)))))
(http/on-close channel (fn [_] (async/close! c))))))
It works, but I'm unsure if this is the right way.
EDIT since <!! is blocking I'm now trying a non-blocking variant in a go-loop
(defn handler
[request]
(let [c (process request)]
(http/with-channel request channel
(async/go-loop [v (<! c)]
(http/send! channel {:status 200
:body v}))
(http/on-close channel (fn [_] (async/close! c))))))
Why not send on the channel in the go block?
(http/with-channel request channel
(go (http/send! channel (<! c))))
<!! is blocking - so there is no real advantage in your code from just directly calling <!! c in the handler:
(defn handler
[request]
(let [c (process request)] ;; long running process that returns a channel
{:status 200
:body (<!! c)}))
Edit in response to question update: The updated code works - this is a fully functioning namespace which works for me:
(ns async-resp.core
(:require [org.httpkit.server :as http]
[clojure.core.async :as async :refer (<! >! go chan go-loop close! timeout)]))
(defn process
[_]
(let [ch (chan)]
(go
(<! (timeout 5000))
(>! ch "TEST"))
ch))
(defn test-handler
[request]
(let [c (process request)]
(http/with-channel request channel
(go-loop [v (<! c)]
(http/send! channel {:status 200
:body v}))
(http/on-close channel (fn [_] (close! c))))))
(defn run
[]
(http/run-server test-handler {}))
As of the current moment in time though, I had to manually add the tools.analyzer.jvm dependency to project.clj - as I get compilation failures using core.async as-is.
Check you're running the latest core.async and analyzer?
While using Clojure proxies, fns passed to proxy should override existing methods or are they called in conjunction with super.method()?
In the following code, RequestHandler.get() is invoked along with the proxy get [].
;see: http://github.com/paulosuzart/JTornado
(ns org.ctornadoweb)
(import '(org.jtornadoweb Web$RequestHandler))
(import '(org.jtornadoweb HttpServer Web$Application))
(let [myHandler (proxy [Web$RequestHandler] []
(get []
(.write "Hi CLJ"))
(post []
(.write "POST")))]
(.listen
(HttpServer.
(.add (Web$Application.) "/" (class myHandler))
false nil false) 8089))
The same happens to the compiled/inheritance version:
; Starts a JTornado HTTP Server and a sample RequestHandler.
; Bit verbose due to compilation directives. Recommendation is to generate
; a set of macros to hide this.
(ns org.ctornadoweb
; Compiled and implements a static main method to start the server
(:import (org.jtornadoweb HttpServer)
(org.jtornadoweb.Web$Application)
(org.jtornadoweb.Web$RequestHandler))
(:gen-class :prefix "server-"))
(gen-class
:name org.ctornadoweb.MyHandler
:extends org.jtornadoweb.Web$RequestHandler
:prefix "do")
(defn do-get [this]
"Handles the HTTP GET method"
(.write "hello clojure"))
(defn do-post [this]
"Handles the HTTP POST method"
(.write (.getArgument "name" "default" false)))
(defn server-main []
"main method"
(.listen
(org.jtornadoweb.HttpServer.
(.add (org.jtornadoweb.Web$Application.) "/" org.ctornadoweb.MyHandler)
false nil false) 8089))
;use (compile 'org.ctornadoweb)
The trace shows the proxy get being invoked and then the super.get, what throws (by default) an exception.
HTTP 405: Method Not Allowed
at org.jtornadoweb.Web$RequestHandler.get(Web.java:72)
at org.ctornadoweb.proxy$org.jtornadoweb.Web$RequestHandler$0.get(Unknown Source)
I tried to find some words about the actual behavior of Clojure Proxies. Can someone give this help?
No, the super method will not be called automatically, though you can explicitly call it with proxy-super.
The following test case shows things working as they should:
user=> (def foo
(proxy [java.util.ArrayList] []
(clear [] (println "not clearing"))))
#'user/foo
user=> (.size foo)
0
user=> (.add foo "hi")
true
user=> (.add foo "bye")
true
user=> (.size foo)
2
user=> (.clear foo)
not clearing
nil
user=> (.size foo)
2
If super.clear() were getting called, the size would show as 0.