Clojure Hystrix exceptions because of too many http requests - http

I need to build a system that needs to make a a ton of external http requests and I have to use Netflix's Hystrix to create fallbacks and re-route exceptions. Right now, I've got a very simple setup:
(hystrix/defcommand fetch-request
{:hystrix/group-key "c0"
:hystrix/command-key "URLFetch"
:hystrix/fallback-fn (fn [url]
{:status 419
:headers {}
:body "failed"})}
[url]
#(http/get url))
(defn test3 []
(let [n 4000
m (range 0 n)
p (partition 300 m)]
(doseq [t p]
(thread
(doseq [x t]
(let [res (fetch-request "http://localhost:3000/comments")]
(match (:status res)
200 (prn x)
:else (prn nil)))
)))
))
When I execute test3, I keep getting nil. If I decrease the value of n, I get a status of 200 (which is what I need). Furthermore, if I just use http/get directly on test3 function instead of fetch-request command, it works without any problem (even when the value of n is above 7000).
Note: The reason I'm using partitions+threads is to parallelize the http requests. If you know a better way to execute large volume of http requests in clojure as fast as possible, that would be really awesome.
Update:
I played w/ a variety of configurations. Most of them didn't yield a different outcome. Most of the reqs are still not getting executed so the fallback is immediately triggered. I tried disabling the circuitBreaker (which I don't want to, that's why I'm using hystrix) to see if it did anything -- and it did. 80% of the requests passed through.
(hystrix/defcommand fetch-request
{:hystrix/group-key "ct0"
:hystrix/command-key "URLFetch"
:hystrix/init-fn
(fn [_ ^com.netflix.hystrix.HystrixCommand$Setter setter]
(.andCommandPropertiesDefaults
setter
^com.netflix.hystrix.HystrixCommandProperties$Setter
(.withCircuitBreakerEnabled
(HystrixCommandProperties/Setter)
false)
; ^com.netflix.hystrix.HystrixCommandProperties$Setter
; (.withExecutionTimeoutInMilliseconds
; (HystrixCommandProperties/Setter)
; 1000))
; (.andCommandPropertiesDefaults
; setter
; (.withExecutionIsolationStrategy
; (HystrixCommandProperties/Setter)
; com.netflix.hystrix.HystrixCommandProperties$ExecutionIsolationStrategy/THREAD)
))
:hystrix/fallback-fn
(fn fetch-req-fallback [url]
{:status 419
:headers {}
:body "failed"})}
[url]
#(http/get url))
Update 2:
Removing the thread block fixes the problem. However, I do need to execute these requests across multiple thread so it doesn't necessarily solve my issue.

Fixed it. I had to modify the threadpool properties.
(hystrix/defcommand fetch-request
{:hystrix/group-key "ct0"
:hystrix/command-key "URLFetch"
:hystrix/thread-pool-key "URLThread"
:hystrix/init-fn
(fn [_ ^com.netflix.hystrix.HystrixCommand$Setter setter]
(.andThreadPoolPropertiesDefaults
setter
(doto (HystrixThreadPoolProperties/Setter)
(.withMaxQueueSize 10)
(.withQueueSizeRejectionThreshold 10)))
(.andCommandPropertiesDefaults
setter
(doto (HystrixCommandProperties/Setter)
(.withExecutionIsolationStrategy
com.netflix.hystrix.HystrixCommandProperties$ExecutionIsolationStrategy/THREAD)
(.withExecutionTimeoutInMilliseconds
1500))))
:hystrix/fallback-fn
(fn fetch-req-fallback [url]
{:status 419
:headers {}
:body "failed"})}
[url]
#(http/get url))

Related

Clojure Cross Origin Error - Totally Lost

I have the following simple server in Clojure using Compojure (which is some flavor of the ring pattern). Everything was working fine in development, and now that I'm in prod I can't get CORS to work for the life of me - I have a wrap-preflight function which seems to work fine, but I keep getting CORS errors in terminal and neither the post or get requests for my comment system work. I am totally lost and very frustrated, I've asked around and no one else seems to know.
Here is the main core.clj code - If anyone has any ideas please let me know. You can see the errors live at thedailyblech.com (not an advert, but maybe it will help debug).
Thank you!
(ns clojure-play.core
(:use org.httpkit.server
[compojure.core :refer :all]
[compojure.route :as route]
[clojure.data.json :as json]
[clojure.tools.logging :only [info]]
[clojure-play.routes :as routes]
[ring.middleware.json :only [wrap-json-body]]
[ring.middleware.cors :refer [wrap-cors]])
(:require [monger.core :as mg]
[monger.collection :as mc]
[clojure.edn :as edn]
[clojure.java.io :as io]
[compojure.handler :as handler])
(:import [org.bson.types ObjectId]
[com.mongodb DB WriteConcern])
(:gen-class))
(println "in the beginning was the command line...")
(defonce channels (atom #{}))
(defn connect! [channel]
(info "channel open")
(swap! channels conj channel))
(defn notify-clients [msg]
(doseq [channel #channels]
(send! channel msg)))
(defn disconnect! [channel status]
(info "channel closed:" status)
(swap! channels #(remove #{channel} %)))
(defn ws-handler [request]
(with-channel request channel
(connect! channel)
(on-close channel (partial disconnect! channel))
(on-receive channel #(notify-clients %))))
(defn my-routes [db]
(routes
(GET "/foo" [] "Hello Foo")
(GET "/bar" [] "Hello Bar")
(GET "/json_example/:name" [] routes/json_example)
(GET "/json_example" [] routes/json_example)
(POST "/email" [] routes/post_email)
(POST "/write_comment" [] (fn [req] (routes/write_comment req db)))
(POST "/update_comment" [] (fn [req] (routes/update_comment req db)))
(GET "/read_comments/:path" [path] (fn [req] (routes/read_comments req db path)))
(GET "/read_comments/:path1/:path2" [path1 path2] (fn [req] (routes/read_comments req db (str path1 "/" path2))))
(GET "/ws" [] ws-handler)))
(defn connectDB []
(defonce connection
(let
[uri "mongodb://somemlabthingy"
{:keys [conn db]} (mg/connect-via-uri uri)]
{:conn conn
:db db}))
{:db (:db connection)
:conn (:conn connection)})
(def cors-headers
"Generic CORS headers"
{"Access-Control-Allow-Origin" "*"
"Access-Control-Allow-Headers" "*"
"Access-Control-Allow-Methods" "GET POST OPTIONS DELETE PUT"})
(defn preflight?
"Returns true if the request is a preflight request"
[request]
(= (request :request-method) :options))
(defn -main
"this is main"
[& args]
(println "hello there main")
(def db (get (connectDB) :db))
(println (read-string (slurp (io/resource "environment/config.edn"))))
(defn wrap-preflight [handler]
(fn [request]
(do
(println "inside wrap-preflight")
(println "value of request")
(println request)
(println "value of handler")
(println handler)
(if (preflight? request)
{:status 200
:headers cors-headers
:body "preflight complete"}
(handler request)))))
(run-server
(wrap-preflight
(wrap-cors
(wrap-json-body
(my-routes db)
{:keywords? true :bigdecimals? true})
:access-control-allow-origin [#"http://www.thedailyblech.com"]
:access-control-allow-methods [:get :put :post :delete :options]
:access-control-allow-headers ["Origin" "X-Requested-With"
"Content-Type" "Accept"]))
{:port 4000}))
The CORS middleware handles the preflight stuff automatically -- you do not need separate middleware for it, nor do you need to produce your own headers etc.
You have it wrapping the routes which is correct -- so CORS-checking happens first, then routing. You should remove your custom preflight middleware and it should work at that point.
We use wrap-cors at work and the only complication we hit was in allowing enough headers (some inserted by production infrastructure, like load balancers). We ended up with this:
:access-control-allow-headers #{"accept"
"accept-encoding"
"accept-language"
"authorization"
"content-type"
"origin"}
For what it's worth, here's what we have for methods:
:access-control-allow-methods [:delete :get
:patch :post :put]
(you do not need :options in there)
After digging around for hours, I found this to be super helpful on an issues post on the ring-cors github, for which documentation was severely lacking.
Using the linked gist, I was able to get past the CORS issues:
; Copied from linked gist
(def cors-headers
"Generic CORS headers"
{"Access-Control-Allow-Origin" "*"
"Access-Control-Allow-Headers" "*"
"Access-Control-Allow-Methods" "GET"})
(defn preflight?
"Returns true if the request is a preflight request"
[request]
(= (request :request-method) :options))
(defn all-cors
"Allow requests from all origins - also check preflight"
[handler]
(fn [request]
(if (preflight? request)
{:status 200
:headers cors-headers
:body "preflight complete"}
(let [response (handler request)]
(update-in response [:headers]
merge cors-headers )))))
; my -main
(defn -main
"Main entry point"
[& args]
(let [port (Integer/parseInt (or (System/getenv "PORT") "8081"))]
(server/run-server
(all-cors
(wrap-defaults #'app-routes site-defaults))
{:port port})
(println "Running on" port)))
This finally allowed me to see the headers properly set in Chrome dev tools and also got rid of the warning on my React front-end.
Access-Control-Allow-Headers: *
Access-Control-Allow-Methods: GET
Access-Control-Allow-Origin: *
Might be worth trying and adding an explicit
(OPTIONS "/*" req handle-preflight)
route to your Compojure routes - in my case that's what made it work.
(RING-MIDDLEWARE-CORS/wrap-cors
:access-control-allow-credentials "true"
:access-control-allow-origin [#".*"]
:access-control-allow-headers #{"accept"
"accept-encoding"
"accept-language"
"authorization"
"content-type"
"origin"}
:access-control-allow-methods [:get :put :post :delete :options])

ClojureScript file preloader - function or pattern to emulate promise?

I'm trying to create a file preloader within ClojureScript. My idea was a pattern like this:
(def urls (atom[]))
(def loaded-resources (atom []))
(def all-resources (promise))
(defn loading-callback []
(if (= (count urls) (count loaded-resources))
(deliver all-resources loaded-resources)))
;; fill urls array
;; start ajax-loading with loading-callback on success
So my main function could go on until it would require the resources and then wait for them, which works well in Clojure.
Unfortunately, promises don't exist in ClojureScript, so how can I work around that issue? There's promesa bringing promises to CLJS based on core.async channels, but it only allows future-like promises that wait for a single function to execute which won't suffice my needs (at least in the way I've been thinking about it yesterday...).
Any suggestions to solve this issue? Maybe use a completely different pattern? I want to keep the code as simple as possible to convince people in my team to try out CLJ/S.
EDIT:
After Alan's second idea:
(def urls (atom[]))
(def loaded-resources (atom []))
(defn loading-callback [data]
(swap! loaded-resources conj data))
(defn load! [post-loading-fn]
(add-watch loaded-resources :watch-loading
(fn [_ _ _ cur]
(if (= (count cur) (count #urls)) (post-loading-fn))))
;; init ajax loading
)
(defn init []
;; fill urls array
(load! main))
(main []
(do-terrific-stuff #loaded-resources))
Meanwhile I had tried to use core.async
(def urls (atom []))
(def loaded-resources (atom []))
(def resource-chan (chan))
(defn loading-callback [data]
(go (>! resource-chan data)))
;; fill url array from main
(load! []
;; init ajax loading
(go-loop []
(when-not (= (count #loaded-resources) (count #urls))
(swap! loaded-resources conj (<! resource-chan))
(recur)))
Not sure which version is better.
I can think of 2 approaches.
Change all-resources to another atom, initialized at nil. Poll it 2x-5x/sec until it is not nil and has the "delivered" result.
Use add-watch to register a callback function to execute when the value is changed. This takes the place of blocking until the value is delivered. It is described here: http://clojuredocs.org/clojure.core/add-watch
They show a good example:
(def a (atom {}))
(add-watch a :watcher
(fn [key atom old-state new-state]
(prn "-- Atom Changed --")
(prn "key" key)
(prn "atom" atom)
(prn "old-state" old-state)
(prn "new-state" new-state)))
(reset! a {:foo "bar"})
;; "-- Atom Changed --"
;; "key" :watcher
;; "atom" #<Atom#4b020acf: {:foo "bar"}>
;; "old-state" {}
;; "new-state" {:foo "bar"}
;; {:foo "bar"}
Assuming your load resource function returns a channel (like cljs-http/get).
In clj, all you need to do is hold on to them to do a "wait-all".
(let [cs (doall (map load-resource urls)) ;; initiate the get
... ;; other initialisation
res (map <!! cs)] ;; wait-all for the resources
(do-other-things res))
In cljs, you can accumulate the responses before you continue:
(go
(let [res (atom [])]
(doseq [c cs]
(swap! res conj (<! c)))
(do-other-things #res)))
JavaScript is a single threaded environment so there is no blocking wait.
If you wish to request multiple resources and continue iff they have all been served, I do recommend using core.async and especially pipeline-async. It has a knob to finetune the parallelism of asynchronous requests. Here is idiomatic ClojureScript code to achieve what you want:
(ns example.core
(:require [cljs.core.async :refer [chan take! put! pipeline-async]
:as async]))
(defn load-resources [urls on-resources]
(let [urls-ch (chan (count urls))
resources-ch (chan)]
;; Create pipeline:
(pipeline-async 10 ;; have at most 10 requests in flight at
;; the same time, finetune as desired
resources-ch
(fn [url done-ch]
;; Pseudo code:
(request-resource
url
(fn [loaded-resource]
(put! done-ch loaded-resource))))
urls-ch)
;; Eagerly aggregate result until results-ch closes, then call back:
(take! (async/into [] resources-ch) on-resources)
;; Start the party by putting all urls onto urls-ch
;; and then close it:
(async/onto-chan urls-ch urls)))

Clojure - core.async interface for apache kafka

I am using clj-kafka, and I am trying to make a core.async interface to it in the REPL.
I am getting some messages, but my structure feels wrong : I either cannot stop receiving messages, or have to launch the go routine again to receive more messages.
Here is my attempt :
(defn consume [topic]
(let [consume-chan (chan)]
(with-resource [c (consumer config)]
shutdown
(go (doseq [m (messages c "test")]
(>! chan message) ;; should I check the return value?
)))
consume-chan)) ;; is it the right place to return a channel ?
(def consume-chan (consume "test"))
;;(close! consume-chan)
(go (>! consume-chan "hi")) ;; manual test, but I have some messages in Kafka already
(def cons-ch (go
(with-resource [c (consumer config)]
shutdown
(doseq [m (messages c "test")]
(>! consume-chan m))))) ;; should I check something here ?
;;(close! cons-ch)
(def go-ch
(go-loop []
(if-let [km (<! consume-chan)]
(do (println "Got a value in this loop:" km)
(recur))
(do (println "Stop recurring - channel closed")))))
;;(close! go-ch)
How do I consume a lazy-sequence of messages with a core.async interface ?
Here's what I would do:
>! and <! return nil if the channel is closed, so make sure that the loop exits when this happens - that way you can easily end the loop from the outside by closing the channel.
Use a try/catch to check for exceptions inside the go block, and make any exception the return value so that they don't get lost.
Check for exceptions on read values, to catch anything from inside the channel.
The go blocks return a channel, and the return value of the code inside the block (like the exceptions from above) will be put on the channel. Check these channels for exceptions, possibly to rethrow.
You can now write to a channel like this:
(defn write-seq-to-channel
[channel
values-seq]
(a/go
(try
(loop [values values-seq]
(when (seq values)
(when (a/>! channel (first values))
(recur (rest values)))))
(catch Throwable e
e))))
and you read like this:
(defn read-from-channel-and-print
[channel]
(a/go
(try
(loop []
(let [value (a/<! channel)]
(when value
(when (instance? Throwable value)
(throw value))
(println "Value read:" value)
(recur))))
(catch Throwable e
e))))
You will now have two channels, so use something like alts! or alts!! to check for your loops exiting. Close the channel when you are done.

Using http-kit long polling with core.async channels

I have some long running process that returns a core.async channel with the result on it when the process has finished.
Now I'd like to return that result using long-polling with HTTP-kit.
Unfortunately I'm a bit confused what the right way of doing so is.
Currently I have a handler (hooked up to a route) that initiates the processing call and sends the result when done:
(defn handler
[request]
(let [c (process request)] ;; long running process that returns a channel
(http/with-channel request channel
(http/send! channel {:status 200
:body (<!! (go (<! c)))))
(http/on-close channel (fn [_] (async/close! c))))))
It works, but I'm unsure if this is the right way.
EDIT since <!! is blocking I'm now trying a non-blocking variant in a go-loop
(defn handler
[request]
(let [c (process request)]
(http/with-channel request channel
(async/go-loop [v (<! c)]
(http/send! channel {:status 200
:body v}))
(http/on-close channel (fn [_] (async/close! c))))))
Why not send on the channel in the go block?
(http/with-channel request channel
(go (http/send! channel (<! c))))
<!! is blocking - so there is no real advantage in your code from just directly calling <!! c in the handler:
(defn handler
[request]
(let [c (process request)] ;; long running process that returns a channel
{:status 200
:body (<!! c)}))
Edit in response to question update: The updated code works - this is a fully functioning namespace which works for me:
(ns async-resp.core
(:require [org.httpkit.server :as http]
[clojure.core.async :as async :refer (<! >! go chan go-loop close! timeout)]))
(defn process
[_]
(let [ch (chan)]
(go
(<! (timeout 5000))
(>! ch "TEST"))
ch))
(defn test-handler
[request]
(let [c (process request)]
(http/with-channel request channel
(go-loop [v (<! c)]
(http/send! channel {:status 200
:body v}))
(http/on-close channel (fn [_] (close! c))))))
(defn run
[]
(http/run-server test-handler {}))
As of the current moment in time though, I had to manually add the tools.analyzer.jvm dependency to project.clj - as I get compilation failures using core.async as-is.
Check you're running the latest core.async and analyzer?

Does a Clojure proxy always invoke super.method()?

While using Clojure proxies, fns passed to proxy should override existing methods or are they called in conjunction with super.method()?
In the following code, RequestHandler.get() is invoked along with the proxy get [].
;see: http://github.com/paulosuzart/JTornado
(ns org.ctornadoweb)
(import '(org.jtornadoweb Web$RequestHandler))
(import '(org.jtornadoweb HttpServer Web$Application))
(let [myHandler (proxy [Web$RequestHandler] []
(get []
(.write "Hi CLJ"))
(post []
(.write "POST")))]
(.listen
(HttpServer.
(.add (Web$Application.) "/" (class myHandler))
false nil false) 8089))
The same happens to the compiled/inheritance version:
; Starts a JTornado HTTP Server and a sample RequestHandler.
; Bit verbose due to compilation directives. Recommendation is to generate
; a set of macros to hide this.
(ns org.ctornadoweb
; Compiled and implements a static main method to start the server
(:import (org.jtornadoweb HttpServer)
(org.jtornadoweb.Web$Application)
(org.jtornadoweb.Web$RequestHandler))
(:gen-class :prefix "server-"))
(gen-class
:name org.ctornadoweb.MyHandler
:extends org.jtornadoweb.Web$RequestHandler
:prefix "do")
(defn do-get [this]
"Handles the HTTP GET method"
(.write "hello clojure"))
(defn do-post [this]
"Handles the HTTP POST method"
(.write (.getArgument "name" "default" false)))
(defn server-main []
"main method"
(.listen
(org.jtornadoweb.HttpServer.
(.add (org.jtornadoweb.Web$Application.) "/" org.ctornadoweb.MyHandler)
false nil false) 8089))
;use (compile 'org.ctornadoweb)
The trace shows the proxy get being invoked and then the super.get, what throws (by default) an exception.
HTTP 405: Method Not Allowed
at org.jtornadoweb.Web$RequestHandler.get(Web.java:72)
at org.ctornadoweb.proxy$org.jtornadoweb.Web$RequestHandler$0.get(Unknown Source)
I tried to find some words about the actual behavior of Clojure Proxies. Can someone give this help?
No, the super method will not be called automatically, though you can explicitly call it with proxy-super.
The following test case shows things working as they should:
user=> (def foo
(proxy [java.util.ArrayList] []
(clear [] (println "not clearing"))))
#'user/foo
user=> (.size foo)
0
user=> (.add foo "hi")
true
user=> (.add foo "bye")
true
user=> (.size foo)
2
user=> (.clear foo)
not clearing
nil
user=> (.size foo)
2
If super.clear() were getting called, the size would show as 0.

Resources