I have the following scenario:
There's some service I use to retrieve some data passing it my input.
Having some input parameters I need to perform N requests against the aforementioned service, gather outputs and do some CPU-heavy task for each output.
I am trying to achieve this using core/async channels.
Here's my attempt (schematically) which kinda works, but it doesn't behave I would like it to.
Will be grateful for any hints on how to improve it.
(defn produce-inputs
[in-chan inputs]
(let input-names-seq (map #(:name %) inputs)]
(doseq [input-name input-names-seq]
(async/go
(async/>! in-chan input-name)))))
(defn consume
[inputs]
(let [in-chan (async/chan 1)
out-chan (async/chan 1)]
(do
(produce-inputs in-chan inputs)
(async/go-loop []
(let [input-name (async/<! in-chan)]
(do
(retrieve-resource-from-service input-name
; response handler
(fn [resp]
(async/go
(let [result (:result resp)]
(async/>! out-chan result)))))
(when input-name
(recur)))))
; read from out-chan and do some heavy work for each entry
(async/go-loop []
(let [result (async/<! out-chan)]
(do-some-cpu-heavy-work result))))))
; entry point
(defn run
[inputs]
(consume inputs))
Is there any way to update it so that at every moment there will be no more than five requests to service (retrieve-resource-from-service) active?
If my explanation isn't clear, please ask questions, I will update it.
You can create another channel to act as a token bucket for limiting the rate of your requests.
See this link for an example of using a token bucket for per-second rate limiting.
To limit the number of simultaneous requests, you can do something along the lines of:
(defn consume [inputs]
(let [in-chan (async/chan 1)
out-chan (async/chan 1)
bucket (async/chan 5)]
;; ...
(dotimes [_ 5] (async/put! bucket :token))
(async/go-loop []
(let [input-name (async/<! in-chan)
token (async/<! bucket)]
(retrieve-resource-from-service
input-name
; response handler
(fn [resp]
(async/go
(let [result (:result resp)]
(async/>! out-chan result)
(async/>! bucket token)))))
(when input-name
(recur))))
;; ...
))
A new channel, bucket, is created, and five items are put into it. Before firing a request, we take a token from the bucket, and put it back once the request is completed. If there are no tokens in the bucket channel, we have to wait until one of the requests is done.
Note: this is just a sketch of the code, you may have to correct it. In particular, if you have any error handlers in your retrieve-resource-from-service function, you should put the token back in case of an error too to avoid eventual deadlocks.
Related
I'm working on a wiki program and using SQLite as the database. I want to create a many-to-many relationship between wiki pages and tags describing those pages. I'm using clojure.java.jdbc to handle the database operations. I would like to enforce foreign key constraints in the page-to-tags cross-reference table. I looked at the information about foreign keys on the SQLite site (https://www.sqlite.org/foreignkeys.html) and believe something like this is what I want;
(def the-db-name "the.db")
(def the-db {:classname "org.sqlite.JDBC"
:subprotocol "sqlite"
:subname the-db-name})
(defn create-some-tables
"Create some tables and a cross-reference table with foreign key constraints."
[]
(try (jdbc/db-do-commands
the-db false
["PRAGMA foreign_keys = ON;"
(jdbc/create-table-ddl :pages
[[:page_id :integer :primary :key]
;...
[:page_content :text]])
(jdbc/create-table-ddl :tags
[[:tag_id :integer :primary :key]
[:tag_name :text "NOT NULL"]])
(jdbc/create-table-ddl :tags_x_pages
[[:x_ref_id :integer :primary :key]
[:tag_id :integer]
[:page_id :integer]
["FOREIGN KEY(tag_id) REFERENCES tags(tag_id)"]
["FOREIGN KEY(page_id) REFERENCES pages(page_id)"]])])
(catch Exception e (println e))))
But attempting to turn the pragma on has no effect.
Just trying to turn the pragma on and check for effect:
(println "Check before:" (jdbc/query the-db ["PRAGMA foreign_keys;"]))
; Transactions on or off makes no difference.
(println "Result of execute!:" (jdbc/execute! the-db
["PRAGMA foreign_keys = ON;"]))
(println "Check after:" (jdbc/query the-db ["PRAGMA foreign_keys;"]))
;=> Check before: ({:foreign_keys 0})
;=> Result of execute!: [0]
;=> Check after: ({:foreign_keys 0})
The results indicate that the library (org.xerial/sqlite-jdbc "3.21.0.1") was compiled to support foreign keys since there were no errors, but trying to set the pragma has no effect.
I found this in the JIRA for the clojure JDBC back in 2012. The described changes have been implemented since then, but the code still has no effect.
Finally found this answer to a Stackoverflow question that pointed to this post back in 2011. That allowed me to cobble together something that did seem to set the pragma. The code below depends on creating a specially configured Connection.
(ns example
(:require [clojure.java.jdbc :as jdbc])
(:import (java.sql Connection DriverManager)
(org.sqlite SQLiteConfig)))
(def the-db-name "the.db")
(def the-db {:classname "org.sqlite.JDBC"
:subprotocol "sqlite"
:subname the-db-name})
(defn ^Connection get-connection
"Return a connection to a SQLite database that
enforces foreign key constraints."
[db]
(Class/forName (:classname db))
(let [config (SQLiteConfig.)]
(.enforceForeignKeys config true)
(let [connection (DriverManager/getConnection
(str "jdbc:sqlite:" (:subname db))
(.toProperties config))]
connection)))
(defn exec-foreign-keys-pragma-statement
[db]
(let [con ^Connection (get-connection db)
statement (.createStatement con)]
(println "exec-foreign-keys-pragma-statement:"
(.execute statement "PRAGMA foreign_keys;"))))
Based on the above, I rewrote the table creation code above as:
(defn create-some-tables
"Create some tables and a cross-reference table with foreign key constraints."
[]
(when-let [conn (get-connection the-db)]
(try
(jdbc/with-db-connection
[conn the-db]
; Creating the tables with the foreign key constraints works.
(try (jdbc/db-do-commands
the-db false
[(jdbc/create-table-ddl :pages
[[:page_id :integer :primary :key]
[:page_content :text]])
(jdbc/create-table-ddl :tags
[[:tag_id :integer :primary :key]
[:tag_name :text "NOT NULL"]])
(jdbc/create-table-ddl :tags_x_pages
[[:x_ref_id :integer :primary :key]
[:tag_id :integer]
[:page_id :integer]
["FOREIGN KEY(tag_id) REFERENCES tags(tag_id)"]
["FOREIGN KEY(page_id) REFERENCES pages(page_id)"]])])
; This still doesn't work.
(println "After table creation:"
(jdbc/query the-db "PRAGMA foreign_keys;"))
(catch Exception e (println e))))
; This returns the expected results.
(when-let [statement (.createStatement conn)]
(try
(println "After creating some tables: PRAGMA foreign_keys =>"
(.execute statement "PRAGMA foreign_keys;"))
(catch Exception e (println e))
(finally (when statement
(.close statement)))))
(catch Exception e (println e))
(finally (when conn
(.close conn))))))
The tables are created as expected. Some of the clojure.java.jdbc functions still don't seem to work as desired though. (See the jdbc/query call in the middle of the listing.) Getting things to always work as expected seems very "manual" having to fall back on java interop. And it seems like every interaction with the database requires using the specially configured Connection returned by the get-connection function.
Is there a better way to enforce foreign key constraints in SQLite in Clojure?
With the advent of next.jdbc you can now do that like so:
(ns dev
(:require [next.jdbc :as jdbc]
[next.jdbc.sql :as sql]))
(with-open [conn (jdbc/get-connection {:dbtype "sqlite" :dbname "test.db"})]
(println (sql/query conn ["PRAGMA foreign_keys"]))
(jdbc/execute! conn ["PRAGMA foreign_keys = ON"])
; jdbc/execute whatever you like here...
(println (sql/query conn ["PRAGMA foreign_keys"])))
This outputs
[{:foreign_keys 0}]
[{:foreign_keys 1}]
I've not played with SqlLite, but would recommend you test with either
H2: Pure java, can run in memory for tests (http://www.h2database.com)
Postgres: Needs install, but is the gold standard for SQL compliance (https://www.postgresql.org)
Also, when debugging it may be easier to use pure SQL strings (see http://clojure-doc.org/articles/ecosystem/java_jdbc/using_sql.html):
(j/execute! db-spec
["update fruit set cost = ( 2 * grade ) where grade > ?" 50.0])
Using pure SQL strings (especially when debugging) can avoid many misunderstandings/pitfalls with JDBC. Also, keep in mind that you may discover a bug in either the Clojure JDBC libs or the DB itself.
I'm not sure SQLite does support the features you described above. If you really want to keep your data being consisted with strict constraints, use PostgeSQL database. I know that working with SQLite seems easier especially when you've just started the project, but believe me, using Postgres really worth it.
Here is an example of post and tags declaration using Postgres that takes lots of details into account:
create table post(
id serial primary key,
title text not null,
body text not null
);
create table tags(
id serial primary key,
text text not null unique
);
create table post_tags(
id serial primary key,
post_id integer not null references posts(id),
tag_id integer not null references tags(id),
unique(post_id, tag_id)
);
Here, the tags table cannot contain two equal tags. That's important to keep only unique tag strings to prevent the table from growing.
The bridge table that links a post with tags has a special constraint to prevent a case when a specific tag is linked to a post several times. Say, if a post has "python" and "clojure" tags attached, you won't be able to add "python" one more time.
Finally, each reference clause when declaring a table creates a special constraint that prevents you from referencing an id that does not exist in a target table.
Installing Postgres and setting it up might be a bit difficult, but nowadays there are such one-click applications like Postgres App that are quite easy to use even if you are not familiar with them.
I am currently working on a route planning robot that will deliver parcels around an office. The office is set up as a graph with weighted edges using ubegraph.
I have a recursive function that can plan the route for a journey between any amount of rooms. However I'm now trying to add 'parcels' to the robot and I'm unsure if the way I'm going about it is correct.
(defn journey [start end]
(alg/pprint-path (alg/shortest-path all-edges {:start-node start, :end-node end, :cost-attr :weight})))
(defn fullpath [& stops]
(doall (map (fn [a b] (journey a b)) stops (rest stops) )))
Above is the recursive function that at the moment takes in a set of stops and then rights out the route using journey. My idea is that instead of stops I pass in the robot to full path and robot will contain parcels and stops.
;;passed into fullpath to plan journey of robot
(def robot [parcels ;;needs to contain a set of parcels
stops]) ;;stops are all of the end's gathered from parcel
The parcels will come from another structure.
;;passed into robot
(def parcel [start
end
delivered])
Is this a viable way of doing it or will it be impossible?
Any help greatly appreciated.
I am using jet for asynchronous ring adapter.
Jet also comes with async http-client which returns a channel whose value's :body is also a channel.
Also, async server route handler can return a map whose :body key can contain a channel. When this channel would be closed, the response would be returned to the client.
I am writing following go code :
(defn- api-call-1 []
(go (-> (jet-client/get "api-url-1")
<!
:body ;; jet http client :body is also a channel.
<!
api-call-1-response-parse)))
(defn- api-call-2 []
(go (-> (jet-client/get "api-url-2")
<!
:body
<!
api-call-2-response-parse)))
(defn route-function []
(let [response-chan (chan)]
(go
(let [api-call-1-chan (api-call-1) ;; using channel returned by go
api-call-2-chan (api-call-2)]
(-> {:api-1 (<! api-call-1-chan)
:api-2 (<! api-call-2-chan)}
encode-response
(>! response-chan)))
(close! response-chan))
;; for not blocking server thread, return channel in body
{:body response-chan :status 200}))
In my route-function, i can not block.
Though this code works fine, Is using go in api-call-1 is bad ?
I found that to use <! in api-call-1 i need to put it in a go block.
Now i use this go block's channel in route-function. Does this look unidomatic ? I am concerned about not exposing api-call-1-response-parse or even :body as channel to the route-function.
What is the right way to structure go block code and functions ?
Should i care about extra go blocks in functions api-call-1/2 ?
What you have looks much like the equivalent code I have in production. This is quite idiomatic, so I think your code is structured correctly.
The fact that core.async parking opperations can't cross function boundaries stems from the fact that it's written as a macro and needs to process the whole chunk of code at once (or at least while it's lexically available). This tends to make all core.async code come out in the pattern you are using.
EDIT: For a working sample, take a look at this demo project.
Disclaimer: I'm a total noob in building java web applications.
I'm trying to use JWT with Clojure but I simply don't know how deal with this "servlet" thing. So far, my idea was:
Create a WApplication with a "Hello World" form.
(defn make-hello-app [env]
(let [wapp (WApplication. env)
root (.getRoot wapp)]
(.setTitle wapp "Hello world")
(.addWidget root (WText. "Hello!!!!"))
wapp))
Create a servlet, inherited from WtServlet.
(def servlet
(proxy [WtServlet] []
(createApplication [env]
(make-hello-app env))))
Start jetty and use the servlet. This is what I don't know how to do. So far, this was my best shot:
(ns jwttest.core
(:use compojure.core)
(:use ring.adapter.jetty)
(:import (org.eclipse.jetty.server Server))
(:import (eu.webtoolkit.jwt WApplication WEnvironment WtServlet WText WPushButton WLineEdit WBreak)))
;; (the funcions above were defined here)
;; create a jetty server
(defn create-a-jetty-server []
(let [connector (doto (SelectChannelConnector.)
(.setPort 8080)
(.setHost "localhost"))
server (doto (Server.)
(.addConnector connector)
(.setSendDateHeader true))]
server))
;; start the application
(defn start-the-app []
(let [server (create-a-jetty-server)]
;; ???? .addServlet ? How?
(.start server)))
In my project.clj I have:
[org.clojure/clojure "1.4.0"]
[eu.webtoolkit/jwt "3.2.0"]
[compojure "1.1.1"]
[ring "1.1.2"]
I know ring can create a servlet from a handler, but in this case I already have a servlet so... what should I do to run this?
Note: I'm basing my code on this very old post made in 2009.
I dug through some of the Jetty API and the Ring/Noir jetty handling code and here's a summary of what I found (and haven't had a chance to test)
Jetty Server has a "setHandler" method which takes a handler (thanks Ring)
There is a ServletHandler class which looks like it fits into the above and which has a number of addServlet like methods which look like they do what you need them to.
You should be able to set the handler to a Servlet handler and go from there.
I am writing a web app that would require the hunchentoot web server. I have almost no working knowledge of hunchentoot, or any web server for that matter, and I am wondering how my app written in Common Lisp would serve pages to a web client. I have seen some excellent examples (e.g. Hunchentoot Primer, Lisp for the Web) esp. the one listed on the Hunchentoot page. Do you know where I can find more of such examples?
Thanks.
I am wondering how my app written in Common Lisp would serve pages to a web client.
Hunchentoot serves all things that are in its *dispatch-table*, which is just a list of dispatch handlers.
The simplest thing to do is to serve a static file. One typical example would be a CSS file:
(push (create-static-file-dispatcher-and-handler "/example.css"
"example.css")
*dispatch-table*)
For a web application, you would most likely want to dynamically create a web page. You do this by defining a function that returns the page as a string (e.g. with CL-WHO), then creating a handler for this function:
(defun foo ()
(with-html-output-to-string ; ...
))
(push (create-prefix-dispatcher "/foo.html" 'foo)
*dispatch-table*)
You can eliminate a lot of boilerplate through macros, by the way:
(defmacro standard-page ((title) &body body)
`(with-html-output-to-string (*standard-output* nil :prologue t :indent t)
(:html :xmlns "http://www.w3.org/1999/xhtml"
:xml\:lang "de"
:lang "de"
(:head
(:meta :http-equiv "Content-Type"
:content "text/html;charset=utf-8")
(:title ,title)
(:link :type "text/css"
:rel "stylesheet"
:href "/example.css"))
(:body
,#body))))
(defmacro defpage (name (title) &body body)
`(progn
(defmethod ,name ()
(standard-page (,title)
,#body))
(push (create-prefix-dispatcher ,(format nil "/~(~a~).html" name) ',name)
*dispatch-table*)))
The examples you have found should be sufficient to get you started, and if you run into problems, read the manual, then ask concrete questions.
define-easy-handler registers the handler you are defining automatically in a global variable which gets checked when a HTTP request arrives (the variable is called *easy-handler-alist*). So it's being taken care of automatically. Do you want to use a handler of a different form than the one defined in the tutorial?
I think there is an example using Hunchentoot in the Elephant distribution (Elephant being a Persistent Object Database for Common Lisp.)