I have a server running Hunchentoot (CentOS and SBCL). When a user submits a specific type of post request, a subprocess is launched (run-program) which can take up to four minutes to complete. If five people perform that specific type of request at the same time, the server runs out of memory and all of the subprocesses break. What technique do you recommend for queueing up the processes and running them one at a time?
You could set up a single worker thread which receives instructions from a message queue. I have used chanl in the past for similar things, but there are several other options.
(defvar *worker-queue* '())
(defvar *worker-queue-mutex* (sb-thread:make-mutex :name "worker-queue-lock"))
(defvar *worker-queue-semaphore* (sb-thread:make-semaphore :name "worker-queue-semaphore" :count 0))
(defvar *worker-thread*)
(defun worker-queue-function ()
(sb-thread:with-mutex (*worker-queue-mutex*)
(let ((popped-worker-queue-item (pop *worker-queue*)))
(do-something-with popped-worker-queue-item))))
(defun make-worker-thread ()
(setq *worker-thread* (sb-thread:make-thread (lambda ()
(loop
(sb-thread:wait-on-semaphore *worker-queue-semaphore*)
(worker-queue-function)))
:name "worker-thread")))
(defun add-item-to-worker-queue (item-to-add-to-worker-queue)
(sb-thread:with-mutex (*worker-queue-mutex*)
(setq *worker-queue* (append *worker-queue* (list item-to-add-to-worker-queue)))
(sb-thread:signal-semaphore *worker-queue-semaphore*))))
Related
when open an R script in emacs invariably an additional buffer opens up called Flymake Log with the following message
Warning [flymake DR.R]: Disabling backend flymake-proc-legacy-flymake because (error Can’t find a suitable init function)
Following this discussion I added the following line to my init.el file:
(remove-hook 'flymake-diagnostic-functions 'flymake-proc-legacy-flymake)
but it didn't solve the problem.
Additionally, when I start an R process using M-x R an ESS buffer opens up, which says
current-prefix-arg=nil
(inferior-ess: waiting for process to start (before hook)
(inferior-ess 3): waiting for process after hook(R): inferior-ess-language-start=options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)
This happens whenever I restart my R session. It's really annoying and distracting when I constantly have to cycle through these redundant buffers. I say they are redundant, because I haven't noticed any way in which the normal R operation would be disrupted.
I'm using emacs 26.3 with ESS 18.10.3 on ubuntu 20.04 with R 3.6.3. Also, please see below my entire init.el file
(require 'package)
(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
(not (gnutls-available-p))))
(proto (if no-ssl "http" "https")))
;; Comment/uncomment these two lines to enable/disable MELPA and MELPA Stable as desired
(add-to-list 'package-archives (cons "melpa" (concat proto "://melpa.org/packages/")) t)
;;(add-to-list 'package-archives (cons "melpa-stable" (concat proto "://stable.melpa.org/packages/")) t)
(when (< emacs-major-version 24)
;; For important compatibility libraries like cl-lib
(add-to-list 'package-archives (cons "gnu" (concat proto "://elpa.gnu.org/packages/")))))
(package-initialize)
(custom-set-variables
;; custom-set-variables was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won't work right.
'(package-selected-packages (quote (dracula-theme ess-smart-underscore ess)))
'(pop-up-windows nil))
(custom-set-faces
;; custom-set-faces was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won't work right.
)
;; Makes *scratch* empty.
(setq initial-scratch-message "")
;; Removes *scratch* from buffer after the mode has been set.
(defun remove-scratch-buffer ()
(if (get-buffer "*scratch*")
(kill-buffer "*scratch*")))
(add-hook 'after-change-major-mode-hook 'remove-scratch-buffer)
;; Removes *messages* from the buffer.
(setq-default message-log-max nil)
(kill-buffer "*Messages*")
;; Removes *Completions* from buffer after you've opened a file.
(add-hook 'minibuffer-exit-hook
'(lambda ()
(let ((buffer "*Completions*"))
(and (get-buffer buffer)
(kill-buffer buffer)))))
;; Don't show *Buffer list* when opening multiple files at the same time.
(setq inhibit-startup-buffer-menu t)
;; Show only one active window when opening multiple files at the same time.
(add-hook 'window-setup-hook 'delete-other-windows)
;; backup in one place. flat, no tree structure
(setq backup-directory-alist '(("" . "~/.emacs.d/backup")))
;; easier switching between windows
(windmove-default-keybindings 'control)
;; color theme
(load-theme 'dracula t)
;; matching parenthesis
(show-paren-mode 1)
;; keybindings for resizing windows
(global-set-key (kbd "S-C-<left>") 'shrink-window-horizontally)
(global-set-key (kbd "S-C-<right>") 'enlarge-window-horizontally)
(global-set-key (kbd "S-C-<down>") 'shrink-window)
(global-set-key (kbd "S-C-<up>") 'enlarge-window)
;; easier switching between buffers
(ido-mode 1)
;; no menu bar
(menu-bar-mode -1)
;; no line wrapping by default
(set-default 'truncate-lines t)
;; Disable Flymake warnings
(remove-hook 'flymake-diagnostic-functions 'flymake-proc-legacy-flymake)
;; disables ESS buffer
(setq ess-write-to-dribble t)
The last line was an attempt at permanently disabling the ESS log buffer.
Otherwise said, I want to rely on epoll (or similar) to write asynchronous network code that looks like regular code that is without relying on callbacks.
The code must look like synchronous code but unlike synchronous code instead of blocking to wait for network io, it must suspend the current coroutine and restart it when the file descriptor is ready.
My initial thought to achieve that was relying on generators and yield. But this was a mistake that was partly mis-guided by the fact that python used to abuse yield from.
Anyway, guile fibers was a great insipiration and I adapted it to chez scheme.
Here is an example server code:
(define (handler request port)
(values 200 #f (http-get "https://httpbin.davecheney.com/ip")))
(untangle (lambda ()
(run-server "127.0.0.1" 8888)))
The handler returns its IP according the httpbin service. The code look synchronous with the help of call/cc actually call/1cc.
untangle will initiate the event loop with a lambda passed as argument!
Here is the definition of run-server:
(define (run-server ip port handler)
(log 'info "HTTP server running at ~a:~a" ip port)
(let* ((sock (socket 'inet 'stream 'ipv4)))
(socket:setsockopt sock 1 2 1) ;; re-use address
(socket:bind sock (make-address ip port))
(socket:listen sock 1024)
(let loop ()
(let ((client (accept sock)))
(let ((port (fd->port client)))
(spawn (lambda () (run-once handler port)))
(loop))))))
As you can see there is no callback. The only thing that is somewhat different from simple synchronous webserver is the spawn procedure that will handle the request in its own coroutine. In particular accept is asynchronous.
run-once will just pass the scheme request to handler and take its 3 values to build the response. Not very interesting. The part that looks synchronous, but is actually asynchronous is http-get above.
I will only explain, how accept works, given http-get requires to introduce custom binary ports, but suffice to say it is the same behavior...
(define (accept fd)
(let ((out (socket:%accept fd 0 0)))
(if (= out -1)
(let ((code (socket:errno)))
(if (= code EWOULDBLOCK)
(begin
(abort-to-prompt fd 'read)
(accept fd))
(error 'accept (socket:strerror code))))
out)))
As you can see it calls a procedure abort-to-prompt that we could call simply pause that will "stop" the coroutine and call the prompt handler.
abort-to-prompt works in cooperation with call-with-prompt.
Since chez scheme doesn't have prompts I emulate it using two one shot continuations call/1cc
(define %prompt #f)
(define %abort (list 'abort))
(define (call-with-prompt thunk handler)
(call-with-values (lambda ()
(call/1cc
(lambda (k)
(set! %prompt k)
(thunk))))
(lambda out
(cond
((and (pair? out) (eq? (car out) %abort))
(apply handler (cdr out)))
(else (apply values out))))))
(define (abort-to-prompt . args)
(call/1cc
(lambda (k)
(let ((prompt %prompt))
(set! %prompt #f)
(apply prompt (cons %abort (cons k args)))))))
call-with-prompt will initiate a continuation a set! global called %prompt which means there is single prompt for THUNK. If the continuation arguments OUT, the second lambda of call-with-values, starts with the unique object %abort it means the continuation was reached via abort-to-prompt. It will call the HANDLER with the abort-to-prompt continuation and any argument passed to call-with-prompt continuation parameter that is the (apply handler (cons k (cdr out))).
abort-to-promp will initiate a new continuation to be able to come back, after the code executes the prompt's continuation stored in %prompt.
The call-with-prompt is at the heart of the event-loop. Here is it, in two pieces:
(define (exec epoll thunk waiting)
(call-with-prompt
thunk
(lambda (k fd mode) ;; k is abort-to-prompt continuation that
;; will allow to restart the coroutine
;; add fd to the correct epoll set
(case mode
((write) (epoll-wait-write epoll fd))
((read) (epoll-wait-read epoll fd))
(else (error 'untangle "mode not supported" mode)))
(scheme:hash-table-set! waiting fd (make-event k mode)))))
(define (event-loop-run-once epoll waiting)
;; execute every callback waiting in queue,
;; call the above exec procedure
(let loop ()
(unless (null? %queue)
;; XXX: This is done like that because, exec might spawn
;; new coroutine, so we need to cut %queue right now.
(let ((head (car %queue))
(tail (cdr %queue)))
(set! %queue tail)
(exec epoll head waiting)
(loop))))
;; wait for ONE event
(let ((fd (epoll-wait-one epoll (inf))
(let ((event (scheme:hash-table-ref waiting fd)))
;; the event is / will be processed, no need to keep around
(scheme:hash-table-delete! waiting fd)
(case (event-mode event)
((write) (epoll-ctl epoll 2 fd (make-epoll-event-out fd)))
((read) (epoll-ctl epoll 2 fd (make-epoll-event-in fd))))
;; here it will schedule the event continuation that is the
;; abort-to-prompt continuation that will be executed by the
;; next call the above event loop event-loop-run-once
(spawn (event-continuation event))))))
I think that is all.
If you are using chez-scheme, there is chez-a-sync. It uses POSIX poll rather than epoll (epoll is linux specific). guile-a-sync2 is also available for guile-2.2/3.0.
I'm relatively new to Common Lisp (SBCL) and Hunchentoot (using Quicklisp). Can someone tell me how I can get this to work? I'm trying to wrap an Hunchentoot server and some paths in a function as a unit. When I run this, only Hunchentoot's index page is available, the paths /a and /b aren't.
(defun app0 (port)
(let ((*dispatch-table* nil) (server (make-instance 'hunchentoot:acceptor :port port)))
(push (hunchentoot:create-prefix-dispatcher "/a" (lambda () "a")) *dispatch-table*)
(push (hunchentoot:create-prefix-dispatcher "/b" (lambda () "b")) *dispatch-table*)
(hunchentoot:start server) server))
There are multiple problems, as far as I can see. First, request handling via *dispatch-table* requires, that the acceptor is of type easy-acceptor, i.e., you will have to
(make-instance 'easy-acceptor ...)
The documentation has the details.
The second problem is, that you rebind the *dispatch-table* during the set-up code, and push new values into this binding. Since the binding is reverted after the let is finished (and since hunchentoot:start works asynchronously), your entries in the *dispatch-table* are effectively lost, when the server is running. Try
(push (hunchentoot:create-prefix-dispatcher "/a" (lambda () "a")) *dispatch-table*)
(push (hunchentoot:create-prefix-dispatcher "/b" (lambda () "b")) *dispatch-table*)
at the top-level (or do something like that in a dedicated set-up function). If you don't like the global *dispatch-table* approach, you can also create a subclass of acceptor, and override acceptor-dispatch-request (and thus, implement any kind of dispatch you like).
Just as a side-note: you do not prefix *dispatch-table*, while you prefix virtually any other symbol from hunchentoot's package. Is this just a copy/paste mistake, or is this also the case in your actual code? If you do not :use the hunchentoot package in wherever package your code happens to live, then you'd also have to qualify the dispatch table as hunchentoot:*dispatch-table*.
Edit (to address the question in the comment section) There is an example in the hunchentoot documentation, which seems to do exactly what you want to do:
(defclass vhost (tbnl:acceptor)
((dispatch-table
:initform '()
:accessor dispatch-table
:documentation "List of dispatch functions"))
(:default-initargs
:address "127.0.0.1"))
(defmethod tbnl:acceptor-dispatch-request ((vhost vhost) request)
(mapc (lambda (dispatcher)
(let ((handler (funcall dispatcher request)))
(when handler
(return-from tbnl:acceptor-dispatch-request (funcall handler)))))
(dispatch-table vhost))
(call-next-method))
(defvar vhost1 (make-instance 'vhost :port 50001))
(defvar vhost2 (make-instance 'vhost :port 50002))
(push
(tbnl:create-prefix-dispatcher "/foo" 'foo1)
(dispatch-table vhost1))
(push
(tbnl:create-prefix-dispatcher "/foo" 'foo2)
(dispatch-table vhost2))
(defun foo1 () "Hello")
(defun foo2 () "Goodbye")
(tbnl:start vhost1)
(tbnl:start vhost2)
(comments present in the documentation removed for brevity). The tbnl is a predefined nickname for package hunchentoot. You can use both interchangeably, though I would recommend, that you pick one and stick to it. Mixing both might generate confusion.
I have some function with loop, each iteration it reads input, on "0" it calls function "exit-and-save", in that function it saves some database and after that I need it to exit the program? What is the command for that? If I use return-from... it just returns from function, if I use return - error, if I use quit, it disconnects from slime. I'm new in common lisp...
(loop for i from 0 to 10
do (progn (format t "~&cycle ~d" i)
(when (> i 5)
(return nil))))
First of all I cannot verify that slime disconnects using (quit), at least not using sbcl at Ubuntu.
CL-USER> (quit)
; Evaluation aborted on NIL.
CL-USER>
"still able to input here"
But if you got some freakish version of slime you could take advantage of the condition system:
(define-condition end-program-condition (simple-error) ())
(defun some-func ()
(error 'end-program-condition))
(defun main-function ()
(handler-case (some-func)
(end-program-condition () "THE END")))
CL-USER> (main-function)
"THE END"
CL-USER> "still can input here"
"still can input here"
It depends on your common lisp implementation, but if using sbcl for example, you could call sb-ext:exit.
Source: http://www.sbcl.org/manual/#Exit
It seems there is NO ANSI standard way to execute an external program and get its output as the following SBCL special code does:
(defmacro with-input-from-program ((stream program program-args environment)
&body body)
"Creates an new process of the specified by PROGRAM using
PROGRAM-ARGS as a list of the arguments to the program. Binds the
stream variable to an input stream from which the output of the
process can be read and executes body as an implicit progn."
#+sbcl
(let ((process (gensym)))
`(let ((,process (sb-ext::run-program ,program
,program-args
:output :stream
:environment ,environment
:wait nil)))
(when ,process
(unwind-protect
(let ((,stream (sb-ext:process-output ,process)))
,#body)
(sb-ext:process-wait ,process)
(sb-ext:process-close ,process))))))
The following CCL code reports "ERROR: value # is not of the expected type (AND CCL::BINARY-STREAM INPUT-STREAM)"
#+clozure
(let ((process (gensym)))
`(let ((,process (ccl:run-program "/bin/sh" (list "-c" (namestring ,program))
:input nil :output :stream :error :stream
:wait nil)))
(when ,process
(unwind-protect
(let ((,stream (ccl::external-process-output-stream ,process)))
,#body)
;(ccl:process-wait (ccl:process-whostate ,process) nil)
(close (ccl::external-process-output-stream ,process))
(close (ccl::external-process-error-stream ,process))))))
I know little CCL. I want to know how i can modify this code to support CCL ?
Any suggestion is appreciated !
Apparently trivial-shell:shell-command doesn't allow exactly what you want (it executes the external command synchronously and returns the whole output).
You could look into CCL's run-program. See:
run-program;
Does there exist standard way to run external program in Common Lisp? (this is a question that is similar to your question);
external-program (suggested in one of the answers in the question above) is supported by Quicklisp and it seems to have better support for executing external programs.
You should use trivial-shell.
Trivial shell is a simple platform independent interface to the underlying Operating System.