Program does not finish when two nodes are used - mpi

When I run my program in one computer, with 4 processes, it will terminate almost instantly. When I try to run it in a cluster of two computers (the cluster was checked and was OK), it just won't finish!
I had done a run a long time ago in the cluster and I remember that it was slower than in the run on the one pc, but it would terminate!
Here is my run.sh:
#!/bin/bash
start=100
end=100
for ((i = $start; i <= $end; ++i )) ;
do
mpiexec -f machinefile -n 4 ./test ../../l_matrices/Lmat_755.mtx 1 755 755 $i $i 2 2 0 0
done
and I did check that two processes are spawned in every node.
Here is my machinefile:
hostname1.gr:2
hostname2.gr:2
What is happening?

(Presuming that the script is correct and the start and end variable values are intentional, this will not do anything meaningful as mpiexec will execute two copies of the same file with the same arguments once)
Check the paths - You have used relative paths, that can lead to problems as the execution happens in the default directory after login, typically, your home directory.

Related

Kill all R processes that hang for longer than a minute

I use crontask to regularly run Rscript. Unfortunately, I need to do this on a small instance of aws and the process may hang, building more and more processes on top of each other until the whole system is lagging.
I would like to write a crontask to kill all R processes lasting longer than one minute. I found another answer on Stack Overflow that I've adapted that I think would solve the problem. I came up with;
if [[ "$(uname)" = "Linux" ]];then killall --older-than 1m "/usr/lib/R/bin/exec/R --slave --no-restore --file=/home/ubuntu/script.R";fi
I copied the task directly from htop, but it does not work as I expect. I get the No such file or directory error but I've checked it a few times.
I need to kill all R processes that have lasted longer than a minute. How can I do this?
You may want to avoid killing processes from another user and try SIGKILL (kill -9) after SIGTERM (kill -15). Here is a script you could execute every minute with a CRON job:
#!/bin/bash
PROCESS="R"
MAXTIME=`date -d '00:01:00' +'%s'`
function killpids()
{
PIDS=`pgrep -u "${USER}" -x "${PROCESS}"`
# Loop over all matching PIDs
for pid in ${PIDS}; do
# Retrieve duration of the process
TIME=`ps -o time:1= -p "${pid}" |
egrep -o "[0-9]{0,2}:?[0-9]{0,2}:[0-9]{2}$"`
# Convert TIME to timestamp
TTIME=`date -d "${TIME}" +'%s'`
# Check if the process should be killed
if [ "${TTIME}" -gt "${MAXTIME}" ]; then
kill ${1} "${pid}"
fi
done
}
# Leave a chance to kill processes properly (SIGTERM)
killpids "-15"
sleep 5
# Now kill remaining processes (SIGKILL)
killpids "-9"
Why imply an additional process every minute with cron?
Would it not be easier to start R with timeout from coreutils, the processes will then be killed automatically after the time you chose.
timeout [option] duration command [arg]…
I think the best option is to do this with R itself. I am no expert, but it seems the future package will allow executing a function in a separate thread. You could run the actual task in a separate thread, and in the main thread sleep for 60 seconds and then stop().
Previous Update
user1747036's answer which recommends timeout is a better alternative.
My original answer
This question is more appropriate for superuser, but here are a few things wrong with
if [[ "$(uname)" = "Linux" ]];then
killall --older-than 1m \
"/usr/lib/R/bin/exec/R --slave --no-restore --file=/home/ubuntu/script.R";
fi
The name argument is either the name of image or path to it. You have included parameters to it as well
If -s signal is not specified killall sends SIGTERM which your process may ignore. Are you able to kill a long running script with this on the command line? You may need SIGKILL / -9
More at http://linux.die.net/man/1/killall

Multithreaded program only runs on a single processor after compiling, how do I troubleshoot?

I am trying to run a compiled program that is supposed to be running on multiple processors. But with the same data, sometimes this program runs in parallel and sometimes it won't (with the identical PBS script file!). I am suspecting that something is wrong with some of the compute nodes that won't let it run on parallel (I don't get to choose the compute node I want). How can I troubleshoot if this is a bug in the program or it is problem with the compute node?
As per the sys admin's adivce, I am using ulimit -s 100000, but this don't change anything. Also, this program is not an mpi program (runs only on a single node, with multiple processors).
The code that I run is as follows:
quorum_error_correct_reads -q 68 \
--contaminant=/data004/software/GIF/packages/masurca/2.3.0rc1/bin/../share/adapter.jf \
-m 1 -s 1 -g 1 -a 3 --thread=32 -w 10 -e 3 \
quorum_mer_db.jf aa.renamed.fastq ab.renamed.fastq ac.renamed.fastq ad.renamed.fastq ae.renamed.fastq af.renamed.fastq ag.renamed.fastq \
--no-discard -o pe.cor --verbose
Thanks for any advice you can offer. I will greatly appreciate your help!
PS: I don't have sudo access.
EDIT: I know it is supposed to be using multiple processors because, when I SSH into the node and do top -c I can see (above command) sometimes running like 3200 % CPU (all the time) and sometimes only 100 % CPU all the time. This is the only step involved and there are no other sub-process within this program. Also, I am using HPC, where I submit the job to a compute node, each with 32 procs, 512GB RAM.

KSH: Block two process from running at the same time

I have two process that running at random time and I want to force them not to ever run at the same time due to reader-writer problem. My thought is whenever a process run, I create a LOCK file, both process has a logic of checking whether a LOCK exist. If LOCK is existed, then sleep for bit and wake up and check it again. Here is a small piece of it
if [[ ! -f ${INPUT_DIR}/LOCK ]]
then
#Create LOCK file
cat /dev/null > ${INPUT_DIR}/LOCK
retcode=${?}
if [[ ${retcode} -ne 0 ]]
then
echo `date` "Error in creating LOCK file by processA.sh - Error code: " ${retcode} >> ${CORE_LOG}
exit
fi
echo `date` "LOCK turns on by processA.sh" >> ${CORE_LOG}
...
rm ${INPUT_DIR}/LOCK
fi
Howver this does not QUITE stop the two process from running at the same time. There are rare time when both process would get pass the first IF checking if the log exist (if both process get invoke at the same time and there was no LOCK exist, very likely that it will get pass the first IF statement), both try to create a LOCK file, since cat /dev/null > ${INPUT_DIR}/LOCK will not generate an error, even when LOCK is already exist. Is there a solution to this?
For the main versions of unix, the preferred solution is to use a lock directory, I would assume this is true for linux, but I haven't had to test it recently.
Creating a directory is an atomic process, and only 1 of the processes will succeed, assuming that you are making a static name like /bin/mkdir -p /tmp/myProjWorkSpace/LOCK. If you need to have information embedded in your lock, then you need a file, and you need sepqrate subdirs per process, possibly add the processID (.$$) to the dir name.
I hope this helps.

write a background process to check process is still active

In UNIX, I have a utility, say 'Test_Ex', a binary file. How can I write a job or a shell script(as a cron job) running always in the background which keeps checking if 'Test_Ex' is still running every 5 seconds(and probably hide this job). If it is running, do nothing. If not, delete a directory at the specified path.
Try this script:
pgrep Test_Ex > /dev/null || rm -r dir
If you don't have pgrep, use
ps -e -ocomm | grep Test_Ex || ...
instead.
Utilities like upstart, originally part of the Ubuntu linux distribution I believe, are good for monitoring running tasks.
The best way to do this is to not do it. If you want to know if Test_Ex is still running, then start it from a script that looks something like:
#!/bin/sh
Test_Ex
logger "Test_Ex died"
rm /p/a/t/h
or
#!/bin/sh
while ! Test_ex
do
logger "Test_Ex terminated unsuccesfully, restarting in 5 seconds"
sleep 5
done
Querying ps regularly is a bad idea, and trying to monitor it from cron is a horrible, horrible idea. There seems to be some comfort in the idea that crond will always be running, but you can no more rely on that than you can rely on the wrapper script staying alive; either one can be killed at any time. Waking up every 10 seconds to query ps is just a waste of resources.

Wait in console [duplicate]

When writing a batch file to automate something on a Windows box, I've needed to pause its execution for several seconds (usually in a test/wait loop, waiting for a process to start). At the time, the best solution I could find uses ping (I kid you not) to achieve the desired effect. I've found a better write-up of it here, which describes a callable "wait.bat", implemented as follows:
#ping 127.0.0.1 -n 2 -w 1000 > nul
#ping 127.0.0.1 -n %1% -w 1000> nul
You can then include calls to wait.bat in your own batch file, passing in the number of seconds to sleep.
Apparently the Windows 2003 Resource Kit provides a Unix-like sleep command (at last!). In the meantime, for those of us still using Windows XP, Windows 2000 or (sadly) Windows NT, is there a better way?
I modified the sleep.py script in the accepted answer, so that it defaults to one second if no arguments are passed on the command line:
import time, sys
time.sleep(float(sys.argv[1]) if len(sys.argv) > 1 else 1)
The timeout command is available from Windows Vista onwards:
c:\> timeout /?
TIMEOUT [/T] timeout [/NOBREAK]
Description:
This utility accepts a timeout parameter to wait for the specified
time period (in seconds) or until any key is pressed. It also
accepts a parameter to ignore the key press.
Parameter List:
/T timeout Specifies the number of seconds to wait.
Valid range is -1 to 99999 seconds.
/NOBREAK Ignore key presses and wait specified time.
/? Displays this help message.
NOTE: A timeout value of -1 means to wait indefinitely for a key press.
Examples:
TIMEOUT /?
TIMEOUT /T 10
TIMEOUT /T 300 /NOBREAK
TIMEOUT /T -1
Note: It does not work with input redirection - trivial example:
C:\>echo 1 | timeout /t 1 /nobreak
ERROR: Input redirection is not supported, exiting the process immediately.
Using the ping method as outlined is how I do it when I can't (or don't want to) add more executables or install any other software.
You should be pinging something that isn't there, and using the -w flag so that it fails after that amount of time, not pinging something that is there (like localhost) -n times. This allows you to handle time less than a second, and I think it's slightly more accurate.
e.g.
(test that 1.1.1.1 isn't taken)
ECHO Waiting 15 seconds
PING 1.1.1.1 -n 1 -w 15000 > NUL
or
PING -n 15 -w 1000 127.1 >NUL
UPDATE
The timeout command, available from Windows Vista and onwards should be the command used, as described in another answer to this question. What follows here is an old answer.
Old answer
If you have Python installed, or don't mind installing it (it has other uses too :), just create the following sleep.py script and add it somewhere in your PATH:
import time, sys
time.sleep(float(sys.argv[1]))
It will allow sub-second pauses (for example, 1.5 sec, 0.1, etc.), should you have such a need. If you want to call it as sleep rather than sleep.py, then you can add the .PY extension to your PATHEXT environment variable. On Windows XP, you can edit it in:
My Computer → Properties (menu) → Advanced (tab) → Environment Variables (button) → System variables (frame)
SLEEP.exe is included in most Resource Kits e.g. The Windows Server 2003 Resource Kit which can be installed on Windows XP too.
Usage: sleep time-to-sleep-in-seconds
sleep [-m] time-to-sleep-in-milliseconds
sleep [-c] commited-memory ratio (1%-100%)
I disagree with the answers I found here.
I use the following method entirely based on Windows XP capabilities to do a delay in a batch file:
DELAY.BAT:
#ECHO OFF
REM DELAY seconds
REM GET ENDING SECOND
FOR /F "TOKENS=1-3 DELIMS=:." %%A IN ("%TIME%") DO SET /A H=%%A, M=1%%B%%100, S=1%%C%%100, ENDING=(H*60+M)*60+S+%1
REM WAIT FOR SUCH A SECOND
:WAIT
FOR /F "TOKENS=1-3 DELIMS=:." %%A IN ("%TIME%") DO SET /A H=%%A, M=1%%B%%100, S=1%%C%%100, CURRENT=(H*60+M)*60+S
IF %CURRENT% LSS %ENDING% GOTO WAIT
You may also insert the day in the calculation so the method also works when the delay interval pass over midnight.
I faced a similar problem, but I just knocked up a very short C++ console application to do the same thing. Just run MySleep.exe 1000 - perhaps easier than downloading/installing the whole resource kit.
#include <tchar.h>
#include <stdio.h>
#include "Windows.h"
int _tmain(int argc, _TCHAR* argv[])
{
if (argc == 2)
{
_tprintf(_T("Sleeping for %s ms\n"), argv[1]);
Sleep(_tstoi(argv[1]));
}
else
{
_tprintf(_T("Wrong number of arguments.\n"));
}
return 0;
}
You can use ping:
ping 127.0.0.1 -n 11 -w 1000 >nul: 2>nul:
It will wait 10 seconds.
The reason you have to use 11 is because the first ping goes out immediately, not after one second. The number should always be one more than the number of seconds you want to wait.
Keep in mind that the purpose of the -w is not to control how often packets are sent, it's to ensure that you wait no more than some time in the event that there are network problems. There are unlikely to be problems if you're pinging 127.0.0.1 so this is probably moot.
The ping command on its own will normally send one packet per second. This is not actually documented in the Windows docs but it appears to follow the same rules as the Linux version (where it is documented).
Over at Server Fault, a similar question was asked, and the solution there was:
choice /d y /t 5 > nul
You could use the Windows cscript WSH layer and this wait.js JavaScript file:
if (WScript.Arguments.Count() == 1)
WScript.Sleep(WScript.Arguments(0)*1000);
else
WScript.Echo("Usage: cscript wait.js seconds");
Depending on your compatibility needs, either use ping:
ping -n <numberofseconds+1> localhost >nul 2>&1
e.g. to wait 5 seconds, use
ping -n 6 localhost >nul 2>&1
or on Windows 7 or later use timeout:
timeout 6 >nul
There is a better way to sleep using ping. You'll want to ping an address that does not exist, so you can specify a timeout with millisecond precision. Luckily, such an address is defined in a standard (RFC 3330), and it is 192.0.2.x. This is not made-up, it really is an address with the sole purpose of not-existing. To be clear, this applies even in local networks.
192.0.2.0/24 - This block is assigned as "TEST-NET" for use in
documentation and example code. It is often used in conjunction with
domain names example.com or example.net in vendor and protocol
documentation. Addresses within this block should not appear on the
public Internet.
To sleep for 123 milliseconds, use ping 192.0.2.1 -n 1 -w 123 >nul
Update: As per the comments, there is also 127.255.255.255.
If you've got PowerShell on your system, you can just execute this command:
powershell -command "Start-Sleep -s 1"
Edit: from my answer on a similar thread, people raised an issue where the amount of time powershell takes to start is significant compared to how long you're trying to wait for. If the accuracy of the wait time is important (ie a second or two extra delay is not acceptable), you can use this approach:
powershell -command "$sleepUntil = [DateTime]::Parse('%date% %time%').AddSeconds(5); $sleepDuration = $sleepUntil.Subtract((get-date)).TotalMilliseconds; start-sleep -m $sleepDuration"
This takes the time when the windows command was issued, and the powershell script sleeps until 5 seconds after that time. So as long as powershell takes less time to start than your sleep duration, this approach will work (it's around 600ms on my machine).
timeout /t <seconds> <options>
For example, to make the script perform a non-uninterruptible 2-second wait:
timeout /t 2 /nobreak >NUL
Which means the script will wait 2 seconds before continuing.
By default, a keystroke will interrupt the timeout, so use the /nobreak switch if you don't want the user to be able to interrupt (cancel) the wait. Furthermore, the timeout will provide per-second notifications to notify the user how long is left to wait; this can be removed by piping the command to NUL.
edit: As #martineau points out in the comments, the timeout command is only available on Windows 7 and above. Furthermore, the ping command uses less processor time than timeout. I still believe in using timeout where possible, though, as it is more readable than the ping 'hack'. Read more here.
Just put this in your batch file where you want the wait.
#ping 127.0.0.1 -n 11 -w 1000 > null
In Notepad, write:
#echo off
set /a WAITTIME=%1+1
PING 127.0.0.1 -n %WAITTIME% > nul
goto:eof
Now save as wait.bat in the folder C:\WINDOWS\System32,
then whenever you want to wait, use:
CALL WAIT.bat <whole number of seconds without quotes>
The Resource Kit has always included this. At least since Windows 2000.
Also, the Cygwin package has a sleep - plop that into your PATH and include the cygwin.dll (or whatever it's called) and way to go!
The usage of ping is good, as long as you just want to "wait for a bit". This since you are dependent on other functions underneath, like your network working and the fact that there is nothing answering on 127.0.0.1. ;-) Maybe it is not very likely it fails, but it is not impossible...
If you want to be sure that you are waiting exactly the specified time, you should use the sleep functionality (which also have the advantage that it doesn't use CPU power or wait for a network to become ready).
To find an already made executable for sleep is the most convenient way. Just drop it into your Windows folder or any other part of your standard path and it is always available.
Otherwise, if you have a compiling environment you can easily make one yourself.
The Sleep function is available in kernel32.dll, so you just need to use that one. :-)
For VB / VBA declare the following in the beginning of your source to declare a sleep function:
private Declare Sub Sleep Lib "kernel32" Alias "Sleep" (byval dwMilliseconds as Long)
For C#:
[DllImport("kernel32.dll")]
static extern void Sleep(uint dwMilliseconds);
You'll find here more about this functionality (available since Windows 2000) in Sleep function (MSDN).
In standard C, sleep() is included in the standard library and in Microsoft's Visual Studio C the function is named Sleep(), if memory serves me. ;-) Those two takes the argument in seconds, not in milliseconds as the two previous declarations.
I like Aacini's response. I added to it to handle the day and also enable it to handle centiseconds (%TIME% outputs H:MM:SS.CC):
:delay
SET DELAYINPUT=%1
SET /A DAYS=DELAYINPUT/8640000
SET /A DELAYINPUT=DELAYINPUT-(DAYS*864000)
::Get ending centisecond (10 milliseconds)
FOR /F "tokens=1-4 delims=:." %%A IN ("%TIME%") DO SET /A H=%%A, M=1%%B%%100, S=1%%C%%100, X=1%%D%%100, ENDING=((H*60+M)*60+S)*100+X+DELAYINPUT
SET /A DAYS=DAYS+ENDING/8640000
SET /A ENDING=ENDING-(DAYS*864000)
::Wait for such a centisecond
:delay_wait
FOR /F "tokens=1-4 delims=:." %%A IN ("%TIME%") DO SET /A H=%%A, M=1%%B%%100, S=1%%C%%100, X=1%%D%%100, CURRENT=((H*60+M)*60+S)*100+X
IF DEFINED LASTCURRENT IF %CURRENT% LSS %LASTCURRENT% SET /A DAYS=DAYS-1
SET LASTCURRENT=%CURRENT%
IF %CURRENT% LSS %ENDING% GOTO delay_wait
IF %DAYS% GTR 0 GOTO delay_wait
GOTO :EOF
I have been using this C# sleep program. It might be more convenient for you if C# is your preferred language:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
namespace sleep
{
class Program
{
static void Main(string[] args)
{
if (args.Length == 1)
{
double time = Double.Parse(args[0]);
Thread.Sleep((int)(time*1000));
}
else
{
Console.WriteLine("Usage: sleep <seconds>\nExample: sleep 10");
}
}
}
}
Even more lightweight than the Python solution is a Perl
one-liner.
To sleep for seven seconds put this in the BAT script:
perl -e "sleep 7"
This solution only provides a resolution of one second.
If you need higher resolution then use the Time::HiRes
module from CPAN. It provides usleep() which sleeps in
microseconds and nanosleep() which sleeps in nanoseconds
(both functions takes only integer arguments). See the
Stack Overflow question How do I sleep for a millisecond in Perl? for further details.
I have used ActivePerl for many years. It is very easy to
install.
Or command line Python, for example, for 6 and a half seconds:
python -c "import time;time.sleep(6.5)"
The best solution that should work on all Windows versions after Windows 2000 would be:
timeout numbersofseconds /nobreak > nul
There are lots of ways to accomplish a 'sleep' in cmd/batch:
My favourite one:
TIMEOUT /NOBREAK 5 >NUL 2>NUL
This will stop the console for 5 seconds, without any output.
Most used:
ping localhost -n 5 >NUL 2>NUL
This will try to make a connection to localhost 5 times. Since it is hosted on your computer, it will always reach the host, so every second it will try the new every second. The -n flag indicates how many times the script will try the connection. In this case is 5, so it will last 5 seconds.
Variants of the last one:
ping 1.1.1.1 -n 5 >nul
In this script there are some differences comparing it with the last one. This will not try to call localhost. Instead, it will try to connect to 1.1.1.1, a very fast website. The action will last 5 seconds only if you have an active internet connection. Else it will last approximately 15 to complete the action. I do not recommend using this method.
ping 127.0.0.1 -n 5 >nul
This is exactly the same as example 2 (most used). Also, you can also use:
ping [::1] -n 5 >nul
This instead, uses IPv6's localhost version.
There are lots of methods to perform this action. However, I prefer method 1 for Windows Vista and later versions and the most used method (method 2) for earlier versions of the OS.
The pathping.exe can sleep less than second.
#echo off
setlocal EnableDelayedExpansion
echo !TIME! & pathping localhost -n -q 1 -p %~1 2>&1 > nul & echo !TIME!
.
> sleep 10
17:01:33,57
17:01:33,60
> sleep 20
17:03:56,54
17:03:56,58
> sleep 50
17:04:30,80
17:04:30,87
> sleep 100
17:07:06,12
17:07:06,25
> sleep 200
17:07:08,42
17:07:08,64
> sleep 500
17:07:11,05
17:07:11,57
> sleep 800
17:07:18,98
17:07:19,81
> sleep 1000
17:07:22,61
17:07:23,62
> sleep 1500
17:07:27,55
17:07:29,06
I am impressed with this one:
http://www.computerhope.com/batch.htm#02
choice /n /c y /d y /t 5 > NUL
Technically, you're telling the choice command to accept only y. It defaults to y, to do so in 5 seconds, to draw no prompt, and to dump anything it does say to NUL (like null terminal on Linux).
You can also use a .vbs file to do specific timeouts:
The code below creates the .vbs file. Put this near the top of you rbatch code:
echo WScript.sleep WScript.Arguments(0) >"%cd%\sleeper.vbs"
The code below then opens the .vbs and specifies how long to wait for:
start /WAIT "" "%cd%\sleeper.vbs" "1000"
In the above code, the "1000" is the value of time delay to be sent to the .vbs file in milliseconds, for example, 1000 ms = 1 s. You can alter this part to be however long you want.
The code below deletes the .vbs file after you are done with it. Put this at the end of your batch file:
del /f /q "%cd%\sleeper.vbs"
And here is the code all together so it's easy to copy:
echo WScript.sleep WScript.Arguments(0) >"%cd%\sleeper.vbs"
start /WAIT "" "%cd%\sleeper.vbs" "1000"
del /f /q "%cd%\sleeper.vbs"
Just for fun, if you have Node.js installed, you can use
node -e 'setTimeout(a => a, 5000)'
to sleep for 5 seconds. It works on a Mac with Node v12.14.0.
You can get fancy by putting the PAUSE message in the title bar:
#ECHO off
SET TITLETEXT=Sleep
TITLE %TITLETEXT%
CALL :sleep 5
GOTO :END
:: Function Section
:sleep ARG
ECHO Pausing...
FOR /l %%a in (%~1,-1,1) DO (TITLE Script %TITLETEXT% -- time left^
%%as&PING.exe -n 2 -w 1000 127.1>NUL)
EXIT /B 0
:: End of script
:END
pause
::this is EOF
This was tested on Windows XP SP3 and Windows 7 and uses CScript. I put in some safe guards to avoid del "" prompting. (/q would be dangerous)
Wait one second:
sleepOrDelayExecution 1000
Wait 500 ms and then run stuff after:
sleepOrDelayExecution 500 dir \ /s
sleepOrDelayExecution.bat:
#echo off
if "%1" == "" goto end
if NOT %1 GTR 0 goto end
setlocal
set sleepfn="%temp%\sleep%random%.vbs"
echo WScript.Sleep(%1) >%sleepfn%
if NOT %sleepfn% == "" if NOT EXIST %sleepfn% goto end
cscript %sleepfn% >nul
if NOT %sleepfn% == "" if EXIST %sleepfn% del %sleepfn%
for /f "usebackq tokens=1*" %%i in (`echo %*`) DO # set params=%%j
%params%
:end
Since others are suggesting 3rd party programs (Python, Perl, custom app, etc), another option is GNU CoreUtils for Windows available at http://gnuwin32.sourceforge.net/packages/coreutils.htm.
2 options for deployment:
Install full package (which will include the full suite of CoreUtils, dependencies, documentation, etc).
Install only the 'sleep.exe' binary and necessary dependencies (use depends.exe to get dependencies).
One benefit of deploying CoreUtils is that you'll additionally get a host of other programs that are helpful for scripting (Windows batch leaves a lot to be desired).

Resources