Snakemake: R script fails (almost) immediately - r

Once more I'm encountering an error in my snakemake workflow that doesn't make any sense to me.
This is the error I get:
[Thu Jan 25 10:47:00 2018] Building DAG of jobs...
[Thu Jan 25 10:47:01 2018] Provided cores: 24
[Thu Jan 25 10:47:01 2018] Rules claiming more threads will be scaled down.
[Thu Jan 25 10:47:01 2018] Job counts:
[Thu Jan 25 10:47:01 2018] count jobs
[Thu Jan 25 10:47:01 2018] 1 merging_seurat
[Thu Jan 25 10:47:01 2018] 1
[Thu Jan 25 10:47:01 2018] Job 0: --- Merging samples using seurat.
Error in setClass("Snakemake", slots = c(input = "list", output = "list", :
unused argument(s) (slots = c(input = "list", output = "list", params = "list", wildcards = "list", threads = "numeric", log = "list", resources = "list", config = "list", rule = "character"))
Execution halted
[Thu Jan 25 10:47:02 2018] Error in rule merging_seurat:
[Thu Jan 25 10:47:02 2018] jobid: 0
[Thu Jan 25 10:47:02 2018] output: merging_seurat/12_top10_heatmap_all_wilcox.pdf, merging_seurat/13_top10_heatmap_all_roc.pdf, merging_seurat/merging_seurat.RData
[Thu Jan 25 10:47:02 2018] RuleException:
[Thu Jan 25 10:47:02 2018] CalledProcessError in line 372 of .../snakemake_pipeline/Snakefile:
[Thu Jan 25 10:47:02 2018] Command ' set -euo pipefail; Rscript .../snakemake_pipeline/scripts/.snakemake.jv8ijpiw.merging_seurat.R ' returned non-zero exit status 1
[Thu Jan 25 10:47:02 2018] File ".../snakemake_pipeline/Snakefile", line 372, in __rule_merging_seurat
[Thu Jan 25 10:47:02 2018] File ".../tools/anaconda3/envs/Seurat/lib/python3.5/concurrent/futures/thread.py", line 55, in run
[Thu Jan 25 10:47:02 2018] Will exit after finishing currently running jobs.
[Thu Jan 25 10:47:02 2018] Exiting because a job execution failed. Look above for error message
[Thu Jan 25 10:47:02 2018] Complete log: .../snakemake_6/.snakemake/log/2018-01-25T104700.498155.snakemake.log
This is the rule in question:
rule merging_seurat:
input: expand("{sample}/molecule_count/counts_wide.tsv", sample=config["samples"]),
output: "merging_seurat/12_top10_heatmap_all_wilcox.pdf",
"merging_seurat/13_top10_heatmap_all_roc.pdf",
"merging_seurat/merging_seurat.RData"
message: "--- Merging samples using seurat."
script: "scripts/merging_seurat.R"
Here we have the top of the R script that is still executed:
sink('merging_seurat/output.txt')
print(installed.packages())
print(sessionInfo())
sink()
And these are the following lines in the R script that are not executed anymore:
library('Seurat')
library('dplyr')
library('org.Hs.eg.db')
Because I logged the packages that are available, I know that the three packages that should be loaded are also installed so it shouldn't fail because of that. In fact, commenting these lines out doesn't change anything, the script still breaks and the log message that should come right after loading the packages is not written to the log file.
Finally, this is the command that I use to run snakemake:
snakemake --use-conda \
--latency-wait 90 \
--rerun-incomplete \
--keep-going \
--timestamp \
--cluster-config SGE.json \
--cluster "qsub -cwd -N {cluster.name} -l h_vmem={cluster.h_vmem},h_stack=256M -o {cluster.stdout}{cluster.name}.o -e {cluster.stderr}{cluster.name}.e -m {cluster.mailtype} -M {cluster.mailuser}" \
-j 8 \
--directory .../snakemake_6
Does anybody have a clue what the error message could mean? unused argument of the snakemake R object is strange because I'm calling multiple parameters later on in the script.
What is even more strange is that I have another R script that works with different packages but has very similar first lines runs fine. I remember that in the beginning I had the same problem with that script (same error message) but don't remember how I solved it.
Any help is well appreciated.

Related

How to Extract logs between 2 timestamps in Unix

I need to extract the logs between two timestamps from a file in Unix. I basically kind of need those logs to be copied and output in a different file so that I can copy them.
Is there an efficient way to do this? The log format looks like this - The timestamp is in a separate line from the actual logs.
Tue 21 Apr 14:00:00 GMT 2020
{"items":[{"cpu.load": "0.94","total.memory": "6039.798 MB","free.memory": "4367.152 MB","used.memory": "1672.646 MB","total.physical.system.memory": "16.656 GB","total.free.physical.system.memory": "3860.197 MB","total.used.physical.system.memory": "12.796 GB","number.of.cpus": "8"}]}
Tue 21 Apr 18:00:00 GMT 2020
{"items":[{"cpu.load": "0.76","total.memory": "6039.798 MB","free.memory": "4352.656 MB","used.memory": "1687.142 MB","total.physical.system.memory": "16.656 GB","total.free.physical.system.memory": "3858.203 MB","total.used.physical.system.memory": "12.798 GB","number.of.cpus": "8"}]}
I am doing this but it only prints out the timestamp and not the actual logs
cat file.txt | awk -F, '{ if ($1>"Fri 21 Aug 14:00:00 GMT 2020" && $1<"Sat 22 Aug 18:00:00 GMT 2020") print }'
Can someone advice.

Convert string Fri, 04 Aug 2017 13:07:44 into 04-08-2017 in unix

I would like to convert the date " Fri, 04 Aug 2017 13:07:44 GMT " into 04-08-2017 .
And code should recursively convert all string format into specified format.
You may use the -d option with date command.
Execute this in your command line to get the desired output.
date -d 'Fri, 04 Aug 2017 13:07:44 GMT' +'%d-%m-%Y'
Edit :
If you want to populate it in a script you may use something like this:
#!/bin/bash
input_date="Fri, 04 Aug 2017 13:07:44 GMT"
output_date=$(date -d "$input_date" +'%d-%m-%Y')
echo $output_date

openstack dashboard gives internal server error

when i try to open dashboard it gives internal server error. It was running fine earlier but suddenly this error occurred.
Here is my apache log :
[Fri Jun 23 16:43:50.321423 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] mod_wsgi (pid=31878): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Fri Jun 23 16:43:50.321468 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] Traceback (most recent call last):
[Fri Jun 23 16:43:50.321485 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 168, in __call__
[Fri Jun 23 16:43:50.321506 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] self.load_middleware()
[Fri Jun 23 16:43:50.321607 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 46, in load_middleware
[Fri Jun 23 16:43:50.321620 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] mw_instance = mw_class()
[Fri Jun 23 16:43:50.321625 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/dist-packages/django/middleware/locale.py", line 23, in __init__
[Fri Jun 23 16:43:50.321633 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] for url_pattern in get_resolver(None).url_patterns:
[Fri Jun 23 16:43:50.321637 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 372, in url_patterns
[Fri Jun 23 16:43:50.321644 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
[Fri Jun 23 16:43:50.321648 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 366, in urlconf_module
[Fri Jun 23 16:43:50.321654 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] self._urlconf_module = import_module(self.urlconf_name)
[Fri Jun 23 16:43:50.321658 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Fri Jun 23 16:43:50.321665 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] __import__(name)
[Fri Jun 23 16:43:50.321669 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/urls.py", line 35, in <module>
[Fri Jun 23 16:43:50.321676 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] url(r'^api/', include('openstack_dashboard.api.rest.urls')),
[Fri Jun 23 16:43:50.321680 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/dist-packages/django/conf/urls/__init__.py", line 28, in include
[Fri Jun 23 16:43:50.321687 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] urlconf_module = import_module(urlconf_module)
[Fri Jun 23 16:43:50.321691 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
[Fri Jun 23 16:43:50.321697 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] __import__(name)
[Fri Jun 23 16:43:50.321701 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/__init__.py", line 34, in <module>
[Fri Jun 23 16:43:50.321708 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] from openstack_dashboard.api import base
[Fri Jun 23 16:43:50.321726 2017] [:error] [pid 31878:tid 140589128623872] [remote 10.0.30.11:41599] ImportError: cannot import name base
If you Google for this error, you will find a bug report on Launchpad, with a long rambling set of comments. The last one says:
From the responses here it seems that the root issue with those encountering this "bug" is that their Horizon installation is missing some dependency, and installing that dependency fixes the issue.
The error reported "ImportError: cannot import name base" is a symptom of the failure to load all dependent modules.
The cause therefore would appear to be incomplete or erroneous installation documentation.
(Or possibly ... failure to follow the installation instructions.)
In short, check that you have installed the correct versions of all of the dependencies for Horizon.
A second bug report may offer more clues.
You say:
It was running fine earlier but suddenly this error occurred.
That might be due to update / patch being applied.

FreeRADIUS Google Dual Factor Authenticator, Pam

Hi I've been following this article to setup FreeRADIUS Google Dual Factor Authenticator
http://www.supertechguy.com/help/security/freeradius-google-auth
Hours of testing I still can't get it to work. If my /etc/pam.d/radiusd looks like the following it works well with
the following command
radtest test test localhost 18120 testing123
#
# /etc/pam.d/radiusd - PAM configuration for FreeRADIUS
#
# We fall back to the system default in /etc/pam.d/common-*
#
#include common-auth
#include common-account
#include common-password
#include common-session
However if it looks like the following
#
# /etc/pam.d/radiusd - PAM configuration for FreeRADIUS
#
# We fall back to the system default in /etc/pam.d/common-*
#
##include common-auth
##include common-account
##include common-password
##include common-session
auth requisite pam_google_authenticator.so forward_pass
auth required pam_unix.so use_first_pass
my log file says the following and auth fails.
rad_recv: Access-Request packet from host 127.0.0.1 port 43185, id=111, length=56
User-Name = "test"
User-Password = "test"
NAS-IP-Address = 127.0.1.1
NAS-Port = 18120
Thu Sep 26 16:38:19 2013 : Info: # Executing section authorize from file /etc/freeradius/sites-enabled/default
Thu Sep 26 16:38:19 2013 : Info: +- entering group authorize {...}
Thu Sep 26 16:38:19 2013 : Info: ++[preprocess] returns ok
Thu Sep 26 16:38:19 2013 : Info: ++[chap] returns noop
Thu Sep 26 16:38:19 2013 : Info: ++[mschap] returns noop
Thu Sep 26 16:38:19 2013 : Info: ++[digest] returns noop
Thu Sep 26 16:38:19 2013 : Info: [suffix] No '#' in User-Name = "test", looking up realm NULL
Thu Sep 26 16:38:19 2013 : Info: [suffix] No such realm "NULL"
Thu Sep 26 16:38:19 2013 : Info: ++[suffix] returns noop
Thu Sep 26 16:38:19 2013 : Info: [eap] No EAP-Message, not doing EAP
Thu Sep 26 16:38:19 2013 : Info: ++[eap] returns noop
Thu Sep 26 16:38:19 2013 : Info: [files] users: Matched entry DEFAULT at line 74
Thu Sep 26 16:38:19 2013 : Info: ++[files] returns ok
Thu Sep 26 16:38:19 2013 : Info: ++[expiration] returns noop
Thu Sep 26 16:38:19 2013 : Info: ++[logintime] returns noop
Thu Sep 26 16:38:19 2013 : Info: [pap] WARNING! No "known good" password found for the user. Authentication may fail because of this.
Thu Sep 26 16:38:19 2013 : Info: ++[pap] returns noop
Thu Sep 26 16:38:19 2013 : Info: Found Auth-Type = PAM
Thu Sep 26 16:38:19 2013 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Thu Sep 26 16:38:19 2013 : Info: +- entering group authenticate {...}
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: using pamauth string <radiusd> for pam.conf lookup
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: function pam_authenticate FAILED for <test>. Reason: Cannot make/remove an entry for the specified session
Thu Sep 26 16:38:19 2013 : Info: ++[pam] returns reject
Thu Sep 26 16:38:19 2013 : Info: Failed to authenticate the user.
Thu Sep 26 16:38:19 2013 : Info: Using Post-Auth-Type Reject
Thu Sep 26 16:38:19 2013 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Thu Sep 26 16:38:19 2013 : Info: +- entering group REJECT {...}
Thu Sep 26 16:38:19 2013 : Info: [attr_filter.access_reject] expand: %{User-Name} -> test
Thu Sep 26 16:38:19 2013 : Debug: attr_filter: Matched entry DEFAULT at line 11
Thu Sep 26 16:38:19 2013 : Info: ++[attr_filter.access_reject] returns updated
Thu Sep 26 16:38:19 2013 : Info: Delaying reject of request 0 for 1 seconds
Thu Sep 26 16:38:19 2013 : Debug: Going to the next request
Thu Sep 26 16:38:19 2013 : Debug: Waking up in 0.9 seconds.
Thu Sep 26 16:38:20 2013 : Info: Sending delayed reject for request 0
Sending Access-Reject of id 111 to 127.0.0.1 port 43185
Thu Sep 26 16:38:20 2013 : Debug: Waking up in 4.9 seconds.
Thu Sep 26 16:38:25 2013 : Info: Cleaning up request 0 ID 111 with timestamp +3
Thu Sep 26 16:38:25 2013 : Info: Ready to process requests.
I'm using Ubuntu latest
Does anyone know what the issue here?
Many Thanks
After so much internet surfing and forum hunting I manage to fix this problem. If anyone else having this issue this might help them :)
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: using pamauth string <radiusd> for pam.conf lookup
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: function pam_authenticate FAILED for <test>. Reason: Cannot make/remove an entry for the specified session
The above line actually means a auth fail, even though it doesn't sound like it, also it could mean that .google_authenticator file in the user's home directory isn't accessible.
FreeRadius log file not help you much with this issue, but have a look through /var/log/secure on CentOS and /var/log/auth.log in Ubuntu. This will explain which is the issue.
Issue with my system was my time was out and my random generated number by Google Dual Factor Authenticator application on my iPhone wasn't valid. I had to install NTP and change my servers time to the correct time which fixed the issue!!!!
hope this help someone else :)
The how-to on Super Tech Guy's page (http://www.supertechguy.com/help/security/freeradius-google-auth) has a typo.
DEFAULT Auth-Type := PAM
should be
DEFAULT Auth-Type = PAM
I don't know why he put a colon in there, but removing it fixed my issue.
This was after I made sure the server had the correct time (and timezone), which it didn't. So thanks for that suggestion too!

Unix command for extracting lines befor and after a particular searched string pattern

How can I search for lines in a file and extract the lines above and below lines of the searched line .
My input is like
Tue Jun 26 14:59:46 2012
Warning ffffffff act_msg_ctms_remove_from_pending_queue: deleting message 44817201 from the queue.
Tue Jun 26 14:59:46 2012
Warning ffffffff Finishing processing record number 44817201
Tue Jun 26 14:59:46 2012
Warning 5000000 activity_queue_manager_finish_cb: unknown activity 120.
Tue Jun 26 14:59:46 2012
Warning ffffffff Activity State Machine priority (2) finished
Tue Jun 26 14:59:46 2012
Warning ffffffff
====================================================
Processing database file "INCOMING_MESSAGES" record number 47810234 from user "(unknown)"
Tue Jun 26 14:59:46 2012
Warning ffffffff ACTIVITY data: rec_num (47810234) size (116)
Tue Jun 26 14:59:46 2012
Warning ffffffff activity status: ACT_SENT
Tue Jun 26 14:59:46 2012
Warning ffffffff MESSAGE body "MVT
QFA6673/26.VHQOS.BNE
EA0541
"
Tue Jun 26 14:59:46 2012
Warning ffffffff Finishing processing record number 47810234
Tue Jun 26 14:59:46 2012
Warning ffffffff Activity State Machine priority (1) finished
Tue Jun 26 14:59:46 2012
Warning ffffffff
End processing record number 47810234
====================================================
And I require my output to be like
/
Tue Jun 26 14:59:46 2012
Warning ffffffff MESSAGE body "MVT
QFA6673/26.VHQOS.BNE
EA0541"
/
My search string would be MVT.
Pls help
For three lines before and after the match
grep -C 3 pattern filename
For more control on number of after and before lines to be displayed for a match, use
grep -A (num of after) -B (num of lines before) pattern filename
From man grep:
-A NUM, --after-context=NUM
Print NUM lines of trailing context after matching lines.
Places a line containing -- between contiguous groups of matches.
-a, --text
Process a binary file as if it were text;
this is equivalent to the --binary-files=text option.
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines.
Places a line containing -- between contiguous groups of matches.
-C NUM, --context=NUM
Print NUM lines of output context.
Places a line containing -- between contiguous groups of matches.
Grep has options to display rows immediately before and after the match. The numbers in the command line below are the appropriate number of rows to display, after and before the match. E.g.
grep -A3 -B5 yoursearchpattern inputfilepattern
man grep is useful for details about the options.
Assuming you have GNU grep, to check you can use --version option:
> grep --version
GNU grep 2.6.3

Resources