Unix command for extracting lines befor and after a particular searched string pattern - unix

How can I search for lines in a file and extract the lines above and below lines of the searched line .
My input is like
Tue Jun 26 14:59:46 2012
Warning ffffffff act_msg_ctms_remove_from_pending_queue: deleting message 44817201 from the queue.
Tue Jun 26 14:59:46 2012
Warning ffffffff Finishing processing record number 44817201
Tue Jun 26 14:59:46 2012
Warning 5000000 activity_queue_manager_finish_cb: unknown activity 120.
Tue Jun 26 14:59:46 2012
Warning ffffffff Activity State Machine priority (2) finished
Tue Jun 26 14:59:46 2012
Warning ffffffff
====================================================
Processing database file "INCOMING_MESSAGES" record number 47810234 from user "(unknown)"
Tue Jun 26 14:59:46 2012
Warning ffffffff ACTIVITY data: rec_num (47810234) size (116)
Tue Jun 26 14:59:46 2012
Warning ffffffff activity status: ACT_SENT
Tue Jun 26 14:59:46 2012
Warning ffffffff MESSAGE body "MVT
QFA6673/26.VHQOS.BNE
EA0541
"
Tue Jun 26 14:59:46 2012
Warning ffffffff Finishing processing record number 47810234
Tue Jun 26 14:59:46 2012
Warning ffffffff Activity State Machine priority (1) finished
Tue Jun 26 14:59:46 2012
Warning ffffffff
End processing record number 47810234
====================================================
And I require my output to be like
/
Tue Jun 26 14:59:46 2012
Warning ffffffff MESSAGE body "MVT
QFA6673/26.VHQOS.BNE
EA0541"
/
My search string would be MVT.
Pls help

For three lines before and after the match
grep -C 3 pattern filename
For more control on number of after and before lines to be displayed for a match, use
grep -A (num of after) -B (num of lines before) pattern filename
From man grep:
-A NUM, --after-context=NUM
Print NUM lines of trailing context after matching lines.
Places a line containing -- between contiguous groups of matches.
-a, --text
Process a binary file as if it were text;
this is equivalent to the --binary-files=text option.
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines.
Places a line containing -- between contiguous groups of matches.
-C NUM, --context=NUM
Print NUM lines of output context.
Places a line containing -- between contiguous groups of matches.

Grep has options to display rows immediately before and after the match. The numbers in the command line below are the appropriate number of rows to display, after and before the match. E.g.
grep -A3 -B5 yoursearchpattern inputfilepattern
man grep is useful for details about the options.
Assuming you have GNU grep, to check you can use --version option:
> grep --version
GNU grep 2.6.3

Related

How to Extract logs between 2 timestamps in Unix

I need to extract the logs between two timestamps from a file in Unix. I basically kind of need those logs to be copied and output in a different file so that I can copy them.
Is there an efficient way to do this? The log format looks like this - The timestamp is in a separate line from the actual logs.
Tue 21 Apr 14:00:00 GMT 2020
{"items":[{"cpu.load": "0.94","total.memory": "6039.798 MB","free.memory": "4367.152 MB","used.memory": "1672.646 MB","total.physical.system.memory": "16.656 GB","total.free.physical.system.memory": "3860.197 MB","total.used.physical.system.memory": "12.796 GB","number.of.cpus": "8"}]}
Tue 21 Apr 18:00:00 GMT 2020
{"items":[{"cpu.load": "0.76","total.memory": "6039.798 MB","free.memory": "4352.656 MB","used.memory": "1687.142 MB","total.physical.system.memory": "16.656 GB","total.free.physical.system.memory": "3858.203 MB","total.used.physical.system.memory": "12.798 GB","number.of.cpus": "8"}]}
I am doing this but it only prints out the timestamp and not the actual logs
cat file.txt | awk -F, '{ if ($1>"Fri 21 Aug 14:00:00 GMT 2020" && $1<"Sat 22 Aug 18:00:00 GMT 2020") print }'
Can someone advice.

Snakemake: R script fails (almost) immediately

Once more I'm encountering an error in my snakemake workflow that doesn't make any sense to me.
This is the error I get:
[Thu Jan 25 10:47:00 2018] Building DAG of jobs...
[Thu Jan 25 10:47:01 2018] Provided cores: 24
[Thu Jan 25 10:47:01 2018] Rules claiming more threads will be scaled down.
[Thu Jan 25 10:47:01 2018] Job counts:
[Thu Jan 25 10:47:01 2018] count jobs
[Thu Jan 25 10:47:01 2018] 1 merging_seurat
[Thu Jan 25 10:47:01 2018] 1
[Thu Jan 25 10:47:01 2018] Job 0: --- Merging samples using seurat.
Error in setClass("Snakemake", slots = c(input = "list", output = "list", :
unused argument(s) (slots = c(input = "list", output = "list", params = "list", wildcards = "list", threads = "numeric", log = "list", resources = "list", config = "list", rule = "character"))
Execution halted
[Thu Jan 25 10:47:02 2018] Error in rule merging_seurat:
[Thu Jan 25 10:47:02 2018] jobid: 0
[Thu Jan 25 10:47:02 2018] output: merging_seurat/12_top10_heatmap_all_wilcox.pdf, merging_seurat/13_top10_heatmap_all_roc.pdf, merging_seurat/merging_seurat.RData
[Thu Jan 25 10:47:02 2018] RuleException:
[Thu Jan 25 10:47:02 2018] CalledProcessError in line 372 of .../snakemake_pipeline/Snakefile:
[Thu Jan 25 10:47:02 2018] Command ' set -euo pipefail; Rscript .../snakemake_pipeline/scripts/.snakemake.jv8ijpiw.merging_seurat.R ' returned non-zero exit status 1
[Thu Jan 25 10:47:02 2018] File ".../snakemake_pipeline/Snakefile", line 372, in __rule_merging_seurat
[Thu Jan 25 10:47:02 2018] File ".../tools/anaconda3/envs/Seurat/lib/python3.5/concurrent/futures/thread.py", line 55, in run
[Thu Jan 25 10:47:02 2018] Will exit after finishing currently running jobs.
[Thu Jan 25 10:47:02 2018] Exiting because a job execution failed. Look above for error message
[Thu Jan 25 10:47:02 2018] Complete log: .../snakemake_6/.snakemake/log/2018-01-25T104700.498155.snakemake.log
This is the rule in question:
rule merging_seurat:
input: expand("{sample}/molecule_count/counts_wide.tsv", sample=config["samples"]),
output: "merging_seurat/12_top10_heatmap_all_wilcox.pdf",
"merging_seurat/13_top10_heatmap_all_roc.pdf",
"merging_seurat/merging_seurat.RData"
message: "--- Merging samples using seurat."
script: "scripts/merging_seurat.R"
Here we have the top of the R script that is still executed:
sink('merging_seurat/output.txt')
print(installed.packages())
print(sessionInfo())
sink()
And these are the following lines in the R script that are not executed anymore:
library('Seurat')
library('dplyr')
library('org.Hs.eg.db')
Because I logged the packages that are available, I know that the three packages that should be loaded are also installed so it shouldn't fail because of that. In fact, commenting these lines out doesn't change anything, the script still breaks and the log message that should come right after loading the packages is not written to the log file.
Finally, this is the command that I use to run snakemake:
snakemake --use-conda \
--latency-wait 90 \
--rerun-incomplete \
--keep-going \
--timestamp \
--cluster-config SGE.json \
--cluster "qsub -cwd -N {cluster.name} -l h_vmem={cluster.h_vmem},h_stack=256M -o {cluster.stdout}{cluster.name}.o -e {cluster.stderr}{cluster.name}.e -m {cluster.mailtype} -M {cluster.mailuser}" \
-j 8 \
--directory .../snakemake_6
Does anybody have a clue what the error message could mean? unused argument of the snakemake R object is strange because I'm calling multiple parameters later on in the script.
What is even more strange is that I have another R script that works with different packages but has very similar first lines runs fine. I remember that in the beginning I had the same problem with that script (same error message) but don't remember how I solved it.
Any help is well appreciated.

Conversion to datetime is yielding error "ValueError: day is out of range for month" How to find

I have a huge dataframe that looks something like this:
Insider Trading Relationship Date \
SEC Form 4
Nov 16 04:06 PM Silverman Gene Director Nov 14
Oct 27 07:00 AM RAKOLTA JOHN JR Director Oct 26
Nov 16 04:09 PM LEIGHTON F THOMSON Chief Executive Officer Nov 15
Nov 02 04:20 PM Blumofe Robert EVP Platform Nov 01
Oct 28 04:03 PM MCCONNELL RICK M President Prods & Development Oct 28
I'm trying to change the index dtype into a datetime dtype via this code
pd.to_datetime(df2.index, format = '%b %d %I:%M %p')
but it's yielding the error:
Traceback (most recent call last):
File "<pyshell#126>", line 1, in <module>
pd.to_datetime(df2.index, format = '%b %d %I:%M %p')
File "C:\Python27\lib\site-packages\pandas\util\decorators.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Python27\lib\site-packages\pandas\tseries\tools.py", line 420, in to_datetime
return _convert_listlike(arg, box, format, name=arg.name)
File "C:\Python27\lib\site-packages\pandas\tseries\tools.py", line 407, in _convert_listlike
raise e
Is there a way I can find the index of where the error is occurring?
It seems I can set errors to coerce which would just return a Nan as the date, but I would like to avoid that.
Thanks!
You are right, just finish the logic. Set to coerce and filter the index against all values being isnull() to find all the incorrect indices.

FreeRADIUS Google Dual Factor Authenticator, Pam

Hi I've been following this article to setup FreeRADIUS Google Dual Factor Authenticator
http://www.supertechguy.com/help/security/freeradius-google-auth
Hours of testing I still can't get it to work. If my /etc/pam.d/radiusd looks like the following it works well with
the following command
radtest test test localhost 18120 testing123
#
# /etc/pam.d/radiusd - PAM configuration for FreeRADIUS
#
# We fall back to the system default in /etc/pam.d/common-*
#
#include common-auth
#include common-account
#include common-password
#include common-session
However if it looks like the following
#
# /etc/pam.d/radiusd - PAM configuration for FreeRADIUS
#
# We fall back to the system default in /etc/pam.d/common-*
#
##include common-auth
##include common-account
##include common-password
##include common-session
auth requisite pam_google_authenticator.so forward_pass
auth required pam_unix.so use_first_pass
my log file says the following and auth fails.
rad_recv: Access-Request packet from host 127.0.0.1 port 43185, id=111, length=56
User-Name = "test"
User-Password = "test"
NAS-IP-Address = 127.0.1.1
NAS-Port = 18120
Thu Sep 26 16:38:19 2013 : Info: # Executing section authorize from file /etc/freeradius/sites-enabled/default
Thu Sep 26 16:38:19 2013 : Info: +- entering group authorize {...}
Thu Sep 26 16:38:19 2013 : Info: ++[preprocess] returns ok
Thu Sep 26 16:38:19 2013 : Info: ++[chap] returns noop
Thu Sep 26 16:38:19 2013 : Info: ++[mschap] returns noop
Thu Sep 26 16:38:19 2013 : Info: ++[digest] returns noop
Thu Sep 26 16:38:19 2013 : Info: [suffix] No '#' in User-Name = "test", looking up realm NULL
Thu Sep 26 16:38:19 2013 : Info: [suffix] No such realm "NULL"
Thu Sep 26 16:38:19 2013 : Info: ++[suffix] returns noop
Thu Sep 26 16:38:19 2013 : Info: [eap] No EAP-Message, not doing EAP
Thu Sep 26 16:38:19 2013 : Info: ++[eap] returns noop
Thu Sep 26 16:38:19 2013 : Info: [files] users: Matched entry DEFAULT at line 74
Thu Sep 26 16:38:19 2013 : Info: ++[files] returns ok
Thu Sep 26 16:38:19 2013 : Info: ++[expiration] returns noop
Thu Sep 26 16:38:19 2013 : Info: ++[logintime] returns noop
Thu Sep 26 16:38:19 2013 : Info: [pap] WARNING! No "known good" password found for the user. Authentication may fail because of this.
Thu Sep 26 16:38:19 2013 : Info: ++[pap] returns noop
Thu Sep 26 16:38:19 2013 : Info: Found Auth-Type = PAM
Thu Sep 26 16:38:19 2013 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Thu Sep 26 16:38:19 2013 : Info: +- entering group authenticate {...}
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: using pamauth string <radiusd> for pam.conf lookup
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: function pam_authenticate FAILED for <test>. Reason: Cannot make/remove an entry for the specified session
Thu Sep 26 16:38:19 2013 : Info: ++[pam] returns reject
Thu Sep 26 16:38:19 2013 : Info: Failed to authenticate the user.
Thu Sep 26 16:38:19 2013 : Info: Using Post-Auth-Type Reject
Thu Sep 26 16:38:19 2013 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Thu Sep 26 16:38:19 2013 : Info: +- entering group REJECT {...}
Thu Sep 26 16:38:19 2013 : Info: [attr_filter.access_reject] expand: %{User-Name} -> test
Thu Sep 26 16:38:19 2013 : Debug: attr_filter: Matched entry DEFAULT at line 11
Thu Sep 26 16:38:19 2013 : Info: ++[attr_filter.access_reject] returns updated
Thu Sep 26 16:38:19 2013 : Info: Delaying reject of request 0 for 1 seconds
Thu Sep 26 16:38:19 2013 : Debug: Going to the next request
Thu Sep 26 16:38:19 2013 : Debug: Waking up in 0.9 seconds.
Thu Sep 26 16:38:20 2013 : Info: Sending delayed reject for request 0
Sending Access-Reject of id 111 to 127.0.0.1 port 43185
Thu Sep 26 16:38:20 2013 : Debug: Waking up in 4.9 seconds.
Thu Sep 26 16:38:25 2013 : Info: Cleaning up request 0 ID 111 with timestamp +3
Thu Sep 26 16:38:25 2013 : Info: Ready to process requests.
I'm using Ubuntu latest
Does anyone know what the issue here?
Many Thanks
After so much internet surfing and forum hunting I manage to fix this problem. If anyone else having this issue this might help them :)
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: using pamauth string <radiusd> for pam.conf lookup
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: function pam_authenticate FAILED for <test>. Reason: Cannot make/remove an entry for the specified session
The above line actually means a auth fail, even though it doesn't sound like it, also it could mean that .google_authenticator file in the user's home directory isn't accessible.
FreeRadius log file not help you much with this issue, but have a look through /var/log/secure on CentOS and /var/log/auth.log in Ubuntu. This will explain which is the issue.
Issue with my system was my time was out and my random generated number by Google Dual Factor Authenticator application on my iPhone wasn't valid. I had to install NTP and change my servers time to the correct time which fixed the issue!!!!
hope this help someone else :)
The how-to on Super Tech Guy's page (http://www.supertechguy.com/help/security/freeradius-google-auth) has a typo.
DEFAULT Auth-Type := PAM
should be
DEFAULT Auth-Type = PAM
I don't know why he put a colon in there, but removing it fixed my issue.
This was after I made sure the server had the correct time (and timezone), which it didn't. So thanks for that suggestion too!

Time zone and BST woes in Europe/London

I am really struggling with my hwclock since the UK changed to British Summer Time (BST) last weekend on my Centos 5.8 KVM guest.
Here's some command outputs..
[root#host ~]# TZ=Europe/London date
Mon Apr 2 08:12:03 EDT 2012
[root#host ~]# TZ=Europe/Jersey date
Mon Apr 2 08:12:11 EDT 2012
[root#host ~]# TZ=Europe/Paris date
Mon Apr 2 14:12:16 CEST 2012
[root#host ~]# TZ=Europe/Rome date
Mon Apr 2 14:12:22 CEST 2012
[root#host ~]# TZ=Europe/Athens date
Mon Apr 2 15:12:27 EEST 2012
[root#host ~]# hwclock -rD
hwclock from util-linux-2.13-pre7
Using /dev/rtc interface to clock.
Last drift adjustment done at 1333367010 seconds after 1969
Last calibration done at 1333367010 seconds after 1969
Hardware clock is on UTC time
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
/dev/rtc does not have interrupt functions. Waiting in loop for time from /dev/rtc to change
...got clock tick
Time read from Hardware Clock: 2012/04/02 12:22:27
Hw clock time : 2012/04/02 12:22:27 = 1333369347 seconds since 1969
Mon 02 Apr 2012 12:22:27 PM UTC -0.422061 seconds
And finally when I ln -sf to Europe/London it just goes back to EDT :(
[root#host ~]# ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
[root#host ~]# date
Mon Apr 2 08:23:02 EDT 2012
:(
Here's also some contents of files:
[root#host ~]# cat /etc/sysconfig/clock
ZONE="Europe/London"
UTC=true
ARC=false
Really struggling here guy and have googled til my eyes popped out but to no avail.
smybolically linking the files doesn't work - they need to be physically copied
[root#xxx]# mv /etc/localtime /etc/localtime.default
[root#xxx]# cp /usr/share/zoneinfo/Europe/London /etc/localtime
[root#xxx]# date
Fri Apr 13 11:35:57 BST 2012
Hope that helps :-)
I had a similar problem on my mailserver runninng CentOS 6.4 64-bit. Mail coming in marked 6 hours forward in time! I set Oslo-time for the server. The clock was correct but it showed the wrong time zone. It should be CEST:
/etc/localtime -> /usr/share/zoneinfo/Europe/Oslo
$date
Thu Sep 19 17:08:42 EDT 2013
Cities in the same time zone showed CEST. Strange. The only way to fix it was to use Copenhagen instead.
localtime -> /usr/share/zoneinfo/Europe/Copenhagen
$ date
Thu Sep 19 17:16:55 CEST 2013
So if nothing works try to use a city in the same time zone.

Resources