I try to show tail of a text file. If file is small, there is no difference. However if file is too big (~5 gB), tailf does not respond. On the other hand tail -f works fine. What is difference between them?
I have faced the same issue. The log file was about 47GB. The tailf just waits almost infinite. But the tail -f begin to print the output within seconds.
I have digged deeper by examining the underlying system calls using strace command. The results given below:
# strace tailf /var/log/messages
(truncated)
stat("/var/log/messages", {st_mode=S_IFREG|0600, st_size=47432599401, ...}) = 0
open("/var/log/messages", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0600, st_size=47432600425, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7dba2d1000
read(3, "Nov 1 03:23:01 hostnameXXXX"..., 4096) = 4096
read(3, "0.31.148.12)\nNov 1 03:54:33 del"..., 4096) = 4096
read(3, "io.c(600) [receiver=3.0.6]\nNov "..., 4096) = 4096
(truncated)
As you can see, the tailf is trying to read (buffer) all the lines from beginning before generating output to the screen.
Check the output of tail -f below, here it is using the system call lseek (C/C++) to directly jump to end of file and start reading from there:
# strace tail -f /var/log/messages
(truncated)
open("/var/log/messages", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0600, st_size=47294167448, ...}) = 0
lseek(3, 0, SEEK_CUR) = 0
lseek(3, 0, SEEK_END) = 47294170917
lseek(3, 47294169088, SEEK_SET) = 47294169088
(truncated)
From the man page:
tailf will print out the last 10 lines of a file and then wait for the
file to grow. It is similar to tail -f but does not access the file
when it is not growing. This has the side effect of not updating the
access time for the file, so a filesystem flush does not occur periodi-
cally when no log activity is happening.
http://linuxcommand.org/man_pages/tailf1.html
If it doesn't access the file directly it will have some difficulties with very lage files, depending on your machines setup.
Related
I am trying to run the following command in ubuntu terminal
patch -p0 -i adjustmentFile.patch
That is giving the following error
patching file ./src/helpStructures/CastaliaModule.cc
patching file ./src/node/communication/mac/tunableMac/TunableMAC.cc
Hunk #2 FAILED at 456.
1 out of 2 hunks FAILED -- saving rejects to file ./src/node/communication/mac/tunableMac/TunableMAC.cc.rej
I tried almost all the ways suggested in the link Hunk #1 FAILED at 1. What's that mean?. However, nothing worked.
Here is my version detail
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Jun 06 2019 17:31:41)
Included patches: 1-1453
The patch file:
diff -r -u ./src/helpStructures/CastaliaModule.cc ./src/helpStructures/CastaliaModule.cc
--- ./src/helpStructures/CastaliaModule.cc 2010-12-09 09:56:47.000000000 -0300
+++ ./src/helpStructures/CastaliaModule.cc 2011-12-20 00:16:39.944320051 -0300
## -180,6 +180,8 ##
classPointers.resourceManager = getParentModule()->getParentModule()->getSubmodule("ResourceManager");
else if (name.compare("SensorManager") == 0)
classPointers.resourceManager = getParentModule()->getSubmodule("ResourceManager");
+ else if (name.compare("Routing") == 0)
+ classPointers.resourceManager = getParentModule()->getParentModule()->getSubmodule("ResourceManager");
else
opp_error("%s module has no rights to call drawPower() function", getFullPath().c_str());
if (!classPointers.resourceManager)
Only in ./src/helpStructures: CastaliaModule.cc~
diff -r -u ./src/node/communication/mac/tunableMac/TunableMAC.cc ./src/node/communication/mac/tunableMac/TunableMAC.cc
--- ./src/node/communication/mac/tunableMac/TunableMAC.cc 2011-03-30 02:14:34.000000000 -0300
+++ ./src/node/communication/mac/tunableMac/TunableMAC.cc 2011-12-19 23:57:43.894686687 -0300
## -405,6 +405,8 ##
void TunableMAC::fromRadioLayer(cPacket * pkt, double rssi, double lqi)
{
TunableMacPacket *macFrame = dynamic_cast <TunableMacPacket*>(pkt);
+ macFrame->getMacRadioInfoExchange().RSSI = rssi;
+ macFrame->getMacRadioInfoExchange().LQI = lqi;
if (macFrame == NULL){
collectOutput("TunableMAC packet breakdown", "filtered, other MAC");
return;
## -454,7 +456,8 ##
}
case DATA_FRAME:{
- toNetworkLayer(macFrame->decapsulate());
+ cPacket *netPkt = decapsulatePacket(macFrame);
+ toNetworkLayer(netPkt);
collectOutput("TunableMAC packet breakdown", "received data pkts");
if (macState == MAC_STATE_RX) {
cancelTimer(ATTEMPT_TX);
Only in ./src/node/communication/mac/tunableMac: TunableMAC.cc~
Patching takes some changes made to a file X, and applies them to a different instance of file X. That is, suppose you start with generation 1 of file X; you make changes to get generation 2-a, and someone else starts with generation 1 to make generation 2-b. Now you want to take his edits that created his generation 2-b, and apply them to your generation 2-a.
If 'his' changes clash with 'your' changes, they cannot be automatically patched.
You'll need to look at the changes being made in hunk 2.
- toNetworkLayer(macFrame->decapsulate());
+ cPacket *netPkt = decapsulatePacket(macFrame);
+ toNetworkLayer(netPkt);
and figure out what you want the result to look like. Someone needs to know what the result is supposed to be. You can't resolve conflicts without knowledge of intent.
My main question is how to split strings on the command line into parameters using a terminal command in Linux?
For example
on the command line:
./my program hello world "10 20 30"
The parameters are set as:
$1 = hello
$2 = world
$3 = 10 20 30
But I want:
$1 = hello
$2 = world
$3 = 10
$4 = 20
$5 = 30
How can I do it correctly?
You can reset the positional parameters $# by using the set builtin. If you do not double-quote $#, the shell will word-split it producing the behavior you desire:
$ cat my_program.sh
#! /bin/sh
i=1
for PARAM; do
echo "$i = $PARAM";
i=$(( $i + 1 ));
done
set -- $#
echo "Reset \$# with word-split params"
i=1
for PARAM; do
echo "$i = $PARAM";
i=$(( $i + 1 ));
done
$ sh ./my_program.sh foo bar "baz buz"
1 = foo
2 = bar
3 = baz buz
Reset $# with word-split params
1 = foo
2 = bar
3 = baz
4 = buz
As an aside, I find it mildly surprising that you want to do this. Many shell programmers are frustrated by the shell's easy, accidental word-splitting — they get "John", "Smith" when they wanted to preserve "John Smith" — but it seems to be your requirement here.
Use xargs:
echo "10 20 30" | xargs ./my_program hello world
xargs is a command on Unix and most Unix-like operating systems used
to build and execute command lines from standard input. Commands such as
grep and awk can accept the standard input as a parameter, or argument
by using a pipe. However, others such as cp and echo disregard the
standard input stream and rely solely on the arguments found after the
command. Additionally, under the Linux kernel before version 2.6.23,
and under many other Unix-like systems, arbitrarily long lists of
parameters cannot be passed to a command,[1] so xargs breaks the list
of arguments into sublists small enough to be acceptable.
(source)
I have a running a fuse fs with options allow_other and umask 0. This gives me a set of files with permissions set to 777. Though when I try to ls -l in the directory containing the files I get the following output:
ls: name: Operation not permitted
ls: tags: Operation not permitted
ls: location: Operation not permitted
ls: ext: Operation not permitted
ls: experiment_id: Operation not permitted
ls: file_path: Operation not permitted
Can anyone tell me why despite having global permissions (777) I am still getting operation not permitted?
On running strace, I get the following traces:
lstat("tags", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lgetxattr("tags", "security.selinux", 0x112ae80, 255) = -1 EPERM (Operation not permitted)
write(2, "ls: ", 4ls: ) = 4
write(2, "tags", 4tags) = 4
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1
) = 1
lstat("location", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lgetxattr("location", "security.selinux", 0x112aea0, 255) = -1 EPERM (Operation not permitted)
write(2, "ls: ", 4ls: ) = 4
write(2, "location", 8location) = 8
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1) = 1
lstat("ext", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lgetxattr("ext", "security.selinux", 0x112aec0, 255) = -1 EPERM (Operation not permitted)
write(2, "ls: ", 4ls: ) = 4
write(2, "ext", 3ext) = 3
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1) = 1
lstat("experiment_id", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lgetxattr("experiment_id", "security.selinux", 0x112aee0, 255) = -1 EPERM (Operation not permitted)
write(2, "ls: ", 4ls: ) = 4
write(2, "experiment_id", 13experiment_id) = 13
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1) = 1
lstat("file_path", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lgetxattr("file_path", "security.selinux", 0x112af00, 255) = -1 EPERM (Operation not permitted)
write(2, "ls: ", 4ls: ) = 4
write(2, "file_path", 9file_path) = 9
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1) = 1
So from the trace, it looks like its trying to get the selinux attribute even though its disabled on my system.
cat /etc//sysconfig/selinux
SELINUX=disabled
SELINUXTYPE=targeted
follow the below steps to resolve the issue. I tried below steps, worked for me
1.Pull down the Apple menu and choose ‘System Preferences’
2.Choose “Security & Privacy” control panel
3.Now select the “Privacy” tab, then from the left-side menu select “Full Disk Access”
4.Click the lock icon in the lower left corner of the preference panel and authenticate with an admin level login
5.Now click the [+] plus button to add an application with full disk access
6.Navigate to the /Applications/Utilities/ folder and choose “Terminal” to grant Terminal with Full Disk Access privileges
7.Relaunch Terminal, the “Operation not permitted” error messages will be gone
Set permissions on the directory that contains the files.
Use strace(1) at least as
strace ls -l
this will show you all the syscalls done by ls and you would recognize which FUSE file-system related syscalls(2) are failing.
Perhaps stat(2) is failing on individual directory entries like tags etc...?
You are probably forgetting to implement some operations in your FUSE.
The problem was with my getxattr implementation. I was returning -1 on error which translated to EPERM, instead I should have returned ENODATA which is more correct error case for my logic. This fixed those errors as well.
https://gowalker.org/github.com/hanwen/go-fuse/fuse
Does anyone know the api for adding users and groups in unix and removing them ? I want to do this programatically.
Thanks,
Frank
I started looking at some system calls and found the following. Note that they are of varying standards, so not all may work on your Unix version:
getpwent
setpwent
putpwent
These however, all assume a password file. Out of curiosity, I straced useradd to find out what he did. Here's a small section of it's output:
# grep -E 'passwd|shadow' useradd.log.20283
...
open("/etc/shadow", O_RDONLY|O_CLOEXEC) = 3
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
open("/etc/passwd.20283", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
link("/etc/passwd.20283", "/etc/passwd.lock") = 0
stat("/etc/passwd.20283", {st_mode=S_IFREG|0600, st_size=6, ...}) = 0
unlink("/etc/passwd.20283") = 0
open("/etc/passwd", O_RDWR) = 4
open("/etc/shadow.20283", O_WRONLY|O_CREAT|O_EXCL, 0600) = 5
link("/etc/shadow.20283", "/etc/shadow.lock") = 0
stat("/etc/shadow.20283", {st_mode=S_IFREG|0600, st_size=6, ...}) = 0
unlink("/etc/shadow.20283") = 0
open("/etc/shadow", O_RDWR) = 5
open("/etc/gshadow.20283", O_WRONLY|O_CREAT|O_EXCL, 0600) = 7
link("/etc/gshadow.20283", "/etc/gshadow.lock") = 0
stat("/etc/gshadow.20283", {st_mode=S_IFREG|0600, st_size=6, ...}) = 0
unlink("/etc/gshadow.20283") = 0
open("/etc/gshadow", O_RDWR) = 7
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 8
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 9
open("/etc/passwd-", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 9
utime("/etc/passwd-", [2010/09/02-07:07:34, 2010/09/02-07:07:34]) = 0
open("/etc/passwd+", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
lstat("/etc/passwd", {st_mode=S_IFREG|0644, st_size=2479, ...}) = 0
rename("/etc/passwd+", "/etc/passwd") = 0
open("/etc/shadow-", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
utime("/etc/shadow-", [2010/09/02-07:07:34, 2010/09/02-07:07:34]) = 0
open("/etc/shadow+", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
lstat("/etc/shadow", {st_mode=S_IFREG|0600, st_size=1429, ...}) = 0
r ename("/etc/shadow+", "/etc/shadow") = 0
open("/etc/gshadow-", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
utime("/etc/gshadow-", [2010/09/02-07:07:34, 2010/09/02-07:07:34]) = 0
open("/etc/gshadow+", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
lstat("/etc/gshadow", {st_mode=S_IFREG|0400, st_size=1069, ...}) = 0
rename("/etc/gshadow+", "/etc/gshadow") = 0
unlink("/etc/shadow.lock") = 0
unlink("/etc/passwd.lock") = 0
unlink("/etc/gshadow.lock") = 0
Although you get a better idea what's going on with the full context, note that it links a temporary file it created (/etc/passwd.20283) to /etc/passwd.lock. useradd does similarly with the shadow, and gshadow files as well.
It's also important to note that useradd made four calls out to nscd. Two of these were for passwd, and two were for group:
execve("/usr/sbin/nscd", ["/usr/sbin/nscd", "nscd", "-i", "passwd"], [/* 0 vars */]) = 0
If there isn't an API (and I can't seem to find one), it may be because there's many more ways to store users than simple passwd files. Indeed, it's possible that the machine has no control at all over the users.
EDIT: I suppose it's also important to note that useradd consulted /etc/nsswitch.conf as well, likely to verify the origin of the user database. Furthermore, userdel behaved almost identically, creating similarly named temporary and lock files.
I tested under Linux using the following command:
strace -o useradd.log -f -ff -s 1024 useradd tempuser
strace may also appear as truss and ktrace on other unix systems.
I found this question while looking for a way to list all groups on a Unix.
But to create and delete users you can programatically call the utility programs useradd, userdel, groupadd, groupdel since you would know the user name been sent. But I guess you will need superuser rights to invoke them. Anyway you can check the process exit code for command status.
useradd xxx // status zero operation completed
useradd xxx // status nine user already exists
Hope it helps.
I am working on a UNIX box, and trying to run an application, which gives some debug logs to the standard output. I have redirected this output to a log file, but now wish to get the lines where the error is being shown.
My problem here is that a simple
cat output.log | grep FAIL
does not help out. As this shows only the lines which have FAIL in them. I want some more information along with this. Like the 2-3 lines above this line with FAIL. Is there any way to do this via a simple shell command? I would like to have a single command line (can have pipes) to do the above.
grep -C 3 FAIL output.log
Note that this also gets rid of the useless use of cat (UUOC).
grep -A $NUM
This will print $NUM lines of trailing context after matches.
-B $NUM prints leading context.
man grep is your best friend.
So in your case:
cat log | grep -A 3 -B 3 FAIL
I have two implementations of what I call sgrep, one in Perl, one using just pre-Perl (pre-GNU) standard Unix commands. If you've got GNU grep, you've no particular need of these. It would be more complex to deal with forwards and backwards context searches, but that might be a useful exercise.
Perl solution:
#!/usr/perl/v5.8.8/bin/perl -w
#
# #(#)$Id: sgrep.pl,v 1.6 2007/09/18 22:55:20 jleffler Exp $
#
# Perl-based SGREP (special grep) command
#
# Print lines around the line that matches (by default, 3 before and 3 after).
# By default, include file names if more than one file to search.
#
# Options:
# -b n1 Print n1 lines before match
# -f n2 Print n2 lines following match
# -n Print line numbers
# -h Do not print file names
# -H Do print file names
use strict;
use constant debug => 0;
use Getopt::Std;
my(%opts);
sub usage
{
print STDERR "Usage: $0 [-hnH] [-b n1] [-f n2] pattern [file ...]\n";
exit 1;
}
usage unless getopts('hnf:b:H', \%opts);
usage unless #ARGV >= 1;
if ($opts{h} && $opts{H})
{
print STDERR "$0: mutually exclusive options -h and -H specified\n";
exit 1;
}
my $op = shift;
print "# regex = $op\n" if debug;
# print file names if -h omitted and more than one argument
$opts{F} = (defined $opts{H} || (!defined $opts{h} and scalar #ARGV > 1)) ? 1 : 0;
$opts{n} = 0 unless defined $opts{n};
my $before = (defined $opts{b}) ? $opts{b} + 0 : 3;
my $after = (defined $opts{f}) ? $opts{f} + 0 : 3;
print "# before = $before; after = $after\n" if debug;
my #lines = (); # Accumulated lines
my $tail = 0; # Line number of last line in list
my $tbp_1 = 0; # First line to be printed
my $tbp_2 = 0; # Last line to be printed
# Print lines from #lines in the range $tbp_1 .. $tbp_2,
# leaving $leave lines in the array for future use.
sub print_leaving
{
my ($leave) = #_;
while (scalar(#lines) > $leave)
{
my $line = shift #lines;
my $curr = $tail - scalar(#lines);
if ($tbp_1 <= $curr && $curr <= $tbp_2)
{
print "$ARGV:" if $opts{F};
print "$curr:" if $opts{n};
print $line;
}
}
}
# General logic:
# Accumulate each line at end of #lines.
# ** If current line matches, record range that needs printing
# ** When the line array contains enough lines, pop line off front and,
# if it needs printing, print it.
# At end of file, empty line array, printing requisite accumulated lines.
while (<>)
{
# Add this line to the accumulated lines
push #lines, $_;
$tail = $.;
printf "# array: N = %d, last = $tail: %s", scalar(#lines), $_ if debug > 1;
if (m/$op/o)
{
# This line matches - set range to be printed
my $lo = $. - $before;
$tbp_1 = $lo if ($lo > $tbp_2);
$tbp_2 = $. + $after;
print "# $. MATCH: print range $tbp_1 .. $tbp_2\n" if debug;
}
# Print out any accumulated lines that need printing
# Leave $before lines in array.
print_leaving($before);
}
continue
{
if (eof)
{
# Print out any accumulated lines that need printing
print_leaving(0);
# Reset for next file
close ARGV;
$tbp_1 = 0;
$tbp_2 = 0;
$tail = 0;
#lines = ();
}
}
Pre-Perl Unix solution (using plain ed, sed, and sort - though it uses getopt which was not necessarily available back then):
#!/bin/ksh
#
# #(#)$Id: old.sgrep.sh,v 1.5 2007/09/15 22:15:43 jleffler Exp $
#
# Special grep
# Finds a pattern and prints lines either side of the pattern
# Line numbers are always produced by ed (substitute for grep),
# which allows us to eliminate duplicate lines cleanly. If the
# user did not ask for numbers, these are then stripped out.
#
# BUG: if the pattern occurs in in the first line or two and
# the number of lines to go back is larger than the line number,
# it fails dismally.
set -- `getopt "f:b:hn" "$#"`
case $# in
0) echo "Usage: $0 [-hn] [-f x] [-b y] pattern [files]" >&2
exit 1;;
esac
# Tab required - at least with sed (perl would be different)
# But then the whole problem would be different if implemented in Perl.
number="'s/^\\([0-9][0-9]*\\) /\\1:/'"
filename="'s%^%%'" # No-op for sed
f=3
b=3
nflag=no
hflag=no
while [ $# -gt 0 ]
do
case $1 in
-f) f=$2; shift 2;;
-b) b=$2; shift 2;;
-n) nflag=yes; shift;;
-h) hflag=yes; shift;;
--) shift; break;;
*) echo "Unknown option $1" >&2
exit 1;;
esac
done
pattern="${1:?'No pattern'}"
shift
case $# in
0) tmp=${TMPDIR:-/tmp}/`basename $0`.$$
trap "rm -f $tmp ; exit 1" 0
cat - >$tmp
set -- $tmp
sort="sort -t: -u +0n -1"
;;
*) filename="'s%^%'\$file:%"
sort="sort -t: -u +1n -2"
;;
esac
case $nflag in
yes) num_remove='s/[0-9][0-9]*://';;
no) num_remove='s/^//';;
esac
case $hflag in
yes) fileremove='s%^$file:%%';;
no) fileremove='s/^//';;
esac
for file in $*
do
echo "g/$pattern/.-${b},.+${f}n" |
ed - $file |
eval sed -e "$number" -e "$filename" |
$sort |
eval sed -e "$fileremove" -e "$num_remove"
done
rm -f $tmp
trap 0
exit 0
The shell version of sgrep was written in February 1989, and bug fixed in May 1989. It then remained unchanged except for an administrative change (SCCS to RCS transition) in 1997 until 2007, when I added the -h option. I switched to the Perl version in 2007.
http://thedailywtf.com/Articles/The_Complicator_0x27_s_Gloves.aspx
You can use sed to print specific lines, lets say you want line 20
sed '20 p' -n FILE_YOU_WANT_THE_LINE_FROM
Done.
-n prevents echoing lines from the file. The part in quotes is a sed rule to apply, it specifies that you want the rule to apply to line 20, and you want to print.
With GNU grep on Windows:
$ grep --context 3 FAIL output.log
$ grep --help | grep context
-B, --before-context=NUM print NUM lines of leading context
-A, --after-context=NUM print NUM lines of trailing context
-C, --context=NUM print NUM lines of output context
-NUM same as --context=NUM