I have a query regarding the execution of a complex command in the makefile of the current system.
I am currently using shell command in the makefile to execute the command. However my command fails as it is a combination of a many commands and execution collects a huge amount of data. The makefile content is something like this:
variable=$(shell ls -lart | grep name | cut -d/ -f2- )
However the make execution fails with execvp failure, since the file listing is huge and I need to parse all of them.
Please suggest me any ways to overcome this issue. Basically I would like to execute a complex command and assign that output to a makefile variable which I want to use later in the program.
(This may take a few iterations.)
This looks like a limitation of the architecture, not a Make limitation. There are several ways to address it, but you must show us how you use variable, otherwise even if you succeed in constructing it, you might not be able to use it as you intend. Please show us the exact operations you intend to perform on variable.
For now I suggest you do a couple of experiments and tell us the results. First, try the assignment with a short list of files (e.g. three) to verify that the assignment does what you intend. Second, in the directory with many files, try:
variable=$(shell ls -lart | grep name)
to see whether the problem is in grep or cut.
Rather than store the list of files in a variable you can easily use shell functionality to get the same result. It's a bit odd that you're flattening a recursive ls to only get the leaves, and then running mkdir -p which is really only useful if the parent directory doesn't exist, but if you know which depths you want to (for example the current directory and all subdirectories one level down) you can do something like this:
directories:
for path in ./*name* ./*/*name*; do \
mkdir "/some/path/$(basename "$path")" || exit 1; \
done
or even
find . -name '*name*' -exec mkdir "/some/path/$(basename {})" \;
Related
I've been using this utility successfully for many years, in many environemnts. But I'm noticing that on one particular environment, it produces very unexpected results.
grep -r 'search-term1' . | grep 'search-term2'
The above code greps recursively for all instances of search-term1, in the current-dir. The results are then piped to another grep, which selects only those lines that also contain search-term2. This works exactly as I would expect.
grep -r 'search-term1' . | grep -r 'search-term2'
The only difference in the above code is that the -r recursive flag in specified in both grep commands. I would expect the behavior to not change for this particular case. After all, the input to the 2nd grep is a pipe-input, and there's nothing further to be found recursively.
I have been using the command successfully, for many years, in many different environments (both unix and mac-os). However, the most recent environment that I started working in (unix), breaks the above behavior. The second piped grep searches for all instances of search-term2, not only in the piped-input, but also all files in my current directory. Because of this, instead of getting only results that contain both search-terms, I get all results in current-dir that contain the 2nd search term.
Is there any reason why this one particular environment produces this odd behavior? Is there any way I can avoid this, while still preserving the -r flag?
FAQ:
Q: Why am I using the -r flag on a piped input?
Ans: I actually have grep saved as an alias, with many different options and flags that I always want to use as a default. The recursive flag is one of them. I would like to always use this alias, instead of having to type out all the flags every time.
Q: If you want to search for all instances matching both search terms, why not do (insert-superior-method-here) instead?
Ans: You're probably right. I'm sure there are things I can change in my usual habits that would workaround this issue. However, as intellectual curiosity, I would like to find out why recursive-greps-on-pipes work as intended on most environments, but not all, and if that can somehow be resolved.
The -r flag to grep changed in grep version 2.11 (release notes to implicitly use the working directory as the input if no file arguments are given.
If no file operand is given, and a command-line -r or equivalent
option is given, grep now searches the working directory.
You aren't giving the second grep any file arguments so it defaults to the current directory despite there being pipe input.
Try grep -r 'search-term1' . | grep -r 'search-term2' - as a workaround.
grep -r 'search-term1' . | grep -r -d skip 'search-term2' may also work around the problem.
Is there a way in a unix shell (specifically Ubuntu) to change directory into the xth directory that was printed from the ls command?
I know you can sort a directory in multiple ways, but using the output from ls to get the xth directory?
An example shell:
$ ls
$ first_dir second_dir third_really_long_and_complex_dir
where I want to move into the third_really_long_and_complex_dir by passing 3 (or 2 in proper array format).
I know I could simply copy and paste, but if I'm already using the keyboard, it would be easier to type something like "cdls 2" or something like that if I knew the index.
The main problem with cd in an interactive session is that you generally want to change the current directory of the shell that is processing the command prompt. That means that launching a sub-shell (e.g. a script) would not help, since any cd calls would not affect the parent shell.
Depending on which shell you are using, however, you might be able to define a function to do this. For example in bash:
function cdls() {
# Save the current state of the nullglob option
SHOPT=`shopt -p nullglob`
# Make sure that */ expands to nothing when no directories are present
shopt -s nullglob
# Get a list of directories
DIRS=(*/)
# Restore the nullblob option state
$SHOPT
# cd using a zero-based index
cd "${DIRS[$1]}"
}
Note that in this example I absolutely refuse to parse the output of ls, for a number of reasons. Instead I let the shell itself retrieve a list of directories (or links to directories)...
That said, I suspect that using this function (or anything to this effect) is a very good way to set yourself up for an enormous mess - like using rm after changing to the wrong directory. File-name auto-completion is dangerous enough already, without forcing yourself to count...
Trying to rename a set of files in a directory with various filetypes, all with one common word, say 'foo', to another word, say 'bar' on a MacBook Pro.
E.g.:
foo.txt
form_foo.plist
home_foo.png
images_foo.zip
->
bar.txt
form_bar.plist
home_bar.png
images_bar.zip
Any ideas?
Use with care:
ls | grep foo | while read -r name; do echo mv "$name" "${name//foo/bar}"; done
That will report the commands it will run when you omit "echo". Inspect
the results, then rerun with "echo" omitted. This makes no attempt to work
on files with newlines in the name, nor does it recurse into subdirectories. If you want to work with files whose name begins with ., add -a to the invocation of ls. For safety's sake, you may want to add -i to the invocation of mv. Certainly make a backup first.
I don't have access to a Mac, but under Ubuntu you can use the rename command for this. Here's the man page in case that command is available
For my developer work I reside in the *nix shell environment pretty much all day, but still can't seem to memorize the name and argument specifics of programs I don't use daily. I wonder how other 'casual amnesiacs' handle this. Do you maintain an big cheat sheet? Do you rehearse the emacs shortcuts when you take your weekly shower? Or is your desk covered under sticky notes?
Using bash_completion is one way of not having to remember the precise syntax of program arguments.
> svn [tab][tab]
--help checkout delete lock pdel propget revert
--version ci diff log pedit proplist rm
-h cleanup export ls pget propset status
add co help merge plist pset switch
annotate commit import mkdir praise remove unlock
blame copy info move propdel rename update
cat cp list mv propedit resolved
If I don't use a command regularly enough to remember what I want, I tend to just use --help or the man pages when I need to.
Or, if I'm lucky, I use CTRL+R and let bash's history search find when I last used it.
Eventually you just remember them, well the set that you use anyway. I used to maintain a README in my home directory when I was starting out but that disappeared many years ago.
One useful command is man -k which you pass a word to and it will return a list of all commands whose man page summary contains that word.
'apropos' is also a very useful command. It will list all commands whose man pages contain the keyword.
I did some havoc on my computer, when I played with the commands suggested by vezult [1]. I expected the one-liner to ask file-names to be removed. However, it immediately removed my files in a folder:
> find ./ -type f | while read x; do rm "$x"; done
I expected it to wait for my typing of stdin:s [2]. I cannot understand its action. How does the read command work, and where do you use it?
What happened there is that read reads from stdin. When you put it at the end of a pipe, it read from that pipe.
So your find becomes
file1
file2
and so on; read reads that and replaces x successively with file1 then file2, and so your loop becomes
rm "file1"
rm "file2"
and sure enough, that rm's every file starting at the current directory ".".
A couple hints.
You didn't need the "/".
It's better and safer to say
find . -type f
because should you happen to type ". /" (ie, dot SPACE slash) find will start at the current directory and then go look starting at the root directory. That trick, given the right privileges, would delete every file in the computer. "." is already the name of a directory; you don't need to add the slash.
The find or rm commands will do this
It sounds like what you wanted to do was go through all the files in all the directories starting at the current directory ".", and have it ASK if you want to delete it. You could do that with
find . -type f -exec rm -i {} \;
or
find . -type f -ok rm {} \;
and not need a loop at all. You can also do
rm -r -i *
and get nearly the same effect, except that it will try to delete directories too. If the directory is empty, that'll even work.
Another thought
Come to think of it, unless you have a LOT of files, you could also do
rm -i `find . -type f`
Now the find in backquotes will become a bunch of file names on the command line, and the '-i' interactive flag on rm will ask the yes or no question.
Charlie Martin gives you a good dissection and explanation of what went wrong with your specific example, but doesn't address the general question of:
When should you use the read command?
The answer to that is - when you want to read successive lines from some file (quite possibly the standard output of some previous sequence of commands in a pipeline), possibly splitting the lines into several separate variables. The splitting is done using the current value of '$IFS', which normally means on blanks and tabs (newlines don't count in this context; they separate lines). If there are multiple variables in the read command, then the first word goes into the first variable, the second into the second, ..., and the residue of the line into the last variable. If there's only one variable, the whole line goes into that variable.
There are many uses. This is one of the simpler scripts I have that uses the split option:
#!/bin/ksh
#
# #(#)$Id: mkdbs.sh,v 1.4 2008/10/12 02:41:42 jleffler Exp $
#
# Create basic set of databases
MKDUAL=$HOME/bin/mkdual.sql
ELEMENTS=$HOME/src/sqltools/SQL/elements.sql
cat <<! |
mode_ansi with log mode ansi
logged with buffered log
unlogged
stores with buffered log
!
while read dbs logging
do
if [ "$dbs" = "unlogged" ]
then bw=""; cw=""
else bw="-ebegin"; cw="-ecommit"
fi
sqlcmd -xe "create database $dbs $logging" \
$bw -e "grant resource to public" -f $MKDUAL -f $ELEMENTS $cw
done
The cat command with a here-document has its output sent to a pipe, so the output goes into the while read dbs logging loop. The first word goes into $dbs and is the name of the (Informix) database I want to create. The remainder of the line is placed into $logging. The body of the loop deals with unlogged databases (where begin and commit do not work), then run a program sqlcmd (completely separate from the Microsoft new-comer of the same name; it's been around since about 1990) to create a database and populate it with some standard tables and data - a simulation of the Oracle 'dual' table, and a set of tables related to the 'table of elements'.
Other scripts that use the read command are bigger (by far), but generally read lines containing one or more file names and some other attributes of relevance, and then apply an appropriate transform to the files using the attributes.
Osiris JL: file * | grep 'sh.*script' | sed 's/:.*//' | xargs wgrep read
esqlcver:read version letter
jlss: while read directory
jlss: read x || exit
jlss: read x || exit
jlss: while read file type link owner group perms
jlss: read x || exit
jlss: while read file type link owner group perms
kb: while read size name
mkbod: while read directory
mkbod:while read dist comp
mkdbs:while read dbs logging
mkmsd:while read msdfile master
mknmd:while read gfile sfile version notes
publictimestamp:while read name type title
publictimestamp:while read name type title
Osiris JL:
'Osiris JL: ' is my command line prompt; I ran this in my 'bin' directory. 'wgrep' is a variant of grep that only matches entire words (to avoid words like 'already'). This gives some indication of how I've used it.
The 'read x || exit' lines are for an interactive script that reads a response from standard input, but exits if the command gets EOF (for example, if standard input comes from /dev/null).