My url (http://myhost.com/getuser/Default.aspx?username=b772643) returns the following line of of info always:
John, Thomas;John.B.Thomas#Company.com
I wish to read this line using a shell or bash script without wget/lynx. I'm in a situation where I cannot use any other utility, the perl language etc.
Curl or wget are obviously better for the job but for the record bash and Unix standard commands (cat & printf) can do the job.
ksh introduced shell network internal handling and this has been adopted by bash.
#!/bin/bash
exec 5<> /dev/tcp/myhost.com/80
cat <&5 &
printf "GET /getuser/Default.aspx?username=b772643 HTTP/1.0\r\n\r\n" >&5
One liner:
(echo 'GET /getuser/Default.aspx?username=b772643' > /dev/tcp/myhost.com/80);
so
curl "http://myhost.com/getuser/Default.aspx?username=b772643"
curl "http://myhost.com/getuser/Default.aspx?username=b772643"| sed 's/\(.*\);\(.*\)/\2 \1/' | while read email name; do echo =$email=$name=; done
You could use :
curl "http://myhost.com/getuser/Default.aspx?username=b772643"
and extract the datas from what is returned :)
Related
Given the following code
pingval=$(ping6 -c 1 "$url")
time=$($pingval| tail -1| awk '{print $4}'|cut -d '/' -f 2)
Could anyone tell me why I'm receiving a PING: command not found error?
My solution to this was to simply use
time=$(echo pingval|......)
I'm presuming you're wrapping this in a shell script? Is the ping6 command in the search path for the shell you're using to execute that script?
Also, stating the obvious: ping6 is also in your path?
which ping6
Following html mail using mailx command is working from shell terminal, but the same command is not working from shell script.
mailx -s "$(echo -e "${sub} TRP OF ${system} \nContent-Type: text/html")" example#gmail.com < TRP.html
I guess it is some small escape character error, but not sure what it is.
Can any one help here?
Perhaps your vars sub / system are only known in your current environment.
When your sript is called mymail, try
. mymail
(Start with a dot),
or first export your vars.
When these suggestions fail, debug:
use set -x or temporary put an "echo -e" in front of your line.
Currently I am using the following command to catch the Error String in the MY_FILE_NAME*.log
Currentdate=`date -u +"%Y/%m/%d"`
YEST=`TZ=XYZ+24 date '+%Y/%m/%d'`
grep -E "$Currentdate|$YEST" MY_FILE_NAME*.log | grep "Type: Error"
This command is generating huge data with the string "Type: Error" with redundancy in the same error type (in my case the same error is displayed like 100 times)
I want the error strings of same type to be displayed only once
If using GNU/Linux try the '-m' switch
grep -m 1 -E "$Currentdate|$YEST" MY_FILE_NAME*.log | grep "Type: Error"
In the GNU version of grep, the '-m ' switch stops reading the input file after matches are found. This feature does not exist in the older Unix grep on which AIX and similar are built.
If on AIX where there is no -m or -B see this StackOvreflow post
I have a list URLs in a file called urls.txt. Each line contains 1 URL. I want to download all of the files at once using cURL. I can't seem to get the right one-liner down.
I tried:
$ cat urls.txt | xargs -0 curl -O
But that only gives me the last file in the list.
This works for me:
$ xargs -n 1 curl -O < urls.txt
I'm in FreeBSD. Your xargs may work differently.
Note that this runs sequential curls, which you may view as unnecessarily heavy. If you'd like to save some of that overhead, the following may work in bash:
$ mapfile -t urls < urls.txt
$ curl ${urls[#]/#/-O }
This saves your URL list to an array, then expands the array with options to curl to cause targets to be downloaded. The curl command can take multiple URLs and fetch all of them, recycling the existing connection (HTTP/1.1), but it needs the -O option before each one in order to download and save each target. Note that characters within some URLs ] may need to be escaped to avoid interacting with your shell.
Or if you are using a POSIX shell rather than bash:
$ curl $(printf ' -O %s' $(cat urls.txt))
This relies on printf's behaviour of repeating the format pattern to exhaust the list of data arguments; not all stand-alone printfs will do this.
Note that this non-xargs method also may bump up against system limits for very large lists of URLs. Research ARG_MAX and MAX_ARG_STRLEN if this is a concern.
A very simple solution would be the following:
If you have a file 'file.txt' like
url="http://www.google.de"
url="http://www.yahoo.de"
url="http://www.bing.de"
Then you can use curl and simply do
curl -K file.txt
And curl will call all Urls contained in your file.txt!
So if you have control over your input-file-format, maybe this is the simplest solution for you!
Or you could just do this:
cat urls.txt | xargs curl -O
You only need to use the -I parameter when you want to insert the cat output in the middle of a command.
xargs -P 10 | curl
GNU xargs -P can run multiple curl processes in parallel. E.g. to run 10 processes:
xargs -P 10 -n 1 curl -O < urls.txt
This will speed up download 10x if your maximum download speed if not reached and if the server does not throttle IPs, which is the most common scenario.
Just don't set -P too high or your RAM may be overwhelmed.
GNU parallel can achieve similar results.
The downside of those methods is that they don't use a single connection for all files, which what curl does if you pass multiple URLs to it at once as in:
curl -O out1.txt http://exmple.com/1 -O out2.txt http://exmple.com/2
as mentioned at https://serverfault.com/questions/199434/how-do-i-make-curl-use-keepalive-from-the-command-line
Maybe combining both methods would give the best results? But I imagine that parallelization is more important than keeping the connection alive.
See also: Parallel download using Curl command line utility
Here is how I do it on a Mac (OSX), but it should work equally well on other systems:
What you need is a text file that contains your links for curl
like so:
http://www.site1.com/subdirectory/file1-[01-15].jpg
http://www.site1.com/subdirectory/file2-[01-15].jpg
.
.
http://www.site1.com/subdirectory/file3287-[01-15].jpg
In this hypothetical case, the text file has 3287 lines and each line is coding for 15 pictures.
Let's say we save these links in a text file called testcurl.txt on the top level (/) of our hard drive.
Now we have to go into the terminal and enter the following command in the bash shell:
for i in "`cat /testcurl.txt`" ; do curl -O "$i" ; done
Make sure you are using back ticks (`)
Also make sure the flag (-O) is a capital O and NOT a zero
with the -O flag, the original filename will be taken
Happy downloading!
As others have rightly mentioned:
-cat urls.txt | xargs -0 curl -O
+cat urls.txt | xargs -n1 curl -O
However, this paradigm is a very bad idea, especially if all of your URLs come from the same server -- you're not only going to be spawning another curl instance, but will also be establishing a new TCP connection for each request, which is highly inefficient, and even more so with the now ubiquitous https.
Please use this instead:
-cat urls.txt | xargs -n1 curl -O
+cat urls.txt | wget -i/dev/fd/0
Or, even simpler:
-cat urls.txt | wget -i/dev/fd/0
+wget -i/dev/fd/0 < urls.txt
Simplest yet:
-wget -i/dev/fd/0 < urls.txt
+wget -iurls.txt
I would like to extract the first part of this hostname testsrv1
from testsrv1.main.corp.loc.domain.com in UNIX, within a shell script.
What command can I use? It would be anything before the first period .
Do you have the server name in a shell variable? Are you using a sh-like shell? If so,
${SERVERNAME%%.*}
will do what you want.
You can use cut:
echo "testsrv1.main.corp.loc.domain.com" | cut -d"." -f1
To build upon pilcrow's answer, no need for new variable, just use inbuilt $HOSTANME.
echo $HOSTNAME-->my.server.domain
echo ${HOSTNAME%%.*}-->my
Tested on two fairly different Linux's.
2.6.18-371.4.1.el5, GNU bash, version 3.2.25(1)-release (i386-redhat-linux-gnu)
3.4.76-65.111.amzn1.x86_64, GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
try the -s switch:
hostname -s
I use command cut, awk, sed or bash variables
Operation
Via cut
[flying#lempstacker ~]$ echo "testsrv1.main.corp.loc.domain.com" | cut -d. -f1
testsrv1
[flying#lempstacker ~]$
Via awk
[flying#lempstacker ~]$ echo "testsrv1.main.corp.loc.domain.com" | awk -v FS='.' '{print $1}'
testsrv1
[flying#lempstacker ~]$
Via sed
[flying#lempstacker ~]$ echo "testsrv1.main.corp.loc.domain.com" | sed -r 's#([^.]*).(.*)#\1#g'
testsrv1
[flying#lempstacker ~]$
Via Bash Variables
[flying#lempstacker ~]$ hostName='testsrv1.main.corp.loc.domain.com'
[flying#lempstacker ~]$ echo ${hostName%%.*}
testsrv1
[flying#lempstacker ~]$
You could have used "uname -n" to just get the hostname only.
You can use IFS to split text by whichever token you want. For domain names, we can use the dot/period character.
#!/usr/bin/env sh
shorthost() {
# Set IFS to dot, so that we can split $# on dots instead of spaces.
local IFS='.'
# Break up arguments passed to shorthost so that each domain zone is
# a new index in an array.
zones=($#)
# Echo out our first zone
echo ${zones[0]}
}
If this is in your script then, for instance, you'll get test when you run shorthost test.example.com. You can adjust this to fit your use case, but knowing how to break the zones into the array is the big thing here, I think.
I wanted to provide this solution, because I feel like spawning another process is overkill when you can do it easily and completely within your shell with IFS. One thing to watch out for is that some users will recommend doing things like hostname -s, but that doesn't work in the BSD userland. For instance, MacOS users don't have the -s flag, I don't think.
Assuming the variable $HOSTNAME exists, so try echo ${HOSTNAME%%.*} to get the top-most part of the full-qualified hostname. Hope it helps.
If interested, the hint is from the below quoted partial /etc/bashrc on a REHL7 host:
if [ -e /etc/sysconfig/bash-prompt-screen ]; then
PROMPT_COMMAND=/etc/sysconfig/bash-prompt-screen
else
PROMPT_COMMAND='printf "\033k%s#%s:%s\033\\" "${USER}" "${HOSTNAME%%.*}" "${PWD/#$HOME/~}"'
fi
;; ... ```