I had been using a proxy for a long time. Now I need to remove it. I have forgotten how I have added the proxy to wget. Can someone please help me get back to the normal wget where it doesn't use any proxy. As of now, I'm using
wget <link> --proxy=none
But I'm facing a problem when I'm installing using a pre-written script. It's painstaking to search all through the scripts and change each command.
Any simpler solution will be very much appreciated.
Thanks
Check your
~/.wgetrc
/etc/wgetrc
and remove proxy settings.
Or use wget --no-proxy command line option to override them.
In case your OS is alpine/busybox then the wget might vary from the one used by #Logu.
There the correct command is
wget --proxy off http://server:port/
Running wget --help outputs:
/ # wget --help
BusyBox v1.31.1 () multi-call binary.
Usage: wget [-c|--continue] [--spider] [-q|--quiet] [-O|--output-document FILE]
[-o|--output-file FILE] [--header 'header: value'] [-Y|--proxy on/off]
[-P DIR] [-S|--server-response] [-U|--user-agent AGENT] [-T SEC] URL...
Retrieve files via HTTP or FTP
--spider Only check URL existence: $? is 0 if exists
-c Continue retrieval of aborted transfer
-q Quiet
-P DIR Save to DIR (default .)
-S Show server response
-T SEC Network read timeout is SEC seconds
-O FILE Save to FILE ('-' for stdout)
-o FILE Log messages to FILE
-U STR Use STR for User-Agent header
-Y on/off Use proxy
Related
I use Wget to download all files in a directory of a site. But it was interrupted. How could I make it to continue?
You want wget -c url, where -c stands for continue and url is the URL of the interrupted file.
I have a URL in my custom module which runs a long script. If i call url via wget it downloads the page content. It doesn't run the script. How to do it?
I would have thought that even though it downloaded the page it would still run the script.
To run without downloading the file use:
wget -O - -q -t 1 http://example.com/path/to/file.php
From memory:
-O and the hyphen are redirecting the output so it's not saved to a file.
-q is for quiet
-t is the number of attempts.
You can use man wget to look any more options up.
I need to identify a daemon process that is writing to a log file periodically. The problem is that I dont have any idea which process is doing the job, and I need to show some progress to the client by tomorrow. Anybody has any clue?
I have already sorted out the daemon processes running in the system with the help of the PPID. Any help would be appreciated.
Also I think it is possible (rarely) for a daemon not to have a PPID as 1. How can we find it out then?
Try the fuser command on your log file, which will display the PIDs of processes using it.
Example:
$ fuser file.log
file.log: 3065
lsof gives a list of open files with the processes.
So lsof | grep <filename> should help you.
You can use auditctl.
# sudo apt-get install auditd
# sudo /sbin/auditctl -w /path/to/file -p war -k hosts-file
-w watch /etc/hosts
-p warx watch for write, attribute change, execute or read events
-k hosts-file is a search key.
# sudo /sbin/ausearch -f /path/to/file | more
Gives output such as
type=UNKNOWN[1327] msg=audit(1459766547.822:130): proctitle=2F7573722F7362696E2F61706163686532002D6B007374617274
type=PATH msg=audit(1459766547.822:130): item=0 name="/path/to/file" inode=141561 dev=08:00 mode=0100444 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1459766547.822:130): cwd="/"
type=SYSCALL msg=audit(1459766547.822:130): arch=c000003e syscall=2 success=yes exit=41 a0=7f3c23034cd0 a1=80000 a2=1b6 a3=8 items=1 ppid=24452 pid=6797 auid=42949672
95 uid=33 gid=33 euid=33 suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=(none) ses=4294967295 comm="apache2" exe="/usr/sbin/apache2" key="hosts-file"
I'd like to download a web pages while supplying URLs from stdin. Essentially one process continuously produces URLs to stdout/file and I want to pipe them to wget or curl. (Think about it as simple web crawler if you want).
This seems to work fine:
tail 1.log | wget -i - -O - -q
But when I use 'tail -f' and it doesn't work anymore (buffering or wget is waiting for EOF?):
tail -f 1.log | wget -i - -O - -q
Could anybody provide a solution using wget, curl or any other standard Unix tool? Ideally I don't won't want to restart wget in the loop, just keep it running downloading URLs as they come.
What you need to use is xargs. E.g.
tail -f 1.log | xargs -n1 wget -O - -q
Use xargs which converts stdin to argument.
tail 1.log | xargs -L 1 wget
Try piping the tail -f through python -c $'import pycurl;c=pycurl.Curl()\nwhile True: c.setopt(pycurl.URL,raw_input().strip()),c.perform()'
This gets curl (well, you probably meant the command-line curl and I'm calling it as a library from a Python one-liner, but it's still curl) to fetch each URL immediately, while still taking advantage of keeping the socket to the server open if you're requesting multiple URLs from the same server in sequence. It's not completely robust though: if one of your URLs is duff, the whole command will fail (you might want to make it a proper Python script and add try / except to handle this), and there's also the small detail that it will throw EOFError on EOF (but I'm assuming that's not important if you're using tail -f).
The effective way is to avoid using xargs, if downloading files from the same web server:
wget -q -N -i - << EOF
http://sitename/dir1/file1
http://sitename/dir2/file2
http://sitename/dir3/file3
EOF
I'm using plink to run a command on a Unix remote machine.
The command is:
ls -1trd testegrep.txt |tail -1 |xargs tail -f| grep 's';
The way I'm sending this command is by using a file with a set of commands like:
plink.exe -ssh -t -l user -pw pwd tst.url.pt -m commands.out
When I run the command this way the plink does not receive any input. It seems that is waiting for input.
But if I run:
plink.exe -ssh -t -l user -pw pwd tst.url.pt "ls -1trd testegrep.txt |tail -1 |xargs tail -f| grep 's';"
I get the expected result.
I'm not using the plink with a file with the command because I choose so. I'm using a test automation software that allows me to run tests on remote hosts and this is the way the tool works.
Any thoughts on what is going wrong?
I tested the command you provided and it worked without problems.
Maybe the problem is related to:
The server's host key is not cached in the registry.
The path to the file is not correct.
The file is empty.
include server hostkey
most importantly, you need to include the unix profile using the -m paramater
You can include all your commands in the same file where the profile is kept also.
$Output = ((plink.exe -hostkey hostkey -l UNAME -i SSHKEY -P 22 -ssh server -batch -m PROFILE) | ? {$_ -ne ""})