I want to download a website using it's address as a download limiter.
To be precise..
The website address is http://das.sdss.org/spectro/1d_26/
It contains around 2700 subsites with data. I want to limit the recursive download of all files in a way that I can download sites from:
http://das.sdss.org/spectro/1d_26/0182/
to
http://das.sdss.org/spectro/1d_26/0500/
Using this tutorial I have made a wget command:
wget64 -r -nH -np -N http://das.sdss.org/spectro/1d_26/{0182..0500}/
but the last bit of address gives me 404 error.
Is there a mistake in my command or is the tutorial faulty?
P.S. I know it's possible to achieve with -I lists but I want to do it this way if it's possible.
Related
I have a mini program/server built on one of my computers (Machine1) and I am trying to create or overwrite a file through cURL on another computer (Machine2). So Machine2 is connected to Machine1. Ive been looking through cURL's documentation for command that will do this but have had no luck and as well on stack overflow.
https://curl.haxx.se/docs/manpage.html
I have also tried the examples on this SO post:
HTTP POST and GET using cURL in Linux
Any idea as to what the command might be through command prompt? (equivalent of a POST command). I have tried so far using -O, -K, -C and a multitude of others which have not worked.
In command line, all you need to do is using curl --form to simulate a multipart/form-data POST request:
curl --form "testfile=#thefilename.jpg" http://<Machine2>/<Path>
testfile is the field name used for form, if you don't care, just use any english word.
# is used here to make file thefilename.jpg get attached in the post as a file upload. Refer to curl man doc.
In server side, URL http://<Machine2>/<Path> should be listened. When curl send the previous POST request, server side program should get it, extract the attached file (thefilename.jpg), and save to disk.
I am trying to use wget to download a file from a http link that is password protected. I am using the following syntax:
wget --http-user=user --http-password=xxxxxx http://......
Am I using the right syntax? Should user and password be surrounded by quotes or double quotes?
I did this a few years ago and luckily found the script in a backup I still have.
I remember it was a two-stage process.
The first step is to get and store the cookie(s):
wget --keep-session-cookies --save-cookies nameofcookiesfile.txt --post-data 'email=my.email#address.com&password=mypassword123' https://web.site.com/redirectLogin -O login.html
The second is to use those cookies to get the file/page you need:
wget --load-cookies nameofcookiesfile.txt -p http://web.site.com/section/ -O savedoutputfile.html -nv
These are the commands exactly as I used them (except I have changed usernames, passwords, filenames and websites). I also came across this link which may be of some assistance particularly the "referer" part:
http://www.linuxscrew.com/2012/03/20/wget-cookies/
Hope this helps or at least gives someone a starting point.
I'm trying to get terminal to upload a file for me, in this case: file.txt
Unfortunately, it won't work, no matter what I try.
#!/bin/bash
HOST=*
USER=*
PASS=*
# I'm 100% sure the host/user/pass are correct.
#Terminal also connects with the host provided
ftp -inv $HOST << EOF
user $USER $PASS
cd /Users/myname/Desktop
get file.txt #which is located on my desktop
bye
EOF
I've tried 100 different scripts but it just won't upload :(
This is the output after saving to an .sh file, chmod +x and sudo the .sh file:
Connected to *hostname*.
220 ProFTPD 1.3.4b Server ready.
331 Password required for *username*
230 User *username* logged in
Remote system type is UNIX.
Using binary mode to transfer files.
550 /Users/myname/Desktop: No such file or directory
local: file.txt remote: file.txt
229 Entering Extended Passive Mode (|||35098|)
550 file.txt: No such file or directory
221 Goodbye.
myname:Desktop Myname$
I've browsed through many other topics about the same issue here, but I just can't figure it out. I've started playing with UNIX since this morning, so excuse me for this (probably) foolish question.
Try:
#!/bin/bash
HOST=*
USER=*
PASS=*
# I'm 100% sure the host/user/pass are correct.
#Terminal also connects with the host provided
cd /Users/myname/Desktop # Go the the local folder where the file is located in
ftp -inv $HOST << EOF
user $USER $PASS
cd /User/$USER/Desktop # Go to the folder in which you want to upload the file
put file.txt #which is located on my desktop
bye
EOF
So use put and make sure your file is the current working directory and the remote directory exists.
You are using get but talk about an upload. Probably you just want to use put?
Anyway, I'm not sure this can be done using a basic ftp client. I'm always using ncftp for things like this. This comes with command line utilities like ncftpput which accept command line arguments and options to perform the task.
Alfe is right, you need to use put <filename> to upload a file to FTP. You can find a quick guide here. It should be possible using the basic FTP tool but I would also recommend ncftp :-)
You need to use put to upload a file.
Is it possible to recursively download files from specific TLDs with wget?
Specifically, I'm trying to download the full text of the Code of Massachusetts Regulations. The actual text of the regulations is stored in multiple files across multiple domains—so I'd like to start the recursive download from the index page, but only follow links to .gov and .us domains.
With help from the wget documentation on spanning hosts, I was able to make this work with the -H and -D flags:
wget -r -l5 -H -D.us,.gov http://www.lawlib.state.ma.us/source/mass/cmr/index.html
Greetings to everyone,
I'm on OSX. I use the terminal a lot as a habit from my Linux old days that I never surpassed. I wanted to download the files listed in this http server: http://files.ubuntu-gr.org/ubuntistas/pdfs/
I select them all with the mouse, put them in a txt files and then gave the following command on the terminal:
for i in `cat ../newfile`; do wget http://files.ubuntu-gr.org/ubuntistas/pdfs/$i;done
I guess it's pretty self explanatory.
I was wondering if there's any easier, better, cooler way to download this "linked" pdf files using wget or curl.
Regards
You can do this with one line of wget as follows:
wget -r -nd -A pdf -I /ubuntistas/pdfs/ http://files.ubuntu-gr.org/ubuntistas/pdfs/
Here's what each parameter means:
-r makes wget recursively follow links
-nd avoids creating directories so all files are stored in the current directory
-A restricts the files saved by type
-I restricts by directory (this one is important if you don't want to download the whole internet ;)