Q. How do I use wget ftp / http client tool to download files from password protected web page?
A. wget command supports username and password combo for both FTP and HTTP file retrieval.
Pass following option to wget command:
--user=userName: Your FTP/HTTP username
--password=passWord : Your HTTP/FTP password
These parameters can be overridden using the --ftp-user and --ftp-password options for FTP connections and the --http-user and --http-password options for HTTP connections.
Download a file called foo.pdf from theos.in:
$ wget --user=vivek --password='myPassword' http://theos.in/protected/area/foo.pdf
WGET for Windows (win32) - current version: 1.11.4
updated April 25 2009
Read below to download and for some help with wget.
license information and compiler details
From the official wget homepage:
"GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without Xsupport, etc."
While you can get Windows binaries from Heiko Herold's page, the binaries here are tweaked a bit so they operate somewhat better on Windows.
The following changes, compared to the official distribution, were retained/added since 1.8.2:
* Statically linked with (masm optimized) OpenSSL 0.9.8k, which makes wget.exe completely stand-alone.
* Compressed with UPX 1.07 for smaller filesize
It seems the rfc1738 problems on Windows (see below) were fixed in wget 1.9, so there is no longer a need to edit the source code.
Wget now supports Secure Socket Layer (SSL, https://...) among other things. Most available binaries are dynamically linked against OpenSSL, and require you to have a couple of dll's in your path. The binary on this site is statically linked with OpenSSL (which makes it larger in size, but stand-alone).
Note the license addendum:
"In addition, as a special exception, the Free Software Foundation gives permission to link the code of its release of Wget with the OpenSSL project's "OpenSSL" library (or with modified versions of it that use the same license as the "OpenSSL" library), and distribute the linked executables. You must obey the GNU General Public License in all respects for all of the code used other than "OpenSSL". If you modify this file, you may extend this exception to your version of the file, but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version."
Furthermore, compiling (statically) with OpenSSL is cumbersome in VC++. If you were to try this yourself, necessary steps would include:
* Getting the OpenSSL source and untarring it somewhere besides wget source.
* Configuring for win32 - this involves Perl (e.g. ActivePerl). (ms\do_masm.bat will help)
* Compiling static libraries (ms\nt.mak, not ms\ntdll.mak)...
* runtime libraries should be matched between recent OpenSSL and wget source distributions, but do check (should be /MT).
* configure.bat --msvc in the wget tree.
* Adding the OpenSSL include and lib directories to compiler include and linker library paths respectively.
Latest version is 1.11.4, compiled with MS Visual C++ and linked with OpenSSL 0.9.8k. Page will be updated with new releases of wget. Wget tends to see a couple of incremental bugfix releases (i.e. 1.11.x). I am currently using wget 1.11.x on a daily basis.
>> wget.exe (401408 bytes) << : win32 binary with OpenSSL support.
wget 1.10.2 (Dec 2 2005)
wget-1.10.2.exe (332800 bytes): win32 binary compiled with MS Visual C++ and with OpenSSL 0.9.7i support.
wget 1.9.1 (Jun 03 2004)
wget-1.9.1.exe (308736 bytes): win32 binary compiled with MS Visual C++ 6.0 and with OpenSSL 0.9.7c support.
wget is a command line program. You start it from the command prompt, either command.com in Windows 9x/Me or cmd.exe in Windows 2000/XP. The command prompt can be found in the Start Menu (Accessories).
wget.exe must be placed in your path (e.g. c:\windows)
To retrieve a file: wget http://users.ugent.be/~bpuype/wget/wget.exe
wget in action...
Display all help: wget --help
Completely mirror a site: wget -mr http://...
Mirror without following links to other servers, parent directories: wget -mrnp http://...
Retrieve a html file and convert relative links to absolute ones: wget -k http://users.ugent.be/~bpuype/wget
-k: 'k'onvert links
Resume partially downloaded files (if supported by the server): wget -c http://...
Read url's from a file and retrieve them: wget -i file_with_urls.txt
Ask for url's (read from stdin): wget -i -. Enter url's on the command line, press enter after each url, and terminate with ^Z (press CTRL-Z) on an empty line.
Don't treat (, *, ? etc. as globbing characters. Use when transfering files with names that contain these characters.
Use passive mode for data connection (try this if you're behind a firewall, NAT box...)
To make wget use a proxy, you must set up an environment variable before using wget. Type this at the command prompt:
...where you use the correct proxy hostname and port for your ISP or network. You can use ftp_proxy to proxy ftp requests.
Turn proxy usage on/off once variable is set; default is on when variable is present.
To retrieve with passwords (http or ftp), you can use the following url syntax:
Additionally, you can also use --http-user, --http-password as well as --ftp-user, --ftp-password:
wget ftp://ftp.example.net/somefile --ftp-user=username --ftp-password=password
If username or password contain non-alphanumeric characters, you need to escape them when passing them in urls (rfc1738 %HH) syntax. For example, with a username of user@domain and password of pass, your url becomes http://user%40domain:firstname.lastname@example.org/somefile. When using escaped urls in batch files, remember that % itself is a special character, and needs to be escaped itself (by using %% instead of %).