Command Center

Q. How do I use wget ftp / http client tool to download files from password protected web page?

A. wget command supports username and password combo for both FTP and HTTP file retrieval.

Pass following option to wget command:
--user=userName: Your FTP/HTTP username
--password=passWord : Your HTTP/FTP password

These parameters can be overridden using the --ftp-user and --ftp-password options for FTP connections and the --http-user and --http-password options for HTTP connections.

Download a file called foo.pdf from theos.in:
$ wget --user=vivek --password='myPassword' http://theos.in/protected/area/foo.pdf





=========================================================================
WGET for Windows (win32) - current version: 1.11.4

updated April 25 2009

Read below to download and for some help with wget.
license information and compiler details
GNU wget

From the official wget homepage:

"GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without Xsupport, etc."

While you can get Windows binaries from Heiko Herold's page, the binaries here are tweaked a bit so they operate somewhat better on Windows.

The following changes, compared to the official distribution, were retained/added since 1.8.2:

* Statically linked with (masm optimized) OpenSSL 0.9.8k, which makes wget.exe completely stand-alone.
* Compressed with UPX 1.07 for smaller filesize

It seems the rfc1738 problems on Windows (see below) were fixed in wget 1.9, so there is no longer a need to edit the source code.
OpenSSL

Wget now supports Secure Socket Layer (SSL, https://...) among other things. Most available binaries are dynamically linked against OpenSSL, and require you to have a couple of dll's in your path. The binary on this site is statically linked with OpenSSL (which makes it larger in size, but stand-alone).

Note the license addendum:

"In addition, as a special exception, the Free Software Foundation gives permission to link the code of its release of Wget with the OpenSSL project's "OpenSSL" library (or with modified versions of it that use the same license as the "OpenSSL" library), and distribute the linked executables. You must obey the GNU General Public License in all respects for all of the code used other than "OpenSSL". If you modify this file, you may extend this exception to your version of the file, but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version."

Furthermore, compiling (statically) with OpenSSL is cumbersome in VC++. If you were to try this yourself, necessary steps would include:

* Getting the OpenSSL source and untarring it somewhere besides wget source.
* Configuring for win32 - this involves Perl (e.g. ActivePerl). (ms\do_masm.bat will help)
* Compiling static libraries (ms\nt.mak, not ms\ntdll.mak)...
* runtime libraries should be matched between recent OpenSSL and wget source distributions, but do check (should be /MT).
* configure.bat --msvc in the wget tree.
* Adding the OpenSSL include and lib directories to compiler include and linker library paths respectively.

Downloads!

Latest version is 1.11.4, compiled with MS Visual C++ and linked with OpenSSL 0.9.8k. Page will be updated with new releases of wget. Wget tends to see a couple of incremental bugfix releases (i.e. 1.11.x). I am currently using wget 1.11.x on a daily basis.

>> wget.exe (401408 bytes) << : win32 binary with OpenSSL support.

MD5: bd126a7b59d5d1f97ba89a3e71425731
SHA1: 457b1cd985ed07baffd8c66ff40e9c1b6da93753

Previous versions
wget 1.10.2 (Dec 2 2005)

wget-1.10.2.exe (332800 bytes): win32 binary compiled with MS Visual C++ and with OpenSSL 0.9.7i support.
wget 1.9.1 (Jun 03 2004)

wget-1.9.1.exe (308736 bytes): win32 binary compiled with MS Visual C++ 6.0 and with OpenSSL 0.9.7c support.
Usage

wget is a command line program. You start it from the command prompt, either command.com in Windows 9x/Me or cmd.exe in Windows 2000/XP. The command prompt can be found in the Start Menu (Accessories).

wget.exe must be placed in your path (e.g. c:\windows)

To retrieve a file: wget http://users.ugent.be/~bpuype/wget/wget.exe

wget screenshot
wget in action...
Basic options

Display all help: wget --help

Completely mirror a site: wget -mr http://...
-m: mirror
-r: recursive
Mirror without following links to other servers, parent directories: wget -mrnp http://...
-np: no-parent

Retrieve a html file and convert relative links to absolute ones: wget -k http://users.ugent.be/~bpuype/wget
-k: 'k'onvert links

Resume partially downloaded files (if supported by the server): wget -c http://...
-c: continue

Read url's from a file and retrieve them: wget -i file_with_urls.txt
-i: input-file
Ask for url's (read from stdin): wget -i -. Enter url's on the command line, press enter after each url, and terminate with ^Z (press CTRL-Z) on an empty line.
FTP

--glob=off
Don't treat (, *, ? etc. as globbing characters. Use when transfering files with names that contain these characters.

--passive-ftp
Use passive mode for data connection (try this if you're behind a firewall, NAT box...)
Proxy

To make wget use a proxy, you must set up an environment variable before using wget. Type this at the command prompt:

set http_proxy=http://proxy.myprovider.net:8080

...where you use the correct proxy hostname and port for your ISP or network. You can use ftp_proxy to proxy ftp requests.

--proxy=on
--proxy=off

Turn proxy usage on/off once variable is set; default is on when variable is present.
Passwords

To retrieve with passwords (http or ftp), you can use the following url syntax:

wget http://username:password@www.example.net/somedir/somefile
wget ftp://username:password@ftp.example.net/somedir/somefile

Additionally, you can also use --http-user, --http-password as well as --ftp-user, --ftp-password:

wget ftp://ftp.example.net/somefile --ftp-user=username --ftp-password=password

If username or password contain non-alphanumeric characters, you need to escape them when passing them in urls (rfc1738 %HH) syntax. For example, with a username of user@domain and password of pass, your url becomes http://user%40domain:pass@www.example.net/somefile. When using escaped urls in batch files, remember that % itself is a special character, and needs to be escaped itself (by using %% instead of %).