web -- transfer files to or from a Web server

SYNOPSIS

web [-Adeirv] [-a user-agent] [-B bufsize] [-h header] [-p password] [-H GSSAPI-SPN | -spn GSSAPI-SPN] [-t seconds | -timeout seconds] [-u user] [-f certfile] [-x proxyurl [-P proxy-passwd] [-U proxy-user]] [-X auth_excl_list] get URL [file]

web [-Adeirv] [-a user-agent] [-B bufsize] [-c content-type] [-H GSSAPI-SPN | -spn GSSAPI-SPN] [-h header] [-p password] [-t seconds | -timeout seconds] [-u user] [-f certfile] [-x proxyurl [-P proxy-passwd] [-U proxy-user]] [-X auth_excl_list] put URL file

web [-AdeirVv] [-a user-agent] [-B bufsize] [-c content-type] [-H GSSAPI-SPN | -spn GSSAPI-SPN] [-h header] [-p password] [-t seconds | -timeout seconds] [-u user] [-f certfile] [-x proxyurl [-P proxy-passwd] [-U proxy-user]] [-X auth_excl_list] post URL file

web [-AdeirVv] [-a user-agent] [-B bufsize] [-H GSSAPI-SPN | -spn GSSAPI-SPN] [-h header] [-p password] [-t seconds | -timeout seconds] [-u user] [-f certfile] [-x proxyurl [-P proxy-passwd] [-U proxy-user]] [-X auth_excl_list] index URL

web [-AdeirVv] [-B bufsize] [-H GSSAPI-SPN | -spn GSSAPI-SPN] [-h header] [-p password] [-t seconds | -timeout seconds] [-u user] [-f certfile] [-x proxyurl [-P proxy-passwd] [-U proxy-user]] [-X auth_excl_list] -k keyword HTTP-arg

DESCRIPTION

The web command provides an interface to HTTP and FTP requests, letting you transfer documents to or from a Web, ftp, or news server. web can access http: URLs, secure https: URLs (using SSL), news: URLs, snews: URLs (using SSL) and ftp: URLs.

For secure URLs (both https: and snews:), a file of standard certificate authorities is loaded by openssl from ROOTDIR/etc/openssl/cert.pem. This file must be populated before openssl will initialize. It is not provided by the PTC MKS Toolkit installation, rather a script ROOTDIR/etc/openssl/mkcabundle.pl is provided to populate the file with a sample set of certificates dynamically downloaded from the mozilla trusted certificates list. You may of course choose to provide your own list of trusted root certificates. You can supply additional certificate authorities by adding a cert.pem file to your home directory or by specifying the -f option.

For ftp: URLs, the URL has the form:

ftp://[username[:password]@]host[:port][/[/|%2Fpath]

If you specify username without password, web prompts you for the password to use. Specifying a user name as part of the URL overrides one given with the -u option and the password associated with it (given as part of the URL or in answer to a prompt) overrides one given with the -p option.

When the path portion of such URLs is preceded by a single slash (/), it is assumed to be relative to either the user's home directory on the ftp server for named logins, or the public ftp directory for anonymous logins. (Note that these directories are dependent on the settings for the ftp server.) When path is preceded by a double slash (//), it is equivalent to issuing a CWD command with no arguments within ftp (to change to the user's home directory) and path is treated as relative to that directory. In most cases (depending on the server settings), this will be equivalent to using a single slash. Finally, when you precede path with /%2F, path is treated as being relative to root of the ftp server. For example, ftp://ftp.myserver.com/%2Fmyfile refers to a file called myfile in the root of the server, while ftp://ftp.myserver.com/myfile refers to a file called myfile in the user's home directory.

Options

-A

enables the passing of your current login credentials to the server (see Authentication below). If authentication fails (possibly, because either the client or server does not support the mechanism used to pass credentials), the user is prompted to provide a password.

-a user-agent

specifies the contents of the user-agent header for the HTTP request. If user-agent is not specified, "MKS Toolkit web" is used.

-B bufsize

Specifies the size of the buffer to be used for requests. This can be used to improve performance when transferring large files with the web command.

-c content-type

specifies the content-type of the document for PUT and POST requests. If content-type is not specified, text/HTML is used for PUT and application/x-www-form-urlencoded is used for POST.

-d

provides extra debugging information.

-e

displays, on the standard output, the status line (for example, "200 OK", "404 Not Found", or "401 Access Denied") to indicate success of failure before any other output from the server. returns an error line to standard output. Normally, web does not display the header line, unless it indicates an error.

-f certfile

specifies an alternative .pem file name to use as a certificate authority. The specified file may have been created with the openssl utility or supplied by a recognized authority.

get URL [file]

fetches the contents of a URL. web stores the contents in a local file, if specified; otherwise, it writes the contents to standard output.

With ftp: URLs, the get command retrieves the file indicated by url in binary mode. If url indicates a directory, get retrieves the directory listing in ASCII mode.

With news: and snews: URLs, the information returned by get depends on the form of the URL specified.

A URL of the form news_server/news_group returns the contents of all message in the specified news group on the given news server.

A URL of the form news_server/news_group/message_num returns the contents of the specified message number from the given newsgroup. message_num can be a range of a message numbers (for example, 10-20), in which case, get returns the contents of all messages in that range.

Finally, a URL of the form news_server/message_ID returns the contents of the message with the specified message identifier. Message identifiers are unique to a message across all news servers and usually have a form similar to ab2qqq$6vh$1@hypnos.mkssoftware.com.

-H GSSAPI-SPN

-spn GSSAPI-SPN

Attempts to connect to the server with the Server Provider Name (SPN) specified by GSSAPI-SPN. When the Negotiate authentication mechanism is being used, the client may need to specify a SPN registered by the server.

When neither of these options is specified, web uses a SPN of HOST/hostname.

You can determine the SPN of a server by using the setspn utility (provided as part of the Windows support tools that can be installed from your Windows CD). The command:

setspn -L accountname

displays all the registered SPNs of the specified accountname. To display the list of registered SPNs associated with a given computer account, one would use:

setspn -L computername$

-h header

adds an arbitrary header to the HTTP request. You must quote headers which contain enclosed whitespace.

With ftp:, news:, and snews: URLs, this option only works when the URL is being accessed through a proxy.

-i

ignores certificate errors. Many sites on the internet do not have completely valid certificates. They may have incorrect information, invalid dates, and so on. This option lets you view information from these sites even though the certificate is invalid.

If the -v option or the -V is also specified, the ignored errors are displayed.

index URL

gets the listing of a given URL directory. Since web uses the HTTP INDEX request for this option, you must have permission to make this request of the URL's web server.

With ftp: URLs, index normally retrieves directory listings in ASCII mode, similar to the get command, although the index command is faster. However, if an ftp: URL is accessed through a proxy, index retrieves the directory listing an HTML Web page.

With news: and snews: URLs, the information returned by index depends on the form of the URL specified.

A URL of the form news_server returns a list of news groups on the specified news server.

A URL of the form news_server/news_group returns a list of all message in the specified news group on the given news server.

A URL of the form news_server/news_group/message_num returns the header information for the specified message number from the given newsgroup. message_num can be a range of a message numbers (for example, 10-20), in which case, index returns the header information for all messages in that range.

Finally, a URL of the form news_server/message_ID returns the header information for the message with the specified message identifier. Message identifiers are unique to a message across all news servers and usually have a form similar to ab2qqq$6vh$1@hypnos.mkssoftware.com.

-k keyword HTTP-argument

sends an arbitrary HTTP keyword, with some form of a URL as an argument.

When working with ftp: URLs, the -k option sends an ftp keyword. For example, -k DELE deletes a file, -k MKD creates a directory, and -k RMD removes a directory.

Because of specific response code, using this option with ftp URLs does not give useful return codes for success or failure. The -v (or -V) option returns all server correspondence and this must be parsed if knowledge of successful completion is required.

-P proxy-passwd

specifies the password to use if the proxy URL specified by -x requires you to log in.

-p password

adds a password to an Authentication header. If you don't use this option with the -u option, web prompts you for a password.

With ftp: URLs, the -u and -p options send a user name and password to the ftp server. If they are not specified, web sends an anonymous user name and e-mail style password if the ftp server allows anonymous connections. Remember, however, that anonymous connections are usually limited to certain access privileges.

post URL file

posts the contents of a file at a URL, using an HTTP POST request. web cannot read a file from standard input, since it needs to know the length of the file to initiate the request.

This command does not work with ftp: URLs.

With news: and snews: URLs, post uses a URL of the form news_server/newsgroups and posts the contents of file to the specified news group on the given news server.

put URL file

puts the contents of a file at a URL, using an HTTP PUT request. web cannot read a file from standard input, since it needs to know the length of the file to initiate the request.

With ftp: URLs, put uploads files in binary mode. The directory path need not exist and will be created if necessary (assuming permissions allow).

-r

automatically redirects when a 3xx error is received.

-t seconds

-timeout seconds

specifies the number of seconds that web has to complete the operation before timing out and issuing an error.

-U proxy-user

specifies the user name to use if the proxy URL specified by -x requires you to log in.

-u username

adds a username to an Authentication header. If you don't also specify a password on the command line (with the -p option), web prompts you for one.

Note:: Not all authentication mechanisms understand domain names. For those that do, you can specify username as domain/user or domain\user. However, for those that do not (such as CRAM-MD5), the domain name should not be included.

-V

is similar to the -v but the verbose output is written to the standard error instead of standard output.

-v

produces verbose output. In particular, the -v option produces the exact HTTP request sent to the server, and the server's exact response. This option also displays details and errors of dealing with https: URLs.

-X auth_excl_list

specifies an authentication exclusion list (auth_excl_list). auth_excl_list is a comma-separated list of authentication mechanisms to ignore when attempting authentication. This list can contain any of the authentication mechanisms listed under Authentication below and is case insensitive.

It is recommended that auth_excl_list be placed within double quotes to ensure the command is always correctly interpreted by the shell or command interpreter being used.

-x proxyurl

routes requests through a proxy. proxyurl is the URL of the proxy.

You can access https: URLs through a proxy if the proxy allows the use of the https: protocol. In this case, the -i option to ignore certificate options is ignored and the proxy determines whether or not to accept the remote server's certificate. The -f option is also ignored and certificates are checked against the database on the proxy server.

Authentication

When web attempts to connect to a server, the following steps are performed:

web first tries to retrieve the page using anonymous authentication, that is, no user name or password are passed to the server. Any -u or -p options are ignored. If the page is successfully retrieved, no further actions are taken.

web attempts to connect using the user name and password specified with -u and -p (prompting for a password, if no -p is given and login credential passing is disabled) by cycling through various authentication mechanisms until the page is successfully retrieved.

When login credential passing is enabled (either with -A or by setting the TK_ENABLE_CREDENTIAL_PASSING variable) and -u is specified, web passes that user name to SSPI and lets Windows determine the appropriate password and cycles through various authentication mechanisms until the page is successfully retrieved.

When login credential passing is enabled and no user name is specified with -u, web uses various methods to determine the user name depending on the authentication mechanism being attempted. For example, with SSPI mechanisms, no credentials are passed and Windows determines what user name should be used while with CRAM-MD5 or BASIC, the current logged in user name is used (along with the domain if necessary).

Normally, login credential passing is disabled for the web utility. However, by setting the environment variable TK_ENABLE_CREDENTIAL_PASSING or by specifying the -A option on the command line, you can enable its use.

The following authentication mechanisms are supported for web:

BASIC

Is a not particularly secure authentication method that uses MIME's base64 encoding scheme (as used by 64encode) to encode the user name and password pair.

DIGEST-MD5

Uses a MD5 checksum (similar to that generated by the md5sum utility) generated from the user name, password, and other information for authentication. This mechanism is only slightly more secure than BASIC.

Negotiate

Enables support for Integrated Windows Authentication, which utilizes Negotiate together with Kerberos as a single sign-on mechanism. It uses a challenge-response dialog between the client and server to allow the client to prove its identity without actually sending a password. This is the most secure method of single sign-on, but not all web servers are configured to support it.

With this authentication method you may need to supply a "Server Principal Name" (or SPN). If you do not specify one, a default value of HOST/hostname is assumed.

If the hostname in the URL is not a fully qualified domain name (FQDN), it is likely that you will need to supply a SPN for the host in question for this authentication method to succeed.

NTLM

Is an authentication mechanism used by Windows, particularly as a integrated single sign-on (SSO) mechanism. It uses a challenge-response dialog between the client and server to allow the client to prove its identity without actually sending a password.

EXAMPLE

The following script fragment checks for a need authorization error return (401); if found, this script prompts the user for a password and retries. You could also extend this script to retry upon redirects.


# On the first call, attempt to get the page either without authentication 
# or using login credential passing if possible. On subsequent calls we
# know we will know by the failure if we need a username and password.

authFlags=-A
while [ true ] ; do
        err=$(web -e $authFlags get $1 $2 2>&1)
        rc=$?
        err=${err#Password:*( )}
        case "$err" in
        20?*)   if [ $rc -ne 0 ] ; then
                        #Trim off the 200 OK line
                        echo "Failure on receive: "$(print -r "$err" | tail +1)
                        break
                fi
                echo "Success"
                exit 0; ### This is the only success case
                ;;
        401*)   echo "Authorization error"
                echo " enter username and password <enter> to quit"
                read NAME PASS
                [ ! "$NAME" ] && exit 1
                authFlags="-u $NAME"
                [ "$PASS" ] && typeset authFlags="$authFlags -p $PASS"
                ;;
        web:*)  print -r $err && exit 1
                ;;
        *)      echo "Command failed, host returned error $err"
                exit 1
                ;;
        esac
done

ENVIRONMENT VARIABLES

TK_ENABLE_CREDENTIAL_PASSING: When set, enables the passing of your current login credentials to the server.

DIAGNOSTICS

Possible exit status values are:

0

Successful completion.

1

Failure due to any of the following:

— local error

— any non-200 return code from the server indicating a server error

— failure to connect

— networking error

— certificate error

2

Usage error.

PORTABILITY

Windows 8.1. Windows Server 2012 R2. Windows 10. Windows Server 2016. Windows Server 2019. Windows 11. Windows Server 2022.

AVAILABILITY

PTC MKS Toolkit for System Administrators
PTC MKS Toolkit for Developers
PTC MKS Toolkit for Interoperability
PTC MKS Toolkit for Professional Developers
PTC MKS Toolkit for Professional Developers 64-Bit Edition
PTC MKS Toolkit for Enterprise Developers
PTC MKS Toolkit for Enterprise Developers 64-Bit Edition

web
transfer files to or from a Web server

Command