Downloading Data with gsqlcmd

Downloading Data with gsqlcmd

gsqlcmd allows downloading files and making HTTP requests.

For example:

gsqlcmd download https://www.nasdaq.com/symbol/AAPL/dividend-history aapl.htm

You can use a mask for output files if the URL contains a detected file name.

For example:

gsqlcmd download https://www.savetodb.com/sitemap.xml *.*

/Check Option

You can use the /check option to test return results.

For example:

gsqlcmd download http://www.savetodb.com/ /check

The result shows page redirect:

http://www.savetodb.com/ 301 https://www.savetodb.com/

Automating

You can use a simple or headered task file to download a list of URLs.

For example, you can place URLs to download in a file like sitemap.txt and use the command:

gsqlcmd download @sitemap.txt *.*

Also, you can omit @ in the task file name in the download mode.

For example:

gsqlcmd download sitemap.txt *.*

Below is a simple batch file that allows downloading sitemap.xml, converting it to sitemap.txt, and downloading its pages:

@echo off

gsqlcmd download https://www.savetodb.com/sitemap.xml *.*

gsqlcmd convert "SELECT loc FROM sitemap.xml" sitemap.txt /noHeaders

gsqlcmd download @sitemap.txt *.*

You can even simplify the batch using a pipe:

gsqlcmd convert "SELECT loc FROM https://www.savetodb.com/sitemap.xml" /noHeaders | gsqlcmd download @ *.*

You can this trick to check pages with the /check option:

gsqlcmd convert "SELECT loc FROM https://www.savetodb.com/sitemap.xml" /noHeaders | gsqlcmd download @ *.* /check

Downloading Multiple Pages

gsqlcmd detects several schemes of the next page URLs.

In such cases, you can use the /pages option to download multiple pages.

For example, you use the following command to download option chains from Yahoo Finance:

gsqlcmd download "https://query1.finance.yahoo.com/v7/finance/options/AAPL" *.json /pages=20

GET, POST, PUT, PATCH, DELETE, HEAD

The download mode has the /method option with the following values:

AUTO, GET, POST, PUT, PATCH, MERGE, DELETE, HEAD

Here is an example of the DELETE method use:

gsqlcmd download https://localhost/crm/contacts?id=1 /method=DELETE

You can use the HTTP request method names as synonyms of the download mode with the specified /method option.

For example:

gsqlcmd DELETE https://localhost/crm/contacts?id=1

Using Authentication

Use the /auth option to specify the authentication method and use the method-specific options to pass authentication values.

For example:

gsqlcmd download https://localhost/crm/contacts /auth=Forms /username=user /password=pass

gsqlcmd supports the following methods: Basic, Windows, Forms, OAuth1, and OAuth2.

See more at HTTP Authentication.

Additional Notes

See also Web Request Options.