Running Whois and Crawling URIs from PowerShell

In a previous post I described the installation of Whois on Windows 11 from Microsoft SysInternals site (https://irthoughts.wordpress.com/2021/12/29/installing-whois-on-windows-11/).

This time I would like to cover how we can run Whois from PowerShell. I assume you have Whois installed and know its current location or path.

I also want to explain how you can crawl all kind of URIs with PowerShell. No additional software is required as Windows comes with PowerShell. You just need to know how to use it. Its PowerShell ISE can help you to learn by doing.

On PowerShell and PowerShell ISE

Windows 11 and previous versions come with PowerShell and PowerShell Integrated Scripting Environment (ISE).

To locate both, go to Start and search for PowerShell. Results might change depending on the Windows version used. A search in my Windows 11 PC found the following:

Windows Power Shell
Windows Power Shell ISE
Windows Power Shell (x86)
Windows Power Shell ISE (x86)

PowerShell is used for automating the management of systems and to develop, test, and implement solutions. PowerShell ISE lets you create, run, and debug commands and scripts.

Checking Versions and Modules Installed

At the time of writing, current versions are 7.1, 7.2 (LTS), 7.3 (prerelease). Version 7 is designed for the cloud and can coexist with Windows PowerShell 5.1. This lets you test and compare between editions before deployment. Migration is safe and easy. Latest versions can be installed from

https://aka.ms/PSWindows

To check your system version, open PowerShell or PowerShell ISE and run the following command line:

Get-Host | Select-Object version

To retrieve additional information about the current version, run the following command line:

$PSVersionTable

To check for default modules installed, run the following command line:

$Env:PSModulePath -split (‘;’)

Running Whois

Open PowerShell or PowerShell ISE. In the command line, type the location of your Whois copy followed by

-v domainname.tld

where domainname is the target domain name and tld is its top-level domain (com, net, gov, edu,…).

For instance to whois yahoo.com, add to your Whois path

-v yahoo.com

Crawling URIs

Crawling URIs with PowerShell or PowerShell ISE is easy. You just need to use the following pattern

(Invoke-WebRequest -Uri ‘uri’).X

where X = Headers, Links, Images, Forms, Scripts…

Then to extract links from a given URI, let say yahoo.com, run the following command line

(Invoke-WebRequest -Uri ‘yahoo.com’).Links

This retrieves links, headers, and additional information about these so you may want to implement some filtering.

Filtering crawling results

To get just web addresses, use X = links.href, like this:

(Invoke-WebRequest -Uri ‘yahoo.com’).links.href

Note that you can also use lowercases.

To retrieve attribute data from links; for instance, try by replacing href with title, innertext, outertext, innerHTML, outerHTML, …

One more thing. You can also try a pattern with X = Headers, Images, Forms, or Scripts. Thus to retrieve scripts run

(Invoke-WebRequest -Uri ‘yahoo.com’).scripts

You can also do some filtering to the results, like this (just try it!)

(Invoke-WebRequest -Uri ‘yahoo.com’).scripts.src

Retrieving Source Codes, Robots text files, etc

Use a command line pattern as before but use X = RawContent. For instance to retrieve yahoo.com source code use

(Invoke-WebRequest -Uri ‘yahoo.com’).RawContent

Finally to retrieve Yahoo’s robots.txt file from yahoo.com use

(Invoke-WebRequest -Uri ‘yahoo.com/robots.txt’).RawContent

Enjoy whoising and crawling others.

Post Data: Sometimes a Transport Layer Security (TLS) error is triggered when using Invoke-WebRequest or Invoke-RestMethod. I will discuss how to avoid this error altogether in my next post.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s