LinkScan for Unix. Reference Manual.

Section 9

File System Scanning and Orphaned Files

LinkScan incorporates the ability to examine the files on your local hard drive and interpret them in a manner very similar to a web server. This capability has two major applications:

• It can dramatically accelerate the scanning of large numbers of static HTML documents.

• It enables the identification of Orphaned Files.

Configuration is inherently significantly more complex when compared to normal HTTP Scanning. In particular, you must configure the following items:

• The mapping between the URL's of your website and the corresponding File System Pathnames
• The mapping between the various file extensions and the file types
• The names of the files that are served by default when a hyperlink points at a directory reference (e.g. <a href="/Products/">)
• Various other options that enable LinkScan to emulate the behavior of your web server and the manner in which it has been configured

If you do not configure the File System Pathnames, LinkScan will automatically use HTTP Scanning. It will also disable the Orphaned File checking.

If you wish to enable Orphaned File checking and use HTTP Scanning, you must configure the File System Pathnames to enable orphan checking. Then, simply set Http = 1.

This is best illustrated by example:

# Map the server root
# http://www.example.com/index.html  <==> /usr/www/htdocs/index.html

Homeurl = http://www.example.com/
Homedir = /usr/www/htdocs/
Homefile = index.html

# http://www.example.com/cgi-bin/    <==> /usr/www/cgi-bin/
# http://www.example.com/~username/  <==> /home/username/public_html/

Alias cgi-bin/ /usr/www/cgi-bin/
Alias ~([^/]+)/ /home/$1/public_html/

# Hide hidden files and directories from the Orphans Report

Noorphans (\.|.*/\.)

# The following are significant (but default) settings

Execute cgi-bin/             # Test cgi-bin/ via HTTP
Execute (?i).*\.(cgi|asp)$   # Test .cgi and .asp files via HTTP

Htmlfiles = html, shtml, htm
Mapfiles = map
Pdffiles = 
Flashfiles = swf
Defaultpages = index.html, index.shtml, index.htm, home.html, home.shtml, home.htm

Indexoptions = 0             # Disallow directory listings
Expandssi = 1                # Expand Server Side Includes
Autohttp = 0                 # Disable automatic HTTP retry
Maxdirlevels = 10            # Don't explore file system beyond 10 levels

On Unix systems only, the Alias directive supports the special !HOME expression:

Alias ~([^/]+)(/|$) !HOME/public_html/

A reference to ~someuser/ will be Aliased to !HOME/public_html/. Then, !HOME will be replaced by the someuser's Home Directory which is determined via a lookup of /etc/passwd.

Remote File Systems

In some cases, the file system directories containing the web site may reside on a physically different computer from LinkScan. In these cases, LinkScan will support Network File System pathnames (subject to any locally imposed security controls).

In other cases, the file system of the remote system may not be visible via the network, quite possibly for security reasons. LinkScan will be unable to scan the remote computer using the File System Scanning Method. You must use HTTP Scanning.

However, it is still possible to enable Orphaned File checking. In summary, you will need to execute a small, self-contained Perl program on the remote computer. It will assemble a "picture" of the file system and save it as a simple ASCII file. That file may be transferred to the LinkScan computer using FTP (or any other more secure technique) and used to perform the orphan analysis in lieu of direct access to the remote server.

1. Fully configure the selected Project as if your were using File System Scanning on your local machine. However, when setting the pathname to the root of the target webserver, (and any associated Aliases) use the pathname conventions applicable to the remote server.

2. In the Project configuration file, force LinkScan to use normal HTTP Scanning by setting:

   
   Http = 1
   

3. Set the Orphanfile setting in the Project configuration file to the full pathname of a file on your local computer. For example:

   
   Orphanfile = /usr/linkscan/someproject/orphans.list
   

4. Transfer the following files to the remote server:

   
   /usr/linkscan/lsfind.pl
   /usr/linkscan/someproject/linkscan.cfg
   

5. On the remote server, execute the lsfind.pl program:

   
   perl lsfind.pl orphans.list
   

6. Transfer the orphans.list file back to the LinkScan machine.

7. Initiate a scan of the target website in the normal manner. LinkScan will use the orphans.list file from the remote server in lieu of scanning the file system on the local server.

LinkScan for Unix. Reference Manual. Section 9. File System Scanning and Orphaned Files
LinkScan Version 12.1
© Copyright 1997-2010 Electronic Software Publishing Corporation (Elsop)
LinkScan™ and Elsop™ are Trademarks of Electronic Software Publishing Corporation