CMAR c-squares mapper and the c-squares Xplanet mapper README file

Accompanies Mapper Version: 3.1.3
Last Updated: 18 September 2008
Copyright under GNU General Public License (GPL) : CSIRO Marine and Atmospheric Research (CMAR), Hobart, TAS 7001, Australia. See included file disclaimer.html for further information on usage
README file authors: Philip Bohm and Tony Rees, CMAR.
README file location (check occasionally for updates):
http://www.marine.csiro.au/csquares/mapper_README.html.

Description

The c-squares mapper (cs_map.pl referred to hereafter as the mapper) is a web-accessible, perl based utility (application) which creates web plots of c-squares (typically, representations of dataset spatial extents or "footprints") on any of a range of user-selectable, pre-prepared base maps, according to one or more strings of c-square codes passed to it via the web, either by a human user or (more typically), a remote application.

The c-squares xplanet mapper (csqXplanet.pl referred to hereafter as the Xmapper) takes large global maps created by the mapper in Mercator projection and wraps them into the orthographic projection. The mapper must be run first.

More general information about c-squares principles, notation, etc. can be found at http://www.marine.csiro.au/csquares/.

 

Version 1 of the c-squares mapper was created in January 2002 by Tony Rees and Miroslaw Ryba to perform mapping functions for various web-based CMAR applications, including MarLIN, CAAB and others (for further examples refer http://www.marine.csiro.au/csquares/examples.htm), and has since been used in a similar manner by a number of applications across the world.

Version 2. The original version of the mapper was upgraded in 2003 by Philip Bohm and Tony Rees to provide a significant range of new functions including active (clickable) maps, support for multiple c-squares strings, decompression of c-squares passed in "compressed" notation, and more, and has since undergone various more minor revisions to provide additional or improved functionality.

Version 3 creates extra maps behind the scenes when a user selects a full size global image. The mapper then provides links to the csquare Xplanet mapper which uses the extra maps to create global views. The basemaps may now be either gif, png or jpg formats. See: http://www.obis.org.au/admin/csquares/help.html

 

Installations of the c-squares mappers at CMAR currently run on UNIX and SUSE Linux and the mapper and Xmapper have also been successfully installed and run on Windows servers.

For a reference installation of the mapper, visit http://www.obis.org.au/cgi-bin/cs_map.pl. Other current known installations are at http://filaman.uni-kiel.de/cgi-bin/cs_map.pl and http://uinen.nrm.se/cgi-bin/cs_map.pl.

A screenshot of a (fairly sophisticated) example map created by a client application is available here. This map (created by the European AquaMaps project) exercises a number of features of the mapper including plotting filled as opposed to outline squares, plotting multiple c-square stings in a range of user-defined colours, including custom graphics and HTML formatting in the supplied header, title and legend, and including clickable maps (in the online version) that return spatially relevant information regarding the point on the map where the user has clicked. Examples of other types of maps that can be created can be located via the c-squares website, http://www.marine.csiro.au/csquares/ and the c-squares sourceforge project page, http://csquares.sourceforge.net/.

 

Contents of the installation package

For distribution, the mapper/Xmapper version last update date is appended to the file name (example: cs_map2005111.pl), in which case the file should be renamed to cs_map.pl and csqXplanet.pl before installation, by stripping the appended date. Other included files support the mapper and are required to produce the maps and aid with web presentation.

The following lists the required files. Other files may be included but are most likely not required.

In the folder csquares ->

cs_map[date].pl (latest version of the cs_map.pl script. Expects csqXplanet.pl to be present)
csqXplanet[date].pl (latest version of the csquares Xplanet mapper script. Designed to be part of this package)
csqlogo.gif
csqlogo.swf
csquare.css
c-square.ico
csquare_print.css
disclaimer.html
funcs.js
xplanet-config.txt
mapper_README.html
maps (folder)

In the folder csquares/maps ->

tmp_maps (folder)  (folder for the maps created by cs_map.pl for return to the user) 
Map_Index  
colouraus7gab.gif    (Example basemap)
colouraus7gab-big.gif (Example of a corresponding 'big' map)
colouraus7gab-big-mask.gif (Example of a corresponding landmask for a 'big' image)
Many more images ......  
Note: not all base maps have a corresponding 'landmask' image

 

 

Hardware Requirements

  1. A web server capable of running CGI programs (web viewable) Windows or Linux.
  2. A basic server with minimum resources may not give satisfactory service when requesting large maps or when large amounts c-squares are to be processed. For example, 90,000 squares are currently rendered in 23 seconds on our reference, dual 3.2 Ghz processor Linux server. Rendering the same number of squares on our original 167Mhz Unix machine would take about 10 minutes. Large global basemaps maps from '-7200' files are 7200 pixels wide and these are resized up to three times for returning to the user in the mapper and for later use by the Xmapper. The resizing is processor intensive and may account for 10 seconds of processing time even on a fast machine.
  3. When called by HTTP "POST" (recommended method), the mapper will potentially cope with unlimited numbers of squares. The only limitation may be the timeout settings of the host machine, web gateway, or browser client if processing is slow.
  4. Ideally the server (and client) will need to connect to the internet via a high speed link. The HTML file size is around 1 Megabyte when the combined c-square strings total 100,000 squares (although many requests would be typically much smaller than this).

Required software

  1. Apache or equivalent web server
  2. Perl
  3. GD package for perl
    -Note: The GD we're referring to is the GD Perl Module, or GD.PM. There is also something else often referred to as GD: the GD library file (or libgd) which is a language independent graphics library (and is used by GD.PM). That library is not useful by itself.
  4. CGI package for perl
  5. Xplanet source code http://prdownloads.sourceforge.net/xplanet/xplanet-1.2.0.tar.gz. Note: this is the source code for both Windows and Linux installation.

Installation procedure for Linux

The executable file that creates the initial maps is cs_map[ date created ].pl, which is a Perl file. It must be first be renamed to cs_map.pl. Rename your old version to enable rollback if required.

The executable file that creates the rotatable globe maps is csXplanet[ date created ].pl. It must be first be renamed to csXplanet.pl. All other included files support the mapper, which are simply served by the hosting web server.

  1. Ensure Perl and the GD and CGI packages are installed on the web server
    GD may not be packaged by default. If this is a new install I have found ActivePerl (http://downloads.vnunet.com/download/cgi+perl/active+perl/_33548.html) to be very easy to add packages to. The mapper needs the GD Perl wrapper installed as a module of Perl.
  2. Install Xplanet via readme in its own package with the following alterations to the source code so that the default shading of the globe will be suppressed. Xplanet will become an executable program in the path of all users after using configure and make. On linux if the install folder location is not in the right path the whole folder can be moved.

    Changes to make:
    in drawProjection.cpp and libmultiple/drawSphere.cpp

    In drawProjection.cpp, at line 127:
    const bool limbDarkening = (options->Projection() == HEMISPHERE || options->Projection() == ORTHOGRAPHIC);

    Set limbDarkening to false.

    const bool limbDarkening = false;

    in libmultiple/drawSphere.cpp, at line 102, there is this block of code:
    for (int k = 0; k < 3; k++) color[k] = static_cast<unsigned char> (color[k] * darkening);
    You can get rid of that block, and then the color won't be affected by limb darkening
  3. In the command window change directory to the main Xplanet folder containing the configure file. type './configure' then 'make'. There should be no problems on linux, however if required the program can be compiled with reduced features for csquares purposes.

    (To see the parameters type:
    $ ./configure -- help)

    To create the make file type for csquares purposes:
    $ ./configure --with-gif --with-jpeg --with-png --without-x --without-xscreensaver --without-pango --without-cspice

  4. By default `make install' will install all the files in `/usr/local/bin', `/usr/local/lib' etc. You can specify an installation prefix other than `/usr/local' using `--prefix', for instance `--prefix=$HOME'. The installation directory is important as the executable needs to be in the path of the user that runs the web server. Basically it should be in the path for all users.
  5. Unzip csquares.zip into a web accessible directory folder called 'csquares'
  6. Rename and move the file cs_map.pl and csqXplanet.pl into a directory where CGI programs can execute (this is usually the folder named 'cgi-bin')
  7. Move the file xplanet-config.txt to the directory xplanet/config (note, in previous releases of the mapper this file was named "csq.txt")
  8. All other files should live in a folder that you created called 'csquares' inside your main 'public_html', 'htdocs' or 'www' folder used by your web server.
  9. If desired, edit the content of the file disclaimer.html to include your agency's name and acronym in the appropriate places.
  10. Edit the variables that describe the file paths to the file cs_map.pl. In cs_map.pl the paths are set for a linux installation. In our cs_map.pl file the variable $BASE_MAPS the path is: /srv/www/htdocs/csquares/maps. You will need to change the /srv/www/htdocs/part to reflect the path to the installed folder 'maps' for example.

    Here is the appearance of the relevant portion of cs_map.pl as currently installed at www.obis.org.au on Linux:
# ------------------------------------------------------------------------------
BEGIN {
  $VERSION   = "3";
  $WWW_ADDRS = "http://www.obis.org.au";                      # home page (used in credits)
  $URL_2_CGI = "$WWW_ADDRS/cgi-bin/";                         # path to this file (and csqXplanet.pl)
  $FILE_NAME = "cs_mapVer3.pl";                               # this file name
  $BASE_MAPS = "/srv/www/htdocs/csquares/maps";               # location of maps and temp map folder on server
  $TEMP_MAPS = "tmp_maps";                                    # writable folder within $BASE_MAPS
  $MAP_INDEX = "$BASE_MAPS/Map_Index2";                       # map index file
  $WEB_FILES = "$WWW_ADDRS/admin/csquares";                   # included files - css, gif, swf, js etc
  $TMPMAPURL = "$WEB_FILES/maps/$TEMP_MAPS";                  # Absolute url to temp maps
  $CSQR_PAGE = "http://www.marine.csiro.au/csquares/about-csquares.htm"; # c-square docs location
  $XPLANCONF = "/usr/local/share/xplanet/config/csq.txt"      # xplanet (3d Mapping) config file
}
# ------------------------------------------------------------------------------
	
Example paths on different platforms:

 

Modify the first "shebang" line in cs_map.pl and csqXplanet.pl as required

The original line looks like :
#!/usr/bin/perl

you must change it to
#!<where your perl interpreter lives>

Modify the file permissions of cs_map.pl to 755 or similar (owner=rwx group=rx others=rx) if required

Modify the file permissions of the temporary map (tmp_maps) folder to enable the writing of files at run time if required.

To enable (the essential) automatic deletion of old temporary maps via cron create a crontab file as the same user as the owner of the 'tmp_maps' folder

Edit the crontab and install with the following line completed with your timing settings and paths:

Linux example only:

# clean out old cs_map.pl maps
15 * * * * find /srv/www/htdocs/csquares/maps/tmp_maps/ -type f -amin +360 -exec rm -f {} \;
16 * * * * find /srv/www/htdocs/csquares/maps/tmp_maps/ -type d -mmin +360 -exec rm -drv {} \;

The first line removes files the second the empty folders. In the previous example the scripts run every hour at 15 and 16 minutes past the hour, cleaning files last accessed 360 minutes ago or longer. To modyfy this type 'man find' in the command line to see the full options available.

High usage of the mappers use a lot of hard disk space. ( 20 uses per hour = 400Mb) Change to suit your requirements.

 

For Windows Installation

The executable file that creates the initial maps is cs_map[ date created ].pl, which is a Perl file. It must be first be renamed to cs_map.pl. Save your old versions by renaming them.

The executable file that creates the global maps is csXplanet[ date created ].pl. It must be first be renamed to csXplanet.pl. All other included files support the mapper, which are simply served by the hosting web server.

  1. Ensure Perl and the GD and CGI packages are installed on the web server
  2. GD may not be packaged by default. If this is a new install I have found ActivePerl to be very easy to add packages to. The mapper needs the GD Perl wrapper installed as a module of Perl.
  3. Install Xplanet via readme in its own package with the following alterations to the source code so that the default shading of the globe will be suppressed.
    1. Changes to make:
      in drawProjection.cpp and libmultiple/drawSphere.cpp

      In drawProjection.cpp, at line 127:
      const bool limbDarkening = (options->Projection() == HEMISPHERE || options->Projection() == ORTHOGRAPHIC);

      Set limbDarkening to false.

      const bool limbDarkening = false;

      in libmultiple/drawSphere.cpp, at line 102, there is this block of code:
      for (int k = 0; k < 3; k++) color[k] = static_cast<unsigned char> (color[k] * darkening);
      You can get rid of that block, and then the color won't be affected by limb darkening.
    2. To successfully compile and run xplanet on windows use Cygwin. The windows executable available from xplanet was built using cygwin. This part of the installation is most likely the most challenging and you may find you encounter different issues or approach this stage using methods you are more familiar with.

      http://cygwin.com/setup.exe

      Setup cygwin and include the the jpg, gif and png image packages.

      Once in the cygwin bash shell, change directory to the folder containing the Xplanet 'configure' file using UNIX style commands.
      Type:
      $ cd /cygdrive/c

      This gets you to the C:\ drive of your window machine for example. Navigate to your folder containing the source code for Xplanet.

    3. For the experienced C compilers, there is a help section specifiying options to the configure file.

      (To see the parameters type:
      $ ./configure -- help
      To print the parameters to a file in the current directory type:
      $ ./configure -- help > help.txt
      Note: open with wordpad. )

      To create the make file type for csquares purposes:
      $ ./configure --with-gif --with-jpeg --with-png --without-x --without-xscreensaver --without-pango --without-cspice

      Watch for errors in the output of the 'configure' and 'make' commands. You may need to delete all but the most recent cygwin1.dll file on your computer. The output of configure will tell you this or alert you to other errors.

      then type
      $ make

      As we have not specified a destination folder for the newly created xplanet.exe file, the new file is in the folder 'src' inside the base xplanet source directory. It should be possible to move this executable to any chosen folder, however the cygwin1.dll (a copy) must be in the same directory. Update xsqXplanet.pl with the path and name of the executable after installing.

  4. unzip csquares.zip into a web accessible directory folder called 'csquares'
  5. rename and move the file cs_map.pl and csqXplanet.pl into a directory where CGI programs can execute
    (this is usually the folder named 'cgi-bin')
  6. Move the file xplanet-config.txt to the directory xplanet/config (note, in previous releases of the mapper this file was named "csq.txt")
  7. All other files should live in a folder that you created called 'csquares' inside your main 'public_html', 'htdocs' or 'www' folder used by your web server.
  8. If desired, edit the content of the file disclaimer.html to include your agency's name and acronym in the appropriate places.
  9. Edit the variables that describe the file paths to the file cs_map.pl. In cs_map.pl the paths are set for a linux installation. In our cs_map.pl file the variable $BASE_MAPS the path is: /srv/www/htdocs/csquares/maps. You will need to change the /srv/www/htdocs/part to reflect the path to the installed folder 'maps' for example see below.

    Here is the appearance of the relevant portion of cs_map.pl as currently installed for testing purposes on my Windows machine (probably not running by the time you read this):
# ------------------------------------------------------------------------------
BEGIN {
$VERSION = "3";
$WWW_ADDRS = "http://C000178-hf.hba.marine.csiro.au"; # home page (used in credits)
$URL_2_CGI = "$WWW_ADDRS/cgi-bin/"; # path to this file (and csqXplanet.pl)
$FILE_NAME = "cs_map.pl"; # this file name
$BASE_MAPS = "C:/csquares/maps"; # location of maps and temp map folder on server
$TEMP_MAPS = "tmp_maps" # writable folder within $BASE_MAPS
$MAP_INDEX = "$BASE_MAPS/Map_Index"; # map index file
$WEB_FILES = "$WWW_ADDRS/csquares"; # included files - css, gif, swf, js etc
$TMPMAPURL = "$WEB_FILES/maps/$TEMP_MAPS"; # Absolute url to temp maps
$CSQR_PAGE = "http://www.marine.csiro.au/csquares/about-csquares.htm"; # c-square docs location
$XPLANCONF = "C:/xplanet/config/csq.txt" # xplanet (3d Mapping) config file
$XPLAN_EXE = "C:/csquares/csq_xplanet.exe"; # the Xplanet executable. needs whole path for windows
}
# ------------------------------------------------------------------------------
Example paths on different platforms:

 

Change the first line in cs_map.pl and csqXplanet.pl The original line looks like :
#!/usr/bin/perl

You must change it to
#!<where your perl interpreter lives>

for example:
#!c:/usr/bin/perl.exe

this may also work
#!perl

Uncomment the lines "+ binmode ....... " wherever it appears in the perl script

Create a Visual Basic script which deletes all the generated maps. This is more involved than previous versions of the mapper as we now have subfolders. The example below is from a microsoft help page and should work across various Windows variants. Also create a task in the scheduler of windows, so that it gets executed when required.

REMEMBER TO CHANGE THE PATH TO YOUR TEMPORARY IMAGE FOLDER (on line 3).

' WARNING: This visual basic script will remove the contents of strFolderName
'

strFolderName = "c:\csquares\maps\tmp_maps"

Dim arrFolders()
intSize = 0

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")



Set colSubfolders = objWMIService.ExecQuery _
    ("Associators of {Win32_Directory.Name='" & strFolderName & "'} " _
        & "Where AssocClass = Win32_Subdirectory " _
            & "ResultRole = PartComponent")

ReDim Preserve arrFolders(intSize)


For Each objFolder in colSubfolders
    GetSubFolders strFolderName
Next

Sub GetSubFolders(strFolderName)
    Set colSubfolders2 = objWMIService.ExecQuery _
        ("Associators of {Win32_Directory.Name='" & strFolderName & "'} " _
            & "Where AssocClass = Win32_Subdirectory " _
                & "ResultRole = PartComponent")

    For Each objFolder2 in colSubfolders2
        strFolderName = objFolder2.Name
        ReDim Preserve arrFolders(intSize)
        arrFolders(intSize) = strFolderName
        intSize = intSize + 1
        GetSubFolders strFolderName
    Next
End Sub

For i = Ubound(arrFolders) to 0 Step -1
    strFolder = arrFolders(i)
    strFolder = Replace(strFolder, "\", "\\")
    Set colFolders = objWMIService.ExecQuery _
        ("Select * from Win32_Directory where Name = '" & strFolder & "'")

    For Each objFolder in colFolders
        errResults = objFolder.Delete
    Next
Next

 

Testing the installed application

Enter the full absolute URL to the script appending and the minimum required parameter 'csq', 'http://www.yourserver/cgi-bin/cs_map.pl?csq=1000' for example (note, by default the URL cited now returns a message rather than a map; to test the map functionality, use any other valid c-square code e.g. 'http://www.yourserver/cgi-bin/cs_map.pl?csq=1001'). The mapper requires a valid csq to run or it will return an error message. This in itself is an indication that the script is running of course.

(For Linux only) If your server is a new installation and you want to run a simpler test of the server and external web calls try creating a file called 'test_var' including the code below.

#!/bin/sh
echo Content-type: text/plain
echo
env

Set the permissions!!

The file must have correct line endings for linux. Most PC code editors can save files with unix line breaks

To test Perl If your server is a new installation and you want to run a simpler test of Perl on the server and external web calls try creating a file called 'test_var' including the code below.

Remeber to change the path to perl on the first 'shebang' line.

#!perl
##
## printenv -- demo CGI program which just prints its environment
##
print "Content-type: text/html; charset=iso-8859-1\n\n";
foreach $var (sort(keys(%ENV))) {
$val = $ENV{$var};
$val =~ s|\n|\\n|g;
$val =~ s|"|\\"|g;
print "${var}=\"${val}<BR>\"\n";
}

 

Set the permissions!!

The file must have correct line endings for linux. Most PC code editors can save files with unix line breaks

Trouble shooting

PERL script throws server errors

File permissions incorrect. must be an executable file. Does the URL point to the correct folder. The cgi-bin folder has special permissions
(consider changing the owner and or group to something owned by the web server)

The file must have correct line endings for linux. Most PC code editors can save files with unix line breaks

PERL script throws perl errors incorrect paths / missing packages. A perl script can be run via the command line. It may be worth hard coding the variable csq within the script to a valid csq value for trouble shooting purposes. my $csq = $params{'csq'} || 1000; , however this should be disabled once testing is finished, in order that normal behaviour is restored (not supplying a value for csq should be trapped as a recognised error in the production version)
Script runs returning error msgs by cs_map.pl incorrect paths / permissions for the 'tmp_maps' folder. Try supplying debug=true to see what throws the error.
Script runs returning missing map / images paths

Customising the Range of Available Base Maps

Individual base maps can be disabled by commenting out the relevant lines in the Map_Index file. Every map should exists in both "normal" and "large" versions, and may also have associated land masks.

Large versions can be identified as '<fileidentifier>[-big or -7200]<file_extension>'. eg worldmap-7200.gif. The Map Index determines priority. The last matching and existing file identified in the Map Index will be the large version. '-7200' in the previous example stands for 7200 pixels wide.

Use the Identifier '-7200' for Large world maps. This identifier tells the software this is a large global map and may be projected into a 3D globe using the csquares Xplanet mapper.

Use the Identifier '-big' for larger size map images of sections of the globe. These images may be any size larger than the 'normal' version.

Note that cs_map.pl resizes all displayed maps at a maximum width of 900px. The proccessing time of cs_map.pl and the size and visibility of rendered csquares is affected by this resizing, so as a rule '-big' images should not be bigger than 900px.

Additional (custom) maps can also be created, however there are a few rules that must be followed in order that the behaviour of the mapper does not become unpredictable (e.g. when selecting the relevant "best fit" map, displaying normal vs. large images in the correct order, etc.). Most of these rules are explained further in the header section of the Map_Index file. Please note, experimentation in this department is at the local installation's risk...

Known Limitations and Bugs

Limitations:

  1. A hard coded limit is imposed for decompressing any squares of finer resolution than (example) 1000:***:***:* (40,000 sub-squares) since the next value is 1,000,000 which would probably choke the server doing the processing.

  2. Squares are plotted sequentially, i.e. csq followed by csq2, csq3, etc. (if present). Earlier plotted squares will be overwritten by later plotted squares if the same square code is sent more than once (i.e., only the last to be sent will be visible - transparency is not supported).

  3. At CMAR, the popular test URL of the form "http://www.yourserver/cgi-bin/cs_map.pl?csq=1000" tends to get heavy use by robots if advertised (e.g. 50% of all maps requested). To prevent undue mapper use, this specific case is trapped in the present cs_map.pl code around line 617, and made to return a human readable message rather than a map. This action can be commented out if not required, and normal maps will then result when square 1000 is the only one supplied.

Bugs: All known bugs to the current release date have been fixed. Please report any others found to the address below.

Release and Update Schedule

This application is updated at irregular intervals, depending on user requests and the availability of CMAR staff time for ongoing development. To be informed of the availability of new releases, it is recommended to join the "c-squares-discuss" mailing list (details below).

General c-squares Information

http://www.marine.csiro.au/csquares

Technical information (functionality and input parameters etc.)

http://www.marine.csiro.au/csquares/mapper-tech.htm

Mapper User Help

http://www.obis.org.au/admin/csquares/help.html

Feedback and Application Support

Limited support for this application is available on a goodwill basis from CMAR whenever possible. Please address any positive or negative feedback, bug reports, suggestions for improvements, etc. to Tony.Rees@csiro.au. A "c-squares-discuss" listserver is also available (for joining instructions see http://www.marine.csiro.au/csquares/about-csquares.htm#listserver) which interested persons should consider joining, to be informed about (and participate in) ongoing development activities regarding the mapper and the c-squares spatial indexing system in general.