See also informal notes prior to Nov 06 installation

Notes for programmers

Style notes

In-line Comments

In-line documentation is intended to explain why steps were taken, rather than just say what the code is doing (hopefully that is obvious). It also aims to label different sections of code, to speed your searching.

%DEV may prefix some comments. This was to remind me of unresolved issues or potential improvements (and indicates they are still not resolved!)

Redundant code

Much of the clutter and complication is required to trap and respond to potential events that might arise (especially those which could crash the system.) It might be that some of the anticipated events are unlikely to ever arise, in which case we can eventually remove the associated code for the sake of clarity.

Crash protection

Often the code can be made yet more robust, by protecting against events which shouldn't (but might) occur. For example, we could always use nanmean rather than mean, even if there should be no NaNs at that stage in the calculation. In such cases I would rather the system crash than have it silently (and maybe frequently) roll on when something has occurred that should not be allowed.

Functions

I use Matlab functions instead of scripts - this is a simple but important step to make code readable and maintainable - no variables change or appear without you seeing it happen!

Matlab version

ArgoRT was developed in Matlab V7.0, and installed in V7.2.

Some matfiles are saved with "-v6" for backwards compatibility. This has little -ve impact (files are bigger.) The use of v6 files could be stopped if noone uses Matlab V6.

• The code of this system should be kept in src/.
• Any manual intervention or investigation which involves creation of code or data files should be kept separate - maybe in work/ or subdirctories work/. That is, don't mix anything else with the source code of this system, or the standard files of this system.
• Experimental or temporary work, or work related to other activities (even if also related to this datastream) should be separated from all the above directories.

html code

the software that outputs html is terse compared with its precursors. This is mainly for speed, as I guess that, as in other languages, each fprint statement has a non-trivial overhead.

Figures

gif files were being generated using the home-grown 'save_fig.m'. This is probably not robust enough, especially as relies on other non-matlab paths and scripts, so now use .png files created directly by Matlab.

Variable names

General rule for Case (sometimes broken, such as in structure field names):

lower case	Normal variables
UPPER CASE	Global and persistent variables
Mixed Case	Fixed values (hard-coded parameters)

Pre-existing names have generally been used within the matlab struct (which is a small risk because new variables, having come about by a different route, might have different values/meanings (deliberately or otherwise.)

Some specific variables

surfpres NOT identical to previous variable called psfc It is measured surface pressure. NOTE: Webb floats have a 5db offset in surface value. This IS now corrected when message is first decoded, so that offset should NOT be considered at any later stage in the processing.

position_accuracy Read from Argos location fixes. Presume will only get the standard characters '0' to '3', but left as char because letter codes are possible. The Argos codes are defined in Argos manual table 2.3.5. The Argo definitions are in Ref table 5. Although the limits don't match, it turns out that the Argos codes 0-3 are best just equated to Argo 0-3.

That is, we end up leaving them untouched!

float, fpp, fp The array of structures containing all profiles for a single float was previously called float, and a single profile structure was called fp. I have kept fp for a single profile structure, but moved towards using fpp for the array of structures, because the word float appears in many places (in descriptive text, labels, and in "ncfloat"). fpp is a much better search target!

float still occurs frequently because of legacy code and it is still the name used in the ArgoRT matfiles.

_calibrate We calibrate S and so call the entire product 'A'=Adjusted. This obliges us to fill the '_ADJUSTED' fields for all profile variables in the netCDF files. I have taken the approach of leaving the matfile '_calibrate' fields empty (for T, Oxy, etc) unless we have calibrated that variable If argoprofile_nc.m, we fill adjusted fields for variable X with X_calibrate if it exists, and otherwise with X_raw. The X_qc field applies to both - we QC the raw data!

Profile and float numbers

Profile numbers

A profile number is read as part of the profile message (referred to as profile_number or sometimes pnum), and this is used to create some filenames (especially the netCDF profile files.) This is also the number reported in web pages. There have been glitches in this (probably only when a message has been scrambled.)

Each profile is stored in the float array of structures in matfiles, so each profile is a particular element of such an array. This alternative profile number, usually referred to as np is robust and often needed within the system, but is often different to pnum.

Workfiles are identified by both numbers, which avoids filename clashes and possibility of confusion.

ID numbers

Floats have WMO, Argos, and our deployment numbers. We aim to work exclusively with WMO numbers.

If other IDs are required, a global variable ARGO_ID_CROSSREF is created by getdbase.m (called at start of a normal processing run) and can be address directly or the function idcrossref.m can be used to translate.

The Argos ID is useful at early stages because it is the only way to identify floats in the Argos messages. Note that these may be reused. We cope with duplicate numbers by having idcrossref return the last "live" ID for any duplicates. That is, if duplicate Argos IDs, the returned WMO will be for "live" floats in preference to "dead" ones, and more recent entries in the database over earlier ones.

top

Random bits of algorithms

Speed Test

This is a non-standard implementation. Note that this test as previously coded would, if it progressed to testing the present #2 fix, always conclude that the #1 fix was bad - so earlier results and reactions were wrong.

Recommend checking if this test failed (whether or not the report is activated.) Use and/or plot first fix positions from relevant profiles.

Some aspects of the system

Argos download

The Argos downloads are, I assume, kept as the primary backup, from which reprocessing can be performed.

These files need to be identifiable by date because:

- operator may wish to check or reprocess something

- without unique identification, backup is awkward and risky

- programs need to know when ftp transfer took place, as this has a bearing on the decision to process a profile now or wait until the next day.

It would be best, but not essential, to have date/time recorded at the start of these files so that there was never doubt nor difficulty in determining just which download it is!

The old system named files by day-of-year - but used doy-9, because want data from 9 days previous! Reprocessing involved changing file dates (using 'touch'). There is no robust way to determine the year of the download file! Now changed to naming by LOCAL year and day of download.

Hex, and decoding it (and ASCII and Decimal)

The Argos transmission is received as ASCII, with the Argo component being displayed as hexadecimal. I have taken the approach of dumping immediately from ASCII hex to decimal.

    dd = hex2dec('ff')   

By default this becomes datatype double, which is fine because that can then be scaled and combined without drama.

Each ASCII hex pair is really an unsigned byte, so uint8 is the natural datatype, which is used where appropriate.

    bb = uint8(dd)  

Where values are not stored on byte boundaries (ie not all groups of bits being multiples of 8 in length), I reduce to 0s and 1s - it looks like binary but is now really vectors of uint8.

   cc = bitget(bb,1:8)  

Could get same result by using a bitmask, but is less clear

    ff = bitand(bb,uint8(241))  

I wrote bin2num to convert (these representations of binary) to decimal.

    ee = bin2num(cc) 

(Matlab routine bin2dec converts ascii binary rep to decimal.)

This approach means less slopping between char and numbers, and takes no more memory (wouldn't be an issue anyway).

Reporting

See Reports for details of reports.

Diagnositic and warning messages, and normal status/activity reports, are all passed to logerr.m

All reports could be just dumped to screen (eg using 'disp'), and captured in a file by redirection of stdout. That approach would fit well with standalone use of any functions.

Instead, reports are written to the file ID in global ARGO_RPT_FID. If this is empty (no file has been opened) then it is written to stdout.

A global structure ARGO_REPORT is also maintained which accumulates reports until it is reset (at start of processing each profile). Presently this is only used to count reports of each severity level. These counts are recorded in the PROC_RECORDS.

netCDF files

The metadata and techinfo files can be appended rather than recreated, as N_CYCLE is the unlimited dimension. We do this primarily for speed.

In a slight departure from official specifications, the trajectory file is made appendable, by setting the fixed dimension N_CYCLE larger than initially required. The trajectory positions are stored in N_MEASUREMENT dimension variables, which is the unlimited dimension. This is as coded and the files have passed the GDAC format checks.

All files in export/ are sent to GDACs at each run, and then deleted. export_argo checks through the processing record for all files required to be sent, complains if any are missing from netcdf/, and copies the rest to export/. As each required file is copied, its counter is incremented. If it has been sent enough times the counter is set to completed (99). If it is a Profile file, it is then deleted from netcdf/ or saved to a backup area (depends on value of system parameter ncsave_dir).

It is organised this way so that:

- extra files can be sent (once) just by placing them in export/

- files that somehow fail to be removed by the system [if that ever happens] when they have been sent will not hang around in export/ and be resent forever.

- incidentally, if the transfer process crashes then its last action [deleting files] will not be performed, which conveniently leave the files in place so they will be also be in the next batch of transfers!

Last updated 24 Nov 2006