1) acquire the data files.

2) convert the files to MedsASCII format:

Data Source	Data Location	Conversion Program
Mership/BOM	/home/mership/proc	/home/mership/programs/mixedSEA
NODC/WOCE	/home/gronell/quest/woceYY(year)	(none -already in Meds-ASCII)
WOCE CD V2	/home/mership/proc/wocecd2	/home/mership/programs/convertnetCDF2MA
Far Seas Fisheries	/home/gronell/farseasfisheries	/home/gronell/farseasfisheries/mixedfarseas
Franklin DPG	/home/mership/proc	/home/mership/programs/mixedSEA
WOA - observed level data	/rwx5/atlas & /rwx3/atlas	sh /rwx3/atlas/convertalltoMA2.sh using convertATLAS2M
Noumea	/home/mership/proc/noumea	/home/mership/programs/mixedSEA
SCRIPPS "e" format data	/home/mership/proc/scrippspx9	/home/mership/programs/mixedSEA

3) Convert the Meds-ASCII files to Direct Access (individual data files for each cast, indexed by unique ID).

        Program used:  /home/gronell/quest/convertM2DA - if priority has not been added to the files yet, it is done at this stage.

            Alternate program: /home/gronell/quest/convertM2DAix1 - this converts only those casts within the designated region containing IX1.

4) Concatenate the constituent files into the master file - output to file called ix1master

Program used:  /home/gronell/quest/concatenateDA2DA - this creates a new unique ID if a cast with the same ID is already present in the database.

5) Backup the master database files.

6)Duplicate check the resulting master file - output to file called ix1masternodupes

Program used:  /home/gronell/quest/duplicateselfDA - note - this takes a long time to run if you are processing large files (120,000 records took 3 days to run).

Alternate program: /home/gronell/quest/duplicateDAaddons - this program compares a file you wish to add to the master database with the contents of the master database, creating an output file called "add....." which can then be concatenated with the master.  Note - the master may be changed if a record exists but is a duplicate of a higher quality record in the file to be added.  In this case, the master record receives a DUR flag and the new record is written to the "add...." file for concatenation.

7) Backup the masternodupes database files.

8) check the duplicates from different years ( in file diffyeardupesix1masternodupes.MA) and reject all true duplicates (DUR flag), leaving only the non-duplicates unflagged.

Programs used: /home/gronell/quest/convertM2DA - to create a file called diffyears

/home/gronell/quest/seaquestIOTA all n diffyears - to qc the data and eliminate true duplicates

/home/gronell/quest/convertDA2M - to extract non-duplicate profiles so they can be added back into the master database.

/home/gronell/quest/convertM2DA - to re-add the retrieved records to the master database

9) Backup the masternodupes database files.

10) run the program to check for extreme temperature gradients.  This program also automatically corrects bad bottle data and rejects wire breaks.

Program used: /home/gronell/quest/checkgradient

11) Backup the masternodupes database files.

12) Hand check the profiles flagged by the gradient checking program.

13) Backup the masternodupes database files.

14) generate the parameters for all casts remaining in the database.

Program used: /home/gronell/quest/stats/calcstatsnetcdf - this program asks you questions about the procedures to be used.  We specify "good" data only, "central" filter, 5. m depth bins, 1. degree temperature bins, don't pre-filter the data, and do interpolate sparse data to 2m resolution.

15) screen the data using the statistics calculated from the parameters generated in the previous step.

Program used: /home/gronell/quest/autoqc - this program writes a flag to each profile that fails one or more tests.  Failure is defined as a parameter with a value of more than 3 standard deviations from the relevant mean.  The number of standard deviations is specified at run time.  It also flags casts from areas with insufficient data to generate a background statistic (and therefore the cast cannot be compared with "expectations").

Alternate program: Eventually there will be a program (/home/gronell/quest/autoseparatefiles) to compare parameters from one file with statistics generated from another file.  This will allow screening of data files before they are actually added to the master database.

16) Backup the screened masternodupes database files.

17) Hand check the profiles flagged by the auto qc program.

Program used: /home/gronell/quest/seaquestAUTO

18) Backup the screened/qc'd FINAL masternodupes database files.

19) Re-generate the parameters for all casts remaining in the database.

Program used: /home/gronell/quest/stats/calcstatsnetcdf - see step 14 for details.

20) Archive the screened/qc'd FINAL masternodupes database files and the final parameter files.  These are now your working files.

There are some other things to be done but they are not formally part of the process at this time.

One is to check for and eliminate all profiles on land.  There is no automatic way to do this as yet.  We use /home/gronell/quest/seaquestIOTA and look for casts that are obviously landlocked.

Another is to rewrite the database to eliminate all duplicate profiles (some are generated by the duplicate checking routines and others might be identified during the hand qc stages).  This is done whenever deemed necessary/wise.  The program used is /home/gronell/rewriteDA2DAnodupes.  Most of these programs either work with both the DA and RA (normal QUEST structure) files or have versions which will owrk on the RA files but we maintain the IOTA processing in the DA format.