| Aus Argo | ArgoRT Doco Page |
The ArgoRT system is run automatically by "cron". Reports are sent to designated operators after each run (typically 4 per day), and a webpage detailing the day's processing is updated daily. Some reports may require further investigation and possibly manual intervention. If intervention is required, the aim is to reject parts of the message which are preventing correct decoding and generation of profiles.
Profiles are transmitted on the GTS (ideally within 24 hours of surfacing), are sent to GDACs in netCDF files, and are ultimately stored locally in one matfile per float. Web pages displaying each new profile are generated every run.
An obligatory picture, dedicated to 1970s software development methods. This shows the gross flow of data, and the two manual intervention detours.
[Argos] --- FTP_GET_ARGOS ---> 6-hrly ASCII - - EXTRACT_ARGOS_MSG
download file |
| |
| - - - - - - +
V
STRIP_ARGOS_MSG }
| } R
v } e
EDIT_WORKFILES - - -> PROCESS_PROFILE } p
^ : } o
| : <-----> FIND_BEST_MSG } r
| : <-----> DECODE_{WEBB,PROVOR} } t
[Workfiles] <--------------- : } s
:-------+---------+ }
+--------- : V V }
| netcdf/ tesac/
V | |
Web Pages V V
EXPORT_ARGO ------> GTS
V
export/
|
V
writeGDAC --------> GDACs
| Processing area | |
|---|---|
| argos_downloads/ | Argos messages will be fed into here |
| cal_data/ | CTD reference files for salinity calibration |
| docs/ | a copy of the documentation set, transferred to your web directory during installation |
| export/ | netcdf files for transfer to GDACs (cleared after every transfer) |
| matfiles/ | the Argo matfiles |
| netcdf/ | per-float netCDF files live and grow here, but temporary copies are placed in export to send. |
| src/ | all ArgoRT software |
| reports/ | ArgoRT processing reports |
| run_logs/ | system reports from ArgoRT auto-runs |
| spreadsheet/ | the Excel spreadsheet and .csv copies |
| tesac/ | TESAC messages for transmission on GTS |
| workfiles/ | intermediate files for examining, correcting or reprocessing profiles. |
| work/ | an area for doing work. extract_argos_msg puts download extracts here. |
| Web area | |
| doc/ | system documentation |
| doc/1../ | software files pages |
| floats/ | pages for each float in own subdirectory |
| processing/ | processing report pages |
| templates/ | components used by software to build pages |
All checking or manual intervention work is performed in Matlab (V7 or higher), so some proficiency in Matlab is required.
I recommend activating the debugger, so that if any exception causes an
error then you can quickly locate the source and cause of the problem.
(If debugger is not active by default, use the following line in each
session OR in your startup.m file:
dbstop if error
For interactive sessions you may optionally use dbstop if warning but it should not be used for auto-runs as we don't want them to hang if plotting routines warn because of some unusual data.
When an error occurs, Matlab pauses in the debugging mode. (The following describes behaviour when using Matlab with the -nodesktop option.)
See where the problem occurred, and move up and down between workspaces:
k>> dbstack
> In decode_provor at 32
In process_profile at 267
In strip_argos_msg at 318
In drive_argo at 30
K>> dbup
In workspace belonging to process_profile.
All variables within the present workspace can be accessed.
Being familiar with all capabilities of the debugger will be of great benefit to you.
A lot of fundamental ArgoRT system info is kept as global variables. The first time a function needs them in the present Matlab session, they are generated. After that they remain in the session.
If you change the database, or the set_argo_sys_params.m file, or the name of processing records or report files, then these things may not be updated until you a) clear the globals OR b) restart Matlab.
You can access these variables for your own purposes by declaring them in the workspace - for example, to get all WMO IDs:
>> whos ARGO_ID_CROSSREF
>> global ARGO_ID_CROSSREF
>> whos ARGO_ID_CROSSREF
Name Size Bytes Class
ARGO_ID_CROSSREF 132x3 3168 double array (global)
>> wmos = ARGO_ID_CROSSREF(:,1);
These variables are used in many places, but most of them can be seen in use in the first part of strip_argos_msg.
They can all be seen with command
>> whos global
Do read the "Filetypes" section on database.
Ideally add new floats before they report in for the first time, with a status of "expected" (see "New Floats" above). Add to the master and calibration sheets at the same time, and keep floats in the same order in both. Note that the spreadsheet structure must not be changed (without modifying programs to match.)
When updated, save both sheets as csv files (argomaster.csv) and argomaster_cal.csv respectively.) Excel may throw up warnings about losses converting to csv - click "yes" and ignore them!
When saved to csv, update web page version by running web_database. If new floats have been added, also update the float selection webpages using web_select_float.
The master sheet status field has a small set of allowed values. This field implements our grey list and enables ArgoRT to ignore bad floats and warn of missing good floats.
| Value | meaning | ArgoRT action |
|---|---|---|
| live | active and ok | process it |
| dead | assumed dead | warn if it does appear |
| unknown | maybe pre-launch, or recently gone missing? | treat as "live" |
| suspect | have reservations about data quality (ie class 4 data) | process, send netCDF, but do not send to GTS |
| expected | temporary status: a float due to be deployed | will be reported when it becomes live (see sections above) |
| evil | Incurably bad float, but it will not die! | ignore it completely |
Normal operation is scheduled by the Unix scheduler called cron. This can be alterred - see "cron" in installation notes.
ArgoRT can be activated manually, but make sure that doing so does not clash with an automatic run. Also, options can be passed to the main calling function strip_argos_msg which are worth considering for non-standard runs.
ArgoRT can be manually activated to process portions or whole ftp downloads by:
To manually activate an ftp download:
src/ftp_get_argos
Manually activate an update of the Processing report webpage with web_processing_report. Note the option to run this report without clearing the 'new' flags - so that the next auto-update will be uneffected.
Manual intervention to remove material which is corrupting message decoding is achieved either by extracting, editing and reprocessing the segment of the ftp download, or, much preferred, by flagging in the workfile.
>> edit_workfile
(Type ? for help) : ?
(Type ? for help) : f 5900865
(Type ? for help) : p42
(Type ? for help) : l
... list through data to find bad bits
(Type ? for help) : r 1:127
(Type ? for help) : t
(Type ? for help) : g 123:127
(Type ? for help) : t
(Type ? for help) : a
(Type ? for help) : q
Note that such profiles may be processed too soon, because of the preceding garbage is initially taken to be the start of profile transmission. Try reprocessing after another day. It may even be worth having another value in the database status field ("new"?). ArgoRT could be modified to warn the operator of the first appearance of the float (but not attempt to process it.) The operator could then change the status to "live" when the first profile has fully arrived.
Another issue with first profiles is that metadata netcdf files need to be at the GDAC before profile files will be accepted. This should be okay - metadata files are built at the same time as the first profile for a float, and are sent seconds beforehand, but mishaps could happen??
Here the download from an "expected" float is worked with to trim out the pre-profile messages. We look at the heads (Argos fixes) with h, and see a discontinuity in times at line 182. Then look at the download from line 150 onwards (l 150). Notice the undecodable block numbers (NaN) prior line 182, and sensible ones beyond. We then reject everything before line 182 (r1:181) and test again. (Repeat "l 150" if you want to see the effect of "r1:181").
The final q command means that we discarded the work done, because I think we give it a bit more time - wait for the next download. We should now change the status in the spreadsheet from "expected" to "live" (and do the normal steps to make that available to ArgoRT). At the next ArgoRT run this float will then be acquired again and profile #1 processed. We will want to repeat the this session then (it will now the profile #1), to clean it up - there will still be junk there but maybe sopme more real profile message.
>> edit_workfile
Start with commands F then P. Eg F 5901170
then P 57
(Type ? for help) : f5901175
Float 5901181: 1 workfiles, profiles 0 to 0
(Type ? for help) : p0
Profile 0 392 lines, 22 headers, 15 Block1
pmeta =
wmo_id: 5901181.00
ftptime: 2454069.35
ftp_fname: 'argos06_334'
np: 0
pnum: 0
ftptime: 29-Nov-2006 20:20:00
(Type ? for help) : t
FBM: 1=248 2=209 3=0 4=1 5=27 6=4 7=16 8=0
Unresolved excessive speeds between fixes
L3 warning Revert to jday(1) - bad ascent_end estimate
JAE est: min=2454068.18 median=2454068.53 max=2454068.86 j1=2454068.17
JAE median: 29-Nov-2006 00:40:18 Jday(1): 28-Nov-2006 16:07:23
Estimate nmsg B1 tim (maxblk 49 reprate 44)
29 8:44<= 5 29 11: 8
29 7:55 7 29 11:31
29 7:31 8 29 11:43
29 6:19 11 29 12:18
29 4:18 16 29 13:17
29 2:41 20 29 14: 4
29 1: 4 24 29 14:50
29 0:40 25 29 15: 2
28 23:27 28 29 15:37
28 23: 3 29 29 15:49
28 21:26 33 29 16:36
28 19:49 37 29 17:23
28 18:37 40 29 17:58
28 18:13 41 29 18:10
28 16:12<= 46 29 19: 9
Float 5901175 Profile 1 Download 28-Nov-2006 20:20:00
N_Fixes 22 N_Gd_Pos 22
P raw N=70 Min= 4.400 Max=1900.100 Mean=571.473
T raw N=70 Min= 2.470 Max= 9.228 Mean= 7.085
S raw N=70 Min= 34.300 Max= 34.665 Mean= 34.489
Type ? for help) : h
3: 0 2006-11-28 16:07:23 -48.7620 129.6050 2
20: 0 2006-11-28 16:30:07 -48.7590 129.6290 2
36: 0 2006-11-28 16:42:57 -48.7590 129.6310 2
57: 0 2006-11-28 17:56:17 -48.7590 129.6560 2
94: 0 2006-11-28 18:24:53 -48.7600 129.6620 1
110: 0 2006-11-28 18:30:01 -48.7610 129.6670 2
125: 0 2006-11-28 19:36:23 -48.7620 129.6840 2
145: 0 2006-11-28 19:51:47 -48.7600 129.6890 2
182: 0 2006-11-29 11:05:50 -48.7900 129.8230 1
193: 0 2006-11-29 11:35:10 -48.8010 129.8500 1
206: 0 2006-11-29 11:39:34 -48.8010 129.8360 1
221: 0 2006-11-29 12:23:56 -48.8020 129.8660 1
242: 0 2006-11-29 13:18:56 -48.8160 129.8750 1
257: 0 2006-11-29 14:06:14 -48.8260 129.8850 1
276: 0 2006-11-29 14:53:32 -48.8370 129.8970 1
287: 0 2006-11-29 14:59:46 -48.8360 129.9060 2
297: 0 2006-11-29 15:44:08 -48.8500 129.9180 2
317: 0 2006-11-29 16:32:54 -48.8570 129.9460 1
345: 0 2006-11-29 17:57:14 -48.8690 129.9520 1
357: 0 2006-11-29 18:06:46 -48.8800 129.9590 1
368: 0 2006-11-29 18:14:50 -48.8740 129.9660 2
381: 0 2006-11-29 19:10:34 -48.8860 129.9820 2
Type ? for help) : l 150
150: 0 NaN ? 7 161 12 36 74 124 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
151: 0 NaN ? 44 162 12 36 74 168 72 0 0 154 156 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
152: 0 NaN ? 33 163 12 36 74 212 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
153: 0 NaN ? 44 164 12 36 75 0 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
154: 0 NaN ? 96 165 12 36 75 44 72 0 1 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
155: 0 NaN ? 148 166 12 36 75 88 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
156: 0 NaN ? 65 167 12 36 75 132 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
157: 0 NaN ? 106 168 12 36 75 176 72 0 1 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
158: 0 NaN ? 44 169 12 36 75 220 72 0 1 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
159: 0 NaN ? 70 170 12 36 76 8 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
160: 0 NaN ? 124 171 12 36 76 52 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
161: 0 NaN ? 74 173 12 36 76 140 72 0 1 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
162: 0 NaN ? 172 174 12 36 76 184 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
163: 0 NaN ? 99 180 12 36 77 192 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
164: 0 NaN ? 50 181 12 36 77 236 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
165: 0 NaN ? 180 183 12 36 78 68 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
166: 0 NaN ? 152 184 12 36 78 112 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
167: 0 NaN ? 153 185 12 36 78 156 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
168: 0 NaN ? 200 187 12 36 78 244 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
169: 0 NaN ? 248 188 12 36 79 32 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
170: 0 NaN ? 158 189 12 36 79 76 72 0 1 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
171: 0 NaN ? 96 190 12 36 79 120 72 0 0 154 156 15 39 173 165 213 74 243 155 3 177 255 230 140 246 106 254 252 251 216 240 NaN
172: 0 NaN ? 168 191 12 36 79 164 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
173: 0 NaN ? 87 192 12 36 79 208 72 0 0 154 159 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
174: 0 NaN ? 59 193 12 36 79 252 72 0 1 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
175: 0 NaN ? 218 194 12 36 80 40 72 0 2 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
176: 0 NaN ? 249 195 12 36 80 84 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
177: 0 NaN ? 139 196 12 36 80 128 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
178: 0 NaN ? 159 197 12 36 80 172 72 0 0 154 158 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
179: 0 NaN ? 22 198 12 36 80 216 72 0 0 154 156 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
180: 0 NaN ? 190 199 12 36 81 4 72 0 0 154 157 15 0 225 39 16 85 12 100 252 78 32 25 115 9 149 1 1 4 39 6 NaN
181: 0 NaN ? 181 200 12 36 81 52 72 0 0 154 157 15 0 225 38 16 84 12 100 252 78 32 25 115 9 197 3 33 68 39 6 NaN
182: 0 2006-11-29 11:05:50 -48.7900 129.8230 1
183: 0 7 10 7 138 24 98 28 84 134 154 23 100 29 2 134 159 22 112 30 56 134 183 21 113 31 28 134 202 20 124 31 213 NaN
184: 0 8 B 188 8 134 219 19 130 32 175 134 237 18 151 33 134 135 0 17 140 34 92 135 25 16 149 34 163 135 24 15 153 34 NaN
185: 0 9 77 9 164 135 13 14 161 34 135 134 252 13 162 34 64 134 234 13 61 33 221 134 209 12 216 33 237 134 207 12 119 NaN
186: 0 11 B 11 11 132 35 22 134 252 10 31 34 226 134 238 11 51 51 51 51 51 51 51 50 35 49 51 51 39 54 34 100 68 NaN
187: 0 13 B 237 13 226 6 155 34 241 134 223 2 57 35 88 134 241 5 217 35 151 134 252 5 109 35 166 134 255 5 9 35 164 NaN
188: 0 15 201 15 222 134 255 2 179 36 12 134 249 2 78 34 0 134 73 1 235 33 233 134 53 1 138 34 25 134 57 1 38 NaN
189: 5 2006-11-29 11:08:02
189: 0 1 70 1 5 12 36 1 70 0 158 43 65 1 121 143 62 24 152 0 0 221 221 23 54 134 50 28 50 153 1 0 50 NaN
190: 0 2 69 2 99 25 150 14 9 166 135 105 74 57 9 166 135 105 74 53 9 224 135 78 70 73 10 29 135 45 66 99 10 NaN
191: 0 4 76 4 11 252 134 128 50 194 12 119 134 106 48 206 12 230 134 81 46 216 13 196 134 68 44 230 14 57 134 46 42 NaN
192: 0 5 197 5 242 14 158 134 23 40 255 15 191 134 21 39 10 16 173 134 16 37 23 17 119 134 2 35 32 19 22 134 22 NaN
193: 0 2006-11-29 11:35:10 -48.8010 129.8500 1
194: 0 16 B 233 16 34 44 134 61 0 199 34 39 134 46 128 92 163 61 151 44 34 1 223 191 253 237 127 255 127 247 191 255 255 NaN
(Type ? for help) : r1:181
(Type ? for help) : t
Profile downloaded too soon after first fix - should not process
FBM: 1=197 2=161 3=0 4=0 5=0 6=15 7=1 8=0
Unresolved excessive speeds between fixes
JAE est: min=2454068.93 median=2454068.93 max=2454068.93 j1=2454068.96
JAE median: 29-Nov-2006 10:21:06 Jday(1): 29-Nov-2006 11:05:50
Estimate nmsg B1 tim (maxblk 16 reprate 44)
29 10:21<* 5 29 11: 8
29 10:21 7 29 11:31
29 10:21 8 29 11:43
29 10:21 11 29 12:18
29 10:21 16 29 13:17
29 10:21 20 29 14: 4
29 10:21 24 29 14:50
29 10:21 25 29 15: 2
29 10:21 28 29 15:37
29 10:21 29 29 15:49
29 10:21 33 29 16:36
29 10:21 37 29 17:23
29 10:21 40 29 17:58
29 10:21 41 29 18:10
29 10:21<* 46 29 19: 9
Float 5901175 Profile 1 Download 28-Nov-2006 20:20:00
N_Fixes 14 N_Gd_Pos 14
P raw N=70 Min= 4.400 Max=1900.100 Mean=571.473
T raw N=70 Min= 2.470 Max= 9.228 Mean= 7.085
S raw N=70 Min= 34.300 Max= 34.665 Mean= 34.489
(Type ? for help) : q
... bare bones as yet ...
To work from ftp downloads, copy and modify drive_argo_manual.
To use workfiles, use run_from_workfile.
You choose whether to use RT, reprocessing, or other processing records file. The first means that this work will appear on the RT processing daily webpage, which is usually not ideal, unless only reprocessing the latest profile for each float.
Note that processing reports generated by reprocessing will be full of warnings to ignore, mainly about overwriting existing files.
netCDF files for any float or profile are easily generated (example below). The Profile files are deleted after sending to GDACs, but the other 3 types are maintained in netcdf/.
To manually send files to the GDACs, generate them and put in export/. They will be sent once (during the next auto-run) then deleted. No confirmation of this is provided.
You may wish to generate netCDF files or try running system components on one profile or a whole float. For example, for 5901171, to recreate the web page (profile #21) and the metadata file:
fp = getargo(5901171,21);
web_profile_plot(fp);
[fpp,db] = getargo(5901171);
metadata_nc(db,fpp);
OR you can get the database record with
db = getdbase(5901171);
[note - here using the convention of fpp for a 'float' array, and fp for a single profile]
Here we run floats through QC tests, in partic to look for activity of test #14 (density inversion). Note that we don't (can't!) put this data back into the matfiles, so we can muck around as much as we like)
for wmo = [5900344 5900859 5901170]
[fpp,db] = getargo(wmo);
fpo = qc_tests(db,fpp);
for ii=1:length(fpp)
if fpo(ii).testsfailed(14)
.. could test s_qc==4, but that might have been set
by other tests ..
density = sw_dens(fpp(ii).s_raw,fpp(ii).t_raw,fpp(ii).p_calibrate);
disp([num2str([wmo ii max(diff(density))])])
end
end
end
Note: getargo expects pnum, not np; see "Profile numbers" in programmer notes.
To see listings of plots of profile, use view_profs. It works best in conjunction with viewing the webpages for a float, as none of the header information is displayed.
This would be a rare occurence (I hope) and should be approached with caution, but matfile_edit can be used if needed to insert or remove profiles from a matfile. See also "matfiles" in File Types page.
Last updated 17 Nov 2006