2

Raw Data

When this option is selected, raw data will be supplied in the data products: no action is taken to modify the data. In general, all scalar data is associated with a quality control flag. For plots, such as the time series scalar plot, data that fail quality control are marked on the plot with coloured data points and flag markers for emphasis.

Oceans 3.0 API filter: dpo_qualityControl=0

Clean Data

Selecting this option will cause any data values with quality control failures (QAQC flags 3, 4 and 6) to be replaced with NaNs. Because NaNs cannot be plotted, quality control failures will be excluded from any plots under the clean option. For all data products, when resampling with the clean option, any data with quality control failures are removed prior to the resampling (this rule applies to all resampling types: average, min/max, etc).

This is the default option for all data products.

Oceans 3.0 API filter: dpo_qualityControl=1

File-name mode field

'clean' is added to the file-name when the quality option is set to clean data.

Resample Type:

• None - no resampling.

          Oceans 3.0 API filter: dpo_resample=none

• Average - the mean value within resample period (fixed-window averaging without overlap). This is also known as a 'box-car' or ensemble average. It is subject to the 70% data completeness QAQC check (see below) with the exception of engineering data or data from irregular or scheduled sampling. Only available with the clean data product option.

          Oceans 3.0 API filter: dpo_resample=average and dpo_average={0, 60, 600, 900, 3600, 86400}

• Min/Max - the most extreme minimum and maximum values within resample period. It is subject to the 70% data completeness QAQC check (except for engineering data or data from irregular or scheduled sampling); QAQC flags are taken from the extreme data points.

     Oceans 3.0 API filter: dpo_resample=minMax and dpo_minMax={0, 60, 600, 900, 3600, 86400}

• Min/Max+Avg - the combination of the min/max and average as described above. The average is always calculated from clean data and will be NaN if there is less than 70% data available after cleaning. QAQC flags for min/max+avg with automatic resampling are the worst flag in the resample period, which includes the 70% check on data completeness (except for engineering data or data from irregular or scheduled sampling). This is the default option for time series scalar plots, other plots, such as the BHT, AGO, profile or staircase plots will have different options and defaults.

     Oceans 3.0 API filter: dpo_resample=minMaxAvg and dpo_minMaxAvg={0, 60, 600, 900, 3600, 86400}

Resample Period:

Visible when an actionable resample type is selected, immediately to the right of the resample type. Current periods offered:

• Automatic: 

  Oceans 3.0 API filter: dpo_average=0

• 1 Minute: 

  Oceans 3.0 API filter: dpo_average=60

• 10 Minute: 

  Oceans 3.0 API filter: dpo_average=600

• 15 Minute: 

  Oceans 3.0 API filter: dpo_average=900

• 1 Hour: 

  Oceans 3.0 API filter: dpo_average=3600

• 1 Day: 

  Oceans 3.0 API filter: dpo_average=86400

When resampling is selected:

• The timestamps in the data series correspond to the centre of each resampling interval. (Data downloaded prior to 13 Feb 2013: timestamps were at beginning of interval). The resample interval always begins and ends at an integer multiple of the resample period, so minutes on the minute, hours on the hour, days on the day, etc.
• If the date/time range on the search has limits that are within a resampling interval, the date/time endpoints are extended to include the entire resampling interval. For example, when daily resampling is selected from 03:00:00.000 on Monday to 20:00:00.000 on Thursday, the date range is extended to 00:00:00.000 on Monday to 23:59.59.999 on Thursday.
• Note that tides are not filtered out in resampled products.
• No anti-alias filtering is done. This is why only averaging and min/max are offered at this time. Box-car / ensemble averaging is an easily understood and ubiquitous process that is effective as a low-pass anti-alias filter. For more information, see this page on data reduction and time-averaging.
• Spatial / mobile data may be resampled, but users are warned against this procedure, as it may be inappropriate to do so. Spatial averages or a geospatial display of the non-resampled data may be a better approach.
• All resampled data products are subject to an additional QAQC check on data completeness (except engineering data or data from irregular or scheduled sampling). If any resample period does not contain at least 70% of the expected data, the QAQC flag for this period will be a failure (6), unless overridden by a manual QAQC flag, see the QAQC page. For live data, it is quite likely that the last resample period will not be complete and will be flagged; this is especially obvious for plots. Future improvements will allow users to modify the data completeness threshold.
• Automatic resampling chooses the most appropriate resample period for min/max or min/max+avg resampling, such that the amount of data returned is adequate for plotting. For short duration plots, it can result in no resampling.

More options will be available in the future as we work to improve the data products. Feedback is welcomed and encouraged. For custom resampling, users can develop their own matlab code in the Oceans 3.0 Sandbox and run it in the ONC computing environment.

File-name mode field

The resample type and period are added to the file-name when resampling is selected. Examples: 'avg1hour', 'MinMax10minute', 'MinMaxAvgAuto15minute', 'MinMaxAvgAuto' (automatic resampling chose no resampling).

State of Ocean and Environment Plots

State of Ocean and Environment plots are designed to give a long-term overview of state variables for either the Ocean or Atmosphere. These plots appear on the website and on the Data Preview tool. They are equivalent to a 'Variables by Location' search with 'All Available' time range, 1 hour averaging for the four or five variables put together into one search request. These searches are run on a schedule, usually once per day. State of Ocean/Environment plots are essentially device-level time series scalar plots, but have a specific colour scheme, as follows: 

Here is an example State of Ocean plot for the Saanich Inlet VENUS Instrument Platform:

State of Ocean and Environment Climate and Anomaly Plots

These plots are based on the same information/configuration as the original State of Ocean/Environment plots, except that they use daily min/max + average data. These plots are available in Data Preview alongside the originals. Here are examples from Saanich Inlet:

The anomaly plot (left) shows a time series of daily averages coloured on whether the value for that date is above the 66 percentile (red) or is below the 33 percentile (blue), where the percentiles are calculated from daily average values for that day of the year. The climatology plot (right) shows the day of year data for each sensor, showing the seasonality of the data. The grey lines in this plot are daily averages from every year, while the black lines are the overall day of year averages computed from the daily averages. The red and blue lines represent the day of year average plus and minus 1 standard deviation, respectively. Day of year standard deviation is also calculated from the daily averages from each day of the year. The magenta line is the current years' daily averages. Leap days are excluded from the climate plot and the day of year averages. The percentiles and standard deviations aren't showed if there is less than 3 years of data on each date, in that case, the anomaly plot will only show daily averages as black lines, the climate plot will omit the red and blue lines and comments will be added to the plots to indicate that this has occurred (it's common for new deployments).

State of Ocean and Environment Climate and Anomaly Data

The data behind these plots are available via links on their display in Data Preview. The file data products (CSV, MAT, ODV) are as documented in time series scalar data products, with the exception of the MAT file, which contains an additional structure called 'dataClimate'. 'dataClimate' is a copy of the 'data' structure, excluding non-climate sensors and replacing the min/max+avg values that exist in the 'data' structure with additional calculated values:

dailyTime, dailyAverage, dailyCount are copies of time, average, count in the normal 'data' structure, with data gaps filled. The code is capable of calculating these variables from hourly or other averaged data (currently we use daily data, so this step is essentially a pass-through). The dailyAverageByYear, dailyCountByYear are a reshape of the dailyAverage and dailyCount data into a matrix that is year by date, where the year is specified by iYear and the date is 1 to 365 (leap days excluded), so the matrix is n years by 365 days. dayOfYearAverage, dayOfYearCount, dayOfYearStdDev, dayOfYear33Percentile, dayOfYear66Percentile are the day of year statistics for each of the 365 years of the year, calculated using the dailyAverageByYear data only, without weighting by dailyCount or recalculation from the raw data. This average of an average approach is normal metrological procedure, as many metrological data sets only report daily min/max average values. The dayOfYearCount is the count of the raw readings on that day of the year, whereas dayOfYearCountDays is the number of days that have data. So for the 15 years of Saanich Inlet, if only 5 years have data on a specific day, that dayOfYearCountDays value is 5, but dayOfYearCount will be ~400000 readings. dayOfYearCountDays is used to determine if there is enough data (minimum 3) for a valid percentile calculation. dailyAverageAbove66Percentile, dailyAverageBelow33Percentile are the values used the anomaly plot, so that they are daily time series of the same size as the daily average data, and their values are either the dailyAverage or NaN if they are not above/below the percentile. The anomaly plot is then simply:

hold on;
plot(dailyTime, dailyAverage)
plot(dailyTime, dailyAverageAbove66Percentile, 'r')
plot(dailyTime, dailyAverageBelow33Percentile, 'b')

so that the red/blue colouring goes over top of the black daily average.

One more addition to the State of Ocean / Environment file products is a special "externally-derived" sensor. The MAT, CSV and ODV file have a "levelled-pressure" sensor in addition to the regular scalar pressure sensor. This purpose of this version of the pressure is to remove the variation between deployments so that a continuous time series is available. The algorithm to derive it is as follows: for each deployment remove the mean from the time series (i.e. zero mean), then concatenate the time series back together and add a fixed offset, calculated from the site depth (from the metadata) converted to pressure using the GSW toolbox. The effect is that the jumps in value at each deployment are removed and the time series is much more suitable for calculating tides, storm surge, etc. Users can do this calculation themselves with the deployment data that exists in the headers of the ODV and CSV files or the metadata structure in the MAT file.