Getting Started: Slocum#
Gather data#
Slocum gliders have 4 types of files. For telemetry data there are *.tbd files for sensor data, and *.sbd for the glider’s attitude and position data. These are called *.ebd and *.tbd respectively, when retrieved from the gliders’ payload post deployment. Modern gliders have compressed version of these, eg *.tcd, *.scd that pyglider should be able to parse. These data files need to be made available in a single directory for pyglider to process. Note that on the glider they are often separated into science/logs and flight/logs.
Slocum gliders also have a sensor cache file *.cac, all of which have randomized names. These are needed by the processing, and are usually stored in a separate cache directory.
You can download example data at https://cproof.uvic.ca/pyglider-example-data/pyglider-example-data.zip which will add a local directory example-data to your current directory.
Make a deployment configuration file#
The processing routines all take a deployment.yaml file as an argument, and information from this is used to fill in metadata and to map sensor names to NetCDF variable names. See Example deployment.yaml, below.
There are four top-levels to the deployment.yaml
metadata: The only field that is necessary here isglider_name. The rest of the fields will be added to the netcdf files as top-level attributesglider_devices: This is a list of the glider devices, and any information about them like make, mode, serial number. This is optional, and again is added to the netcdf top-level attributesnetcdf_variables: These are necessary, and map from sensor name (e.g.source: GPCTD_CONDUCTIVITY) to a data variable name (e.g.conductivity). The fields other thansource:are optional for the processing to run, and are placed in the attributes of the netCDF variable. However, note that many of these attributes are necessary for CF compliance.profile_variables: This is a mapping for variables that are per-profile, rather than timeseries. They include variables like a mean position and time for the profile, and a mean derived ocean velocities.
Process#
The example script is relatively straight forward if there is no intermediate processing. See Example processing script, below.
Data comes from an input directory, and is translated into a single CF-compliant netCDF timeseries file using the package dbdreader. Finally individual profiles are saved and a 2-D 1-m grid in time-depth is saved.
:::{note}
There is a version that does not require dbdreader to do the initial conversion from the Dinkum format to netCDF. However it is quite slow, particularly for full-resolution datasets, and less robust. We suggest using the slocum.raw_to_timeseries.
:::
It is possible that between these steps the user will want to add any screening steps, or adjustments to the calibrations. PyGlider does not provide those steps, but is designed so they are easy to add.
Example deployment.yaml#
metadata:
# https://github.com/ioos/ioosngdac/wiki/NGDAC-NetCDF-File-Format-Version-2
acknowledgement: Funding from Fisheries and Oceans Canada, Canadian Foundation
for Innovation, BC Knowledge Development Fund
comment: "Calvert Island test deeployment June 2019."
contributor_name: James Pegg, Jody Klymak, Tetjana Ross, Cailin Burmaster
contributor_role: Lead Technician, Principal Investigator, Co-PI, Technician
creator_email: jklymak@uvic.ca
creator_name: Jody Klymak
creator_url: http://cproof.uvic.ca
# R: real time. P: provisional
data_mode: 'R'
# date_created etc: added automatically
# numbers must be eenclosed by strings so YAML keeps as strings
deployment_id: '1'
deployment_name: 'dfo-rosie713-20190615'
deployment_start: '2019-06-15'
deployment_end: '2019-06-30'
format_version: IOOS_Glider_NetCDF_v2.0.nc
# id filled automatically...
glider_name: dfo-rosie
glider_serial: '713'
glider_model: Slocum G3 Deep
glider_instrument_name: slocum
glider_wmo: "999999"
institution: C-PROOF
keywords: "AUVS, Autonomous Underwater Vehicles, Oceans, Ocean Pressure,
Water Pressure, Oceans, Ocean Temperature, Water Temperature,
Oceans, Salinity/Density, Conductivity, Oceans,
Salinity/Density, Density, Oceans, Salinity/Density, Salinity"
keywords_vocabulary: GCMD Science Keywords
license: "This data may be redistributed and used without restriction or
warranty"
metadata_link: "https://cproof.uvic.ca"
Metadata_Conventions: CF-1.8, Unidata Dataset Discovery v1.0
naming_authority: "ca.uvic.cproof"
platform_type: "Slocum Glider"
processing_level: "Data provided as is with no expressed or implied
assurance of quality assurance or quality control."
project: SaanichInletTest19
project_url: http://cproof.uvic.ca
publisher_email: jklymak@uvic.ca
publisher_name: Jody Klymak
publisher_url: http://cproof.uvic.ca
references: cproof toolbox URL
# https://www.nodc.noaa.gov/General/NODC-Archive/seanamelist.txt
sea_name: Coastal Waters of Southeast Alaska and British Columbia
source: Observational data from a profiling glider.
standard_name_vocabulary: CF STandard Name Table v72
summary: Manufacturer test in Saanich Inlet.
transmission_system: IRRIDIUM
wmo_id: "999999"
glider_devices:
pressure:
make: Micron
model: Pressure
serial: '104702'
ctd:
make: Seabird
model: SlocumCTD
serial: '9507'
long_name: Seabird SlocumCTD
make_model: Seabird SlocumCTD
factory_calibrated: " "
calibration_date: " "
calibration_report: " "
comment: " "
optics:
make: Wetlabs
model: FLBBCDSLC
serial: '5059'
oxygen:
make: AADI
model: Optode4831
serial: '665'
# map between glider variables and netcdf variables. This shouldn't
# change too much.
netcdf_variables:
# Time and Place:
time:
source: sci_m_present_time
long_name: Time
standard_name: time
calendar: gregorian
units: seconds since 1970-01-01T00:00:00Z
axis: T
observation_type: "measured"
latitude:
source: m_lat
long_name: latitude
standard_name: latitude
units: degrees_north
axis: Y
comment: "Estimated between surface fixes"
observation_type: measured
platform: platform
reference: WGS84
valid_max: 90.0
valid_min: -90.0
coordinate_reference_frame: urn:ogc:crs:EPSG::4326
longitude:
source: m_lon
long_name: longitude
standard_name: longitude
units: degrees_east
axis: X
comment: "Estimated between surface fixes"
observation_type: measured
platform: platform
reference: WGS84
valid_max: 180.0
valid_min: -180.0
coordinate_reference_frame: urn:ogc:crs:EPSG::4326
heading:
source: m_heading
long_name: glider heading angle
standard_name: platform_orientation
units: degrees
pitch:
source: m_pitch
long_name: glider pitch angle
standard_name: platform_pitch_angle
units: degrees
roll:
source: m_roll
long_name: glider roll angle
standard_name: platform_roll_angle
units: degrees
# profile info:
waypoint_latitude:
source: c_wpt_lat
long_name: waypoint latitude
standard_name: latitude
units: degree_north
waypoint_longitude:
source: c_wpt_lon
long_name: waypoint longitude
standard_name: longitude
units: degree_east
# data parameters
conductivity:
source: sci_water_cond
long_name: water conductivity
standard_name: sea_water_electrical_conductivity
units: S m-1
instrument: instrument_ctd
valid_min: 0.
valid_max: 10.
observation_type: "measured"
accuracy: 0.0003
precision: 0.0001
resolution: 0.00002
temperature:
source: sci_water_temp
long_name: water temperature
standard_name: sea_water_temperature
units: Celsius
instrument: instrument_ctd
valid_min: -5.0
valid_max: 50.0
observation_type: "measured"
accuracy: 0.002
precision: 0.001
resolution: 0.0002
pressure:
source: sci_water_pressure
long_name: water pressure
standard_name: sea_water_pressure
units: dbar
conversion: bar2dbar
valid_min: 0.0
valid_max: 2000.0
positive: "down"
reference_datum: "sea-surface"
instrument: "instrument_ctd"
observation_type: "measured"
accuracy: 1.0
precision: 2.0
resolution: 0.02
comment: "ctd pressure sensor"
# optics:
chlorophyll:
source: sci_flbbcd_chlor_units
long_name: chlorophyll
standard_name: concentration_of_chlorophyll_in_sea_water
units: mg m-3
cdom:
source: sci_flbbcd_cdom_units
long_name: CDOM
units: ppb
backscatter_700:
source: sci_flbbcd_bb_units
long_name: 700 nm wavelength backscatter
units: "1"
# Oxygen
oxygen_concentration:
source: sci_oxy4_oxygen
long_name: oxygen concentration
standard_name: mole_concentration_of_dissolved_molecular_oxygen_in_sea_water
units: umol l-1
# derived water speed:
# water_velocity_eastward:
# source: m_final_water_vx
# long_name: mean eastward water velocity in segment
# standard_name: barotropic_eastward_sea_water_velocity
# units: m s-1
# coordinates: time depth latitude longitude
#
# water_velocity_northward:
# source: m_final_water_vy
# long_name: mean northward water velocity in segment
# standard_name: barotropic_northward_sea_water_velocity
# units: m s-1
# coordinates: time depth latitude longitude
profile_variables:
# variables for extract_L1timeseries_profiles processing step...
profile_id:
comment: Sequential profile number within the trajectory. This value is unique in each file that is part of a single trajectory/deployment.
long_name: 'Profile ID'
valid_max: 2147483646
valid_min: 1
profile_time:
comment: Timestamp corresponding to the mid-point of the profile
long_name: Profile Center Time
observation_type: calculated
platform: platform
standard_name: time
profile_time_start:
comment: Timestamp corresponding to the start of the profile
long_name: Profile Start Time
observation_type: calculated
platform: platform
standard_name: time
profile_time_end:
comment: Timestamp corresponding to the end of the profile
long_name: Profile End Time
observation_type: calculated
platform: platform
standard_name: time
profile_lat:
comment: Value is interpolated to provide an estimate of the latitude at the mid-point of the profile
long_name: Profile Center Latitude
observation_type: calculated
platform: platform
standard_name: latitude
units: degrees_north
valid_max: 90.0
valid_min: -90.0
profile_lon:
comment: Value is interpolated to provide an estimate of the latitude at the mid-point of the profile
long_name: Profile Center Longitude
observation_type: calculated
platform: platform
standard_name: longitude
units: degrees_east
valid_max: 180.0
valid_min: -180.0
u:
comment: The depth-averaged current is an estimate of the net current measured while the glider is underwater. The value is calculated over the entire underwater segment, which may consist of 1 or more dives.
long_name: Depth-Averaged Eastward Sea Water Velocity
observation_type: calculated
platform: platform
standard_name: eastward_sea_water_velocity
units: m s-1
valid_max: 10.0
valid_min: -10.0
v:
comment: The depth-averaged current is an estimate of the net current measured while the glider is underwater. The value is calculated over the entire underwater segment, which may consist of 1 or more dives.
long_name: Depth-Averaged Northward Sea Water Velocity
observation_type: calculated
platform: platform
standard_name: northward_sea_water_velocity
units: m s-1
valid_max: 10.0
valid_min: -10.0
lon_uv:
comment: Not computed
long_name: Longitude
observation_type: calculated
platform: platform
standard_name: longitude
units: degrees_east
valid_max: 180.0
valid_min: -180.0
lat_uv:
comment: Not computed
long_name: Latitude
observation_type: calculated
platform: platform
standard_name: latitude
units: degrees_north
valid_max: 90.0
valid_min: -90.0
time_uv:
comment: Not computed
long_name: Time
standard_name: time
calendar: gregorian
units: seconds since 1970-01-01T00:00:00Z
observation_type: calculated
instrument_ctd:
comment: pumped CTD
calibration_date: "2017-12-24"
calibration_report: 20171224_Seabird_SlocumCTD_SN9446_calibrations.pdf
factory_calibrated: "yes"
long_name: Seabird Glider Payload CTD
make_model: Seabird GPCTD
platform: platform
serial_number: "9446"
type: platform
Example processing script#
# -*- coding: utf-8 -*-
import logging
import os
import pyglider.ncprocess as ncprocess
import pyglider.slocum as slocum
import pyglider.utils as pgutils
logging.basicConfig(level='INFO')
binarydir = './realtime_raw/'
rawdir = './realtime_rawnc/'
cacdir = './cac/'
sensorlist = './dfo-rosie713_sensors.txt'
deploymentyaml = './deploymentRealtime.yml'
l1tsdir = './L0-timeseries/'
profiledir = './L0-profiles/'
griddir = './L0-gridfiles/'
scisuffix = 'tbd'
glidersuffix = 'sbd'
do_direct = True
# only do this for a real run, or something like this
real = False
if real:
os.system('rsync -av cproof@sfmc.webbresearch.com:/var/opt/sfmc-dockserver/' +
'stations/dfo/gliders/ ~/gliderdata/slocum_dockserver/')
os.system('rsync -av ~/gliderdata/slocum_dockserver/rosie_713/from-glider/* ' +
binarydir)
os.system('rm ' + rawdir + 'dfo* ' + rawdir + 'TEMP*.nc ' + l1tsdir + '* ' +
profiledir + '* ' + griddir + '* ')
if do_direct:
# turn *.sdb and *.tbd into timeseries netcdf files
outname = slocum.binary_to_timeseries(
binarydir, cacdir, l1tsdir, deploymentyaml, search='*.[s|t]bd',
profile_filt_time=20, profile_min_time=20)
else:
# turn *.EBD and *.DBD into *.ebd.nc and *.dbd.nc netcdf files.
slocum.binary_to_rawnc(
binarydir, rawdir, cacdir, sensorlist, deploymentyaml,
incremental=True, scisuffix=scisuffix, glidersuffix=glidersuffix)
# merge individual neetcdf files into single netcdf files *.ebd.nc and *.dbd.nc
slocum.merge_rawnc(
rawdir, rawdir, deploymentyaml,
scisuffix=scisuffix, glidersuffix=glidersuffix)
# Make level-1 timeseries netcdf file from th raw files...
outname = slocum.raw_to_timeseries(
rawdir, l1tsdir, deploymentyaml,
profile_filt_time=100, profile_min_time=300)
if True:
# make profile netcdf files for ioos gdac...
ncprocess.extract_timeseries_profiles(outname, profiledir, deploymentyaml)
# make grid of dataset....
outname2 = ncprocess.make_gridfiles(outname, griddir, deploymentyaml)
pgutils.example_gridplot(outname2, './gridplot2.png', ylim=[150, 0],
toplot=['potential_temperature', 'salinity',
'oxygen_concentration', 'chlorophyll', 'cdom'])