Imagery

ESD gliders may be equipped with either a ‘glidercam’ or a shadowgraph camera (see details here). This page describes the management and processing of these images once they come off of the glider.

Raw Imagery

Once the gliders are recovered, the images are copied from the memory cards and uploaded to GCP as described here. Specifically, raw images are uploaded to the bucket amlr-gliders-imagery-raw-dev, along with associated config files from the deployment. Image metadata files generated during the glider data processing are also written to the deployment folder within this bucket. Project/year/deployment folder structures mirror that of the glider data stored in amlr-gliders-deployments-dev.

Viewing Images

There are several ways to view raw or processed images. If you click down to an image itself, Google Cloud provides an image preview at the bottom of the ‘object’ page. However, you cannot view more than one image at a time. It is a goal of the ESD to build a cloud-native image viewer application to easily view multiple images at a time, but this is not a current project.

All raw imagery have been mirrored in VIAME-Web-AMLR. Thus, if you are familiar with VAIME-Web and accessing ESD’s deployment, then you can view imagery using the VIAME-Web annotator.

Finally, you may also use the gcloud CLI to download images to your computer, and then view them locally using your preferred program.

Shadowgraph Image Processing

Shadowgraph image processing workflow, v1

Shadowgraph image processing workflow, v2

Gliders deployed with shadowgraph cameras can easily collect more than 200,000 images during a single deployment. Therefore, it is impractical to have humans review and annotate all these images by hand, and these images must be processed using AI/ML methods.

Currently, ESD development of these methods is (mostly) on hold. This section describes the current status of these methods, as well as anticipated future directions.

There are three main steps to processing shadowgraph images; where and how these steps happen for ESD are further described below.

  1. Preprocessing: flat-fielding, masking, other processing as needed.
  2. Segmentation: Detect objects, and create regions of interest (ROIs; i.e. blobs or chips). This is done using the preprocessed image.
  3. Classification: Use a trained ML model to classify the ROIs. Have a trained biologist review and/or correct these predictions

Preprocessing and Segmentation

All preprocessing and object detection/segmentation steps are performed by code in the amlr-shadowgraph repo, developed by Randy Cutter based on work done by Ohman/Ellen for the Zooglider. Currently, all processing must be done through a single notebook: sg-flatfield-segm-loop.ipynb. There are active efforts to move this functionality from the Notebook to a Python module for easier use and development.

For instructions on how to run this code, or for more information about the algoroithms used, see the repo readme.

All currently processed imagery can be found in amlr-gliders-imagery-proc-dev, which like the raw imagery is mirrored to and thus viewable using ESD’s deployment of VIAME-Web. The amlr-shadowgraph code currently writes output to the amlr-imagery-proc-dev bucket; current efforts include generalizing this code to write directly to the amlr-gliders-imagery-proc-dev bucket. This output includes the following folders:

  • regions: Region of interest (ROI) blobs, created using Ohman/Ellen methods. This folder contains Dir#### folders, matching the raw data.
  • regions-tmser: Region of interest (ROI) blobs, created using an adapted version of Oregon State’s Threshold MSER In Situ Plankton Segmentation. Most deployments do not have this output, as this method has not been tuned to ESD imagery.
  • jpgorig-regions: jpg of the original (i.e., raw) image, with ROI boundaries from the Ohman/Ellen segmentation methods outlined in red.
  • images-imgff: Flat-fielded images, used in the segmentation
  • images-ffPCG: Flat-fielded images, which also have had pixel gamma correction factors applied.

Classification

Describe ISIIS image library

Describe models that Randy has trained

Describe running these models on ESD ROIs

Describe validation work, that Jen is doing

Future Directions

Back to top