public final class ImageSegmenter

Performs segmentation on images.

The API expects a TFLite model with TFLite Model Metadata..

The API supports models with one image input tensor and one output tensor. To be more specific, here are the requirements.

  • Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
    • image input of size [batch x height x width x channels].
    • batch inference is not supported (batch is required to be 1).
    • only RGB inputs are supported (channels is required to be 3).
    • if type is kTfLiteFloat32, NormalizationOptions are required to be attached to the metadata for input normalization.
  • Output image tensor (kTfLiteUInt8/kTfLiteFloat32)
    • tensor of size [batch x mask_height x mask_width x num_classes], where batch is required to be 1, mask_width and mask_height are the dimensions of the segmentation masks produced by the model, and num_classes is the number of classes supported by the model.
    • optional (but recommended) label map(s) can be attached as AssociatedFile-s with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the class name, i.e. ColoredLabel.getlabel() of the results. The display name, i.e. ColoredLabel.getDisplayName(), is filled from the AssociatedFile (if any) whose locale matches the `display_names_locale` field of the `ImageSegmenterOptions` used at creation time ("en" by default, i.e. English). If none of these are available, only the `index` field of the results will be filled.

An example of such model can be found on TensorFlow Hub..

Nested Classes

class ImageSegmenter.ImageSegmenterOptions Options for setting up an ImageSegmenter. 

Public Methods

static ImageSegmenter
createFromBuffer(ByteBuffer modelBuffer)
Creates an ImageSegmenter instance with a model buffer and the default ImageSegmenter.ImageSegmenterOptions.
static ImageSegmenter
createFromBufferAndOptions(ByteBuffer modelBuffer, ImageSegmenter.ImageSegmenterOptions options)
Creates an ImageSegmenter instance with a model buffer and ImageSegmenter.ImageSegmenterOptions.
static ImageSegmenter
createFromFile(Context context, String modelPath)
Creates an ImageSegmenter instance from the default ImageSegmenter.ImageSegmenterOptions.
static ImageSegmenter
createFromFile(File modelFile)
Creates an ImageSegmenter instance from the default ImageSegmenter.ImageSegmenterOptions.
static ImageSegmenter
createFromFileAndOptions(File modelFile, ImageSegmenter.ImageSegmenterOptions options)
Creates an ImageSegmenter instance from ImageSegmenter.ImageSegmenterOptions.
static ImageSegmenter
createFromFileAndOptions(Context context, String modelPath, ImageSegmenter.ImageSegmenterOptions options)
Creates an ImageSegmenter instance from ImageSegmenter.ImageSegmenterOptions.
List<Segmentation>
segment(long frameBufferHandle, ImageProcessingOptions options)
List<Segmentation>
segment(TensorImage image)
Performs actual segmentation on the provided image.
List<Segmentation>
segment(MlImage image)
Performs actual segmentation on the provided MlImage.
List<Segmentation>
segment(TensorImage image, ImageProcessingOptions options)
Performs actual segmentation on the provided image with ImageProcessingOptions.
List<Segmentation>
segment(MlImage image, ImageProcessingOptions options)
Performs actual segmentation on the provided MlImage with ImageProcessingOptions.

Inherited Methods

Public Methods

public static ImageSegmenter createFromBuffer (ByteBuffer modelBuffer)

Creates an ImageSegmenter instance with a model buffer and the default ImageSegmenter.ImageSegmenterOptions.

Parameters
modelBuffer a direct ByteBuffer or a MappedByteBuffer of the segmentation model
Throws
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error
IllegalArgumentException if the model buffer is not a direct ByteBuffer or a MappedByteBuffer

public static ImageSegmenter createFromBufferAndOptions (ByteBuffer modelBuffer, ImageSegmenter.ImageSegmenterOptions options)

Creates an ImageSegmenter instance with a model buffer and ImageSegmenter.ImageSegmenterOptions.

Parameters
modelBuffer a direct ByteBuffer or a MappedByteBuffer of the segmentation model
options
Throws
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error
IllegalArgumentException if the model buffer is not a direct ByteBuffer or a MappedByteBuffer

public static ImageSegmenter createFromFile (Context context, String modelPath)

Creates an ImageSegmenter instance from the default ImageSegmenter.ImageSegmenterOptions.

Parameters
context
modelPath path of the segmentation model with metadata in the assets
Throws
IOException if an I/O error occurs when loading the tflite model
IllegalArgumentException if an argument is invalid
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error

public static ImageSegmenter createFromFile (File modelFile)

Creates an ImageSegmenter instance from the default ImageSegmenter.ImageSegmenterOptions.

Parameters
modelFile the segmentation model File instance
Throws
IOException if an I/O error occurs when loading the tflite model
IllegalArgumentException if an argument is invalid
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error

public static ImageSegmenter createFromFileAndOptions (File modelFile, ImageSegmenter.ImageSegmenterOptions options)

Creates an ImageSegmenter instance from ImageSegmenter.ImageSegmenterOptions.

Parameters
modelFile the segmentation model File instance
options
Throws
IOException if an I/O error occurs when loading the tflite model
IllegalArgumentException if an argument is invalid
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error

public static ImageSegmenter createFromFileAndOptions (Context context, String modelPath, ImageSegmenter.ImageSegmenterOptions options)

Creates an ImageSegmenter instance from ImageSegmenter.ImageSegmenterOptions.

Parameters
context
modelPath path of the segmentation model with metadata in the assets
options
Throws
IOException if an I/O error occurs when loading the tflite model
IllegalArgumentException if an argument is invalid
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error

public List<Segmentation> segment (long frameBufferHandle, ImageProcessingOptions options)

Parameters
frameBufferHandle
options

public List<Segmentation> segment (TensorImage image)

Performs actual segmentation on the provided image.

ImageSegmenter supports the following TensorImage color space types:

Parameters
image a UINT8 TensorImage object that represents an RGB or YUV image
Returns
  • results of performing image segmentation. Note that at the time, a single Segmentation element is expected to be returned. The result is stored in a List for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error
IllegalArgumentException if the color space type of image is unsupported

public List<Segmentation> segment (MlImage image)

Performs actual segmentation on the provided MlImage.

Parameters
image an MlImage to segment.
Returns
  • results of performing image segmentation. Note that at the time, a single Segmentation element is expected to be returned. The result is stored in a List for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error
IllegalArgumentException if the storage type or format of the image is unsupported

public List<Segmentation> segment (TensorImage image, ImageProcessingOptions options)

Performs actual segmentation on the provided image with ImageProcessingOptions.

ImageSegmenter supports the following TensorImage color space types:

ImageSegmenter supports the following options:

Parameters
image a UINT8 TensorImage object that represents an RGB or YUV image
options the options configure how to preprocess the image
Returns
  • results of performing image segmentation. Note that at the time, a single Segmentation element is expected to be returned. The result is stored in a List for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error
IllegalArgumentException if the color space type of image is unsupported

public List<Segmentation> segment (MlImage image, ImageProcessingOptions options)

Performs actual segmentation on the provided MlImage with ImageProcessingOptions.

ImageSegmenter supports the following options:

Parameters
image an MlImage to segment.
options the options configure how to preprocess the image.
Returns
  • results of performing image segmentation. Note that at the time, a single Segmentation element is expected to be returned. The result is stored in a List for later extension to e.g. instance segmentation models, which may return one segmentation per object.
Throws
IllegalStateException if there is an internal error
RuntimeException if there is an otherwise unspecified error
IllegalArgumentException if the color space type of image is unsupported