README.adoc

Semantic Segmentation

Image Semantic Segmentation based on the state-of-art DeepLab Tensorflow model.

Semantic Segmentation is the process of associating each pixel of an image with a class label, (such as flower, person, road, sky, ocean, or car). Unlike the Instance Segmentation, which produces instance-aware region masks, the Semantic Segmentation produces class-aware masks. For implementing Instance Segmentation consult the Object Detection Service instead.

The JsonMapperFunction permits converting the List<ObjectDetection> into JSON objects, and the ObjectDetectionImageAugmenter allow to augment the input image with the detected bounding boxes and segmentation masks.

Usage

Add the semantic-segmentation dependency to your pom (use the latest version available):

<dependency>
    <groupId>org.springframework.cloud.fn</groupId>
    <artifactId>semantic-segmentation-function</artifactId>
    <version>${revision}</version>
</dependency>

<dependency>
    <groupId>org.springframework.cloud.fn</groupId>
    <artifactId>object-detection-function</artifactId>
    <version>${revision}</version>
</dependency>

Following snippet demos how to use the PASCAL VOC model to apply mask to an input image

SemanticSegmentation segmentationService = new SemanticSegmentation(
  "https://download.tensorflow.org/models/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz#frozen_inference_graph.pb", // (1)
  true); // (2)

byte[] inputImage = GraphicsUtils.loadAsByteArray("classpath:/images/VikiMaxiAdi.jpg"); // (3)

byte[] imageMask = segmentationService.masksAsImage(inputImage); // (4)
BufferedImage bi = ImageIO.read(new ByteArrayInputStream(imageMask));
ImageIO.write(bi, "png", new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_masks.png"));

byte[] augmentedImage = segmentationService.augment(inputImage); // (5)
IOUtils.write(augmentedImage, new FileOutputStream("./semantic-segmentation-function/target/VikiMaxiAdi_augmented.jpg"));

1. Download the PASCAL 2012 trained model directly from the web. The frozen_inference_graph.pb is the name of the model file inside the tar.gz archive.

2. Cache the downloaded model locally

3. Load the input image as byte array

4. Read get the segmentation mask as separate image

5. Blend the segmentation mask on top of the original image

Models

Based on the training datasets, three groups of pre-trained models provided:

	DeepLab models trained on PASCAL VOC 2012
	DeepLab models trained on Cityscapes
	DeepLab models trained on ADE20K

Select the model you want to use, copy its archive download Url and add a #frozen_inference_graph.pb fragment to it. Later fragment is the frozen model’s file name inside the archive

Tip	Download the archive and uncompress the frozen_inference_graph.pb for required model. Then use the file://<local-file-name>; URI schema.

Also, convenience there are a couple of models, extracted from the archive and uploaded to bintray:

PASCAL VOC 2012 (default)	https://dl.bintray.com/big-data/generic/deeplabv3_mnv2_pascal_train_aug_frozen_inference_graph.pb
CITYSCAPE	https://dl.bintray.com/big-data/generic/deeplabv3_mnv2_cityscapes_train_2018_02_05_frozen_inference_graph.pb
ADE20K	https://dl.bintray.com/big-data/generic/deeplabv3_xception_ade20k_train_2018_05_29_frozen_inference_graph.pb

References:

• Semantic Image Segmentation with DeepLab in TensorFlow

• DeepLab Project

• How to re-train DeepLab Segmentation models using Transfer Learning