Published on Tue Oct 15 2019

SegSort: Segmentation by Discriminative Sorting of Segments

Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, Liang-Chieh Chen

We present the SegSort, as a first attempt using deep learning for unsupervised semantic segmentation. When supervision is available, SegSort shows consistent improvements over conventional approaches based on pixel-wise softmax training.

0
0
0
Abstract

Almost all existing deep learning approaches for semantic segmentation tackle this task as a pixel-wise classification problem. Yet humans understand a scene not in terms of pixels, but by decomposing it into perceptual groups and structures that are the basic building blocks of recognition. This motivates us to propose an end-to-end pixel-wise metric learning approach that mimics this process. In our approach, the optimal visual representation determines the right segmentation within individual images and associates segments with the same semantic classes across images. The core visual learning problem is therefore to maximize the similarity within segments and minimize the similarity between segments. Given a model trained this way, inference is performed consistently by extracting pixel-wise embeddings and clustering, with the semantic label determined by the majority vote of its nearest neighbors from an annotated set. As a result, we present the SegSort, as a first attempt using deep learning for unsupervised semantic segmentation, achieving performance of its supervised counterpart. When supervision is available, SegSort shows consistent improvements over conventional approaches based on pixel-wise softmax training. Additionally, our approach produces more precise boundaries and consistent region predictions. The proposed SegSort further produces an interpretable result, as each choice of label can be easily understood from the retrieved nearest segments.

Fri Nov 24 2017
Computer Vision
Deep Extreme Cut: From Extreme Points to Object Segmentation
This paper explores the use of extreme points in an object as input to obtain precise object segmentation. We do so by adding an extra channel to the image in the input of a convolutional neural network. The CNN learns to transform this information into a segmentation of an object.
0
0
0
Thu Feb 11 2021
Computer Vision
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals
Unsupervised semantic segmentation is an important problem in computer vision. This problem remains rather unexplored, with a few exceptions. In this paper, we make a first attempt to tackle the problem on datasets that have been traditionally utilized for the supervised case.
0
0
0
Mon Apr 09 2018
Computer Vision
Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learning
This paper tackles the problem of video object segmentation. The proposed method supports different kinds of user input such as segmentation mask in the first frame.
0
0
0
Wed Mar 29 2017
Artificial Intelligence
LabelBank: Revisiting Global Perspectives for Semantic Segmentation
Semantic segmentation requires a detailed labeling of image pixels by object category. Information derived from local image patches is necessary to describe the detailed shape of individual objects. Global inference of image content can instead capture the general semantic concepts present.
0
0
0
Mon May 03 2021
Computer Vision
Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning
weakly supervised segmentation requires assigning a label to every pixel. This task is challenging as coarse annotations (tags, boxes) lack precise pixel localization. We propose 4 types of contrastiverelationships between pixels and segments in the feature space.
1
0
0
Tue Jul 26 2016
Computer Vision
Region-based semantic segmentation with end-to-end training
We propose a novel method for semantic segmentation, the task of labeling each pixel in an image with a semantic class. Our method combines the advantages of the two main competing paradigms. It improves the state-of-the-art in terms of class-average accuracy.
0
0
0