Introduction

On Feb 21st, Michael and I tested the autonomous steering and cruise control system. Unfortunately, the testing was largely unsuccessful. We encounter many issues with the system and testing conditions. This prompted me to think about the robustness of our systems, specifically the semantic segmentation system.

IMG-7775
Testing on Feb 21st. Thanks to Ms. Richardson

Shadows

Convolutional neural networks are notoriously bad at handling shadows. I suspect that our segmentation network relies on colors and shapes to perform pixel-wise classification. To make matter worse, the training dataset doesn’t have many shadowy training images. I ran some test on shadowy images of asphalt pavements.

Left is the original image, which has a shadow on the left. The right is the segmentation results. The circled region indicates that the network considers the shadow both as road and also as an obstacle.

sd-1.jpg
result3_0.png

The shadow on this road is not significant enough to interfere with the segmentation predict. While struggling to classify the edges of the road precisely, the network performed very well in general.

sd-3.jpg
result3_2.png

Classifying this road is the most difficult. As you can see, the network has no problem classifying the sky or trees. But when it comes to segmenting the road, it thinks everything is a part of a large obstacle. If this were a real situation, the vehicle would stop and get stuck in the middle of the road.

Wet Road

Unfortunately, the network performed extremely poorly when it encountered wet road yesterday during testing. I suspect the reflect of the sun and the strange colors of the road.

result3_6.png
test_7.JPG

 .

The network classified only successfully segmented parts of the trees and sky. The whole entire road is classified as an obstacle.

result3_1.png
test_2.jpg

Poor segmentation results. Interestingly, the network considers the dry asphalt as road, but the wet road as an obstacle. As usual, the trees and the sky are classified correctly.

Glares from the Sun

Another potential issue that arose yesterday was related to the glare of the sun. We were facing west yesterday afternoon and the sun was pretty much shinning directly on the camera. There is no specific evidence that points to this issue in our system, but it’s very likely that glares will cause issues.

sn-1.jpg
result3_0.png

The results turned out better than expected. First of all, the network can segment roads correctly with the glare. However, it missed parts of the tree. If the golf cart were driving down this path, the mis-classification would not affect the performance.

result3_2.png
sn-3

The road was classified relatively well. However, the network was confused about the differences between the cars and the trees.

Conclusion

The robustness of the network is not disappointing. It does have lots of issues with shadows, which is a general issue that plagues most ConvNets.

Frankly, we don’t have a good solution to the current problem of miss-classification under certain circumstances. The most straightforward solution would be to diversify the dataset and include more images with shadows and reflections. However, creating that dataset is very difficult. The second solution is to use a better network for segmentation, such as Mask RCNN. The tradeoff for better accuracy is speed and memory. Using a better network would mean upgrading the current vehicle computer.

For now, the simple ENet segmentation network works just fine. If you are interested in the source code, please visit the Github repo.

Please don’t hesitate to contact me if you have any questions, comments or concerns. Thanks for stopping by!

If you liked the post, please leave a like. You might enjoy these posts as well:

10 thoughts on “The Robustness of the Semantic Segmentation Network

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s