On Feb 21st, Michael and I tested the autonomous steering and cruise control system. Unfortunately, the testing was largely unsuccessful. We encounter many issues with the system and testing conditions. This prompted me to think about the robustness of our systems, specifically the semantic segmentation system.

Testing on Feb 21st. Thanks to Ms. Richardson


Convolutional neural networks are notoriously bad at handling shadows. I suspect that our segmentation network relies on colors and shapes to perform pixel-wise classification. To make matter worse, the training dataset doesn’t have many shadowy training images. I ran some test on shadowy images of asphalt pavements.

Left is the original image, which has a shadow on the left. The right is the segmentation results. The circled region indicates that the network considers the shadow both as road and also as an obstacle.


The shadow on this road is not significant enough to interfere with the segmentation predict. While struggling to classify the edges of the road precisely, the network performed very well in general.


Classifying this road is the most difficult. As you can see, the network has no problem classifying the sky or trees. But when it comes to segmenting the road, it thinks everything is a part of a large obstacle. If this were a real situation, the vehicle would stop and get stuck in the middle of the road.

Wet Road

Unfortunately, the network performed extremely poorly when it encountered wet road yesterday during testing. I suspect the reflect of the sun and the strange colors of the road.



The network classified only successfully segmented parts of the trees and sky. The whole entire road is classified as an obstacle.


Poor segmentation results. Interestingly, the network considers the dry asphalt as road, but the wet road as an obstacle. As usual, the trees and the sky are classified correctly.

Glares from the Sun

Another potential issue that arose yesterday was related to the glare of the sun. We were facing west yesterday afternoon and the sun was pretty much shinning directly on the camera. There is no specific evidence that points to this issue in our system, but it’s very likely that glares will cause issues.


The results turned out better than expected. First of all, the network can segment roads correctly with the glare. However, it missed parts of the tree. If the golf cart were driving down this path, the mis-classification would not affect the performance.


The road was classified relatively well. However, the network was confused about the differences between the cars and the trees.


The robustness of the network is not disappointing. It does have lots of issues with shadows, which is a general issue that plagues most ConvNets.

Frankly, we don’t have a good solution to the current problem of miss-classification under certain circumstances. The most straightforward solution would be to diversify the dataset and include more images with shadows and reflections. However, creating that dataset is very difficult. The second solution is to use a better network for segmentation, such as Mask RCNN. The tradeoff for better accuracy is speed and memory. Using a better network would mean upgrading the current vehicle computer.

For now, the simple ENet segmentation network works just fine. If you are interested in the source code, please visit the Github repo.

Please don’t hesitate to contact me if you have any questions, comments or concerns. Thanks for stopping by!

If you liked the post, please leave a like. You might enjoy these posts as well:

Posted by:NeilNie

Student at Columbia University, School of Engineering and Applied Sciences. Prev. software engineer intern at Apple. More than six years of experience developing iOS and macOS applications. Experienced in electrical engineering/microcontrollers. From publishing several apps to presented a TEDx talk in machine learning. Striving to use my knowledge, skills, and passion to positively impact the world.

11 replies on “The Robustness of the Semantic Segmentation Network

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.