Introduction
On Feb 21st, Michael and I tested the autonomous steering and cruise control system. Unfortunately, the testing was largely unsuccessful. We encounter many issues with the system and testing conditions. This prompted me to think about the robustness of our systems, specifically the semantic segmentation system.
Shadows
Convolutional neural networks are notoriously bad at handling shadows. I suspect that our segmentation network relies on colors and shapes to perform pixel-wise classification. To make matter worse, the training dataset doesn’t have many shadowy training images. I ran some test on shadowy images of asphalt pavements.
Left is the original image, which has a shadow on the left. The right is the segmentation results. The circled region indicates that the network considers the shadow both as road and also as an obstacle.
The shadow on this road is not significant enough to interfere with the segmentation predict. While struggling to classify the edges of the road precisely, the network performed very well in general.
Classifying this road is the most difficult. As you can see, the network has no problem classifying the sky or trees. But when it comes to segmenting the road, it thinks everything is a part of a large obstacle. If this were a real situation, the vehicle would stop and get stuck in the middle of the road.
Wet Road
Unfortunately, the network performed extremely poorly when it encountered wet road yesterday during testing. I suspect the reflect of the sun and the strange colors of the road.
.
The network classified only successfully segmented parts of the trees and sky. The whole entire road is classified as an obstacle.
Poor segmentation results. Interestingly, the network considers the dry asphalt as road, but the wet road as an obstacle. As usual, the trees and the sky are classified correctly.
Glares from the Sun
Another potential issue that arose yesterday was related to the glare of the sun. We were facing west yesterday afternoon and the sun was pretty much shinning directly on the camera. There is no specific evidence that points to this issue in our system, but it’s very likely that glares will cause issues.
The results turned out better than expected. First of all, the network can segment roads correctly with the glare. However, it missed parts of the tree. If the golf cart were driving down this path, the mis-classification would not affect the performance.
The road was classified relatively well. However, the network was confused about the differences between the cars and the trees.
Conclusion
The robustness of the network is not disappointing. It does have lots of issues with shadows, which is a general issue that plagues most ConvNets.
Frankly, we don’t have a good solution to the current problem of miss-classification under certain circumstances. The most straightforward solution would be to diversify the dataset and include more images with shadows and reflections. However, creating that dataset is very difficult. The second solution is to use a better network for segmentation, such as Mask RCNN. The tradeoff for better accuracy is speed and memory. Using a better network would mean upgrading the current vehicle computer.
For now, the simple ENet segmentation network works just fine. If you are interested in the source code, please visit the Github repo.
Please don’t hesitate to contact me if you have any questions, comments or concerns. Thanks for stopping by!
If you liked the post, please leave a like. You might enjoy these posts as well:
- Deep learning steering prediction
- Visualizing the Steering Model with Attention Maps
- Successfully Tested the Autonomous Steering System for the Self-Driving Golf Cart
- Predicting Steering Angle with Deep Learning — Part 2
- Predicting Steering Angle with Deep Learning — Part 1
- On the Road — Testing the Autonomous Golf Cart
- Testing the Steering System with NVIDIA Jetson
- Semantic segmentation
Neil Nie 
11 replies on “The Robustness of the Semantic Segmentation Network”