Large-scale Continual Road Inspection:
Visual Infrastructure Assessment in the Wild

This work develops a method to inspect the quality of pavement conditions based on images captured from moving vehicles. This task is challenging because the appearance of road surfaces varies tremendously, depending on the construction materials (e.g., concrete, asphalt), the weather conditions (e.g., rain, snow), the illumination conditions (e.g., sunny, shadow), and the interference of other structures (e.g., manholes, road marks). This problem is amplified by the lack of a sufficiently large and diverse dataset for training a pavement classifier. Our first contribution in this paper is the development of a method to create a large-scale dataset of pavement images. Specifically, using map and GPS information, we match the ratings by government inspectors found in public databases to Google Street View images, creating a dataset containing more than 700K images from 70K street segments. We use the dataset to develop a deep-learning method for road assessment, which is based on Convolutional Neural Networks, Fisher Vector encoding, and UnderBagging random forests. This method achieves an accuracy of 58.2% and significantly outperforms various other texture classification methods.

The dataset is divided into 72 small packages.

We provide the ground truth files used in our paper in Matlab .mat file. It is a 71601*4 matrix. The first column is the index of the street, the second column is the number of images in this street segment. The third column is a fine grained rating of the street segment while the last column is the rating we used in the paper where 2 means GOOD, 1 means FAIR and 0 means POOR.

For example, row 10335 is [10547, 5, 6, 1], which means image 10574_1.jpg, 10574_2.jpg, ..., 10574_5.jpg are FAIR condition.

As the images in Google Street View keep being updated, as well as the publicly available records, you may want to create a dataset on your own. The pavement condition records can be found at NYC Open Data website. When this webpage was set up, the time of the last update was Aug 8, 2017. You can download the records in your preferred format.

The code to fetch images from Google Street View API is here under the MIT license. Please note that, the provided code is newer than the code used in the paper. The new code can also grab the timestamp of the image. So you can tell the real time gap between the time when the image was taken and when the corresponding road was rated. It is worth noting that you need to link a Google account to the street view service and obtain a token to use this code. Please refer to the code for details.

If using the dataset or code, please cite:
Large-scale Continual Road Inspection: Visual Infrastructure Assessment in the Wild, Ke Ma and Minh Hoai and Dimitris Samaras. Proceedings of British Machine Vision Conference (BMVC), 2017 [ BibTex ]

This work was supported by the Stony Brook University SensorCAT, FRA DTFR5315C00011, a gift from Adobe, the Partner University Fund 4DVision project, and New York State Infrastructure, Transportation and Security Center (ITSC). The authors would like to thank Professor Anil Yazici from the Stony Brook University Department of Civil Engineering for fruitful discussion and feedback.

If you have any question, please send email to kemmaATcsDOTstonybrookDOTedu.