Abstract. Traffic sign detection and recognition can be divided in threemain problems: Automatic location, detection and categorization of trafficsigns. Basically, most of the approaches in locating and detecting oftraffic signs are based on color information extraction. But the main issueof using color information is to select the most proper color spaceto assure robust color analysis without influencing the exterior environment.Given the strong dependence on weather conditions, shadows andtime of the day some authors focus on shape-based sign detection (e.
g.Hough transform, ad-hoc models based on Canny edges or convex hulls).Recognition of traffic signs has been addressed by a number of popularclassification techniques ranging from simple template matching (e.g.cross-correlation similarity) to sophisticated Machine learning techniques(e.g. support vector machines, boosting, random forest, etc.
) are implementedto assure a straightforward outcome necessary for a real end-usersystem. Moreover, extending the traffic sign analysis from isolated framesto videos can allow significant reduction in the number of false alarm ratiosas well as to increase the precision and the accuracy of the detectionand recognition process.Keywords: Traffic sign detection and recognition(TSDR), Color-baseddescription,Shape-based description, Uncontrolled , Environments , Multiclassclassification1 IntroductionEmbedded and Intelligent Automated system for vehicles for safety in transportationhas been the limelight of research in the Computer Vision and PatternRecognition community for the more than three decades. In Pacilik 1, a timeline from the currently popular methods of Traffic Sign Detection and RecognitionSystem to the paramount study of it in Japan, 1984 can be outlined.
Foryears, researchers have been addressing the difficulties of detecting and recognizingtraffic signs. The most common automated systems belonging to trafficsigns detection and recognition comprise of one or two video cameras mountedon the front of the vehicle (e.g. a geo van). Recently, some geo vans also haveanother camera at the rear end and/or the side of the vehicle recording the signsbehind or alongside the vehicle.
The cars are retrofitted with a PC system foracquiring the videos, or specialized hardware for driving assistance applications.Road signs have specific properties that distinguish them from other outdoor objects.Operating systems for the automated recognition system of road signs aredesigned to identify these properties.
Traffic sign recognition systems have threemain parts: Location of the region of interest and color segmentation. Detectionby verification of the hypothesis of the presence of the sign (e.g. equilateral triangles,circles, etc.). Categorization/Recognition of the type of traffic sign andthen detection of the signs from outdoor images is the most complex step inthe automated traffic sign recognition system 2.
Many issues make the problemof the automatic detection of traffic signs difficult such as changeable lightconditions which are difficult to control (lighting varies according to the timeof the day, season, cloud cover and other weather conditions); presence of otherobjects on the road (traffic signs are often surrounded by other objects producingpartial occlusions, shadows, etc.). The research takes a precarious turn whentrying to think the possibilities that can cause false positives since the algorithmhas to take camera distance and view angle of the recorded image along withpossible deformation caused by external factors to the signs appearance intoaccount Hence, any robust automatic detection and recognition system mustprovide straightforward results that are not affected by perspective distortion,lighting changes, partial occlusions or shadows 3. Ideally, the system shouldalso provide additional information on the lack of visibility, poor conditions andpoor placement of traffic signs. This paper discusses the research progress of realtime Traffic Sign Detection and Recognition (TSDR). Also it reviews the papersbased on three main parts of TSDR namely color segmentation, shape detectionand recognition.2 Core Back ground Study2.1 Color SegmentationSign detection using color is based on the five typical colors defined in standardtraffic signs (red, blue, yellow, white and black).
Robust color segmentation especiallyconsidering non-homogeneous illumination is given priority, since errorsin segmentation may be propagated in the following steps of the system.2.2 Shape DetectionDetection of Traffic Signs via its shape follows the defining algorithm of shapedetection i.e. to finding the contours and approximating it to reach a final decisionbased on the number of contours. But there are some slight difficultiesin shaped based detection. Notable shape based detection problems for TrafficSigns are discussed below.
Traffic signs are mostly designed in basic shapes likecircle, triangle, pentagon etc. to make it easily visible. But similar shaped objectsalso exist in the surroundings that are not traffic signs. Traffic Signs areprone to physical damage and being obstructed from view. The size of trafficsigns compared to its real size depends on factors such as the distance betweenthe camera lens and traffic sign. The camera view might also be disoriented verticallyor horizontally.
Moreover factors like small object size of traffic sign inimages and slanted angle of view create difficulties in the detection phase due tochange in aspect ratio. Variation in daylight or colors do not affect shape baseddetection. Small roughly distinguishable traffic signs in images makes it ratherdifficult to approximate contours and so robust edge detection and recognitionalgorithms are necessary.2.3 Learning Based DetectionDetection of Traffic Signs based on Deep Learning is a fairly new approachas conventional means of Detection are far too static for real time detection.Artificial Neural Network algorithms are widely used to collect a large data setof traffic signs that have been preprocessed to detect objects accurately. Moretraining data results in further accuracy increase of the method. Drawbacks ofdeep learning detection notably fuzzy logic and neural networks are that theyrequire a large amount of resources during the learning process.
But the pros offast and accurate detection and recognition is well worth the time consuminglearning phase.2.4 RecognitionThe detection phase results in the output of a number of detected shapes thatare referred to as “candidate objects” in some research papers. These candidateobjects contain the deciphered traffic sign shapes. Candidate objects are sent toa recognizer and then to the classifier which comes to the decision whether theinput is rejected, false positives or actual traffic signs. The detected object areidentified according to their sign codes. A good recognition system must meetsome criteria to be called efficient. Some are mentioned below:1.
Ability to differentiate false positives and rejected objects from real ones ina short amount of time2. Robustness in defining size, position and geometrical status i.e. vertical orhorizontal orientation, of the traffic sign in the image3. Wary of noise4. Requiring Low Computational cost and time for real time applications5. Ability to be trained with large dimensions of data set with prior informationon road signs to match with3 Previous work3.1 Review Based on MethodsColor information is the staple method used in segmentation of image 7-12.
Poor lighting, strong illumination and adverse weather conditions reduce the performanceof color information based road sign detection. These problems wereovercome by using Color models such as HSV 14, 9, 11, 25, 26, YUV 13and CIECAM9715. Segmentation was done by Shadeed et al. 13 by implementingthe U and V chrominance channels of YUV space where U is regardedas positive and V as negative for red colors. The hue channel of HSV color space,in combination with YUV space information was used to segment red coloredtraffic signs.
Gao et al. 15 applied a quad-tree histogram method to segmentthe image based on the hue and chroma values of the CIECAM97 color model.The CIECAM97 color model consisting of hue and chroma values were used byGao et al. 15 to segment images implementing the quad-tree histogram method.Thresholding is a common practice in segmentation. The hue channel of HSVcolor space was thresholded by Malik et al.
9 for color segmentation of red trafficsigns. Some research papers prioritize shape information over color information.Grayscale images are used in shape information method. Loy and Zelinksy 17theorized a method to highlight points if interest that detect octagonal, squareand triangular traffic signs using local radial symmetry.Extraction of feature vectors from segmented region of interest (ROI) is necessaryfor the recognition process. Scale Invariant Feature Transform (SIFT),SURF and Binary Robust Invariant Scalable Keypoints (BRISK) feature descriptorwere implemented by the authors in 19 along with the comparisonof these feature vectors in contrast with others. Histogram Oriented Gradient(HOG) feature vector is another method in which 20 classified traffic signs.Classifier models are created to train feature vectors that can distinguish betweendifferent traffic signs in supervised learning paradigm.
Classifiers such asANN 6, Adaboost 7,24, SVM 12,21, 23 are prominent candidates in therecognition of traffic signs. A novel ROI extraction method, called High ContrastExtraction in combination with occlusion robust recognition method basedon Extended Sparse Presentation Classification (ESRC) was introduced in therecognition process in 20.3.
2 Review Based on FrameworksEven though Colour Feature and Neural Networks 24 use HSV, noise reductionetc. in pre-processing to minimalize error, there can be change in color ofthe physical object i.e. the color of traffic sign might change after transitionto HSV and Color Segmentation. Color Probability Map and Artificial NeuralNetworks27 method use minimum computational resource to provide with highquality images to detect and validate which require high memory space.
In AutoAssociative Neural Networks 28 change of orientation, weather conditions, lightingand also speed of the vehicle will decrease accuracy in real time conditions.Deep Convolutional Neural Networks, Real Time Detection and Recognition 29accuracy is dependent solely on weather and illumination conditions along withthe number of data sets used to train the algorithm. Selective Search basedConvolutional Neural Network 30 allows data of different sizes; after color segmentationit enables specific region searching with faster computation times butdecreases Speed of Deep Learning CNN algorithm as there are too many regionsto train and most of them are not real signs but false positives of detection whichneeds to be further improved. Region Based Convolutional Neural Networks(RCNN)and Region Proposal Network 31 are highly efficient since it shares convolutionsacross individual proposals.
It also performs bounding box regressionto further enhance the quality of the proposed regions. It enables fast and rapiddetection for detection and recognition phase. Even though it is very fast it isa three stage training phase for R-CNN and it requires a large amount of spacealong with sufficient GPU power.
Random Forests and multiple features extraction34 method is robust and tolerant to noise for using random forest comparedto other classifiers but the main drawback of this method is basic thresholding.Census Transform and Multilevel Support Vector Machine 35 method displayshigh illumination-invariant accuracy in detection and recognition but in case ofurban area its accuracy decreases. Principle Component Analysis (PCA) andMulti-Layer Perception Network (MLP) using Morphological 38 classificationsare disadvantageous because it can not detect damaged signs. Adaptive neurofuzzy inference system(ANFIS) 43 method is independent of color segmentationprocess. It reduces the computational cost and also produces a higher recognitionsuccess rate but is extremely vulnerable to the illumination change. BilateralChinese Transform (BCT), Vertex and Bisector Transform(VBT) 36 reduce theROI but their accuracy rate declines considerably.
3.3 Review Based on Experimental ResultsIn 28 Color Segmentation Accuracy is 93.3% and its detection rate for test datasets if 14 out of 15 in daylight and 19 out of 20 in shadow respectively. AutoAssociative Neural Networks (AANN) has a 100% accuracy in daylight but 4 falsepositives must be added to the 14 test data sets to recognize 14. For Shadow itis 94.7% with 5 false positives added to the data set of 19 to detect 18 finally.
At 29 Training with only Positive samples i.e. real signs and after that mixingtraining data with 25,000 samples of real signs and 78,000 false positives, the rateof learning is 0.01 per 100000 iterations with accuracy of 92.
63% where 39209training data was used and 1000 random test cases were selected as input. In 301918 images were used as data sets for detection with 91.69% accuracy, 684 falsepositives out of 20 million window frames in 1918 images were observed.
2520images for classification in 32×32 size was taken from detection and randomlyused for training. Accuracy was observed at 93.77%. At 31 proposed methodhad a 98.76% success rate. Compared to the propose Multi fusion Multi Classifiermethod Complementary features had a success rate 98.65%, Multi-scale CNN98.31%, SRGE 98.
19%, Random forests 96.14%, LDA on HOG2 95.68%.4 ObservationThe environment for Traffic Sign Detection and Recognition system is an adverseone with many unaccounted variables mostly physical. Some of the problems anddifficulties while the system is active and working are discussed below:1. Recognition becomes difficult when the traffic signs are exposed to sunlightand air resulting in the color of the signs to fade2. Adverse conditions and natural disasters that affect visibility namely fog,rain, storms etc.3.
Constant change in sunlight brightness depending of time, the change ifseasons and lack of light due to shadows created by other objects in thesurroundings4. Changing light conditions along with viewing geometry, illuminant colorand illumination geometry affect color information which is very sensitiveto change.5. Presence of obstacles blocking the view of the camera such as buildings,vehicles, pedestrians, vegetation etc.6. Presence of objects shaped similar to traffic sign shapes like circular, triangular,pentagonal7.
Physical damage, image distortion etc. may create false positives or negativeresults8. Size of the sign varies according to the distance between the camera and thesign. Traffic signs may appear at a different angle of view due to the imagingorientation and cause discrepancies to the calculation of the size9. Acquired images often suffer from motion blur due to vibration and movementof a running vehicle 24. Prediction of this motion blur beyond a certainthreshold is not feasible because vehicular movement has variant speed andacceleration which is unknown to the recognition process. It is possible tomake an assertion about the movement of objects in the future if the motionis continuous and unchanged.10.
Sign boards can appear to have bright white or nearwhite spots due to firstsurface reflection from the light sources. In first surface reflection the lightis reflected prior to penetrating to a depth where certain wavelengths areabsorbed, thereby imparting a color associated with the sign. This is calledhighlight.11. Real time application application makes it hard to maintain both accuracywith constantly changing fps as the ROI changes every second.12.
In Night Time without high resolution camera real time application lead tothe presence of noise13. Vandalism of sign boards by people who put stickers or write on them ordamage the signs by changing the pictograms within it making it unrecognizable.14. Different countries use different colors and different pictograms, a standarddatabase for evaluation of existent classification methods is unavailable5 DiscussionIt is extremely hard for the detection and recognition of road and traffic signsframeworks to have high robustness of color segmentation, high insensitivityto noise and brightness variations and invariant to geometrical effects such astranslation, inplane, outplane rotations and scaling changes in the real timeimage. Due to the continuously changing frame it is quite difficult to produce aresult that is both nearly accurate and has a low computational time.6 ConclusionThe primary purpose of the research is to develop a TSDR system in real time.By selecting a proper threshold for color segmentation and shape based detectionthen developing a fast classifier to make it applicable in real time is the mainobjective of this research.