资源描述
Computers and electronics in agriculture36 (2002) 215 -223Computer vision based system for apple surface defect detectionQingzhong Lia, Maohua Wangb, Corresponding authorE-mail address: mhwpublic.bta.net.cn (M. Wang).0168-1699/02/$ - see front matter 2002 Elsevier Science B.V. All rights reserved. PII: S0168- 1 699(02)00093-5, Weikang Gua11 Department of Information and Electronic Engineering, Zhejiang University, Hangzhou, Peoples Republicof Chinab Research Centre for Precision Agriculture, China Agricultural University, Beijing, Peoples Republicof ChinaAbstractA novel automated apple surface defect sorting experimental system based on computer image technology has been developed. The hardware system has the advantage of being able to inspect simultaneously four sides of each apple on the sorting line. The methods, including image background removal, defects segmentation and identification for stem-end and calyx areas, were developed. The results show that the experimental hardware system is practical and feasible，and that the proposed algorithm of defect detection is effective. 2002 Elsevier Science B.V. All rights reserved.Keywords: Machine vision; Apple; Surface defect1. IntroductionChina is a large agricultural country. Its annual apple production is over 17 million tons. Much of the sorting and grading process, however, is still not automated. Hand inspection of fruit is tedious and can cause eye fatigue; it is also subject to sorting errors due to different judgment by different persons. Although some quality inspection procedures such as color, size, and shape are performed by automated systems in western countries, the automation of the defect sorting process is still a challenging subject due to the complexity of the problem. Currently there are two main problems blocking the implementation of automatic apple grading. One is how to acquire the whole surface image of an apple by cameras at an on-line speed. The other is how to quickly identify the stem and calyx. To solve the first problem, Growe and Delwiche (1996), Tao (1996) used a roller conveyer system. The drawback of this method was that the camera above the conveyor cannot inspect the two end sides of the horizontal axes of the rolling fruits. For the second problem, Throop et al. (1997) developed two kinds of orienting devices. These devices were used to rotate apples of different varieties along the stemcalyx axes. But the results showed that the varieties that were successfully oriented with one system would not orient using the other device. Yang (1993) used structured lighting to identify the stem and calyx of apples. The major problem with the structured lighting is the misclassification of laser lines on the image. Wen aad Tao (1998) successfully developed a dual-camera NIR/MIR imaging method for apple defect recognition and stem-calyx identification. But the MIR camera is too expensive to use in China.The objective of the work described in this paper was to develop an experimental system that can inspect four sides of each apple，simultaneously, at on-line throughput (over three to four fruits per s) and the corresponding methods for effective defects segmentation and recognition.2. System setup overviewA system capable of inspecting four directions of each apple at on-line throughput was developed. The setup of the system is shown in Fig. 1. It consisted of a feeding unit，an apple uniform spacing unit，a machine vision system，and a sorting conveyor. The basic feeding conveyor transported the apples to the uniform spacing conveyor. Then, the apples were fed to the machine vision system for the defect inspection. Finally，the automatic sorting unit accomplished the apple grading operation.The machine vision system included a cup type conveyor, a lighting chamber for the desired spectrum and light distribution for fruit illumination, two cameras, and an image grabbing card with four input channels inserted in a microcomputer (processor speed: 500 MHz). . As an experimental system, the fruit-feeding system and the automatic sorting system were not constructed in the first stage of the research.To achieve a basically complete inspection of apples on the fruit sorting line，two identical monochromatic cameras were mounted above and below the conveyor, respectively. The setup of the vision system is shown in Fig. 2. The image sensors in the cameras had an actual resolution of 580 horizontal and 350 vertical TV lines. Each camera was synchronized to another timing source and had a variable electronic shutter. Identical 8.5 mm focal length C-mount lenses were attached to the cameras，with interference band-pass optical filters (840 nm) attached to the outside of each lens. The conveyor was composed of fruit cups without bottoms as shown in Fig. 2. Two mirrors were fixed on both sides of the conveyor; thus the camera above the conveyor took three side views of an apple, i.e. top and two sides. The camera below the conveyor took a bottom view of the fruit. Moreover, this imaging system was able to inspect several apples on the conveyor simultaneously. This scheme had the advantage of being able to inspect simultaneously four sides of each apple while it was traveling on the conveyor.3. Algorithm descriptionThe algorithm developed for the surface defect detection mainly included modules of image preprocessing, defect segmentation, stem-calyx recognition, and defect area calculation and grading. The algorithm is shown schematically in Fig. 3.3.1. Image background removal through a method of subtractionThe image backgrounds in the mirror and on the conveyor were different, so it was impossible to segment the parts of fruit by a simple threshold process. Therefore, a subtracting method was used, as depicted below:where g(x, y) is the image after its background has been removed, f(x, y）is the original image, b(x, y) is the background image, and T is the threshold.3.2 Defects segmentation by using reference apple imagesApples under inspection had substantially spherical shapes, resulting in curved distributed image intensity. This curved distribution caused the intensity values of the normal surface near the boundary to be lower than the intensity of the defect patches on the surface of the fruit. It is difficult to use any simple global threshold segmentation algorithm for defect extraction. Local adaptive methods could be used for defect segment extraction. However, the processing time prevents their practical use in real-time fruit sorting operations. Based on the reference image of an apple, Li and Wang (1999) developed a method to accomplish defect segmentation for a curved fruit image. In this method, a reference fruit image (RFI) was generated first and the original fruit image for inspection was then normalized to achieve the normalized reference fruit image (NRFI). Finally by subtracting normalized original fruit image (NOFI) from the NRFI and then by simple threshold processing, the defects could be extracted easily.3.3 Stemcalyx identification based on fractal features and artificial neural networkDuring the defect inspection process, it is difficult to distinguish the stem and calyx from true defects，because they are similar to defective spots in the image. Based on fractal dimensions and neural networks (NN), the authors of this paper developed a novel method to distinguish the stem-calyx concave area from true defects.Fractal is a term used to describe the shape and appearance of the objects, which have the properties of self-similarity and . scale invariance. Fractal dimension is a scale independent measure of the degree of surface roughness or boundary irregularity. Although the intensity of stem-calyx and true defects are similar, their fractal features may be different. Moreover，fractal analysis in the frequency domain only depends on the frequency distribution of the image surface. These fractal textural features would be independent of the variation of ambient light intensity and orientation of the apples being sorted. So this method is suitable for apple sorting operations where apples are in random orientations. The image distribution can be regarded as a three-dimensional curved surface. Based on the above consideration, five fractal dimensions including one traditional fractal dimension and four oriented fractal dimensions were selected as the features of the image spots produced by stem-calyx concave area or true defects. The four oriented fractal dimensions (D1, D12, D3, D4) are shown in Fig. 4. In fact，the oriented fractal dimensions were the fractal dimensions of the curves in the corresponding directions (Fig. 5). The five fractal dimensions are calculated by the method derived by Li and Wang (2000). The digital image can be depicted as: Z=f(x, y), where (x，y) is the coordinates of a pixel; Z is the gray value. Assuming the area of the image is M x M the x-y plane of the image is divided into grids with area . The maximum and the minimum of gray values in the grid are expressed as u(i,j) and b(i,j)，respectively. And their difference is d= u(i,j)- b(i,j).Then the total nonempty box number (N) for all the grids is calculated as：For all the given a data set from a series of points log , log N can be obtained. Through linear regression of the points (log , log N), the minus slope of the regression line gives the estimated fractal dimension. The four oriented dimensions can be estimated by using a similar method.A feedforward backpropagation (BP) NN algorithm was used to classify stem- calyx from true defect areas. The feedforward network structure was suitable for handling nonlinear relationships between input and output variables of prediction- D4 D3 D2related problems. The designed BP network is shown in Fig. 6. The NN model had five input nodes, one hidden layer with four hidden nodes，and one output node. During the training process, the weights of the network were updated after each pass through all the training samples. The convergence of the learning was judged by two conditions: whether the mean squared error for all training samples were smaller than a tolerance value, and whether the output errors for each training sample were smaller than another predefined tolerance valueFig. 5. Oriented fractal curve. Hidden layer3.4 Real-time implementation of apple surface defect detectionThe real-time implementation of apple surface defect detection is divided into two stages. The first is the segmentation of doubtful spot areas, including defects and stem-calyx areas，by the method described in Section 3.2. The segmentation results show that the stem-calyx areas are often with bigger areas. So in the second stage, the segmented spots with areas bigger than a given value are further processed for distinguishing stem-calyx concave areas from defects by the method presented in Section 3.3.4 Tests and resultsThe algorithm was used to detect defects and stem-calyx areas in forty samples of Fuji apples. Some results are shown in Fig. 7，where (a), (c) , (e), and (g) are the(a)(b)(c)(d)(e)(f)(g)(h)Fig. 7. Defects segmentation results. (a) (c), (e), (g) Original image; (b), (c), (f), (h) segmented defects.original image of the apples to be inspected, and (b), (d)，（f)，and (h) are the defect segmentation results. These results show that the defects and stemcalyx areas were basically extracted. The segmented spots with area bigger than a given value were further processed for distinguishing stem-calyx concave area from defects by the method in Section 3.3. Table 1 lists some results of the stem-calyx recognition by the BP network. If the output value of the network is near 1，the detected patch is the stem-calyx area. Similarly, if the output value is near 0，the detected patch is a true defect area. The test results show that the accuracy of the network classifier was over 93%. The Number 16 and 23 defect patches in Table 1 were rotten areas and the degree of rot was so high that their surfaces were concave. The results show that the input fractal features are effective for classifying concave surfaces from normal fruit surfaces. Because the stem-calyx patches are usually concave in shape, the proposed method for stem-calyx recognition is feasible. The processing time for the defect detection and grading for one apple was 320 ms with microcomputer (processor speed: 500 MHz)5.ConclusionsThe results show that the input fractal features are effective for classifying concave surfaces from the normal fruit surfaces. Because the stem-calyx particles are usually concave in shape the proposed method for stem-calyx recognition is feasible.The system has the advantage of being able to inspect, simultaneously, four aspects of each apple on a sorting line. Furthermore, based on the reference image of an apple, the developed method of defect segmentation can extract most of the surface defects on apples at a speed commensurate with the requirements of a practical grading system，which is the objective of further research.AcknowledgementsIt is gratefully acknowledged that this work is supported under University Doctoral Course Special Fund (Project No. 950801).ReferencesGrowe,T.G., Delwiche, M.J., 1996. Real-time defect detection in fruitpart I: design concepts and development of prototype hardware. Trans. ASAE 39 (6) ,22992308.Li, Q. ,Wang, M. ,1999. Study on high-speed apple surface defect segment algorithm based on computer vision. Proceedings of International Conference on Agricultural Engineering (99-ICAE) ,Beijing, Peoples Republic of China, 14-17 December 1999,pp. V27-31.Li, Q., Wang, M., 2000. A fast identification method for fruit surface defect based on fractal characters. J.Image Graphics (China) 5 (2) ,144-148Tao, Y. ,1996. Spherical transform of fruit images for on-line defect extraction of mass objects. Opt. Eng. 35 (2), 344-350.Throop, J.A. ,Aneshansley, D.J., Upchurch, B.L. ,1997. Apple orientation on automatic sorting equipment. Proceedings of the Sensors for Nondestructive Testing International Conference, NRAES, Ithaca,NY, pp. 328-342.Yang, Q. ,1993. Finding stalk and calyx of apples using structured lighting. Comput. Electron. Agric. 8, 31-42.Wen, Z. ,Tao，Y. ,1998. Method of dual-camera NIR/MIR image for fruit sorting. ASAE paper 983043. St. Joseph, MI. 基于计算机视觉系统对苹果表面的缺陷探测Qingzhong Lia, Maohua Wangb, Weikang GuaDepartment of Information and Electronic Engineering，Zhejiang University，Hangzhou, Peoples Republic of ChinaResearch Centre for Precision Agriculture, China Agricultural University, Beijing, Peoples Republic of China摘要:一种基于计算机图像处理，对苹果表面缺陷进行探测的视觉技术得到发展。 硬件系统能同时检查分类线上每个苹果的四边。其方法包括图像的背景移除，缺 陷的分类，茎和花萼的辨别。实验结果表示其系统是实际可行的，探测表面缺陷 的运算法则也被证明是有效的。 2002 Elsevier Science B.V版权所有。关键词：计算机视觉系统，苹果，表面探测 1绪论中国是一个农业大国，其一季的苹果产量达到1700万吨，但是其等级分 类并没有实现自动化。人工检查是非常乏味的，而且会引起视觉疲劳。另外， 每个人判别标准各不相同。尽管，在一些西方国家可以通过计算机自动检查 水果的质量，如：颜色，尺寸和外形，但由于环境的复杂性其自动控制方法 还是具有一定的挑战性。一般苹果的自动分级有两个主要问题：一是怎样通 过照相机及时获得苹果的完整图像，二是如何快速辨别茎和花萼。为解决第 一个问题，Growe and Delwiche (1996), Tao (1996)使用了一种滚轴运输系统。这 种方法的缺点是运送装置上方的照相机不能同时检查旋转中苹果的两个边缘。为 解决第二个问题，Throopetal.(1997)发明了两种相应的装置。这些装置使苹果沿着 垄干旋转不同的角度。但是结果显示其中一个装置的分级方法不能在另外的装置上 实行。Yang (1993)发明了一种照明装置来辨别苹果的茎干和花萼。其照明装置的 主要问题是图像处理中光线的误分类。Wen and Tao (1998)成功发明了一种双重照 相NIR/MIR成像法来识别苹果的茎和花萼。但是在中国使用MIR照相机花费太 昂贵了。本论文的目的是介绍一种实验装置和实验方法，使其在生产线上（每秒 3至4个苹果）能检查苹果的四侧，并有效的进行识别和分级。2关于系统装置的总体看法图1实验装置示意图一种在生产线上能同时检查苹果四侧的装置被发明，其机构形式如图1。 它由输送装置，形似于苹果的空间间隔，机械视觉系统和信息传输装置组成。 输送装置将苹果分到每一小格，再反馈给视觉系统以检查苹果表面。最终， 分类系统将苹果分成各个等级。机械视觉系统包括一个杯状的运送装置，一个能发出不同光谱和对水果表面 色泽进行分类的装置，两个照相机，一个连接于微型电子计（处理速度500MHz) 带有四个输入通道的图像抓取装置。作为一个实验系统，水果输送装置和自动分 类系统在第一阶段没有被建立。.为完成苹果的初步检查，两个相同的单色照相机被分别安置在输送装置的下 方。其视觉系统的机构如图2。照相机里的摄像传感器的分辨率为580X350。每 个照相机和另一个是同步的，并有多个电子快门。同样的8. 5mm焦距的透镜被安 装在照相机内，光学带通滤波器(840nm)安装在每个透镜外。传送装置由多个无 底的杯状物体组成（如图2)，两边固定有两面镜子，因此，其上方的照相机可 以拍摄到苹果的三面（顶部和两边)，而底部的照相机可以拍摄到苹果的仰视图。 此外，这个系统可以同时检查运送机上的数个苹果。其好处就是可以同时检查运 送机上苹果的四面。3.运算法则的描述 苹果表面检测的运算法则主要包括图像预处理、缺点分割、茎和花萼的识别、缺陷区域的计算和分类。运算法则示意图如图3。图3缺陷识别运算法则流程图3.1 图像背景的移除运送机两边镜子里的图像背景是不同的，所以不能通过简单的处理将水果的图像进行分割。因此，一种减法法则被运用，如下所述。g(x, y)是移除背景图像后的图像，f(x, y)是原始图像，b(x, y)是背景图像，T是初始值。3.2参考苹果图像进行区域分割在检验时，由于苹果形状，造成了图像强度的曲线分布。这导致边界 附近的表面亮度比有缺陷的表面亮度更低，因此难以使用简单的运算法则 辨别出真正的缺陷表面。参考苹果的图像，Li and Wang (1999)发明了一种方法完成水果曲线图像缺陷区域的分割。在这个方法中，首先产生一个水果的参 考图像，经过检验后，使其变成规格化后的水果参考图像。经过初始阶段处理后， 表面的缺点会被更容易地识别。3. 3基于几何学和人工神经网络辨别茎和萼在缺陷识别过程中，因为茎和萼的图像与缺陷类似，导致系统难以对它们进 行识别。基于几何学和人工神经网络，本文作者发明了一种方法来区分茎和萼的 凹面区域与真正的表面缺陷。不规则碎片形：一种几何形状，被以越来越小的比 例反复折叠而产生不能被标准几何所定义的不标准的形状和表面。虽然茎和花萼 与真实的缺点表面很相似，但它们不规则碎片形状还是不同的。而且，在频域中， 不规则碎片形状分析只依赖于表面的频率分布。组织上的这些不规则碎片形状特 征与周围光强和苹果的方位无关。因此，此方法可以对任意方位的苹果进行分类。 图像的分布形状可看作是三位曲线表面。基于上述考虑，包括传统的不规则碎片 形状和四导向的不规则碎片形状的五维不规则碎片被用来描述茎-花萼弯曲区域 和真实缺陷区域图像的特征。四导向的不规则碎片形尺寸（Dl，D2，D3，D4)如 图4所示。事实上，导向的不规则碎片形尺寸是对应方向（图5)的曲线不规则 碎片形尺寸。五个不规则碎片形尺寸被Li and Wang (2000)的方法计算出来。 其数字式可写为：Z=f(X，y)，（x，y)是象素的坐标，Z是灰度值。假定图像的 分辨率为MM，在x-y平面内，图像被分成数个的区域。栅格灰度的最大值和最小值分别表达为u（i，j）和b（i，j），它们的差值为d= u（i，j）- b（i，j），非空的栅格总数为：对于所有给定的S取其对数得到一系列的点，可获得logN。通过点（log，logN)的线，其斜率为负，由此估算出不规则碎片形的尺寸。用近似法 求出另外四个不规则碎片形的尺寸。NN (神经网络）运算法则用来前馈来自真实缺陷区域的花萼。前馈网络结构 可适当预测输入和输出间的非线性关系的相关问题。被设计的BP网络结构如图 6所示。NN模型有五个输入点，一个含有四个隐藏输入点的层和一个输出点。在 程序学习过程中，网络结构的权重不断被刷新。其收敛性有两个条件：是否每个 区域误差比允许误差值更小，是否每个样品的输出误差比预先确定的允许误差值 小。图6关于茎-萼缺陷分类的神经网络3. 4苹果表面缺陷的即时检查苹果表面缺陷的即时检查分为两个阶段。第一阶段是3. 2中描述的方法对可 疑缺陷区域的分割（包括茎、萼和缺陷表面），其结果显示莲-萼区域时常为较 大的区域。在第二阶段中，对较大的分割区域进一步处理以区别真正的表面缺陷。 4.实验和结果运算法则被用于探测四个富士苹果的表面缺陷和茎-萼区域。结果如图7， (a)、(c)、(e)和（g)是苹果的原始探测图像。（b)、（d)、（f)和（h)是 缺陷区域分割后的结果。这些结果显示缺陷区域和茎-萼区域基本上被提取出来， 大于指定值的分割区域被进一步分割。表1为BP对茎-萼区域的识别结果。如果 神经网络的输出值接近1，则为茎-萼区域。如果神经网络的输出值接近0，则为 真实的缺陷区域。实验结果显示不规则碎片形状对正常的水果凹入表面和缺陷表 面的分类是非常有效的。因为茎-花萼的片在外形上通常是凹的，被提出的方法对 茎-花萼的识别是可行的。对每个苹果的表面缺陷的识别和分级的时间为320ms， 微电子计处理速度为500Hz。(a) (b) (c) (d) (e) (f) (g) (h)图7缺陷区域和茎-萼区域（a）、（c）、（e）和（g）是苹果的原始探测图像。（b）、（d）、（f）和（h）是缺陷区域分割后的结果序号输入区域神经网络的输出1茎-萼31310. 98542茎-萼38380. 99493茎-萼32320. 63934茎-萼37370. 98545茎-萼39390. 95266茎-萼40400. 97837茎-萼44440. 86398莲-萼44440. 98409莲-萼44440. 831410茎-萼46460. 972511茎-萼48480. 887412茎-萼48480.931413茎-萼43430. 995614茎-萼47470. 943415茎-萼47470.837316缺陷区域42420. 994817缺陷区域49490. 059318缺陷区域54540. 035619缺陷区域61610. 053920缺陷区域62620. 274321缺陷区域81810. 265922缺陷区域48480. 099423缺陷区域55550.819424缺陷区域54540. 031225缺陷区域63630. 088326缺陷区域66660. 209327缺陷区域51510.215028缺陷区域62620. 074329缺陷区域84840. 2016 表1 BP对茎-萼区域和缺陷区域的识别5.结论结果显示不规则碎片形状特征对正常的水果凹入表面和缺陷表面的分类是非 常有效的。因为茎-花萼外形通常是凹的，本文提出的方法是切实可行的。而且， 此系统具有同时检查生产线上每个苹果表面的四个方位。基于苹果的原始图像和 缺陷区域的分割方法，根据等级分类系统相应的要求来检查一个苹果真正的缺陷 区域，是较进一步的研究目的。感谢特别感谢支持此工作的大学博士学业特别基金会（Project No. 950801)。参考书目1，Growe， T.G.，Delwiche. M.J.,1996. Real-time defect detection in fruitpart I: design concepts and development of prototype hardware. Trans. ASAE 39 (6),2299-2308.2，Li，Q. ， Wang, M.，1999. Study on high-speed apple surface defect segment algorithm based on computer vision. Proceedings of International Conference on Agricultural Eigneering（99-ICAE) ，Beijing，Peoples Republic of China，14-17 December 1999，pp. V27-31.3，Li，Q.，Wang， M.，2000. A fast identification method for fruit surface defect based on fractal characters. J.Image Graphics (China) 5 (2), 144-148.4，Tao, Y.，1996. Spherical transform of fruit images for on-line defect extraction of mass objects. Opt. Eng.35 (2), 344-350.
展开阅读全文