Computer Vision Enhancing Public Works

STEP 1. Background

Locating in the coastal area, the amounts of rainfall in Hong Kong soar significantly during the summertime. In order to ensure the city operates smoothly in the rainy season, we need to avoid the pipe blockage issue caused by the rain and prevent the situation of flooding. Therefore, it is important to build up a great stormwater drainage system. The pipe and sewage underground would be the key to stormwater drainage in this concrete jungle. Not only do the government need to keep constructing and renewing the drainage system, but they also need to repair and check the existing sewage and pipes with the technologies of computer vision in public works.

Therefore, the government performs regular drainage system checks and tracks down the damaged system, so that they can be repaired and renewed on time.

STEP 2. Case Pain Point

During the regular drainage system check, the staff members will put a camera robot into the pipes, it will move along the pipes and record the interior situation. After the robot reached the endpoint, the staff members will take out the robot, extract the footage, and analyze the pipe status. The defect locations, types, and levels will be recorded, elaborated, and then followed up by the repair workers.

Since the above process will be handled by different staff, the government realized there are two pain points in this way of drainage system check:

• Time-consuming: Pipes that are waiting to be checked to scatter around different districts in Hong Kong. There are enormous images needed to be analyzed. Besides, during the recording process, the staff members need to watch the entire footage one by one, see if there is any problem in every frame, and mark down the damage types and levels. There are over a thousand frames in one video, so it takes a lot of time and effort to review the videos.
• Manual error: Since there are a lot of videos that need to be analyzed, adding that the pipe structures are similar, the footages are often duplicated, without sufficient lights, it is easy for the staff members to be exhausted and misanalyze the defects. More times and human resources are required to double-check the results

Due to the above factors, the government would like to apply computer vision into this process, to perform a more effective footage analysis, increase efficiency and accuracy. To conduct the proof of concept, the government invited Motherapp to participate in this project.

STEP 3. The Solution

We adopted Powerareana – Build Your Own AI, a computer vision analytic system developed by Motherapp, to implement the drainage system analysis.

Steps:

1. Annotation and labeling: We selected the 5 most common defect types to be our target of recognition for this project. We also prepare at least 500 frame samples for each defect type. We let our system study the shapes, traits, and common locations of different types of defects, so that our system may automatically recognize the defect area through the footage.
2. Image analysis: After the annotation and labeling stage, the setup of the computer vision system is finished. Staff members only need to upload the drainage footage to the system for it to run automatically. The system will perform segmentation on the video, scanning every pixel out of every frame, to recognize the existence, types, and levels of the defects. Finally, the system will export all the data in the format of .csv, to assist the staff members with further works.
3. Evaluation: We conducted computer vision tests for over 10 footages to test the effectiveness of computer vision. After acquiring the data files, we compare them with the reports made by the DSD, to mark down the recognized, missed, and false-positive defects that the computer vision system has detected.

Challenges:

Since one of the defects “Broken” has a relatively low appearance frequency, it is difficult to find more than 500 defect examples for the system to learn. The lack of sample learning will lead to the decrease of the system’s accuracy, which is an undesirable outcome. We actively tried to figure out ways to let the computer vision system learn with limited samples, and finally decided to extract different frames that show the same broken defects for the system to learn, so as to increase the sample amounts.

STEP 4. The Results

Proof of Concept (POC)

The aim of this project is to provide the government and the related department with real computer vision analytic data, to see if computer vision can replace staff members to perform manual, repetitive tasks. The data acquired from this project can be used as references in the future so that the related departments can decide which method to use when it comes to the draining systems, or other public facility checking and renewal.

In this project, we obtained up to 85% of artificial intelligent accuracy (successfully recognized the defect types and levels). Although it is not accurate enough to fully replace humans, the outcome and the data are still very useful in assisting the checking process. Before starting the manual check, the artificial intelligence system can filter out the footage of the drainage first, and sort out all the suspicious frames that might include defects. The staff members can then focus on checking the labeled parts of the pipes according to the sorted frames. With the filtering of the AI system, the total playing time of the footage can be reduced by 64.2%, which would be 1/3 of the original time.

We call this working model the “Human-in-the-loop” model, which means that during the AI learning process, human is able to adjust the parameters of the system according to their performance in order to obtain more accurate results. The whole process is interactive, the machines can learn from the manual adjustment, and humans can acquire a more satisfying outcome with the help of the machines.

The Build Your Own AI analytic system also applied the concept of “Human-in-the-loop”. Different from the traditional system development, the parameter is hard to be adjusted after the setup is finished, complex codes are required to make changes. Build Your Own AI system is flexible, coding is not needed for both the annotation and labeling, and the image analysis stage. Those who are not equipped with programming skills can also make good use of computer vision AI. Click here to know more about the specialty of Build Your Own AI.

智慧城市技術 – 電腦視覺的地底應用

第一步. 個案背景

香港位處沿岸地區，每逢夏季，降雨量都會大增。為了能讓城市在雨季中順暢地運作，避免因雨水積聚及淤塞而造成水浸等風險，疏導雨水的工作十分重要。在高樓大廈林立的市區，地下的渠道網絡 便是將市區的雨水疏導至排水設施的關鍵。政府除了要不斷建設和更新排洪系統設施外，維護已有的設施及管道亦同樣重要。

因此，政府會定期為各區的渠管進行例行檢查，檢查渠管狀況及尋找出有損壞的管道，及時維修及更新。

第二步. 個案痛點

在定期檢查中，工作人員會將攝影機機械人放置到需要檢查的渠管中。攝影機會沿着渠管內部行駛，錄製渠管內部狀態。及後工作人員會取出攝影機並調出影像，再從錄製下的影片中檢視渠管狀態。工作人員會記錄下渠管損毀 (defect) 位置/ 種類及損壞級別；將這些狀態資訊整合並製作成報告；然後交由工程人員跟進。

由於上述整個檢查過程都會由不同人員人手進行，政府發現以這種方式檢查渠管有兩大痛點：

• 費時：全港各區都遍佈着需要檢查的渠管，要分析的渠管影像數量不少。而且，在檢視渠管錄像過程中，工作人員需要將每一條影片從頭到尾觀看一遍，每幀每幀檢查渠管狀態並將發現問題的影格 (連同損壞種類與程度) 逐一紀錄下來。而每條影片都有數以千計的影格數，可想而知檢視影片的過程是可等花費時間及功夫。
• 人工錯誤：由於工作人員需要檢視的影片數眾多；加上渠管構造相似，影片畫面變化不大；影片光線不足，難以確認損壞種類；紀錄工作的重覆性高等等這些都是會造成工作人員視角疲勞的原因。視覺疲勞會增加工作人員出錯的機會，及後需要花費更多時間/ 人力去核對及確認分析結果。

基於以上兩大主因，政府希望可以應用智慧城市常用到的電腦視覺技術，代替人手完成檢視影片工作，提升效率及準繩度。為了對這個想法進行概念驗證 (Proof of concept/ POC)，政府找來了 Motherapp 團隊去進行這次項目。

第三步. 團隊方案

我們採用了自主開發的 Powerarena – Build your own AI 電腦視角分析系統 (Computer Vision in Public Works) 去進行這個專項。

工作步驟：

1. 學習階段 (Annotation/ Labelling)：我們挑選了 5 種最常見的損壞類型 (Defect types) 作為這次專項的辨識目標，並為每種損壞類型準備了最少 500 則影像示範 (Frame samples)。讓系統去學習不同損壞類型的形狀特徵及常見位置，在日後可以自行從影片中透過圖像分析找出這些損壞處。
2. 圖像分析 (Image analysis)：在系統完成辨認階段後，影像分析的前置作業已準備完畢。工作人員只需要將渠管錄像上傳至系統，系統便會自動運行影片，進行 Object segmentation (物件分割) 工作，機器會掃描每一影片格 (frame) 的畫面上的每一像素，以辦認存在損壞部份的影片格、損壞類型及級別。加載完畢後，系統會將這些數據可以集合成以 .csv 檔案顯示的分析結果，供工作人員下載及取用。
3. 評估人工智能分析成效 (Evaluation)：我們以十多條過往分析過的渠管錄像作測試，評估採用電腦視角分析的成效。在取得電腦視覺系統的分析數據檔案後，對照了以往由渠務處發出的分析報告，紀錄電腦所成功辨別的損壞處 (Recognized) 、辨別不到的損壞處 (Missed) 及其他電腦辨別到的潛在損壞處 (False Positive)。

工作挑戰：

由於其中一種損壞類型 “Broken” (破損) 的出現頻率相對較低，較難找到 500 例不同的損壞處樣本去讓系統學習。學習樣本不足會降低整個電腦視覺分析的準繩度，這是我們並不希望發生的。我們積極在有限的損壞類型實例中尋找可以增加學習樣本的方法，最後決定從同一處 “Broken” 損壞片段中找出包含不同角度的影片格 (frames)，讓系統去學習，彌補學習樣本數量上的不足。

第四步. 最終成果

概念驗證 (Proof of concept/ POC)

此次項目為政府及相關部門提供了真實的電腦視覺系統分析數據，去驗證人工智能分析能否代替人手去處理這些內容重覆單調的工作。是次項目所取得的資料可以成為日後的參考數據，供相關部門去決定應採用哪種方式去檢查渠管/ 其他公共設施。

這一項目中，我們取得了接近 85% 的人工智能準確度 (即成功辨認損毀類型及其等級)，雖然未足以完全代替人手，但人工分析所得出的數據仍十分有用，它能提升人手檢查工作的效率。在人手檢查開始前，人工智能系統能先過濾渠管錄像，從錄像片段中辨認懷疑存在問題的管道部分及問題的所在位置 (以影片格數標記)。工作人員便可跟據過濾得出的結果，集中檢查被人工智能系統標出的管道部分。在機器過濾下，錄像播放總時長能在電腦的幫助下縮短 64.2% ，幾乎是原本播放時間的 1/3。

電腦分析系統會成為工作人員的工作夥伴，先由機器分析，再由工作人員去核對機器的分析結果。在一同完成渠管檢查工作的過程中，工作人員能更容易發現到經常出現分析錯誤的類型甚或至錯誤成因。從而針對這些錯誤類型和錯誤原因重新調整電腦分析系統的辨識目標 / 辨識參數，協助人工智能更快更準確地辨認目標。應用人工智能及人為監督 (Human-in-the-loop)，在大規模檢測中，相關部門可以大大提升人手檢查工作的效率。

我們會稱這種工作模式為 Human-in-the-loop (人為監督 / 人在回路)。意指在人工智能學習的過程中，人可以跟據機器的表現，手動調節對機器的指令或運作參數，以達到更好的結果。這個這程是相互的，機器可以透過人為調整得到提升；而人可以隨着機器的提升得到更滿意的結果。

這次項目中我們所選用的 Build your own AI 分析系統，加上套用 Human-in-the-loop 的概念，試驗 Computer Vision in Public Works 的成效。有別於傳統系統開發，運作參數在設置完成後便難以更改，需要程序人員重新編寫程序碼。Build your own AI 系統設計彈性，標註階段 (Annotation/ Labeling stage) 及分析階段 (Image analysis stage) 整個過程都不會涉及任何需要編寫程序碼的步驟，即使是沒有學習過任何程式碼的人也能輕鬆應用電腦視覺 AI。按此了解更多關於 Build your own AI 系統的運作特點。