{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T05:34:10Z","timestamp":1736228050647,"version":"3.32.0"},"reference-count":45,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,1,26]],"date-time":"2020-01-26T00:00:00Z","timestamp":1579996800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Data Lab Innovation Centre","award":["16270"]},{"name":"Oil and Gas Innovation Centre","award":["18PR-16"]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"Regulatory requirements for sub-sea oil and gas operators mandates the frequent inspection of pipeline assets to ensure that their degradation and damage are maintained at acceptable levels. The inspection process is usually sub-contracted to surveyors who utilize sub-sea Remotely Operated Vehicles (ROVs), launched from a surface vessel and piloted over the pipeline. ROVs capture data from various sensors\/instruments which are subsequently reviewed and interpreted by human operators, creating a log of event annotations; a slow, labor-intensive and costly process. The paper presents an automatic image annotation framework that identifies\/classifies key events of interest in the video footage viz. exposure, burial, field joints, anodes, and free spans. The reported methodology utilizes transfer learning with a Deep Convolutional Neural Network (ResNet-50), fine-tuned on real-life, representative data from challenging sub-sea environments with low lighting conditions, sand agitation, sea-life and vegetation. The network outputs are configured to perform multi-label image classifications for critical events. The annotation performance varies between 95.1% and 99.7% in terms of accuracy and 90.4% and 99.4% in terms of F1-Score depending on event type. The performance results are on a per-frame basis and corroborate the potential of the algorithm to be the foundation for an intelligent decision support framework that automates the annotation process. The solution can execute annotations in real-time and is significantly more cost-effective than human-only approaches.<\/jats:p>","DOI":"10.3390\/s20030674","type":"journal-article","created":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T12:41:11Z","timestamp":1580128871000},"page":"674","source":"Crossref","is-referenced-by-count":16,"title":["Automatic Annotation of Subsea Pipelines Using Deep Learning"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8279-9973","authenticated-orcid":false,"given":"Anastasios","family":"Stamoulakatos","sequence":"first","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9284-1899","authenticated-orcid":false,"given":"Javier","family":"Cardona","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8833-3173","authenticated-orcid":false,"given":"Chris","family":"McCaig","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"given":"David","family":"Murray","sequence":"additional","affiliation":[{"name":"N-Sea, RS 4301 Zierikzee, The Netherlands"}]},{"given":"Hein","family":"Filius","sequence":"additional","affiliation":[{"name":"N-Sea, RS 4301 Zierikzee, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6206-2229","authenticated-orcid":false,"given":"Robert","family":"Atkinson","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1849-5788","authenticated-orcid":false,"given":"Xavier","family":"Bellekens","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5132-4572","authenticated-orcid":false,"given":"Craig","family":"Michie","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9093-5245","authenticated-orcid":false,"given":"Ivan","family":"Andonovic","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5091-2567","authenticated-orcid":false,"given":"Pavlos","family":"Lazaridis","sequence":"additional","affiliation":[{"name":"Department of Engineering and Technology School of Computing and Engineering, Huddersfield HD1 3DH, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8436-8325","authenticated-orcid":false,"given":"Andrew","family":"Hamilton","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4184-2397","authenticated-orcid":false,"given":"Md Moinul","family":"Hossain","sequence":"additional","affiliation":[{"name":"School of Engineering and Digital Arts, University of Kent, Canterbury, Kent CT2 7NT, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7256-0897","authenticated-orcid":false,"given":"Gaetano","family":"Di Caterina","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9150-6805","authenticated-orcid":false,"given":"Christos","family":"Tachtatzis","sequence":"additional","affiliation":[{"name":"Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow G11XW, UK"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/S0952-1976(97)00001-8","article-title":"Robust real-time detection of an underwater pipeline","volume":"11","author":"Zingaretti","year":"1998","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Jacobi, M., and Karimanzira, D. (2013, January 10\u201314). Underwater pipeline and cable inspection using autonomous underwater vehicles. Proceedings of the 2013 MTS\/IEEE OCEANS, Bergen, Norway.","DOI":"10.1109\/OCEANS-Bergen.2013.6608089"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Jacobi, M., and Karimanzira, D. (2014). Multi Sensor Underwater Pipeline Tracking with AUVs, IEEE. 2014 Oceans\u2014St. John\u2019s.","DOI":"10.1109\/OCEANS.2014.7003013"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Narimani, M., Nazem, S., and Loueipour, M. (2009, January 11\u201314). Robotics vision-based system for an underwater pipeline and cable tracker. Proceedings of the OCEANS 2009-EUROPE, Bremen, Germany.","DOI":"10.1109\/OCEANSE.2009.5278327"},{"key":"ref_5","unstructured":"Conte, G., Zanoli, S., Perdon, A.M., Tascini, G., and Zingaretti, P. (1996, January 23\u201326). Automatic analysis of visual data in submarine pipeline inspection. Proceedings of the OCEANS 96 MTS\/IEEE Conference Proceedings. The Coastal Ocean- Prospects for the 21st Century, Fort Lauderdale, FL, USA."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1007\/s001380100065","article-title":"A vision system for an underwater cable tracker","volume":"13","author":"Ortiz","year":"2002","journal-title":"Mach. Vis. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"140","DOI":"10.3182\/20090916-3-BR-3001.0041","article-title":"Experimental Evaluation of a Particle Filter-based Approach for Visually Tracking Undersea Cables","volume":"42","author":"Ortiz","year":"2009","journal-title":"IFAC Proc. Vol."},{"key":"ref_8","unstructured":"Lazinica, A. (2006). An Active Contour and Kalman Filter for Underwater Target Tracking and Navigation. Mobile Robots: Towards New Applications, I-Tech Education and Publishing."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.ijepes.2017.12.016","article-title":"Automatic autonomous vision-based power line inspection: A review of current status and the potential role of deep learning","volume":"99","author":"Nguyen","year":"2018","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhang, W., Witharana, C., Li, W., Zhang, C., Li, X., and Parent, J. (2018). Using deep learning to identify utility poles with crossarms and estimate their locations from google street view images. Sensors, 18.","DOI":"10.3390\/s18082484"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Jalil, B., Leone, G.R., Martinelli, M., Moroni, D., Pascali, M.A., and Berton, A. (2019). Fault Detection in Power Equipment via an Unmanned Aerial System Using Multi Modal Data. Sensors, 19.","DOI":"10.3390\/s19133014"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Miao, X., Liu, X., Chen, J., Zhuang, S., Fan, J., and Jiang, H. (2019). Insulator detection in aerial images for transmission line inspection using single shot multibox detector. IEEE Access.","DOI":"10.1109\/ACCESS.2019.2891123"},{"key":"ref_13","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2020, January 12). Faster R-CNN, Available online: http:\/\/xxx.lanl.gov\/abs\/1506.01497."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1016\/j.oceaneng.2017.08.024","article-title":"A novel approach for defect detection on vessel structures using saliency-related features","volume":"149","author":"Ortiz","year":"2018","journal-title":"Ocean Eng."},{"key":"ref_15","first-page":"500","article-title":"Towards Visual Detection, Mapping and Quantification of Posidonia Oceanica using a Lightweight AUV","volume":"49","author":"Campos","year":"2016","journal-title":"IFAC-PapersOnLine"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"60956","DOI":"10.1109\/ACCESS.2018.2875412","article-title":"Deep Semantic Segmentation in an AUV for Online Posidonia Oceanica Meadows Identification","volume":"6","year":"2018","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Petraglia, F.R., Campos, R., Gomes, J.G.R.C., and Petraglia, M.R. (2017, January 28\u201331). Pipeline tracking and event classification for an automatic inspection vision system. Proceedings of the 2017 IEEE International Symposium on Circuits and Systems (ISCAS), Baltimore, MD, USA.","DOI":"10.1109\/ISCAS.2017.8050761"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fang, H., and Duan, M. (2014). Submarine Pipelines and Pipeline Cable Engineering. Offshore Operation Facilities, Elsevier.","DOI":"10.1016\/B978-0-12-396977-4.00006-8"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1757","DOI":"10.1016\/j.patcog.2004.03.009","article-title":"Learning multi-label scene classification","volume":"37","author":"Boutell","year":"2004","journal-title":"Pattern Recognit."},{"key":"ref_20","unstructured":"Sinha, R.K., Pandey, R., and Pattnaik, R. (2018). Deep Learning For Computer Vision Tasks: A review. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis. (IJCV)"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017). Focal Loss for Dense Object Detection. arXiv.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_25","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017). Mask R-CNN. arXiv.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_27","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_28","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv."},{"key":"ref_29","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML\u201910), Haifa, Israel."},{"key":"ref_30","unstructured":"Nwankpa, C., Ijomah, W., Gachagan, A., and Marshall, S. (2018). Activation Functions: Comparison of trends in Practice and Research for Deep Learning. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1080\/01621459.1975.10479865","article-title":"The Predictive Sample Reuse Method with Applications","volume":"70","author":"Geisser","year":"1975","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A Survey on Transfer Learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Li, F.-F. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_34","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 9). Automatic Differentiation in PyTorch. Proceedings of the NIPS Autodiff Workshop, Long Beach, CA, USA."},{"key":"ref_35","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference for Learning Representations, San Diego, CA, USA."},{"key":"ref_36","unstructured":"Reddi, S.J., Kale, S., and Kumar, S. (May, January 30). On the Convergence of Adam and Beyond. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Smith, L.N. (2017, January 24\u201331). Cyclical Learning Rates for Training Neural Networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.","DOI":"10.1109\/WACV.2017.58"},{"key":"ref_38","unstructured":"Smith, L.N. (2015). Cyclical Learning Rates for Training Neural Networks. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.neucom.2015.09.116","article-title":"Deep learning for visual understanding: A review","volume":"187","author":"Guo","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_40","unstructured":"Sorower, M.S. (2010). A Literature Survey on Algorithms for Multi-Label Learning, Oregon State University. Technical Report."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Gharroudi, O., Elghazel, H., and Aussem, A. (2015, January 9\u201311). Ensemble Multi-label Classification: A Comparative Study on Threshold Selection and Voting Methods. Proceedings of the 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), Vietri sul Mare, Italy.","DOI":"10.1109\/ICTAI.2015.64"},{"key":"ref_42","unstructured":"Flach, P.A., and Kull, M. (2015, January 12\u201314). Precision-Recall-Gain Curves: PR Analysis Done Right. Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS\u201915), Cambridge, MA, USA."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pone.0118432","article-title":"The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets","volume":"10","author":"Saito","year":"2015","journal-title":"PLoS ONE"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1819","DOI":"10.1109\/TKDE.2013.39","article-title":"A Review on Multi-Label Learning Algorithms","volume":"26","author":"Zhang","year":"2014","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1023\/A:1009982220290","article-title":"An Evaluation of Statistical Approaches to Text Categorization","volume":"1","author":"Yang","year":"1999","journal-title":"Inf. Retr."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/3\/674\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T01:41:05Z","timestamp":1736214065000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/3\/674"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,26]]},"references-count":45,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["s20030674"],"URL":"https:\/\/doi.org\/10.3390\/s20030674","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,1,26]]}}}