Transforming the manufacturing environment from manually operated production units to unsupervised robotic machining centres requires a presence of reliable in-process monitoring system. In this paper, we demonstrate a technique for automatic endpoint detection of weld seam removal in a robotic abrasive belt grinding process with the help of a vision system using deep learning. The paper presents the results of the first investigative stage of semantic segmentation of weld seam removal states using encoder-decoder convolutional neural networks (EDCNN). An experimental investigation using four different weld seam states on mild steel work coupon are trained using the VGG-16 network based on encoder-decoder architecture. The results demonstrate the potential of the developed vision based methodology as a tool for endpoint prediction of the weld seam removal in real time during a compliant abrasive belt grinding process. The prediction system based on semantic segmentation is able to monitor weld profile geometry evolution taking into account the varying belt grinding parameters during machining which will allow further process optimisation