Precise finishing operations such as chamfering and filleting are characterized by relatively low contact forces and low material removal. For such processes, conventional automation approaches like pre-programmed position or force control without adaptations are not suitable to obtain fine surface finishing with high profile accuracy. As a result, polishing tasks are still mainly carried out manually by skilled operators. In this paper, we propose an adaptive framework capable of polishing a wide range of materials including hard metals like titanium using a collaborative robot. We propose an iterative learning controller based on impedance control that adapts both position and forces simultaneously in each iteration to regulate the polishing process. The proposed controller can track the desired profile without any a priori knowledge of the forces required to polish different materials. In addition, we introduce a novel mathematical model to generate the complex filleting toolpath based on Lissajous curves. Trials are carried out in finishing tasks such as chamfering and filleting using a collaborative industrial robot to validate the novel framework. Surface roughness and profile measurements show that our adaptive controller can obtain fine polishing output in various materials such as titanium, aluminum, and wood