Treffer: Foundation model-based control and simulation for robotic arms

The use of robotic arms has transformed industries like manufacturing, healthcare, and logistics, enabling complex operations usually performed by human hands to be easily automated. While current control methods are still pre-programmed and based on pre-defined trajectories, they are less flexible and adaptive than needed in more dynamic real-world environments. This current project evaluates incorporating the latest artificial intelligence models, mostly CLIP (Contrastive Language–Image Pretraining) and LLaVA (Large Language and Vision Assistant), into a more interactive and adjustable robotic arm controller system. This project aims to use these base models to produce a simulation setup in which robotic arms can be manipulated through vision and text input, minimizing hardcoding and real-time adjustment allowance. This project utilizes CoppeliaSim(V-REP), a powerful robotic simulation software that creates a platform to design various robotic arms and movements. The initial phase of this project involved with intensive literature review and research on reinforcement learning algorithms and their integration using machine learning models CLIP and LLaVA. The second phase focused on configuring the codes previously created that need to be enhanced and ways to connect the external Python environment, Visual Studio Code to CoppeliaSim. The third phase involves the incorporation of a machine learning model for image processing and enhancing deep reinforcement learning models, which allows the robotic arm to imitate learning. Looking ahead, additional work will focus on refining the system's multi-agent capability, optimizing the interface between external coding platforms and CoppeliaSim, and real-time testing to analyze how the system would operate under real conditions. The study can significantly enhance the operation of robotic arms, cost-cutting in programming, increase efficiency in task implementation, and make more intelligent and responsive robotic systems in various industries. ; Bachelor's degree