TY - CONF A1 - Niemueller, Tim A1 - Ferrein, Alexander A1 - Reuter, Sebastian A1 - Jeschke, Sabina A1 - Lakemeyer, Gerhard T1 - The RoboCup Logistics League as a Holistic Multi-Robot Smart Factory Benchmark T2 - Proceedings of the IROS 2015 Open forum on evaluation of results, replication of experiments and benchmarking in robotics research N2 - With autonomous mobile robots receiving increased attention in industrial contexts, the need for benchmarks becomes more and more an urgent matter. The RoboCup Logistics League (RCLL) is one specific industry-inspired scenario focusing on production logistics within a Smart Factory. In this paper, we describe how the RCLL allows to assess the performance of a group of robots within the scenario as a whole, focusing specifically on the coordination and cooperation strategies and the methods and components to achieve them. We report on recent efforts to analyze performance of teams in 2014 to understand the implications of the current grading scheme, and derived criteria and metrics for performance assessment based on Key Performance Indicators (KPI) adapted from classic factory evaluation. We reflect on differences and compatibility towards RoCKIn, a recent major benchmarking European project. Y1 - 2015 UR - https://opus.bibliothek.fh-aachen.de/opus4/frontdoor/index/index/docId/7534 ER -