Together with medical experts from the regional hospital Villach (LKH Villach, Austria), we manually annotated hours of raw endoscopic gynecologic surgery . The videos show endometriosis treatment and myoma resection of over 100 patients. Based on a discussion with the medical experts, we identified two main semantic aspects for individual scenes: action and anatomy. Action denotes video scenes that feature significant interaction with the patient’s tissue and organs, using different instruments. These scenes represent the main physical work for the surgeon and are valuable for documentation or teaching certain operation techniques. In total we identified eight sub-classes for actions (e.g., suture, cutting, injection). The anatomy class denotes scenes which feature little or almost no surgical actions apart from moving tissue and organs. The main purpose of such scenes is documentation, in particular the assessment of pathologies on specific organs and their treatment. The identified anatomy sub-classes in the context of endometriosis treatment and myoma resection are uterus, ovaries, oviduct, liver and colon. The cleaned ground truth dataset comprises nine hours of annotated video material from 111 different recordings.