BACKGROUND: The Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery have implemented "milestones" to evaluate residents during their progression in medical education. The purpose of this study was to determine whether a validated evaluation tool correlates with surgical experience, year in training, and progression over time. DESIGN: This was a retrospective study of already collected curriculum assessment data where 2 unbiased, blinded orthopedic surgeons evaluated resident performance on basic diagnostic knee arthroscopy using the Arthroscopic Surgical Skills Evaluation Tool (ASSET) over 3 years. Residents also gained arthroscopy experience through a structured arthroscopy curriculum and clinical experience. SETTING: The study was conducted at the TRIA Orthopaedic Center (Bloomington, Minnesota, USA), an institutional site for The University of Minnesota orthopedic surgery residency program. PARTICIPANTS: Eleven orthopedic surgery residents at postgraduate years 2 to 5 were evaluated using the ASSET. RESULTS: The Pearson's Correlation Coefficient was used to validate both the number of arthroscopic procedures performed by residents (r=0.946) and their level in training (r=0.89). Residents who were re-evaluated after undergoing the arthroscopy curriculum throughout the year displayed significant increases in total ASSET scores (p < 0.01). CONCLUSION: Resident performance on the ASSET correlated with arthroscopic experience based on year-in training. More importantly, performance improved with additional years of training, demonstrating validity over time. The data also demonstrates interobserver reliability. Due to these correlations between exposure to surgery and score on the ASSET, we believe the tool could serve as a suitable means for assessing residents' technical proficiency as required by The Accreditation Council for Graduate Medical Education program guidelines.