BACKGROUND: Each year there are an estimated 4.2-6.7 amputations per 10,000 workers in the metal fabrication trades in the United States. The Minnesota Machine Guarding Study evaluates the effectiveness of a peer-based technical and educational intervention designed to reduce exposure to amputation hazards among workers in small machining/metal working shops. The study reported here involved the development and evaluation of methods for measuring machine safety, which will be used in the intervention study. METHODS: Using OSHA regulations, ANSI standards, and industry best practices, we developed 23 machine-safety scorecards. The safety scores were dependent on the presence or absence of guards, other safety devices and implements, and the presence or absence of acceptable administrative programs. Inter-rater reliability was assessed for the evaluation of eight types of commonly used metal fabrication machines. RESULTS: Of the 23 most common types of machines, there were a sufficient number of machines to evaluate inter-rater reliability for eight types. Three raters in four shops assessed fifty-nine machines. Overall, the kappa statistic ranged from 0.57 to 0.84, indicating good to very good concordance between raters. In general, machines did not fare well with regard to compliance with current standards. DISCUSSION: The ability to assess machine-related hazards is important in industries where it is difficult to identify and count injuries in a timely fashion. Machinists and safety professionals may use this scoring system as a means of reproducible assessment of machine safety.