Background Job evaluation has been widely used to establish a foundation for internal equity and other human resource functions. The United Kingdom adopts the National Health Service (NHS) Job Evaluation scheme to determine the pay bands for most NHS staff and ensure equal pay for work of equal value. The challenges of recruiting and retaining physicians in Chinese public hospitals have heightened the need for a reliable job evaluation system to ensure the equality of physician compensation. The aim of this study was to construct job evaluation factors of physicians in Chinese public hospitals based on the NHS Job Evaluation scheme and to examine the reliability and validity of the established system. Methods Questionnaire surveys and statistical analyses were used to determine the job evaluation factors for physicians. The preliminary screening of the evaluation factors was based on a literature review, focused interviews with physicians and the Delphi method. Based on the results of preliminary screening, a questionnaire with 25 factors was designed to survey physicians regarding the importance of each factor in physician job evaluation. After the pretest, final questionnaire data were collected from 900 physicians by adopting a stratified sampling from 6 tertiary public hospitals in Beijing. A principal component analysis was used for factor extraction and structural validity analysis. The reliability was determined using Cronbach's alpha. Results The results of the principal component analysis showed that the 25 physician job evaluation factors were grouped into the 5 dimensions of Task Characteristics, Knowledge, Responsibility, Effort/Environment, and Skills. The Cronbach's alpha coefficients of the five dimensions ranged from 0.841 to 0.909, which indicated a high level of reliability. The result of the factor analysis indicated fair structural validity. The content validity was established by building onto the NHS Job Evaluation scheme and other well-established job evaluation systems. Conclusions Our study indicates that the Chinese version of physician job evaluation is an instrument with fair reliability and validity, which fully reflects the characteristics of physicians in Chinese public hospitals. This system can provide an important basis for developing a physician compensation plan and ensuring internal equity in healthcare organizations.