Background Any form of long-term physical or mental impairment might negatively influence health-related quality of life (HRQoL). HRQoL, as an independent concept, covers a wide range of characteristics that includes physical, mental, social, and spiritual functions. People with disabilities are continuously exposed to multiple barriers that deteriorate their HRQoL. It also creates impairment in performing physical activities. However, experts opine regular physical exercise as an intervention to help disabled people. This research aims to investigate the association between disability and physical activity with HRQoL among the adult population in Australia. Design A retrospective cohort study. Methods This study utilized the most recent 19 waves of data (2002-2020) from the nationally representative Household, Income and Labour Dynamics in Australia (HILDA) survey. Component summary scores such as physical component summary (PCS) and mental component summary (MCS), and SF-6D utility scores were utilized to measure HRQoL. Random-effects GLS regression technique was fitted to estimate the association between disability and physical activity with HRQoL, after adjusting for a range of socio-demographic and health-related characteristics. Results Disability was negatively associated with the PCS (-5.95), MCS (-2.70) and SF-6D (-0.060) compared with non-disabled counterparts. However, respondents engaged in the recommended level of physical activity had substantial gain in PCS (b = 0.96), MCS (1.57), and SF-6D (0.021) scores. Besides, the results showed that performing the recommended level of physical activity in the presence of disability has lessen the negative effect of disability/ positive moderating effect of physical activity on PCS, MCS, and SF-6D scores by 1.84 points, 0.82 points, and 0.013 percentage points, respectively. Conclusion This study found an inverse association between disability and HRQoL among Australian adults. However, physical activity was associated with improved HRQoL. Therefore, public health interventions, such as the orientation of physical activities, have a higher potential to dwindle the burden regarding HRQoL.