Background Adults with diabetes or obesity are more likely to be physically inactive than healthy adults. Physical activity is essential in the management of both diseases, necessitating targeted interventions in these groups. This study analysed physical inactivity (defined as not taking part in leisure-time physical activity) in over 100,000 adults in Germany considering their body mass index and the presence of diabetes. Furthermore, the relationship between specific socio-demographic factors with physical inactivity was investigated, particularly focussing diabetic and obese people, to refine the identification of risk-groups for targeted interventions on physical activity promotion. Methods Data from 13 population-based health surveys conducted in Germany from 1997 to 2018 were used. The relevant variables extracted from these datasets were merged and employed in the analyses. We included data from 129,886 individuals in the BMI analyses and 58,311 individuals in the diabetes analyses. Logistic regression analyses were performed to identify the importance of six socio-demographic variables (age, sex/gender, education, income, employment, and migration) for the risk of physical inactivity. Results Obese and diabetic people reported a higher prevalence of physical inactivity than those who were not affected. Logistic regression analyses revealed advanced age, low education level, and low household income as risk factors for physical inactivity in all groups. A two-sided migration background and unemployment also indicated a higher probability of physical inactivity. Conclusion Similar socio-demographic barriers appear to be important determinants of physical inactivity, regardless of BMI status or the presence of diabetes. However, physical activity promoting interventions in obese and diabetic adults should consider the specific disease-related characteristics of these groups. A special need for target group specific physical activity programmes in adults from ethnic minorities or of advanced age was further identified.