|
 |
| ORIGINAL ARTICLE |
|
| Year : 2016 | Volume
: 60
| Issue : 2 | Page : 99-106 |
|
|
Evaluation of quality of multivariable logistic regression in Indian medical journals using multilevel modeling approach
Rajeev Kumar1, Abhaya Indrayan2, Pragti Chhabra3
1 Biostatistician, Department of Biostatistics and Medical Informatics, University College of Medical Sciences, New Delhi, India
2 Ex-Head of the Department and Professor, Department of Biostatistics and Medical Informatics, University College of Medical Sciences, New Delhi, India
3 Director Professor, Department of Community Medicine, University College of Medical Sciences, New Delhi, India
| Date of Web Publication | 23-Jun-2016 |
Correspondence Address:
Rajeev Kumar
Department of Biostatistics and Medical Informatics, University College of Medical Sciences, Room No. 401, Dilshad Garden, New Delhi - 110 095
India
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/0019-557X.184538
 Abstract | | |
Background: Availability of user-friendly statistical software has increased the application of multivariable logistic regression (MLR) in the medical journal many fold. The reporting quality in terms of checking assumptions, model building strategies, proper coding, and report format need proper care and attention to communicate correct and reliable model results. Objective: The objective of this article is to evaluate the quality of MLR article based on 10-point well establish criteria and to study the factors that may influence the quality. Methods: Study included PubMed indexed Indian medical journals as on March 2010 and published at least ten original articles that applied MLR during 10 years was included in the study. Multilevel modeling was applied to assess the role of journal and article attributes on MLR quality. Results: Twelve out of 39 Indian PubMed indexed journals fulfilled the inclusion criterion. Of a total 5599 original articles in these journals, 262 (4.68%) applied MLR in their study. Conformity of linear gradient assumption for continuous covariate was the least fulfilled criterion. One-third of the MLR articles involved statistician or epidemiologist as co-author, and almost same number of MLR articles' first author was from outside India. The trend of 10-point criteria remained consistent although the number of MLR articles increased over the period. The average quality score was 3.78 (95% confidence interval: 2.97-4.60) out of a possible 10. Larger sample size, involvement of statistician as co-author, non-Indian as the first author, and use of SAS/STATA software increased the quality of MLR articles. Conclusions: The quality of MLR articles in Indian medical journals is lagging behind as compared to the quality of MLR articles published from the United States and Europe medical journals. Joint effort of editors, reviewers, and authors are required to improve the quality of MLR in Indian journals so that the reader gets the correct results. Keywords: Intra-class correlation, multilevel modeling, multivariable logistic regression, quality
How to cite this article:
Kumar R, Indrayan A, Chhabra P. Evaluation of quality of multivariable logistic regression in Indian medical journals using multilevel modeling approach. Indian J Public Health 2016;60:99-106 |
How to cite this URL:
Kumar R, Indrayan A, Chhabra P. Evaluation of quality of multivariable logistic regression in Indian medical journals using multilevel modeling approach. Indian J Public Health [serial online] 2016 [cited 2023 Mar 27];60:99-106. Available from: https://www.ijph.in/text.asp?2016/60/2/99/184538 |
| Introduction | |  |
Availability of high-speed computational facilities and user-friendly statistical softwares seem to have significantly increased application of multivariable regression models in medical literature. [1],[2] Dichotomous endpoint or binary outcome is often encountered in biomedical research such as diseased-healthy, survivor-nonsurvivor, and case-control. Among the available binary link functions, multivariable logistic regression (MLR) is most frequently applied regression model [1] because (i) sigmoid shape of logistic regression that appeals in most of the epidemiology conditions; (ii) easy interpretation as compared with other binary link function contenders; (iii) the association can be expressed in odds ratio (OR) which is easily understandable by the medical professionals; (iv) no restriction on scales of predictors or explanatory variables, for example, MLR can accommodate, nominal, ordinal and metric scales; (v) in rare disease scenario OR nearly equals relative risk; and (vi) wide availability of MLR in commercial statistical softwares.
No regression model is perfect because regression models are always associated with uncertainty and underlying assumptions but researcher can increase the parsimony of the model by proper testing of associated inherent assumptions, adequate interpretation of coefficients, and complete reporting of model results. Several papers have evaluated the MLR assumptions, validation, and reporting quality in different fields of medicine in the non-Indian medical journals based on the well-established 10-point criteria and stressed the need to improve the reporting and testing of these assumptions. [1],[3],[4],[5],[6] In Indian journals, studies have been conducted to evaluate the reporting quality of elementary statistical methods and basic statistics. [7] No study has been conducted on quality of the application of multivariable regression methods except one study conducted on a few selected Indian medical journals that revealed that quality and testing of assumptions of MLR articles is poor in our journals. [8]
This study has been conducting to evaluate the MLR quality in PubMed indexed Indian journals to get broader aspect about MLR quality. Furthermore, the trend of quality scores over 10 years and effect of five article and three journal covariates that can influence the quality of MLR model has also been studied using two-level multilevel approach because the quality of MLR articles within journals may be correlated.
| Materials and Methods | | |
Inclusion criteria
Original MLR articles published in English language and fulfilling the following criteria were included:
- Original MLR articles published in PubMed indexed journals from January 1, 2000, to December 31, 2009. The cut off year 2000 was selected because as our extensive literature search, the first article that evaluated MLR articles and highlighting the deficiencies appeared in the year 1999 [9]
- Journals having at least ten MLR original articles over the 10-year period were selected. The cut off of ten articles was chosen to have a reasonable number for valid comparison across the journals. The 10-year period was stratified in 5 blocks of two calendar years for investigating the trend of quality over time.
Search method
Original articles were identified from the query made in PubMed search bar using search string "logistic regression" because it is a wider term that minimizes the selection bias. It may be possible that our string search may filter less MLR articles than actual, but we had to rely on the electronic search because it was extremely difficult to search manually for MLR in all the issues of 39 journals published in a 10-year period.
Data extraction
Original MLR articles from the eligible journals were evaluated by using 10-point well-established criteria of Bagley et al. [4] and described in [Table 1]. List of MLR articles was prepared in MS-Excel. An exercise was done to check the reliability of scoring based on 10-point criteria. For this, fifty randomly selected articles out of eligible for this study were independently evaluated by experienced biostatistician and epidemiologist (RK and PC). Inter-rater agreement between RK and PC for each criterion was assessed by Kappa statistics and varied from 0.79 to 0.9. The third author (AI) renowned biostatistician was consulted in case of any clarification required or solving the discrepancies. | Table 1: Criteria used to examine the assumptions and reporting quality of multivariable logistic regression in articles reviewed in the Indian Medical Journals
Click here to view |
Scoring and assumptions
Each of ten criteria when fulfilled by an article was assigned score 1 and 0 otherwise. Many MLR articles did not use any continuous covariate. In this situation, testing of linear gradient criteria was not applicable. The score was standardized to incorporate the not applicable criteria for linear gradient so that all had a similar common base and were comparable. Equal weightage was given to each criterion. Total score obtained had a feature of quasi-interval scale hence regular parametric test could be applied if the score is normally distributed. Higher score represented better quality of the MLR article. If more than one MLR model was applied in an article, the first model reported or the MLR of the primary outcome was selected for evaluation. It is difficult to identify from the MLR article that which author has not tested the criteria or which author tested but not reported due words limitation. Thus, if an article did not mention about a particular logistic assumption, it was considered to indicate that the assumption was not tested.
Journal and article characteristics
Three journal and five article level covariates, namely Journal's specialty, number of MLR article per issue, and impact factor for the journal; and number of authors, name of software used, involvement of statistician or epidemiologist as co-author, sample size, and nationality of the first author were examined. The detailed description is described in the Supplement Material [Additional file 1].
Validity assessment and blinding of article
The single blinding procedure was applied to maintain the anonymity of the authors of the articles and name of journals to avoid possible bias in scoring. The masking and blinding were done by a person not interested in this study.
Ethical consideration
The permission to conduct the study was obtained from the Ethics Committee of our Institution.
Statistical analysis
Statistical analysis was carried out using STATA -11, College Station; TX: StatCrop LP, USA and Multilevel version 2.1. [10] The proportion of each criterion was determined and 95% confidence interval (CI) was calculated using the exact binomial method. [11] The trend of each criterion over 5 block-years was evaluated using Chi-square for trend. [12] Normality of quality score was checked using skewness and kurtosis test and also verified by Q-Q plots and Box-Whiskers plot. The two-level random-intercept model was fitted to account for the clustering within the journals, exploring the effect of article and journal covariates on the quality of MLR, and assessing the journal variability and a robust analytic tool since it generates valid and reliable estimates of fixed effect parameters even for very small clustering. [13] The multilevel model assumptions, namely normality among the intercepts, linearity effect of continuous variable (i.e., sample size), and homogeneity of variance across the journals was verified. The clustering effect was determined by intra-class correlation (ICC) obtained from the unconditional model. Univariable two-level model for each of article covariate was first evaluated. The variables with P < 0.25 in univariable models were included in backward approach to get the final model and P < 0.10 was considered to retain the covariate. [14] Sample size in each study was log transformed due to its highly right skewed distribution and centered for grand mean. The number of authors was also centered for grand mean to get intercept meaningful.
| Results | | |
Twelve journals out of 39 PubMed indexed Indian journals fulfilled the eligible criteria for inclusion in this study. Two hundred and sixty-two (4.68%) out of total 5599 original articles published during the 10-year period from 2000 to 2009 applied MLR in their study. The maximum number of original articles on MLR was published in Indian Journal of Pediatrics (IJP), whereas the highest proportion of MLR articles was found in Journal of Postgraduate Medicine. The mean quality score was nearly consistent across the block year (P = 0.778). The overall mean quality score was 3.81 ± 1.58 [Figure 1]. The percentage of MLR articles to total original articles increased from 2.72% in 2000-2001 to 6.15% in 2004-2005, which is more than double and thereafter remained constant, albeit absolute numbers of MLR articles increased from 30 to 70 [Figure 1]. This increase varied widely across the journals.
None of the MLR articles fulfilled all the 10 or even 9 criteria. Only 8% MLR articles fulfilled 6-8 assumption criteria, 70% fulfilled 3-5, and 21% fulfilled 1-2, and 1.5% article did not fulfill any criteria. Median number of authors was 5 and varied from 2 to 11. Median sample size was 338 and varied from 140 to 1015. The least fulfilled criterion was conformity of linear gradient and most fulfilled was mention of P value, OR and 95% CI [Table 2]. The P value alone was reported in 98% of MLR articles and along with OR in 95% of MLR articles. Fourteen percent of MLR articles did not report the 95% CI of OR and there was a significant association (P < 0.001) of not reporting CI and over fitting that occurs due to disproportionately low sample size.
Three-fourths of MLR articles (78.24%) reported the name of the software used for MLR analysis. Among these, SPSS software was most prevalent (81%) followed by STATA 11% and then SAS 5%, while remaining 11% used other softwares. These percentages add up to more than 100 because some articles used two or more statistical softwares. Statistician or epidemiologist involvement as co-author was only in one-third of MLR articles and in 27.5% of MLR articles the first author was non-Indian. | Figure 1: Mean quality score in relation to the block year, number in column indicates the percentage of multivariable logistic regression articles to total
Click here to view |
 | Table 2: Number and percentage of articles that met the recommended criteria
Click here to view |
The trend over the five block years for any criteria was insignificant. However, three criteria, namely the sample size, P value, OR and 95% CI, and coding of covariate showed improvement over the previous blocks increased from 60% to 70%, 80% to 91.4% and from 43% to 60%, respectively, from 2000-2001 to 2008-2009 but the trend was not statistically significant.
Multilevel modeling results
The box-whisker plot revealed the distribution of quality score for Indian Journal of Community Medicine (IJCM) and Journal of the Association of Physicians of India has right and left skewed, respectively. The combined overall quality score distribution and other remaining journals shows nearly have symmetric pattern [Figure 2]. The Skewness and Kurtosis statistical test revealed that combined overall quality score did not violate the normality condition (P = 0.117). | Figure 2: Box-whisker plot depicting the distribution of quality score across the journals
Click here to view |
The average quality score was 3.78 (95% CI: 2.97-4.60) and ICC within journals was 0.069 (95% CI: 0.016-0.199). Although ICC was small but likelihood ratio test showed significance and data also have a hierarchical nature. Small ICC reflects that heterogeneity within the journal about model strategy, reporting, and testing of assumptions. This ICC can also be interpreted as 7% variability of the total variance in quality score is due to journals variability, and 93% is due to articles. The likelihood ratio test showed a significance (P = 0.0026) between multilevel and single level model. Only Indian Journal of Gastroenterology has significantly lower than overall average quality score and IJP and National Medical Journal of India had significantly high-quality score with the overall average quality score. The significance was determined by the mean shrunken residuals and its 95% CIs. [14]
Univariable multilevel results
Five categories of software used were collapsed them into two categories (SAS/STATA (SAS institute Inc. Cary, NC, USA) vs. Others (SPSS Inc, Chicago, IL, USA)) because the quality score of STATA and SAS users were significant with other three categories of softwares users. Sample size showed a significant quadratic relation with a quality score with a positive linear coefficient and negative quadratic coefficient. These reflect that the quality improves as sample size increases but acceleration rate decreases as sample size increases. Involvement of statistician, nationality of the first author, sample size, and software used had P < 0.25 in the univariable analysis [Table 3]. | Table 3: Results of random intercept model for univariable (separate for covariate) and multivariable analysis
Click here to view |
The multilevel model showed that the involvement of statistician explained 53% journal variance and 4% article variance. The reason of explaining the high journal variance was an unequal distribution of statistician involvement across the journals and proportion varied from 0% (Neurology India) to 100% (IJCM). [14] Sample size also had unequal distribution and median sample size was 338 and varied from 142 to 835 across the journals.
Multivariable multilevel results
When the journal and article covariates were simultaneously entered into the multilevel model, sample size, SAS/STATA users, involvement of statistician and nationality of the first author turn out significant (P < 0.10). The positive coefficient indicated that the presence of covariate improved the quality of MLR. The nationality of the first author had a P = 0.255 in the univariable two-level model but significance (P = 0.029) at two-level multivariable which revealed that selection based on univariable significant (P < 0.05) can skip an important covariate such as this. These variables explained 34% of article variance and 64% of journal level variance. Journal level covariates were not found to have a significant effect on quality and these were removed from the final model [Table 3]. The interaction between journal variable was not studied due to small sample size. The intercepts residuals followed a normal distribution and variance of residuals across the journals was homogeneous which indicate that the assumptions of the multilevel model were not violated.
| Discussion | | |
Advance computing power has increased the application of advanced biostatistical methods many fold. In our study, 4.7% of original articles applied MLR. This rate was reported as 15.6% in Turkish cardiology journals, [15] 8.4% in an article published in pulmonary and intensive care, [16] 6.7% in 10 Chinese leading Medline indexed medical journals published in 2008, [17] and 6% in transplant journals, [6] respectively.
Forty-five of MLR articles selected the potential variables for multivariable analysis using univariable analysis cut off P < 0.05. This is widely accepted that variable based on significance of univariable analysis P < 0.05 is an incorrect procedure because it increases the chance of biased results and instability in model results and may reject an important variable that may become significant only after adjustment like nationality of first author in our study. [18] This practice was more commonly followed in Indian medical journals as well as in American cancer journals (49%) and in two top Chinese journals (40%). [19] One way to deal with problem is relaxing the cut off from P < 0.05 to <0.25 for univariable analysis as suggested in the literature. [20]
Five out of ten well-established criteria were fulfilled in <11% of MLR articles, while the remaining five criteria were fulfilled in more than 50% of MLR articles [Table 2]. Present results indicate that we are far behind in fulfillment of the criteria of MLR as compared to the studies published on European and the Unites States journals. [1],[3],[4],[6]
The involvement of statistician as co-author improved the quality by 0.3 unit and significant at 10% (P = 0.093). Only 36% of MLR articles involved the statistician as co-author. This shows the lower participation of statistician and epidemiologist as co-author in MLR articles. The SAS/STATA users had a higher mean quality score by 0.6 units than users of SPSS and other softwares (Epi Info, MedCalc, etc.). Our objective is not to criticize or advertise any of the statistical software but to state facts as they exist on the ground. These two softwares have features to test the assumptions of MLR, for example, testing the conformity of linear gradient assumptions of continuous variable and model diagnostic. In addition, nonstatistician find difficulty in applying these two softwares compared to SPSS, which is very easy to handle by new users and health professionals. The most common softwares in Indian journals was SPSS, followed by STATA and SAS, whereas study conducted in Journal of American Medical Association found just the reverse pattern, namely SAS, STATA, then SPSS. [21]
Slight deviation in linearity assumption of the continuous covariate in MLR does not affect much, but J-shape and U-shape relationships produce wrong inference about continuous covariate. Categorization of continuous variable is usually a wrong practice and leads to loss of power, loss of information, and other serious disadvantages. [22],[23] Dichotomizing of continuous variable is strongly condemned by statisticians because it loses one-third of data information. [24] Categorization of continuous covariate is justified only when covariate is highly skewed or has nonlinear relationship, but latter problem can now easily handle by the spline method. [25]
The present study is not free from limitations. The study includes MLR articles published up to December 31, 2009, which might be quite old but our experience shows there is not much change in the MLR quality in Indian journals. The percentage of MLR was estimated on the basis of the electronic search. It is not impossible that some of the articles used MLR but not found in the electronic search. It may be possible that authors have applied the criteria but have failed to report. For example, interaction(s) may have been tested but not reported. Thus, our results are based on what author has been reported in the articles. Some articles have used MLR as a secondary analysis; their quality may be low. The cross-level interactions and random slope in multilevel method were not included due to small sample size of journal and convergence problem. In addition, the random selection of level-2 (journals) was not possible because all the eligible journals were included in the study.
The strength of the present study lies in relatively large sample size compared with other studies and inclusion of major Indian journals. Nevertheless, it is small for multilevel modeling analysis. Besides estimating the influence of covariates, this study covers good quality Indian medical journals and sample size is more than double than our previous study. In addition, multilevel model was applied to find the covariates that influenced the MLR quality. This is the first such attempt in India. Thus, the results are believable and true for nearly all Indian medical journals.
| Conclusions and Recommendations | | |
Multivariable models are important and need correct and complete reporting. This will not only help the reader but also to the reviewer to evaluate the model finding and rely on the model results. The reporting format and other cautions for MLR are described elsewhere. [6],[26] The study results are mainly dependent on how the good the model fitted by the author/s, not reporting the sufficient information(s) leave the readers in a dilemma. These information can be provided in little space. Nowadays most of the journals are electronic and these information can be provided as Supplement Material so that interested reader(s) can download the desired information. We advocate that author should report these information(s) if tested. Thus, reporting of these criteria is warranted to prove the parsimony of the model result. We would also suggest that when editor removes the model information he/she should write one line "model assumption(s) like conformity of linear gradient and collinearity have been tested and information deleted due to word constraint." This indicates that author is aware about the assumptions and other relevant criteria.
Furthermore, the editors should encourage the statistical perspective, statistical reviews, etc., in Indian journals like published Indian Pediatric. [27],[28] These publications will brush-up the medical and statistical professionals about the old and latest developments as well as encourage the young professionals to apply such methods.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Mikolajczyk RT, DiSilvestro A, Zhang J. Evaluation of logistic regression reporting in current obstetrics and gynecology literature. Obstet Gynecol 2008;111(2 Pt 1):413-9.  |
| 2. | Müllner M, Matthews H, Altman DG. Reporting on statistical methods to adjust for confounding: A cross-sectional survey. Ann Intern Med 2002;136:122-6. |
| 3. | Moss M, Wellman DA, Cotsonis GA. An appraisal of multivariable logistic models in the pulmonary and critical care literature. Chest 2003;123:923-8. |
| 4. | Bagley SC, White H, Golomb BA. Logistic regression in the medical literature: Standards for use and reporting, with particular attention to one medical domain. J Clin Epidemiol 2001;54:979-85. |
| 5. | Ottenbacher KJ, Ottenbacher HR, Tooth L, Ostir GV. A review of two journals found that articles using multivariable logistic regression frequently did not report commonly recommended assumptions. J Clin Epidemiol 2004;57:1147-52. |
| 6. | Kalil AC, Mattei J, Florescu DF, Sun J, Kalil RS. Recommendations for the assessment and reporting of multivariable logistic regression in transplantation literature. Am J Transplant 2010;10:1686-94. |
| 7. | Karan J, Kantharia N, Yadav P, Bhardwaj P. Reporting statistics in clinical trials published in Indian journals: A survey. Pak J Med Sci 2010;26:212-6. |
| 8. | Kumar R, Indrayan A, Chhabra P. Reporting quality of multivariable logistic regression in selected Indian medical journals. J Postgrad Med 2012;58:123-6. [ PUBMED]  |
| 9. | Khan KS, Chien PF, Dwarakanath LS. Logistic regression models in obstetrics and gynecology literature. Obstet Gynecol 1999;93:1014-20. |
| 10. | Rasbash J, Charlton C, Browne WJ, Healy M, Cameron, B. MLwiN Version 2.1. Centre for Multilevel Modelling, University of Bristol; 2009. |
| 11. | Clopper CJ, Pearson ES. The use of confidence or fiduciallimits illustrated in the case of binomial. Biometrika 1934;26:404-13. |
| 12. | Armitage P, Berry G. Statistical Method in Medcial Research. 3 rd ed. Oxford: Blackwell Science; 2001. |
| 13. | Merlo J, Yang M, Chaix B, Lynch J, Råstam L. A brief conceptual tutorial on multilevel analysis in social epidemiology: Investigating contextual phenomena in different groups of people. J Epidemiol Community Health 2005;59:729-36. |
| 14. | Hox JJ. Multilevel Analysis: Techniques and Applications. 2 nd ed. New York: Taylor and Francis; 2010. |
| 15. | Tanboga IH, Kurt M, Isik T, Kaya A, Ekinci M, Aksakal E, et al. Assessment of multivariate logistic regression analysis in articles published in Turkish cardiology journals. Turk Kardiyol Dern Ars 2012;40:129-34. |
| 16. | Tetrault JM, Sauler M, Wells CK, Concato J. Reporting of multivariable methods in the medical literature. J Investig Med 2008;56:954-7. |
| 17. | Jin Z, Yu D, Zhang L, Meng H, Lu J, Gao Q, et al. A retrospective survey of research design and statistical analyses in selected Chinese medical journals in 1998 and 2008. PLoS One 2010;5:e10822. |
| 18. | Harrell F, Lee KL, Mark DB. Tutorial in biostatistics multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361-87. |
| 19. | Liao H, Lynn HS. A survey of variable selection methods in two Chinese epidemiology journals. BMC Med Res Methodol 2010;10:87. |
| 20. | Hosmer DW, Stanley L. Applied Logistic Regression. New York: John Wiley & Sons; 2000. |
| 21. | Arnold LD, Braganza M, Salih R, Colditz GA. Statistical trends in the Journal of the American Medical Association and implications for training across the continuum of medical education. PLoS One 2013;8:e77301. |
| 22. | Dinero TE. Seven reasons why you should not categorize continuous data. J Health Soc Policy 1996;8:63-72. |
| 23. | Turner EL, Dobson JE, Pocock SJ. Categorisation of continuous risk factors in epidemiological publications: A survey of current practice. Epidemiol Perspect Innov 2010;7:9. |
| 24. | Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ 2006;332:1080. |
| 25. | Durrleman S, Simon R. Flexible regression models with cubic splines. Stat Med 1989;8:551-61. |
| 26. | Kumar R, Chhabra P. Cautions required during planning, analysis and reporting of multivariable logistic regression. Curr Med Res Pract 2014;4:31-9. |
| 27. | Kumar R, Indrayan A. Receiver operating characteristic (ROC) curve for medical researchers. Indian Pediatr 2011;48:277-87. |
| 28. | Indrayan A, Satyanarayana L. 11. Statistical relationships and the concept of multiple regression. Indian Pediatr 2001;38:43-59. |
[Figure 1], [Figure 2]
[Table 1], [Table 2], [Table 3]
|
