Result: Systematic Review and Reliability Generalization Meta-Analysis of the Video Game Dependency Scale

Objective: Video Game Dependency Scale (VGDS) evaluates degrees of online and offline gaming disorder. In this study, we examined the average reliability of VGDS scores and whether they provide values that justify their use, characterized the variation in reliability estimates across studies, and identified study characteristics that explain heterogeneity in reliability estimates. Method: A reliability generalization meta-analysis based on the REGEMA guidelines was conducted. From the 19 selected articles that applied VGDS, 14 (16 independent samples) reported 17 reliability estimates (16 alpha coefficients and 1 test-retest reliability coefficient). A random-effects model was applied in the statistical analyses. Results: The average reliability of the total VGDS scores assessed as internal consistency by Cronbach's alpha coefficient was 0.925 (95% CI [.901,.942]). The results also found these coefficients to be significantly heterogeneous. Conclusions: According to the psychometric theory, VGDS scores can be considered reliable for exploring gaming disorder in field research and clinical practice. Significance statement: This study provides evidence for the score reliability of the VGDS, a relevant tool for assessing Internet gaming disorder. It highlights the importance of addressing score reliability and advancing measurement and assessment in practice and research fields.

aid-jclp2270390217>3.0.co;2-9</span> </p>

Fam, J. Y. (2018). Prevalence of internet gaming disorder in adolescents: A meta-analysis across three decades. Scandinavian Journal of Psychology, 59 (5), 524 – 531. https://doi.org/10.1111/sjop.12459

Haktanir, A., Stephen Lenz, A., & Furkan Kurnaz, M. (2024). Reliability generalization meta-analysis of the child PTSD symptom scale for DSM-5 (CPSS-5). Measurement and Evaluation in Counseling and Development, 57 (2), 186 – 198. https://doi.org/10.1080/07481756.2024.2318552

Hartung, J., & Knapp, G. (2001). On tests of the overall treatment effect in the meta-analysis with normally distributed responses. Statistics in Medicine, 20 (12), 1771 – 1782. https://doi.org/10.1002/sim.791

Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ (Clinical Research ed.), 327 (7414), 557 – 560. https://doi.org/10.1136/bmj.327.7414.557

Henson, R. K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting "Reliability Generalization" (RG) studies. Measurement and Evaluation in Counseling and Development, 35, 113 – 127.

Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21 (11), 1539 – 1558. https://doi.org/10.1002/sim.1186

Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business and Psychology, 27 (1), 99 – 114. https://doi.org/10.1007/s10869-011-9231-8

Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006). Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychological Methods, 11 (2), 193 – 206. https://doi.org/10.1037/1082-989X.11.2.193

Kalkbrenner, M. T. (2024). Choosing between Cronbach's coefficient alpha, McDonald's coefficient omega, and coefficient H : Confidence intervals and the advantages and drawbacks of interpretive guidelines. Measurement and Evaluation in Counseling and Development, 57 (2), 93 – 105. https://doi.org/10.1080/07481756.2023.2283637

*Kewitz, S., Leo, K., Rehbein, F., & Lindenberg, K. (2023). Assessment of hazardous gaming in children and its dissimilarities and overlaps with internet gaming disorder. Frontiers in Psychiatry, 14, 1226799. https://doi.org/10.3389/fpsyt.2023.1226799

*Kewitz, S., Vonderlin, E., Wartberg, L., & Lindenberg, K. (2021). estimated prevalence of unreported IGD cases in routine outpatient children and adolescent psychotherapy. International Journal of Environmental Research and Public Health, 18 (13), 6787. https://doi.org/10.3390/ijerph18136787

*Kindt, S., Szász-Janocha, C., Rehbein, F., & Lindenberg, K. (2019). School-related risk factors of internet use disorders. International Journal of Environmental Research and Public Health, 16 (24), 4938. https://doi.org/10.3390/ijerph16244938

King, D. L., Chamberlain, S. R., Carragher, N., Billieux, J., Stein, D., Mueller, K., Potenza, M. N., Rumpf, H. J., Saunders, J., Starcevic, V., Demetrovics, Z., Brand, M., Lee, H. K., Spada, M., Lindenberg, K., Wu, A. M. S., Lemenager, T., Pallesen, S., Achab, S., ... Delfabbro, P. H. (2020). Screening and assessment tools for gaming disorder: A comprehensive systematic review. Clinical Psychology Review, 77, 101831. https://doi.org/10.1016/j.cpr.2020.101831

King, D. L., Haagsma, M. C., Delfabbro, P. H., Gradisar, M., & Griffiths, M. D. (2013). Toward a consensus definition of pathological video-gaming: A systematic review of psychometric assessment tools. Clinical Psychology Review, 33 (3), 331 – 342. https://doi.org/10.1016/j.cpr.2013.01.002

Király, O., Sleczka, P., Pontes, H. M., Urbán, R., Griffiths, M. D., & Demetrovics, Z. (2017). Validation of the ten-Item Internet Gaming Disorder Test (IGDT-10) and evaluation of the nine DSM-5 internet gaming disorder criteria. Addictive Behaviors, 64, 253 – 260. https://doi.org/10.1016/j.addbeh.2015.11.005

Knapp, G., & Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22 (17), 2693 – 2710. https://doi.org/10.1002/sim.1482

Langan, D. (2022). Assessing heterogeneity in random-effects meta-analysis. Methods in Molecular Biology (Clifton, N.J.), 2345, 67 – 89. https://doi.org/10.1007/978-1-0716-1566-9_4

Lemmens, J. S., Valkenburg, P. M., & Gentile, D. A. (2015). The Internet gaming disorder scale. Psychological Assessment, 27 (2), 567 – 582. https://doi.org/10.1037/pas0000062

Lemmens, J. S., Valkenburg, P. M., & Peter, J. (2009). Development and validation of a game addiction scale for adolescents. Media Psychology, 12 (1), 77 – 95. https://doi.org/10.1080/15213260802669458

Lenz, A. S., Ault, H., Balkin, R. S., Minton, C. B., Erford, B. T., Hays, D. G., Kim, B. S. K., & Li, C. (2022). Responsibilities of users of standardized tests (RUST-4E). Measurement and Evaluation in Counseling and Development, 55 (4), 227 – 235. https://doi.org/10.1080/07481756.2022.2052321

*Leo, K., Kewitz, S., Wartberg, L., & Lindenberg, K. (2021). Depression and social anxiety predict internet use disorder symptoms in children and adolescents at 12-month follow-up: Results From a longitudinal study. Frontiers in Psychology, 12, 787162. https://doi.org/10.3389/fpsyg.2021.787162

*Li, Y., Tang, Y., Huang, S., Tan, L., Huang, Q., Chen, X., Lin, S., Hao, J., Liao, Z., & Shen, H. (2023). Role of gaming devices associated with internet gaming disorder in China: Cross-sectional study. JMIR Serious Games, 11, e40130. https://doi.org/10.2196/40130

*Liao, Z., Chen, X., Huang, S., Huang, Q., Lin, S., Li, Y., Tang, Y., & Shen, H. (2022). Exploring the associated characteristics of Internet gaming disorder from the perspective of various game genres. Frontiers in Psychiatry, 13, 1103816. https://doi.org/10.3389/fpsyt.2022.1103816

*Liao, Z., Huang, Q., Huang, S., Tan, L., Shao, T., Fang, T., Chen, X., Lin, S., Qi, J., Cai, Y., & Shen, H. (2020). Prevalence of internet gaming disorder and its association with personality traits and gaming characteristics among chinese adolescent gamers. Frontiers in Psychiatry, 11, 598585. https://doi.org/10.3389/fpsyt.2020.598585

Lindenberg, K., Halasy, K., & Schoenmaekers, S. (2017). A randomized efficacy trial of a cognitive-behavioral group intervention to prevent Internet Use Disorder onset in adolescents: The PROTECT study protocol. Contemporary Clinical Trials Communications, 6, 64 – 71. https://doi.org/10.1016/j.conctc.2017.02.011

Lindenberg, K., Kindt, S., & Szász-Janocha, C. (2022). Effectiveness of cognitive behavioral therapy-based intervention in preventing gaming disorder and unspecified internet use disorder in adolescents: A cluster randomized clinical trial. JAMA Network Open, 5 (2), e2148995. https://doi.org/10.1001/jamanetworkopen.2021.48995

López-López, J. A., Botella, J., Sánchez-Meca, J., & Marín-Martínez, F. (2013). Alternatives for mixed-effects meta-regression models in the reliability generalization approach: A simulation study. Journal of Educational and Behavioral Statistics, 38, 443 – 469. https://doi.org/10.3102/1076998612466142

López-López, J. A., Marín-Martínez, F., Sánchez-Meca, J., Van den Noortgate, W., & Viechtbauer, W. (2014). Estimation of the predictive power of the model in mixed-effects meta-regression: A simulation study. The British Journal of Mathematical and Statistical Psychology, 67 (1), 30 – 48. https://doi.org/10.1111/bmsp.12002

McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23 (3), 412 – 433. https://doi.org/10.1037/met0000144

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17 (3), 437 – 455. https://doi.org/10.1037/a0028085

Meerkerk, G. J., Van Den Eijnden, R. J., Vermulst, A. A., & Garretsen, H. F. (2009). The Compulsive Internet Use Scale (CIUS): some psychometric properties. Cyberpsychology & Behavior: The Impact of the Internet, Multimedia and Virtual Reality on Behavior and Society, 12 (1), 1 – 6. https://doi.org/10.1089/cpb.2008.0181

Merino-Soto, C., & Angulo-Ramos, M. (2021). Metric studies of the Compliance Questionnaire on Rheumatology (CQR): A case of validity induction? Reumatologia Clinica, 18 (8), 497 – 498. https://doi.org/10.1016/j.reumae.2021.03.006

Merino-Soto, C., & Angulo-Ramos, M. (2020). Validity induction: Comments on the study of Compliance Questionnaire for Rheumatology. Revista Colombiana de Reumatología, 28, 312 – 313. https://doi.org/10.1016/j.rcreu.2020.05.005

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. McGraw Hill.

Oka, T., Hamamura, T., Miyake, Y., Kobayashi, N., Honjo, M., Kawato, M., Kubo, T., & Chiba, T. (2021). Prevalence and risk factors of internet gaming disorder and problematic internet use before and during the COVID-19 pandemic: A large online survey of Japanese adults. Journal of Psychiatric Research, 142, 218 – 225.

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ (Clinical Research ed.), 372, n71. https://doi.org/10.1136/bmj.n71

Paulus, F. W., Ohmann, S., von Gontard, A., & Popow, C. (2018). Internet gaming disorder in children and adolescents: A systematic review. Developmental Medicine and Child Neurology, 60 (7), 645 – 659. https://doi.org/10.1111/dmcn.13754

Pinsoneault, T. B. (2007). Detecting random, partially random, and nonrandom Minnesota Multiphasic Personality Inventory-2 protocols. Psychological Assessment, 19 (1), 159 – 164. https://doi.org/10.1037/1040-3590.19.1.159

Pontes, H. M., & Griffiths, M. D. (2015). Measuring DSM-5 internet gaming disorder: Development and validation of a short psychometric scale. Computers in Human Behavior, 45, 137 – 143. https://doi.org/10.1016/j.chb.2014.12.006

Pontes, H. M., Király, O., Demetrovics, Z., & Griffiths, M. D. (2014). The conceptualisation and measurement of DSM-5 Internet Gaming Disorder: The development of the IGD-20 Test. PloS One, 9 (10), e110137. https://doi.org/10.1371/journal.pone.0110137

Raudenbush, S. W. (2009). Analyzing effect sizes: Random-effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 295 – 315). Russell Sage Foundation.

Rehbein, F., Kleimann, M., & Mössle, T. (2010). Prevalence and risk factors of video game dependency in adolescence: Results of a German nationwide survey. Cyberpsychology, Behavior and Social Networking, 13 (3), 269 – 277. https://doi.org/10.1089/cyber.2009.0227

*Rehbein, F., Kliem, S., Baier, D., Mößle, T., & Petry, N. M. (2015). Prevalence of Internet gaming disorder in German adolescents: Diagnostic contribution of the nine DSM-5 criteria in a state-wide representative sample. Addiction, 110 (5), 842 – 851. https://doi.org/10.1111/add.12849

Reidl-Martínez, L. M. (2013). Confiabilidad en la medición [Reliability of measurement]. Investigación en Educación Médica, 2 (6), 107 – 111.

Rubio-Aparicio, M., Badenes-Ribera, L., Sánchez-Meca, J., Fabris, M. A., & Longobardi, C. (2020). A reliability generalization meta-analysis of self-report measures of muscle dysmorphia. Clinical Psychology: Science and Practice, 27 (1), 1 – 24. https://doi.org/10.1111/cpsp.12303

Sánchez-Meca, J., López-Pina, J. A., & López López, J. A. (2009). Generalización de la fiabilidad: Un enfoque meta-analítico aplicado a la fiabilidad [Reliability generalization: A meta-analytic approach to reliability coefficients]. Fisioterapia, 31, 262 – 270. https://doi.org/10.1016/j.ft.2009.05.005

Sánchez-Meca, J., Marín-Martínez, F., López-López, J. A., Núñez-Núñez, R. M., Rubio-Aparicio, M., López-García, J. J., López-Pina, J. A., Blázquez-Rincón, D. M., López-Ibáñez, C., & López-Nicolás, R. (2021). Improving the reporting quality of reliability generalization meta-analyses: The REGEMA checklist. Research Synthesis Methods, 12 (4), 516 – 536. https://doi.org/10.1002/jrsm.1487

Santos, G. (2017). Validez y Confiabilidad del Cuestionario de Calidad de Vida SF-36 en Mujeres con LUPUS, Puebla [Bachelor's Thesis, Universidad Autónoma de Puebla]. https://repositorioinstitucional.buap.mx/

Stevens, M. W., Dorstyn, D., Delfabbro, P. H., & King, D. L. (2021). Global prevalence of gaming disorder: A systematic review and meta-analysis. The Australian and New Zealand Journal of Psychiatry, 55 (6), 553 – 568. https://doi.org/10.1177/0004867420962851

*Szász-Janocha, C., Vonderlin, E., & Lindenberg, K. (2020). Treatment outcomes of a CBT-based group intervention for adolescents with Internet use disorders. Journal of Behavioral Addictions, 9 (4), 978 – 989. https://doi.org/10.1556/2006.2020.00089

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53 – 55. https://doi.org/10.5116/ijme.4dfb.8dfd

Tejeiro Salguero, R. A., & Morán, R. M. (2002). Measuring problem video game playing in adolescents. Addiction (Abingdon, England), 97 (12), 1601 – 1606. https://doi.org/10.1046/j.1360-0443.2002.00218.x

Tejeiro, R. A., Espada, J. P., Gonzálvez, M. T., & Christiansen, P. (2016). Psychometric properties of the Problem Video Game Playing scale in adults. Revue Européenne de Psychologie Appliquée, 66 (1), 9 – 13. https://doi.org/10.1016/j.erap.2015.11.004

Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174 – 195.

Vacha-Haase, T. (1998). Reliability Generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6 – 20.

Vacha-Haase, T., & Thompson, B. (2011). Score reliability: A retrospective look back at twelve years of Reliability Generalization (RG) studies. Measurement and Evaluation in Counseling and Development, 44, 159 – 168.

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metaphor package. Journal of Statistical Software, 36 (3), 1 – 48. https://doi.org/10.18637/jss.v036.i03

Viechtbauer, W., López-López, J. A., Sánchez-Meca, J., & Marín-Martínez, F. (2015). A comparison of procedures to test for moderators in mixed-effects meta-regression models. Psychological Methods, 20, 360 – 374. https://doi.org/10.1037/met0000023

Wang, H., & Cheng, C. (2021). New perspectives on the prevalence and associated factors of gaming disorder in Hong Kong community adults: A generational approach. Computers in Human Behavior, 114, 106574. https://doi.org/10.1016/j.chb.2020.106574

Wang, R., Yang, S., Yan, Y., Tian, Y., & Wang, P. (2021). Internet Gaming Disorder in Early Adolescents: Gender and Depression Differences in a Latent Growth Model. Healthcare, 9 (9), 1188. https://doi.org/10.3390/healthcare9091188

Worldwide Health Organization (2018). ICD-11 International Classification of Diseases 11th Revision. Accessed on January 2 nd. https://icd.who.int/en

*Zhang, L., Luo, T., Hao, W., Cao, Y., Yuan, M., & Liao, Y. (2022). Gaming disorder symptom questionnaire: the development and validation of a screening tool for ICD-11 gaming disorder in adolescents. Frontiers in Psychiatry, 13, 848157. https://doi.org/10.3389/fpsyt.2022.848157

Zhu, J., Qian, R., Zhong, H., Li, Y., Liu, X., & Ma, J. (2022). Factors influencing the addiction characteristics of non-suicidal self-injurious behaviors in adolescents: A case-control study. Frontiers in Psychiatry, 13, 1033242. https://doi.org/10.3389/fpsyt.2022.1033242

</div>

By Júlia Gisbert-Pérez; Elena Cejalvo; Manuel Martí-Vilar and Laura Badenes-Ribera

Reported by Author; Author; Author; Author

Júlia Gisbert-Pérez graduated in Psychology from the University of Valencia with an extraordinary award and has completed a master's degree in Psychological Intervention in Social Areas at the University of Valencia. She completed an expert course in Psychology and Esports at the Official College of Psychologists of Madrid (Spain). She is a beneficiary of a scholarship for university teacher training granted by the Ministry of Science, Innovation and Universities of Spain (FPU22/03053).

Elena Cejalvo graduated in Psychology from the University of Valencia and has completed a master's degree in Psychological Intervention in Social Areas at the University of Valencia. She realized her practicum at the Valencian Centre for Psychotheraphy in 2019 and Cotlas association in 2021. Currently, she is working as a professor at the International University of Valencia. She has also realized volunteering in the penitentiary centre of Picassent in 2019 and in Plena Inclusión Valencian Community in 2021.

Manuel Martí-Vilar graduated in Psychology in 1994 and PhD in Psychology in 2000 at the University of Valencia. Extraordinary award of degree and doctorate. Full professor of Basic Psychology since 2002 at the University of Valencia. He has conducted research stays at the NFER (UK) and universities in Spain, Portugal, and Chile. He teaches in the degree of Psychology and has master's degrees in Psychological Intervention in Social Areas, Special Education, and Socio-Sanitary Caregivers. He has directed 10 doctoral theses and is member of editorial committees of international research journals. Research line: moral psychology (moral reasoning, moral emotions, prosocial behaviors, and university social responsibility).

Laura Badenes-Ribera , PhD in Psychology. Currently, she serves as a Professor in the Department of Methodology of the Sciences of Behavior at the University of Valencia and is the Head of the M.Sc. degree program in Health and Social Attention to Dependency at the same university. She holds degrees in Law, Criminology, and Psychology. Additionally, she has obtained a Master's degree in Advances in Investigation and Treatments in Psychopathology and Health, Sanitary General Psychology, and Criminology and Security. She is a member of the research group ARMAQoL dedicated to the investigation and application of advanced methodologies to enhance the understanding of quality of life. Her research areas encompass statistical meta-analysis, gaming disorder, sexual prejudice, violence, and quality of life. She has made significant contributions to the field with multiple publications in esteemed journals, including trauma, violence and abuse, sexuality research and social policy, and psychology of violence, among others.