Bibliometrically and systematically analyzing automated writing evaluation for English learning
Abstract
Automated writing evaluation is highly discussed in artificial intelligence for English learning. It is necessary to explore the effect of automated writing evaluation on learning English as a second language. This study combined bibliometric analysis and systematic review to explore the use of automated writing evaluation for learning English as a second language. VOSviewer was used to identify the highly discussed topics, the top ten cited authors, organizations, countries, references, and sources in the studies on automated writing evaluation. Fifty-six peer-reviewed articles were selected according to the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols. The analysis revealed that automated writing evaluation is helpful, but its effectiveness varies according to the types of feedback, and it cannot replace the role of human feedback yet. Teachers’ roles are significant in integrating automated writing evaluation into the classroom. Future research could focus on the specific ways to integrate automated writing evaluation into the classroom better.
References
[1] Allen LK, Likens AD, McNamara DS (2019). Writing flexibility in argumentative essays: A multidimensional analysis. Reading and Writing: An Interdisciplinary Journal 32(6): 1607–1634. doi: 10.1007/s11145-018-9921-y
[2] Bai L, Hu G (2017). In the face of fallible AWE feedback: How do students respond? Educational Psychology 37(1): 67–81. doi: 10.1080/01443410.2016.1223275
[3] Bridgeman B, Ramineni C (2017). Design and evaluation of automated writing evaluation models: Relationships with writing in naturalistic settings. Assessing Writing 34: 62–71. doi: 10.1016/j.asw.2017.10.001
[4] Butterfuss R, Roscoe RD, Allen LK, et al. (2022). Strategy uptake in writing pal: Adaptive feedback and instruction. Journal of Educational Computing Research 60(3): 696–721. doi: 10.1177/07356331211045304
[5] Chen CFE, Cheng WYE (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology 12(2): 94–112.
[6] Chen M, Cui Y (2022). The effects of AWE and peer feedback on cohesion and coherence in continuation writing. Journal of Second Language Writing 57: 100915. doi: 10.1016/j.jslw.2022.100915
[7] Chen Z, Chen W, Jia J, Le H (2022). Exploring AWE-supported writing process: An activity theory perspective. Language Learning & Technology 26(2): 129–148. doi: 10125/73482
[8] Diklil S, Bleyle S (2014). Automated Essay Scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing 22: 1–17. doi: 10.1016/j.asw.2014.03.006
[9] Du Y, Gao H (2022). Determinants affecting teachers’ adoption of AI-based applications in EFL context: An analysis of analytic hierarchy process. Education and Information Technologies 27(7): 9357–9384. doi: 10.1007/s10639-022-11001-y
[10] Fan N (2023). Exploring the effects of automated written corrective feedback on EFL students’ writing quality: A mixed-methods study. Sage Open 13(2). doi: 10.1177/21582440231181296
[11] Feng HH, Chukharev-Hudilainen E (2022). Genre-based AWE system for engineering graduate writing: Development and evaluation. Language Learning & Technology 26(2): 58–77. doi: 10125/73479
[12] Foster S (2019). What barriers do students perceive to engagement with automated immediate formative feedback. Journal of Interactive Media in Education 2019(1): 15. doi: 10.5334/jime.516
[13] Fu QK, Zou D, Xie H, Cheng G (2022). A review of AWE feedback: Types, learning outcomes, and implications. Computer Assisted Language Learning. doi: 10.1080/09588221.2022.2033787
[14] Gao J (2021). Exploring the feedback quality of an automated writing evaluation system Pigai. International Journal of Emerging Technologies in Learning (iJET) 16(11): 322–330. doi: 10.3991/ijet.v16i11.19657
[15] Godwin-Jones R (2022). Partnering with AI: Intelligent writing assistance and instructed language learning. Language Learning & Technology 26(2): 5–24. doi: 10125/73474
[16] Grimes D, Warschauer M (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning, and Assessment 8(6).
[17] Guo Q, Feng R, Hua Y (2021). How effectively can EFL students use automated written corrective feedback (AWCF) in research writing? Computer Assisted Language Learning 35(9): 2312–2331. doi: 10.1080/09588221.2021.1879161
[18] Han T, Sari E (2022). An investigation on the use of automated feedback in Turkish EFL students’ writing classes. Computer Assisted Language Learning. doi: 10.1080/09588221.2022.2067179
[19] Han Y, Zhao S, Ng LL (2021). How technology tools impact writing performance, lexical complexity, and perceived self-regulated learning strategies in EFL academic writing: A comparative study. Frontiers in Psychology 12: 752793. doi: 10.3389/fpsyg.2021.752793
[20] Huang X, Zou D, Cheng G, et al. (2023). Trends, research issues and applications of artificial intelligence in language education. Educational Technology & Society 26(1): 112–131. doi: 10.30191/ETS.202301_26(1).0009
[21] Jiang L, Yu S, Wang C (2020). Second language writing instructors’ feedback practice in response to automated writing evaluation: A sociocultural perspective. System 93: 102302. doi: 10.1016/j.system.2020.102302
[22] Koltovskaia S (2020). Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing 44: 100450. doi: 10.1016/j.asw.2020.100450
[23] Lai YH (2010). Which do students prefer to evaluate their essays: Peers or computer program. British Journal of Educational Technology 41(3): 432–454. doi: 10.1111/j.1467-8535.2009.00959.x
[24] Lang F, Li S, Zhang S (2019). Research on reliability and validity of mobile networks-based automated writing evaluation. International Journal of Mobile Computing and Multimedia Communications 10(1): 18–31. doi: 10.4018/IJMCMC.2019010102
[25] Liao HC (2016). Enhancing the grammatical accuracy of EFL writing by using an AWE-assisted process approach. System 62: 77–92. doi: 10.1016/j.system.2016.02.007
[26] Li J (2022). English writing feedback based on online automatic evaluation in the era of big data. Mobile Information Systems 2022: 9884273. doi: 10.1155/2022/9884273
[27] Li J, Link S, Hegelheimer V (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing 27: 1–18. doi: 10.1016/j.jslw.2014.10.004
[28] Lim K, Song J, Park J (2022). Neural automated writing evaluation for Korean L2 writing. Natural Language Engineering 29(5): 1341–1363. doi: 10.1017/S1351324922000298
[29] Link S, Mehrzad M, Rahimi M (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning 35(4): 605–634. doi: 10.1080/09588221.2020.1743323
[30] Li R (2021). Modeling the continuance intention to use automated writing evaluation among Chinese EFL learners. Sage Open 11(4). doi: 10.1177/21582440211060782
[31] Liu S, Yu G (2022). L2 learners’ engagement with automated feedback: An eye-tracking study. Language Learning & Technology 26(2): 78–105. doi: 10125/73480
[32] Li Z, Feng HH, Saricaoglu A (2017). The short-term and long-term effects of AWE feedback on ESL students’ development of grammatical accuracy. Calico Journal 34(3): 355–375. doi: 10.1558/cj.26382
[33] Li Z (2021). Teachers in automated writing evaluation (AWE) system-supported ESL writing classes: Perception, implementation, and influence. System 99: 102505. doi: 10.1016/j.system.2021.102505
[34] Li Z, Link S, Ma H, et al. (2014). The role of automated writing evaluation holistic scores in the ESL classroom. System 44: 66–78. doi: 10.1016/j.system.2014.02.007
[35] Lu X (2019). An empirical study on the artificial intelligence writing evaluation system in China CET. Big Data 7(2): 121–129. doi: 10.1089/big.2018.0151
[36] Mehrabi-Yazdi O (2018). Short communication on the missing dialogic aspect of an automated writing evaluation system in written feedback research. Journal of Second Language Writing 41: 92–97. doi: 10.1016/j.jslw.2018.05.004
[37] Miranty D, Widiati U (2021). An automated writing evaluation (AWE) in higher education: Indonesian EFL students’ perceptions about Grammarly use across student cohorts. Pegem Journal of Education and Instruction 11(4): 126–137. doi: 10.47750/pegegog.11.04.12
[38] Mohsen MA (2022). Computer-mediated corrective feedback to improve L2 writing skills: A meta-analysis. Journal of Educational Computing Research 60(5): 1253–1276. doi: 10.1177/07356331211064066
[39] Nazari N, Shabbir MS, Setiawan R (2021). Application of Artificial Intelligence powered digital writing assistant in higher education: Randomized controlled trial. Heliyon 7(5): E07014. doi: 10.1016/j.heliyon.2021.e07014
[40] Ngo TTN, Chen HHJ, Lai KKW (2022). The effectiveness of automated writing evaluation in EFL/ESL writing: A three-level meta-analysis. Interactive Learning Environments. doi: 10.1080/10494820.2022.2096642
[41] Page MJ, McKenzie JE, Bossuyt PM, et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 372: n71. doi: 10.1136/bmj.n71
[42] Palermo C, Thomson MM (2018). Teacher implementation of Self-Regulated Strategy Development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology 54: 255–270. doi: 10.1016/j.cedpsych.2018.07.002
[43] Palermo C, Wilson J (2020). Implementing automated writing evaluation in different instructional contexts: A mixed-methods study. Journal of Writing Research 12(1): 63–108. doi: 10.17239/jowr-2020.12.01.04
[44] Parra GL, Calero SX (2019). Automated writing evaluation tools in the improvement of the writing skill. International Journal of Instruction 12(2): 209–226. doi: 10.29333/iji.2019.12214a
[45] Petchprasert A (2021). Utilizing an automated tool analysis to evaluate EFL students’ writing performances. Asian-Pacific Journal of Second and Foreign Language Education 6: 1. doi: 10.1186/s40862-020-00107-w
[46] Potter A, Wilson J (2021). Statewide implementation of automated writing evaluation: Analyzing usage and associations with state test performance in grades 4–11. Educational Technology Research and Development 69(3): 1557–1578. doi: 10.1007/s11423-021-10004-9
[47] Qian L, Yang Y, Zhao Y (2021). Syntactic complexity revisited: Sensitivity of China’s AES-generated scores to syntactic measures, effects of discourse-mode and topic. Reading and Writing 34: 681–704. doi: 10.1007/s11145-020-10087-5
[48] Ranalli J (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning 31(7): 653–674. doi: 10.1080/09588221.2018.1428994
[49] Ranalli J (2021). L2 student engagement with automated feedback on writing: Potential for learning and issues of trust. Journal of Second Language Writing 52: 100816. doi: 10.1016/j.jslw.2021.100816
[50] Reynolds BL, Kao CW, Huang Y (2021). Investigating the effects of perceived feedback source on second language writing performance: A quasi-experimental study. Asia-Pacific Education Researcher 30(6): 585–595. doi: 10.1007/s40299-021-00597-3
[51] Roscoe RD, Wilson J, Johnson AC, Mayra CR (2017). Presentation, expectations, and experience: Sources of student perceptions of automated writing evaluation. Computers in Human Behavior 70: 207–221. doi: 10.1016/j.chb.2016.12.076
[52] Sari E, Han T (2022). Using generalizability theory to investigate the variability and reliability of EFL composition scores by human raters and e-rater. Porta Linguarum: An International Journal of Foreign Language and Learning 38: 27–45. doi: 10.30827/portalin.vi38.18056
[53] Saricaoglu A, Bilki Z (2021). Voluntary use of automated writing evaluation by content course students. ReCALL 33(3): 265–277. doi: 10.1017/S0958344021000021
[54] Shamseer L, Moher D, Clarke M, et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. British Medical Journal 349: g7647. doi: 10.1136/bmj.g7647
[55] Shi H, Vahid A (2022). A systematic review of automated writing evaluation systems. Education and Information Technologies 28: 771–795. doi: 10.1007/s10639-022-11200-7
[56] Stevenson M, Phakiti A (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing 19: 51–65. doi: 10.1016/j.asw.2013.11.007
[57] van Eck NJ, Waltman L (2017). Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics 111(2): 1053–1070. doi: 10.1007/s11192-017-2300-7
[58] Wang Q (2022). The use of semantic similarity tools in automated content scoring of fact-based essays written by EFL learners. Education and Information Technologies 27(9): 13021–13049. doi: 10.1007/s10639-022-11179-1
[59] Wang YJ, Shang HF, Briody P (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning 26(3): 234–257. doi: 10.1080/09588221.2012.655300
[60] Warschauer M, Grimes D (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal 3(1): 22–36. doi: 10.1080/15544800701771580
[61] Warschauer M, Ware P (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research 10(2): 157–180. doi: 10.1191/1362168806lr190oa
[62] Wilken JL (2018). Perceptions of L1 glossed feedback in automated writing evaluation: A case study. Calico Journal 35(1): 30–48. doi: 10.1558/cj.26383
[63] Wilson J, Ahrendt C, Fudge EA, et al. (2021). Elementary teachers’ perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education 168: 104208. doi: 10.1016/j.compedu.2021.104208
[64] Wilson J, Czik A (2016). Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education 100: 94–109. doi: 10.1016/j.compedu.2016.05.004
[65] Woodworth J, Barkaoui K (2020). Perspectives on using automated writing evaluation systems to provide written corrective feedback in the ESL classroom. TESL Canada Journal 37(2): 234–247. doi: 10.18806/tesl.v37i2.1340
[66] Xu J, Zhang S (2022). Understanding AWE feedback and English writing of learners with different proficiency levels in an EFL classroom: A sociocultural perspective. The Asia-Pacific Education Researcher 31(4): 357–367. doi: 10.1007/s40299-021-00577-7
[67] Yu Z (2020). Extending the learning technology acceptance model of WeChat by adding new psychological constructs. Journal of Educational Computing Research 58(6): 1121–1143. doi: 10.1177/0735633120923772
[68] Yu Z (2022). A meta-analysis and bibliographic review of the effect of nine factors on online learning outcomes across the world. Education and Information Technologies 27(2): 2457–2482. doi: 10.1007/s10639-021-10720-y
[69] Yu Z, Zhu Y, Yang Z, Chen W (2019). Student satisfaction, learning outcomes, and cognitive loads with a mobile learning platform. Computer Assisted Language Learning 32(4): 323–341. doi: 10.1080/09588221.2018.1517093
[70] Zhai N, Ma X (2023). The effectiveness of automated writing evaluation on writing quality: A meta-analysis. Journal of Educational Computing Research 61(4): 875–900. doi: 10.1177/07356331221127300
[71] Zhang J, Zhang LJ (2022). The effect of feedback on metacognitive strategy use in EFL writing. Computer Assisted Language Learning. doi: 10.1080/09588221.2022.2069822
[72] Zhang Z (2017). Student engagement with computer-generated feedback: A case study. ELT Journal 71(3): 317–328. doi: 10.1093/elt/ccw089
[73] Zhang Z (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing 43: 100439. doi: 10.1016/j.asw.2019.100439
[74] Zhang Z, Hyland K (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing 36: 90–102. doi: 10.1016/j.asw.2018.02.004
Copyright (c) 2023 author(s)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The author(s) warrant that permission to publish the article has not been previously assigned elsewhere.
Author(s) shall retain the copyright of their work and grant the Journal/Publisher right for the first publication with the work simultaneously licensed under:
OA - Creative Commons Attribution License (CC BY 4.0). This license allows for the copying, distribution and transmission of the work, provided the correct attribution of the original creator is stated. Adaptation and remixing are also permitted.
This broad license intends to facilitate free access to, as well as the unrestricted reuse of, original works of all types.
_%E5%89%AF%E6%9C%AC1.png){kind=logo}
