AI chatbot scores significantly lower than human students in accounting exams
In a surprising turn of events, ChatGPT, an AI chatbot, has failed to outperform human students in accounting examinations. Researchers from Brigham Young University (BYU), US, and 186 other universities published their findings in the journal Issues in Accounting Education, revealing the AI chatbot’s score of 47.4 percent, compared to the students’ overall average of 76.7 percent.
The Performance Gap
The study found that while ChatGPT scored higher than the student average in 11.3 percent of the questions, doing particularly well on accounting information systems (AIS) and auditing, it struggled with tax, financial, and managerial assessments. One possible reason for the poor performance, according to the researchers, could be the mathematical processes required to solve such questions.
Question Type Analysis
According to the findings, ChatGPT demonstrated greater accuracy rates on true/false questions, scoring 68.7% correctly, and on multiple-choice questions, with 59.5% accuracy. However, its accuracy rate ranged between 28.7% and 39.1% when it came to short-answer questions, indicating some difficulties in this area.
The researchers also observed some interesting trends, such as the AI chatbot’s struggle to understand mathematical questions, leading to illogical errors like adding two numbers in a subtraction problem or dividing numbers incorrectly. Furthermore, ChatGPT was seen providing explanations for its incorrect answers and sometimes even fabricating facts.
Despite these shortcomings, the researchers called ChatGPT’s performance “impressive” and a “game changer that will change the way everyone teaches and learns – for the better.
Implications for Education
The study’s co-author and fellow BYU accounting professor Melissa Larson said, “It’s an opportunity to reflect on whether we are teaching value-added information or not. This is a disruption, and we need to assess where we go from here. Of course, I’m still going to have TAs, but this is going to force us to use them in different ways.”
This revelation comes shortly after ChatGPT was reported to have flunked the Joint Entrance Examination (JEE) Advanced, a highly competitive engineering entrance exam in India. The AI program reportedly failed miserably after solving just 11 questions in the exam.
AI Limitations and Future Potential
The performance of ChatGPT in these examinations highlights the limitations of AI systems, particularly in areas requiring mathematical reasoning and comprehension. However, it also offers a chance to reflect on how these technologies can be improved and adapted to better serve educational purposes.
In the future, AI chatbots like ChatGPT could potentially play a significant role in revolutionizing education and learning methodologies. As AI continues to advance, researchers and educators will need to collaborate to ensure that these tools are used effectively and ethically to augment human learning and understanding.
While ChatGPT’s performance in accounting exams may have fallen short of human students, it serves as a valuable reminder of the current limitations of AI technology and the potential for future improvement.