Natural language processing for automatic evaluation of free-text answers — a feasibility study based on the European Diploma in Radiology examination

ConclusionThis study showed the successful design of an NLP-based approach for automatic evaluation of free-text answers in the EDiR examination. Thus, as a future field of application, NLP could work as a decision-support system for reviewers and support the design of examinations being adjusted to the requirements of an automated, NLP-based review process.Clinical relevance statementNatural language processing can be successfully used to automatically evaluate free-text answers, performing better with more structured question-answer formats. Furthermore, this study provides a baseline for further work applying, e.g., more elaborated NLP approaches/large language models.Key points• Free-text answers require manual evaluation, which is time-consuming and potentially error-prone.• We developed a simple NLP-based approach — requiring only minimal effort/modeling — to automatically analyze and mark free-text answers.• Our NLP engine has the potential to support the manual evaluation process.• NLP performance is better on a more structured question-answer format.Graphical Abstract
Source: Insights into Imaging - Category: Radiology Source Type: research