Researchers explore concept of locally run LLMs

In this study, the Vicuna model was used as is, with no additional training or adjustments, they added.The group tested Vicuna using two publicly available and de-identified data sets, one comprised of 3,269 chest x-ray reports (MIMIC-CXR, developed by a group at the Massachusetts Institute of Technology) and the other an NIH data set consisting of 25,596 reports.Using two prompts for two tasks, the researchers asked Vicuna to identify and label the presence or absence of 13 specific findings on the reports, including cardiomegaly, edema, fracture, lung lesion, and pneumonia, for instance. The researchers then compared the LLM’s performance with two widely used natural language processing labelers (non-LLMs), namely CheXpert and CheXbert.According to the findings, a statistical analysis showed that Vicuna’s output achieved moderate to substantial agreement with the labelers on the MIMIC-CXR data set (κ median, 0.57 ) and NIH data set (κ median, 0.52). In addition, Vicuna performed at par (a median area under the curve [AUC] of 0.84) with both labelers on nine of 11 findings.“Our study demonstrated that the LLM’s performance was comparable to the current reference standard. With the right prompt and the right task, we were able to achieve agreement with currently used labeling tools,” Summers said.Ultimately, Summers noted that LLMs like Vicuna could be run locally to extract features from the text of radiology x-ray reports and combine them with features from imag...

Source: AuntMinnie.com Headlines - October 10, 2023 Category: Radiology Authors: Will Morton Tags: Artificial Intelligence Source Type: news