J Hand Surg Glob Online. 2024 Nov 8;7(1):37-40. doi: 10.1016/j.jhsg.2024.10.002. eCollection 2025 Jan.
ABSTRACT
PURPOSE: Many adults in the United States possess basic or below basic health literacy skills, making it essential for patient education materials (PEMs) to be presented at or below a sixth-grade reading level. We evaluate the readability of PEMs generated by ChatGPT 3.5 and 4.0 for common hand conditions.
METHODS: We used Chat Generative Pre-Trained Transformer (ChatGPT) 3.5 and 4.0 to generate PEMs for 50 common hand pathologies. Two consistent questions were asked to minimize variability: 1. "Please explain [Condition] to a patient at a sixth-grade reading level, including details on anatomy, symptoms, doctors' examination, and treatment (both surgical and nonsurgical)." 2. "Create a detailed patient information sheet for the general patient population at a sixth-grade reading level explaining [Condition], including points such as anatomy, symptoms, physical examination, and treatment (both surgical and nonsurgical)." Before asking the second question, a priming phase was conducted where ChatGPT 3.5 and 4.0 were presented with a text sample written at a sixth-grade reading level and informed that this was the desired output level. Multiple readability tests were used to evaluate the output, with a consensus reading level created from the results of all eight readability scores. Statistical analyses were performed using SAS 9.4.
RESULTS: ChatGPT 4.0 successfully produced 28% of its responses at the appropriate reading level following the priming phase, compared to none by ChatGPT 3.5. ChatGPT 4.0 showed superior performance across all readability metrics.
CONCLUSIONS: ChatGPT 4.0 is a more effective tool than ChatGPT 3.5 for generating PEMs at a sixth-grade reading level for common hand conditions.
CLINICAL RELEVANCE: The results suggest that Artificial Intelligence could significantly enhance patient education and health literacy with further refinement.
PMID:39991597 | PMC:PMC11846566 | DOI:10.1016/j.jhsg.2024.10.002