The SymptoChat Clinical Validation Study (Phase 1) was conducted using a structured vignette-based approach to evaluate the accuracy of SymptoChat’s AI diagnostic tool. The study involved 25 licensed medical professionals from the US and Europe, who were presented with a series of clinical vignettes—each detailing hypothetical patient cases with various symptoms.
These vignettes were designed to simulate real-world clinical scenarios, enabling the evaluation of the AI’s diagnostic performance against established medical expertise. The study’s findings indicated that SymptoChat’s AI achieved an accuracy rate of approximately 92% in diagnosing the presented cases.
This validation phase was a critical step in advancing SymptoChat’s development, leading to its registration and approval on the App Store. The testing process, conducted alongside experienced physicians, continues as the SymptoChat team works on developing new functionalities to enhance the app’s capabilities.
1. Pneumonia
Patient: 45-year-old female with persistent cough, fever, and shortness of breath.
Physician Diagnosis:
- Majority Diagnosis: Pneumonia
- Variations included “Bronchitis” and “Pleural Effusion.”
SymptoChat AI Diagnosis: Pneumonia
AI Accuracy Rating by Doctors:
- 5/5: 75% of doctors rated the AI diagnosis as highly accurate.
- 4/5: 13% of doctors rated it accurate with minor suggestions for improvement.
- 3/5: 13% of doctors felt the AI missed certain aspects of the diagnosis or possible alternatives.
2. Ureteral Stones (Kidney Stones)
Patient: 35-year-old male with severe lower back pain and blood in urine.
Physician Diagnosis:
- Majority Diagnosis: Kidney Stones
- Variations included “Renal Colic” and “Urinary Obstruction.”
SymptoChat AI Diagnosis: Ureteral Stones
AI Accuracy Rating by Doctors:
- 5/5: 54% of doctors rated the AI diagnosis as highly accurate.
- 4/5: 38% rated it accurate but noted it could improve on follow-up questioning.
- 3/5: 8% thought the AI missed some differential diagnoses.
3. Cystitis (UTI)
Patient: 28-year-old female with frequent urination and mild burning sensation.
Physician Diagnosis:
- Majority Diagnosis: Cystitis or UTI
- Variations included “Mild Bladder Infection” and “Interstitial Cystitis.”
SymptoChat AI Diagnosis: Cystitis (UTI)
AI Accuracy Rating by Doctors:
- 5/5: 63% of doctors rated the AI as highly accurate.
- 4/5: 29% found the diagnosis generally accurate but felt minor factors were overlooked.
- 3/5: 8% thought the AI missed less common but relevant conditions.