Evaluating the interrater agreement and acceptability of a new reference tool for assessing respiratory rate in children under five with cough and/or difficulty breathing

Authors: Ann-Sophie Stratil, Charlotte Ward, Tedila Habte, Alice Maurel, Max Antson, Elina Naydenova, Kevin Baker

A new reference tool which facilitates manual annotation of breaths based on videos was developed to improve diagnosis of suspected cases and appropriate treatment.

Background

Manual assessment of respiratory rate (RR) in children is unreliable, but remains the main method to diagnose pneumonia in low-resource settings. While automated RR counters offer a potential solution, there is currently no gold standard to validate these diagnostic aids. A video-based reference tool is proposed that allows users to annotate breaths and distortions including movement periods, allowing the exclusion of distortions from the computation of RR measures similar to how new diagnostic aids account for distortions automatically. This study evaluated the interrater agreement and acceptability of the new reference tool.

Methods

Annotations were based on previously recorded reference videos of children under five years old with cough and/or difficulty breathing (n = 50). Five randomly selected medical experts from a panel of ten annotated each video. RR measures (breaths per minute, bpm) were computed as the number of annotated certain breaths divided by the length of calm periods after removing annotated distorted periods.

Results

Reviewers showed good interrater agreement on continuous RR {standard error of measurement (SEM) [4.8 (95 percent CI 4.4–5.3)]} and substantial agreement on classification of fast breathing (Fleiss kappa, 0.71). Agreement was lowest in the youngest age group [< two months: SEM 6.2 (5.4–7.4) bpm, k 0.48; 2–11 months: 4.7 (4.0–5.8) bpm, k 0.84; 12–59 months: 2.6 (2.2–3.1) bpm, k 0.8]. Reviewers found the functionalities of the tool helpful in annotating breaths, but remained uncertain about the validity of their annotations.

Conclusions

Before the new tool can be considered a reference standard for RR assessments, interrater agreement in children younger than two months must be improved.

Published in Journal of Tropical Pediatrics