Inter-transcriber Reliability
The inter-transcriber reliability report provides several metrics of agreement between blind transcribers in a Phon session.
Usage
To open the inter-transcriber reliability wizard select Session Editor window menu. The menu item will be disabled if the editor is in blind mode or there are less than two transcribers for the session.
from theThe first step of the wizard allows selection of domain for comparison (group or word), the transcribers for comparison, and which metrics will be included. Standard options for ignoring diacritics are also available. After clicking Next a HTML report is produced as described below and displayed as the last step of the wizard. Options to save the report and export report tables will be available from the toolbar and context menu.
Report Outline
- Levenshtein Distance
Levenshtein Distance calculates the number of typed-edits required to turn t1 into t2 (and vice versa)
- Phone Agreement
Phone agreement is calculated as the average of percent phones correct (PPC) between t1 and t2:
- Consonant Agreement (optional, default:
yes)
Consonant agreement is caculated as the average of percent consonants correct (PCC) between t1 and t2:
- Vowel Agreement (optional, default: yes)
Vowel agreement is calculated as is calculated as the average of percent vowels correct (PVC) between t1 and t2:
- Diacritic Agreement (optional, default:
no)
Diacritic agreement is caculated as the average of percent diacritics correct (PDC) between t1 and t2:
- Number Tone Agreement (optional, default:
no)
Number tone agreement is calculated as the averager of percent number tones correct (PNTC) between t1 and t2:
The final row of each table will include an average of each selected metric. A summary table continaing the last row of each transcriber table will also be added to the beginning of the report.