Inter-transcriber Reliability

The inter-transcriber reliability report provides several metrics of agreement between blind transcribers in a Phon session.

Usage

To open the inter-transcriber reliability wizard select Session > Inter-transcriber Reliability from the Session Editor window menu. The menu item will be disabled if the editor is in blind mode or there are less than two transcribers for the session.

The first step of the wizard allows selection of domain for comparison (group or word), the transcribers for comparison, and which metrics will be included. Standard options for ignoring diacritics are also available. After clicking Next a HTML report is produced as described below and displayed as the last step of the wizard. Options to save the report and export report tables will be available from the toolbar and context menu.

Report Outline

A table will be produced for each pair of selected transcribers. For each row of the table, which corresponds to a group/word in the session, the following items will be calculated (t1 is the transcript for transcriber 1, t2 is the transcript for transcriber 2):
  • Levenshtein Distance

    Levenshtein Distance calculates the number of typed-edits required to turn t1 into t2 (and vice versa)

  • Phone Agreement

    Phone agreement is calculated as the average of percent phones correct (PPC) between t1 and t2:

  • Consonant Agreement (optional, default: yes)

    Consonant agreement is caculated as the average of percent consonants correct (PCC) between t1 and t2:

  • Vowel Agreement (optional, default: yes)

    Vowel agreement is calculated as is calculated as the average of percent vowels correct (PVC) between t1 and t2:

  • Diacritic Agreement (optional, default: no)

    Diacritic agreement is caculated as the average of percent diacritics correct (PDC) between t1 and t2:

  • Number Tone Agreement (optional, default: no)

    Number tone agreement is calculated as the averager of percent number tones correct (PNTC) between t1 and t2:

Note: The *Agreement columns are calculated as an average because the percent correct function is not commutative.

The final row of each table will include an average of each selected metric. A summary table continaing the last row of each transcriber table will also be added to the beginning of the report.