Cohen's Kappa and Other Interrater Agreement Measures

Loading...

2 Raters (Nominal)

Degree of agreement of the nominal scale by two raters

Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.

Your data needs to have the header (variable names) in the first row. Missing values should be indicated by a period (.) or NA.


                
                

Contingency table


                

Cohen's kappa


                

If the data has more categories than binary (e.g., yes and no) with an ordinal structure (e.g., A > B > C, low < medium < high), consider reporting a weighted Kappa (Weights: squared).

Criteria (Strength of agreement):
< 0.20 Poor
0.21–0.40 Fair
0.41–0.60 Moderate
0.61–0.80 Good (Substantial)
0.81–1.00 Very good (Almost perfect)


Krippendorff's alpha reliability coefficient


                

Agreement plot



R session info

              

2 Raters (Ordinal)

Degree of agreement of the nominal scale by two raters

Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.

Your data needs to have the header (variable names) in the first row. Missing values should be indicated by a period (.) or NA.


                
                

Contingency table


                

Cohen's kappa


                

Criteria (Strength of agreement):
< 0.20 Poor
0.21–0.40 Fair
0.41–0.60 Moderate
0.61–0.80 Good (Substantial)
0.81–1.00 Very good (Almost perfect)


Rank correlation coefficient


                

Krippendorff's alpha reliability coefficient


                

Plot





R session info

              

3 or More Raters (Nominal)

Degree of agreement of the nominal scale by three or more raters

Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.

Your data needs to have the header (variable names) in the first row and the person's IDs in the first column. Missing values should be indicated by a period (.) or NA.


                
                

Categories selected by raters


                

Fleiss' Kappa


                

Criteria (Strength of agreement):
< 0.40 Poor
0.40–0.75 Intermediate to Good
Over 0.75 Excellent


Krippendorff's alpha reliability coefficient


                

Plot





R session info

              

3 or More Raters (Ordinal)

Degree of agreement of the ordinal scale by three or more raters

Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.

Your data needs to have the header (variable names) in the first row and the person's IDs in the first column. Missing values should be indicated by a period (.) or NA.


                
                

Categories selected by raters


                

Fleiss' Kappa


                

Criteria (Strength of agreement):
< 0.40 Poor
0.40–0.75 Intermediate to Good
Over 0.75 Excellent


Kendall's coefficient of concordance for ranks (W)


                

Krippendorff's alpha reliability coefficient


                

Plot





R session info

              
Note

This web application is developed with Shiny.


List of Packages Used
library(shiny)
library(shinyAce)
library(irr)
library(psych)
library(vcd)
library(lattice)
library(reshape2)
library(DescTools)

Code

Source code for this application is based on "The handbook of Research in Foreign Language Learning and Teaching" (Takeuchi & Mizumoto, 2012).

The code for this web application is available at GitHub.

If you want to run this code on your computer (in a local R session), run the code below:
library(shiny)
runGitHub("kappa","mizumot")


Citation in Publications

Mizumoto, A. (2015). Langtest (Version 1.0) [Web application]. Retrieved from http://langtest.jp


Article

Mizumoto, A., & Plonsky, L. (2015). R as a lingua franca: Advantages of using R for quantitative research in applied linguistics. Applied Linguistics, Advance online publication. doi:10.1093/applin/amv025


Recommended

To learn more about R, I suggest this excellent and free e-book (pdf), A Guide to Doing Statistics in Second Language Research Using R, written by Dr. Jenifer Larson-Hall.

Also, if you are a cool Mac user and want to use R with GUI, MacR is defenitely the way to go!


Author

Atsushi MIZUMOTO, Ph.D.
Associate Professor of Applied Linguistics
Faculty of Foreign Language Studies /
Graduate School of Foreign Language Education and Research,
Kansai University, Osaka, Japan



Code for "Cohen's Kappa and Other Interrater Agreement Measures"
by Atsushi Mizumoto

show with app
library(shiny)
library(shinyAce)
library(irr)
library(psych)
library(vcd)
library(lattice)
library(reshape2)
library(DescTools)



shinyServer(function(input, output) {


#----------------------------------------------------
# 1. 2 Raters (Nominal) ------
#----------------------------------------------------

    data1 <- reactive({
        
        dat <- read.csv(text=input$text1, sep="", na.strings=c("","NA","."))
        
          dat[,1] <- as.character(dat[,1])
          dat[,2] <- as.character(dat[,2])
          z <- (c(dat[,1], dat[,2]))
          lvls <- unique(z)
          levels(dat[,1]) <- unique(lvls)
          levels(dat[,2]) <- unique(lvls)
          a <- with(dat, table(dat[,1], dat[,2]))
          names(dimnames(a)) <- colnames(dat)
          x <- a
          x <- addmargins(x)
          print(x)
        
          cat("\n", "----------", "\n") 
         
          # percentage agreement
          pctagree <- agree(dat)
          cat("\n")  
          print(pctagree)  
        
          # percentage
          pct <- round(a/sum(a),2)
          cat("\n")  
          print(pct)
        
    })
    
    output$data1.out <- renderPrint({
          data1()
    })
    
    
    
    
    
    test1 <- reactive({
        
        dat <- read.csv(text=input$text1, sep="", na.strings=c("","NA","."))
        
          res1 <- kappa2(dat)
          print(res1)  
          
          cat("\n")
          cat("95% confidence interval (CI) of kappa", "\n")
        
          dat[,1] <- as.character(dat[,1])
          dat[,2] <- as.character(dat[,2])
          z <- (c(dat[,1], dat[,2]))
          lvls <- unique(z)
          levels(dat[,1]) <- unique(lvls)
          levels(dat[,2]) <- unique(lvls)
          a <- with(dat, table(dat[,1], dat[,2]))
          names(dimnames(a)) <- colnames(dat)
          kappa <- cohen.kappa(a)
          res2 <- kappa[8]$confid[1,]
          print(res2)
        
          cat("\n", "----------", "\n") 
        
          cat("\n")
        
          res3 <- kappa2(dat, "squared")
          print(res3)
        
          cat("\n")
          cat("95% confidence interval (CI) of kappa", "\n")
        
          res4 <- kappa[8]$confid[2,]
          print(res4)
       
    })
    
    output$test1.out <- renderPrint({
        test1()
    })





    Krippendorff1 <- reactive({
  
        dat <- read.csv(text=input$text1, sep="", na.strings=c("","NA","."))
  
        options(warn=-1)
        kripp.alpha(t(dat), method="nominal")
  
    })

    output$Krippendorff1.out <- renderPrint({
        Krippendorff1()
    })





    makepPlot1 <- function(){
        
        dat <- read.csv(text=input$text1, sep="", na.strings=c("","NA","."))
        
          dat[,1] <- as.character(dat[,1])
          dat[,2] <- as.character(dat[,2])
          z <- (c(dat[,1], dat[,2]))
          lvls <- unique(z)
          levels(dat[,1]) <- unique(lvls)
          levels(dat[,2]) <- unique(lvls)
          a <- with(dat, table(dat[,1], dat[,2]))
          names(dimnames(a)) <- colnames(dat)
                
          agreementplot(t(a))
        
    }
    
    output$pPlot1 <- renderPlot({
        print(makepPlot1())
    })
    
    
    
  
  
     info1 <- reactive({
        info1 <- paste("This analysis was conducted with ", strsplit(R.version$version.string, " \\(")[[1]][1], ".", sep = "")
        info2 <- paste("It was executed on ", date(), ".", sep = "")
        cat(sprintf(info1), "\n")
        cat(sprintf(info2), "\n")
    })
    
    output$info1.out <- renderPrint({
        info1()
    })










#----------------------------------------------------
# 2. 2 Raters (Ordinal) ------
#----------------------------------------------------

    data2 <- reactive({
        
        dat <- read.csv(text=input$text2, sep="", na.strings=c("","NA","."))
                
        dat[,1] <- as.character(dat[,1])
        dat[,2] <- as.character(dat[,2])
        z <- (c(dat[,1], dat[,2]))
        lvls <- unique(z)
        levels(dat[,1]) <- unique(lvls)
        levels(dat[,2]) <- unique(lvls)
        a <- with(dat, table(dat[,1], dat[,2]))
        names(dimnames(a)) <- colnames(dat)
        x <- a
        x <- addmargins(x)
        print(x)
        
        cat("\n", "----------", "\n") 
        
        # percentage agreement
        pctagree <- agree(dat)
        cat("\n")  
        print(pctagree)  
        
        # percentage
        pct <- round(a/sum(a),2)
        cat("\n")  
        print(pct)

        })
    
        output$data2.out <- renderPrint({
            data2()
        })
    
    
    
    
    
    test2 <- reactive({
        
        dat <- read.csv(text=input$text2, sep="", na.strings=c("","NA","."))
        
        res1 <- kappa2(dat, "squared")
        print(res1)  
        
        cat("\n")
        cat("95% confidence interval (CI) of kappa", "\n")
        
        dat[,1] <- as.character(dat[,1])
        dat[,2] <- as.character(dat[,2])
        z <- (c(dat[,1], dat[,2]))
        lvls <- unique(z)
        levels(dat[,1]) <- unique(lvls)
        levels(dat[,2]) <- unique(lvls)
        a <- with(dat, table(dat[,1], dat[,2]))
        names(dimnames(a)) <- colnames(dat)
        kappa <- cohen.kappa(a)
        res2 <- kappa[8]$confid[2,]
        print(res2)
            
    })
    
    output$test2.out <- renderPrint({
        test2()
    })
    
    
    
    
    correl2 <- reactive({
  
      dat <- read.csv(text=input$text2, sep="", na.strings=c("","NA","."))
      
        type <- switch(input$method,
                     Spearman = "spearman",
                     Kendall = "kendall")
      
        r <- cor(dat[,1], dat[,2], method = type, use = "complete")
        n <- length(dat[,1])
      
        if (input$method == "Kendall") {
          # [Kendall CI] http://www.stat.umn.edu/geyer/5601/examp/corr.html
          conf.level <- 0.95
          signs <- sign(outer(dat[,1], dat[,1], "-") * outer(dat[,2], dat[,2], "-"))
          tau <- mean(signs[lower.tri(signs)])
          cvec <- apply(signs, 1, sum)
          nn <- length(cvec)
          sigsq <- (2 / (nn * (nn - 1))) *
                    (((2 * (nn - 2)) / (nn * (nn - 1))) * var(cvec)
                    + 1 - tau^2)
          zcrit <- qnorm((1 + conf.level) / 2)
          ci <- tau + c(-1, 1) * zcrit * sqrt(sigsq)
        
        } else {
          ci <- round(r.con(r, n), 3)
          # [Spearman CI] http://www.statsdirect.com/help/default.htm#nonparametric_methods/spearman.htm
        }
      
        pvl <- cor.test(dat[,1], dat[,2], method = type)
      
        if (input$method == "Spearman") {
              cortype <- c("Spearman's ρ =")
        } else {
              cortype <- c("Kendall's τ =")
        }
      
        cat("Correlation between", colnames(dat)[1], "and", colnames(dat)[2], ":", "\n",
            cortype, round(r, 3), "\n",
           "95% CI [lower, upper] =", round(ci, 3), "\n",
           "p-value =", round(pvl$p.value, 3), "\n")
  
    })

    output$correl2.out <- renderPrint({
        correl2()
    })





    Krippendorff2 <- reactive({
  
        dat <- read.csv(text=input$text2, sep="", na.strings=c("","NA","."))
  
        options(warn=-1)
        kripp.alpha(t(dat), method="ordinal")
        
    })

    output$Krippendorff2.out <- renderPrint({
         Krippendorff2()
    })





    makepPlot2 <- function(){
        
        dat <- read.csv(text=input$text2, sep="", na.strings=c("","NA","."))
          
          dat[,1] <- as.character(dat[,1])
          dat[,2] <- as.character(dat[,2])
          z <- (c(dat[,1], dat[,2]))
          lvls <- unique(z)
          levels(dat[,1]) <- unique(lvls)
          levels(dat[,2]) <- unique(lvls)
          a <- with(dat, table(dat[,1], dat[,2]))
          names(dimnames(a)) <- colnames(dat)
        
          agreementplot(t(a))        
    
    }
    
    output$pPlot2 <- renderPlot({
        print(makepPlot2())
    })
    
    
    
    
      
    makemPlot2 <- function(){
        
        dat <- read.csv(text=input$text2, sep="", na.strings=c("","NA","."))
                
        person <- c(1:nrow(dat))
        x <- data.frame(person, dat)
        x[,1] <- factor(x[,1])
        data.long <- melt(x, idvars=x[,1])
        
        xyplot(value~variable, group=person, type="b", data=data.long, 
              main = list(label="Agreement of ratings given by each rater", cex=1.5), 
              xlab=list(label=""), ylab=list(label="Rating", cex=1.5), cex=1.5,
              scales = list(y=list(at=c(0:round(max(dat, na.rm=TRUE)+1,0)), cex=1.5), x=list(cex=1.5))
        )
        
        
    }
    
    output$mPlot2 <- renderPlot({
        print(makemPlot2())
    })





    info2 <- reactive({
        info1 <- paste("This analysis was conducted with ", strsplit(R.version$version.string, " \\(")[[1]][1], ".", sep = "")
        info2 <- paste("It was executed on ", date(), ".", sep = "")
        cat(sprintf(info1), "\n")
        cat(sprintf(info2), "\n")
    })
    
    output$info2.out <- renderPrint({
        info2()
    })










#----------------------------------------------------
# 3. 3 or More Raters (Nominal)
#----------------------------------------------------

    data3 <- reactive({
                
        dat <- read.csv(text=input$text3, sep="", na.strings=c("","NA","."))
        
        freq.c <- function(x) {
          uniq <- c()
          for (i in 1:length(x)) {
            x[,i] <- as.character(x[,i]) 
            fr <- sort(unique(x[,i]))
            uniq <- c(uniq, fr)
          }
          lbls <- sort(unique(uniq))
        }
        ctg <- freq.c(dat)
        
        l <- apply(dat, 2, table)
        
        if (is.matrix(l)) { # if 0 observation DOES NOT exist
          
          x <- l 
          x <- addmargins(x)
          print(x)
        
        } else { # if 0 observation exists
          
          x2 <- lapply(l, FUN=function(X) X[ctg])
          mat <- do.call("cbind", x2)
          row.names(mat) <- ctg
          mat[is.na(mat)] <- 0
          x <- addmargins(mat)
          print(x)
        
        }
        
    })
      
    output$data3.out <- renderPrint({
        data3()
    })
      
      
      
      
      
    test3 <- reactive({
        
        dat <- read.csv(text=input$text3, sep="", na.strings=c("","NA","."))
        
        res1 <- kappam.fleiss(dat)
        print(res1)
        
        cat("\n")
        cat("95% confidence interval (CI) of kappa", "\n")
        
        res2 <- KappaM(dat, method = "Fleiss", conf.level=0.95)
        print(res2)
        
    })
      
    output$test3.out <- renderPrint({
        test3()
    })
      
      
      
    

    Krippendorff3 <- reactive({
        
        dat <- read.csv(text=input$text3, sep="", na.strings=c("","NA","."))
        
        options(warn=-1)
        kripp.alpha(t(dat), method="nominal")
        
    })
      
    output$Krippendorff3.out <- renderPrint({
        Krippendorff3()
    })
      
      
      
      
      
    makepPlot3 <- function(){
        
        dat <- read.csv(text=input$text3, sep="", na.strings=c("","NA","."))  
        
        dat <- as.data.frame(lapply(dat,as.numeric))
        person <- c(1:nrow(dat))
        dat <- data.frame(person, dat)
        dat[,1] <- factor(dat[,1])
        data.long <- melt(dat, idvars=dat[,1])
        
        xyplot(value~variable, group=person, type="b", data=data.long, 
               main = list(label="Agreement of ratings given by each rater", cex=1.5), 
               xlab=list(label=""), ylab=list(label="Rating", cex=1.5), cex=1.5,
               scales = list(y=list(at=c(0:round(max(data.long$value, na.rm=TRUE)+1,0)), cex=1.5), x=list(cex=1.5))
              )
    }

    output$pPlot3 <- renderPlot({
        print(makepPlot3())
    })
      
      
      
      
      
    makemPlot3 <- function(){
  
        dat <- read.csv(text=input$text3, sep="", na.strings=c("","NA","."))
  
        freq.c <- function(x) {
          uniq <- c()
          for (i in 1:length(x)) {
             x[,i] <- as.character(x[,i]) 
             fr <- sort(unique(x[,i]))
             uniq <- c(uniq, fr)
          }
            lbls <- sort(unique(uniq))
        }
        ctg <- freq.c(dat)
  
        l <- apply(dat, 2, table)
  
        if (is.matrix(l)) {
            mat <- l
        } else {
            x2 <- lapply(l, FUN=function(X) X[ctg])
            mat <- do.call("cbind", x2)
            row.names(mat) <- ctg
            mat[is.na(mat)] <- 0.1 # change 0 observatios into 0.1 to show in a graph
        }
          
        par(mar=c(3, 5, 2, 8), xpd=TRUE)
        barplot(t(mat), beside=T, yaxt="n", ylab="Frequency", col=c(1:ncol(mat)))
        my.at <- round(c(0:max(mat),0))
        axis(2, at = my.at, las=1, labels = my.at)
        legend("topright", inset=c(-0.2, 0), legend = colnames(mat), fill=c(1:ncol(mat))) # Add legend to top right, outside plot region
        
    }

    output$mPlot3 <- renderPlot({
        print(makemPlot3())
    })





    info3 <- reactive({
        info1 <- paste("This analysis was conducted with ", strsplit(R.version$version.string, " \\(")[[1]][1], ".", sep = "")
        info2 <- paste("It was executed on ", date(), ".", sep = "")
        cat(sprintf(info1), "\n")
        cat(sprintf(info2), "\n")
    })
    
    output$info3.out <- renderPrint({
        info3()
    })










#----------------------------------------------------
# 4. 3 or More Raters (Ordinal)
#----------------------------------------------------

    data4 <- reactive({
  
       dat <- read.csv(text=input$text4, sep="", na.strings=c("","NA","."))
       
      freq.c <- function(x) {
        uniq <- c()
         for (i in 1:length(x)) {
            x[,i] <- as.character(x[,i]) 
            fr <- sort(unique(x[,i]))
            uniq <- c(uniq, fr)
          }
        lbls <- sort(unique(uniq))
      }
      ctg <- freq.c(dat)
  
      l <- apply(dat, 2, table)
  
      if (is.matrix(l)) { # if 0 observation DOES NOT exist
    
      x <- l 
      x <- addmargins(x)
      print(x)
    
      } else { # if 0 observation exists
    
      x2 <- lapply(l, FUN=function(X) X[ctg])
      mat <- do.call("cbind", x2)
      row.names(mat) <- ctg
      mat[is.na(mat)] <- 0
      x <- addmargins(mat)
      print(x)
    
      }
  
    })

    output$data4.out <- renderPrint({
      data4()
    })





  test4 <- reactive({
  
    dat <- read.csv(text=input$text4, sep="", na.strings=c("","NA","."))
  
    res1 <- kappam.fleiss(dat)
    print(res1)
  
    cat("\n")
    cat("95% confidence interval (CI) of kappa", "\n")
  
    res2 <- KappaM(dat, method = "Fleiss", conf.level=0.95)
    print(res2)
  
  })

  output$test4.out <- renderPrint({
    test4()
  })





  W <- reactive({
    
    dat <- read.csv(text=input$text4, sep="", na.strings=c("","NA","."))
    
    kendall(dat, TRUE)
    
  })
  
  output$W.out <- renderPrint({
    W()
  })





  Krippendorff4 <- reactive({
  
    dat <- read.csv(text=input$text4, sep="", na.strings=c("","NA","."))
  
      options(warn=-1)
      kripp.alpha(t(dat), "ordinal")
  
  })

  output$Krippendorff4.out <- renderPrint({
      Krippendorff4()
  })





  makepPlot4 <- function(){
  
  dat <- read.csv(text=input$text4, sep="", na.strings=c("","NA","."))  
  
    dat <- as.data.frame(lapply(dat,as.numeric))
    person <- c(1:nrow(dat))
    dat <- data.frame(person, dat)
    dat[,1] <- factor(dat[,1])
    data.long <- melt(dat, idvars=dat[,1])
  
    xyplot(value~variable, group=person, type="b", data=data.long, 
          main = list(label="Agreement of ratings given by each rater", cex=1.5), 
          xlab=list(label=""), ylab=list(label="Rating", cex=1.5), cex=1.5,
          scales = list(y=list(at=c(0:round(max(data.long$value, na.rm=TRUE)+1,0)), cex=1.5), x=list(cex=1.5))
    )
  }

  output$pPlot4 <- renderPlot({
      print(makepPlot4())
  })





  makemPlot4 <- function(){
  
  dat <- read.csv(text=input$text4, sep="", na.strings=c("","NA","."))
  
  freq.c <- function(x) {
    uniq <- c()
    for (i in 1:length(x)) {
      x[,i] <- as.character(x[,i]) 
      fr <- sort(unique(x[,i]))
      uniq <- c(uniq, fr)
    }
    lbls <- sort(unique(uniq))
  }
  ctg <- freq.c(dat)
  
  l <- apply(dat, 2, table)
  
  if (is.matrix(l)) {
    mat <- l
  } else {
    x2 <- lapply(l, FUN=function(X) X[ctg])
    mat <- do.call("cbind", x2)
    row.names(mat) <- ctg
    mat[is.na(mat)] <- 0.1 # change 0 observatios into 0.1 to show in a graph
  }
    
  par(mar=c(3, 5, 2, 8), xpd=TRUE)
  barplot(t(mat), beside=T, yaxt="n", ylab="Frequency", col=c(1:ncol(mat)))
  my.at <- round(c(0:max(mat),0))
  axis(2, at = my.at, las=1, labels = my.at)
  legend("topright", inset=c(-0.2, 0), legend = colnames(mat), fill=c(1:ncol(mat))) # Add legend to top right, outside plot region
  
}

output$mPlot4 <- renderPlot({
  print(makemPlot4())
})





  info4 <- reactive({
    info1 <- paste("This analysis was conducted with ", strsplit(R.version$version.string, " \\(")[[1]][1], ".", sep = "")
    info2 <- paste("It was executed on ", date(), ".", sep = "")
    cat(sprintf(info1), "\n")
    cat(sprintf(info2), "\n")
  })

  output$info4.out <- renderPrint({
    info4()
  })






})
library(shiny)
library(shinyAce)


shinyUI(bootstrapPage(


    headerPanel("Cohen's Kappa and Other Interrater Agreement Measures"),



########## Adding loading message #########

tags$head(tags$style(type="text/css", "
#loadmessage {
position: fixed;
top: 0px;
left: 0px;
width: 100%;
padding: 10px 0px 10px 0px;
text-align: center;
font-weight: bold;
font-size: 100%;
color: #000000;
background-color: #CCFF66;
z-index: 105;
}
")),

conditionalPanel(condition="$('html').hasClass('shiny-busy')",
tags$div("Loading...",id="loadmessage")),

########## Added up untill here ##########



    mainPanel(
        tabsetPanel(position = "left", selected = "2 Raters (Nominal)",

        tabPanel("2 Raters (Nominal)",

            h2("2 Raters (Nominal)"),

            h4("Degree of agreement of the nominal scale by two raters"),

            p('Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.'),

            p(HTML("<b><div style='background-color:#FADDF2;border:1px solid black;'>Your data needs to have the header (variable names) in the first row. Missing values should be indicated by a period (.) or NA.</div></b>")),

            aceEditor("text1", value="Rater1\tRater2\nB\tB\nA\tA\nC\tC\nB\tB\nB\tC\nC\tC\nB\tC\nA\tA\nC\tC\nC\tC\nC\tC\nC\tC\nB\tB\nA\tA\nB\tB\nC\tC\nA\tB\nC\tC\nA\tA\nB\tB", mode="r", theme="cobalt"),

            br(),

            h3("Contingency table"),
            verbatimTextOutput("data1.out"),

            br(),

            h3("Cohen's kappa"),
            verbatimTextOutput("test1.out"),
            
            p('If the data has more categories than binary (e.g., yes and no) with an ordinal structure (e.g., A > B > C, low < medium < high), consider reporting a weighted Kappa (Weights: squared).', br()),
            
            p('Criteria (Strength of agreement):', br(),
              '   < 0.20  Poor', br(),
              '0.21–0.40  Fair', br(),
              '0.41–0.60  Moderate', br(),
              '0.61–0.80  Good (Substantial)', br(),
              '0.81–1.00  Very good (Almost perfect)', br()
            ),
            
            br(),
            
            h3("Krippendorff's alpha reliability coefficient"),
            verbatimTextOutput("Krippendorff1.out"),
            
            br(),

            h3("Agreement plot"),

            plotOutput("pPlot1", height = "550px"),
                    
            br(),
            br(),

            strong('R session info'),
            verbatimTextOutput("info1.out")
            ),









        tabPanel("2 Raters (Ordinal)",

            h2("2 Raters (Ordinal)"),

            h4("Degree of agreement of the nominal scale by two raters"),

            p('Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.'),

            p(HTML("<b><div style='background-color:#FADDF2;border:1px solid black;'>Your data needs to have the header (variable names) in the first row. Missing values should be indicated by a period (.) or NA.</div></b>")),

            aceEditor("text2", value="Rater1\tRater2\n1\t1\n1\t1\n1\t1\n1\t1\n1\t1\n2\t2\n2\t2\n2\t2\n2\t2\n2\t2\n3\t3\n3\t3\n3\t3\n3\t3\n3\t3\n1\t2\n1\t3\n1\t3\n1\t2\n1\t2\n2\t1\n2\t3\n2\t3\n2\t1\n2\t1\n3\t1\n3\t2\n3\t1\n3\t2\n3\t2", mode="r", theme="cobalt"),

            br(),

            h3("Contingency table"),
            verbatimTextOutput("data2.out"),
            
            br(),

            h3("Cohen's kappa"),
            verbatimTextOutput("test2.out"),
            
            p('Criteria (Strength of agreement):', br(),
              '   < 0.20  Poor', br(),
              '0.21–0.40  Fair', br(),
              '0.41–0.60  Moderate', br(),
              '0.61–0.80  Good (Substantial)', br(),
              '0.81–1.00  Very good (Almost perfect)', br()
            ),
            
            br(),
            
            h3("Rank correlation coefficient"),
            
            radioButtons("method", "Type of correlation coefficient:",
                         list("Spearman's rank correlation coefficient (Spearman's rho)" = "Spearman",
                              "Kendall's tau rank correlation coefficient (Kendall's tau)" = "Kendall")),
            
            verbatimTextOutput("correl2.out"),    
       
            br(),
            
            h3("Krippendorff's alpha reliability coefficient"),
            verbatimTextOutput("Krippendorff2.out"),
            
            br(),
            
            h3("Plot"),
            plotOutput("pPlot2", height = "550px"),

            br(),
            br(),

            plotOutput("mPlot2"),

            br(),
            br(),

            strong('R session info'),
            verbatimTextOutput("info2.out")
            ),










        tabPanel("3 or More Raters (Nominal)",

            h2("3 or More Raters (Nominal)"), 
                 
            h4("Degree of agreement of the nominal scale by three or more raters"),                 
                     
            p('Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.'),

            p(HTML("<b><div style='background-color:#FADDF2;border:1px solid black;'>Your data needs to have the header (variable names) in the first row and the person's IDs in the first column. Missing values should be indicated by a period (.) or NA.</div></b>")),

            aceEditor("text3", value="rater1\trater2\trater3\trater4\trater5\nA\tB\tB\tB\tB\nA\tA\tC\tC\tC\nC\tC\tC\tC\tC\nA\tA\tA\tA\tC\nA\tA\tA\tC\tC\nA\tB\tB\tB\tB\nA\tA\tA\tA\tA\nB\tB\tB\tB\tC\nA\tC\tC\tC\tC\nA\tA\tA\tC\tC",mode="r", theme="cobalt"),
            
            br(),

            h3("Categories selected by raters"),
            verbatimTextOutput("data3.out"),

            br(),

            h3("Fleiss' Kappa"),
            verbatimTextOutput("test3.out"),
            
            p('Criteria (Strength of agreement):', br(),
              '   < 0.40  Poor', br(),
              '0.40–0.75  Intermediate to Good', br(),
              'Over 0.75  Excellent', br()
            ),
            
            br(),
            
            h3("Krippendorff's alpha reliability coefficient"),
            verbatimTextOutput("Krippendorff3.out"),

            br(),
            
            h3("Plot"),

            plotOutput("pPlot3"),

            br(),
            br(),

            plotOutput("mPlot3"),
            
            br(),
            br(),
            
            
            strong('R session info'),
            verbatimTextOutput("info3.out")
            ),
        
        
        
        
        
        
        
        
        
        
        tabPanel("3 or More Raters (Ordinal)",
                 
                 h2("3 or More Raters (Ordinal)"), 
                 
                 h4("Degree of agreement of the ordinal scale by three or more raters"),                 
                 
                 p('Note: Input values must be separated by tabs. Copy and paste from Excel/Numbers.'),
                 
                 p(HTML("<b><div style='background-color:#FADDF2;border:1px solid black;'>Your data needs to have the header (variable names) in the first row and the person's IDs in the first column. Missing values should be indicated by a period (.) or NA.</div></b>")),
                 
                 aceEditor("text4", value="rater1\trater2\trater3\trater4\trater5\n1\t2\t2\t2\t2\n1\t1\t3\t3\t3\n3\t3\t3\t3\t3\n1\t1\t1\t1\t3\n1\t1\t1\t3\t3\n1\t2\t2\t2\t2\n1\t1\t1\t1\t1\n2\t2\t2\t2\t3\n1\t3\t3\t3\t3\n1\t1\t1\t3\t3",mode="r", theme="cobalt"),
                 
                 br(),
                 
                 h3("Categories selected by raters"),
                 verbatimTextOutput("data4.out"),
                 
                 br(),
                 
                 h3("Fleiss' Kappa"),
                 verbatimTextOutput("test4.out"),
                 
                 p('Criteria (Strength of agreement):', br(),
                   '   < 0.40  Poor', br(),
                   '0.40–0.75  Intermediate to Good', br(),
                   'Over 0.75  Excellent', br()
                 ),
                 
                 br(),
                 
                 h3("Kendall's coefficient of concordance for ranks (W)"),
                 verbatimTextOutput("W.out"),    
                 
                 br(),
                 
                 h3("Krippendorff's alpha reliability coefficient"),
                 verbatimTextOutput("Krippendorff4.out"),
                 
                 br(),
                 
                 h3("Plot"),
                 
                 plotOutput("pPlot4"),
                 
                 br(),
                 br(),
                 
                 plotOutput("mPlot4"),
                 
                 br(),
                 br(),
                 
                 
                 strong('R session info'),
                 verbatimTextOutput("info4.out")
        ),
        
        



        
        tabPanel("About",

            strong('Note'),
            p('This web application is developed with',
            a("Shiny.", href="http://www.rstudio.com/shiny/", target="_blank"),
            ''),

            br(),

            strong('List of Packages Used'), br(),
            code('library(shiny)'),br(),
            code('library(shinyAce)'),br(),
            code('library(irr)'),br(),
            code('library(psych)'),br(),
            code('library(vcd)'),br(),
            code('library(lattice)'),br(),
            code('library(reshape2)'),br(),
            code('library(DescTools)'),br(),
            
            br(),

            strong('Code'),
            p('Source code for this application is based on',
            a('"The handbook of Research in Foreign Language Learning and Teaching" (Takeuchi & Mizumoto, 2012).', href='http://mizumot.com/handbook/', target="_blank")),

            p('The code for this web application is available at',
            a('GitHub.', href='https://github.com/mizumot/rep-chi', target="_blank")),

            p('If you want to run this code on your computer (in a local R session), run the code below:',
            br(),
            code('library(shiny)'),br(),
            code('runGitHub("kappa","mizumot")')
            ),

            br(),

            strong('Citation in Publications'),
            p('Mizumoto, A. (2015). Langtest (Version 1.0) [Web application]. Retrieved from http://langtest.jp'),

            br(),

            strong('Article'),
            p('Mizumoto, A., & Plonsky, L. (2015).', a("R as a lingua franca: Advantages of using R for quantitative research in applied linguistics.", href='http://applij.oxfordjournals.org/content/early/2015/06/24/applin.amv025.abstract', target="_blank"), em('Applied Linguistics,'), 'Advance online publication. doi:10.1093/applin/amv025'),

            br(),

            strong('Recommended'),
            p('To learn more about R, I suggest this excellent and free e-book (pdf),',
            a("A Guide to Doing Statistics in Second Language Research Using R,", href="http://cw.routledge.com/textbooks/9780805861853/guide-to-R.asp", target="_blank"),
            'written by Dr. Jenifer Larson-Hall.'),

            p('Also, if you are a cool Mac user and want to use R with GUI,',
            a("MacR", href="https://sites.google.com/site/casualmacr/", target="_blank"),
            'is defenitely the way to go!'),

            br(),

            strong('Author'),
            p(a("Atsushi MIZUMOTO,", href="http://mizumot.com", target="_blank"),' Ph.D.',br(),
            'Associate Professor of Applied Linguistics',br(),
            'Faculty of Foreign Language Studies /',br(),
            'Graduate School of Foreign Language Education and Research,',br(),
            'Kansai University, Osaka, Japan'),

            br(),

            a(img(src="http://i.creativecommons.org/p/mark/1.0/80x15.png"), target="_blank", href="http://creativecommons.org/publicdomain/mark/1.0/"),

            p(br())

)
)
)
))
Code license: GPL-3