Loading and processing textual data

For this tutorial, we will use Charles Darwin’s On the Origin of Species by means of Natural Selection which we download from the Project Gutenberg archive (see Stroube 2003). Thus, Darwin’s Origin of Species forms the basis of our analysis. You can use the code below to download this text into R (but you have to have access to the internet to do so).

origin <- gutenberg_works(gutenberg_id == "1228") %>%
  gutenberg_download(meta_fields = "gutenberg_id", 
                     mirror = "http://mirrors.xmission.com/gutenberg/") %>%
  dplyr::filter(text != "")

First 10 text elements of Charles Darwin's Origin.

The table above shows that Darwin’s Origin of Species requires formatting so that we can use it. Therefore, we collapse it into a single object (or text) and remove superfluous white spaces.

origin <- origin$text %>%
  paste0(collapse = " ") %>%
  str_squish()

First 1000 characters of Charles Darwin's Origin

The result confirms that the entire text is now combined into a single character object.

Creating simple concordances

Now that we have loaded the data, we can easily extract concordances using the kwic function from the quanteda package. The kwic function takes the text (x) and the search pattern (pattern) as it main arguments but it also allows the specification of the context window, i.e. how many words/elements are show to the left and right of the key word (we will go over this later on).

kwic_natural <- kwic(x = origin, pattern = "selection")

First 10 concordances for the keyword natural in Charles Darwin's Origin.

We can easily extract the frequency of the search term (selection) using the nrow or the length functions which provide the number of rows of a tables (nrow) or the length of a vector (length).

nrow(kwic_natural)

## [1] 412

length(kwic_natural$keyword)

## [1] 412

The results show that there are 414 instances of the search term (selection) but we can also find out how often different variants (lower case versus upper case) of the search term were found using the table function. This is especially useful when searches involve many different search terms (while it is, admittedly, less useful in the present example).

table(kwic_natural$keyword)

## 
## selection Selection SELECTION 
##       369        39         4

To get a better understanding of the use of a word, it is often useful to extract more context. This is easily done by increasing size of the context window. To do this, we specify the window argument of the kwic function. In the example below, we set the context window size to 10 words/elements rather than using the default (which is 5 word/elements).

kwic_natural_longer <- kwic(x = origin, pattern = "selection", window = 10)

First 10 concordances for the keyword natural in Charles Darwin's Origin with extended context (10 elements).

  kwic_nature <- kwic(x = origin, pattern = "nature")

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  # inspect
  kwic_natural %>%
  as.data.frame() %>%
  head(10)

  ##    docname from  to                                  pre   keyword                             post   pattern
  ## 1    text1   44  44          Species BY MEANS OF NATURAL SELECTION         , OR THE PRESERVATION OF selection
  ## 2    text1  275 275              EXISTENCE . 4 . NATURAL SELECTION                    . 5 . LAWS OF selection
  ## 3    text1  411 411            and Origin . Principle of Selection anciently followed , its Effects selection
  ## 4    text1  421 421 Effects . Methodical and Unconscious Selection          . Unknown Origin of our selection
  ## 5    text1  436 436         favourable to Man's power of Selection          . CHAPTER 2 . VARIATION selection
  ## 6    text1  522 522         EXISTENCE . Bears on natural selection               . The term used in selection
  ## 7    text1  616 616                . CHAPTER 4 . NATURAL SELECTION        . Natural Selection : its selection
  ## 8    text1  619 619        . NATURAL SELECTION . Natural Selection        : its power compared with selection
  ## 9    text1  626 626        its power compared with man's selection        , its power on characters selection
  ## 10   text1  647 647               on both sexes . Sexual Selection           . On the generality of selection

  kwic_nature %>%
  as.data.frame() %>%
  nrow()

  ## [1] 261

  kwic_origin <- kwic(x = origin, pattern = "origin")

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  # inspect
  kwic_origin %>%
  as.data.frame() %>%
  head(5)

  ##   docname from  to                                 pre keyword                               post pattern
  ## 1   text1   37  37         definitive edition . On the  Origin             of Species BY MEANS OF  origin
  ## 2   text1  351 351         DETEAILED CONTENTS . ON THE  ORIGIN        OF SPECIES . INTRODUCTION .  origin
  ## 3   text1  391 391     between Varieties and Species .  Origin     of Domestic Varieties from one  origin
  ## 4   text1  407 407     Pigeons , their Differences and  Origin . Principle of Selection anciently  origin
  ## 5   text1  424 424 and Unconscious Selection . Unknown  Origin      of our Domestic Productions .  origin

`

Extracting more than single words

While extracting single words is very common, you may want to extract more than just one word. To extract phrases, all you need to so is to specify that the pattern you are looking for is a phrase, as shown below.

kwic_naturalselection <- kwic(origin, pattern = phrase("natural selection"))

## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

First 10 concordances for the keyphrase natural selection in Charles Darwin's Origin with extended context (10 elements).

Of course you can extend this to longer sequences such as entire sentences. However, you may want to extract more or less concrete patterns rather than words or phrases. To search for patterns rather than words, you need to include regular expressions in your search pattern.

  kwic_naturalhabitat <- kwic(x = origin, pattern = phrase("natural habitat"))

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  # inspect
  kwic_naturalhabitat %>%
  as.data.frame() %>%
  head(10)

  ## [1] docname from    to      pre     keyword post    pattern
  ## <0 Zeilen> (oder row.names mit Länge 0)

  kwic_naturalhabitat %>%
  as.data.frame() %>%
  nrow()

  ## [1] 0

  kwic_theorigin <- kwic(x = origin, pattern = phrase("the origin"))

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  # inspect
  kwic_theorigin %>%
  as.data.frame() %>%
  head(5)

  ##   docname from   to                           pre    keyword                        post    pattern
  ## 1   text1   36   37   the definitive edition . On the Origin      of Species BY MEANS OF the origin
  ## 2   text1  350  351 INDEX DETEAILED CONTENTS . ON THE ORIGIN OF SPECIES . INTRODUCTION . the origin
  ## 3   text1 1617 1618     . Concluding remarks . ON THE ORIGIN OF SPECIES . INTRODUCTION . the origin
  ## 4   text1 1679 1680        to throw some light on the origin   of species - that mystery the origin
  ## 5   text1 1910 1911    conclusions that I have on the origin      of species . Last year the origin

`

Searches using regular expressions

Regular expressions allow you to search for abstract patterns rather than concrete words or phrases which provides you with an extreme flexibility in what you can retrieve. A regular expression (in short also called regex or regexp) is a special sequence of characters that stand for are that describe a pattern. You can think of regular expressions as very powerful combinations of wildcards or as wildcards on steroids. For example, the sequence [a-z]{1,3} is a regular expression that stands for one up to three lower case characters and if you searched for this regular expression, you would get, for instance, is, a, an, of, the, my, our, etc, and many other short words as results.

There are three basic types of regular expressions:

• regular expressions that stand for individual symbols and determine frequencies

• regular expressions that stand for classes of symbols

• regular expressions that stand for structural properties

The regular expressions below show the first type of regular expressions, i.e. regular expressions that stand for individual symbols and determine frequencies.

Regular expressions that stand for individual symbols and determine frequencies.

The regular expressions below show the second type of regular expressions, i.e. regular expressions that stand for classes of symbols.

Regular expressions that stand for classes of symbols.

The regular expressions that denote classes of symbols are enclosed in [] and :. The last type of regular expressions, i.e. regular expressions that stand for structural properties are shown below.

Regular expressions that stand for structural properties.

To include regular expressions in your KWIC searches, you include them in your search pattern and set the argument valuetype to "regex". The search pattern "\\bnatu.*|\\bselec.*" retrieves elements that contain natu and selec followed by any characters and where the n in natu and the s in selec are at a word boundary, i.e. where they are the first letters of a word. Hence, our serach would not retrieve words like unnatural or deselect. The | is an operator (like +, -, or *) that stands for or.

# define search patterns
patterns <- c("\\bnatu.*|\\bselec.*")
kwic_regex <- kwic(origin, patterns, valuetype = "regex")

First 10 concordances for the regular expression \bnatu.*.

  kwic_exu <- kwic(x = origin, pattern = ".*exu.*", valuetype = "regex")

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  # inspect
  kwic_exu %>%
  as.data.frame() %>%
  head(10)

  ##    docname  from    to                               pre keyword                             post pattern
  ## 1    text1   646   646               and on both sexes .  Sexual    Selection . On the generality .*exu.*
  ## 2    text1   806   806 variable than generic : secondary  sexual characters variable . Species of .*exu.*
  ## 3    text1 29294 29294               and on both sexes .  Sexual    Selection . On the generality .*exu.*
  ## 4    text1 31953 31953      like every other structure . _Sexual       Selection_ . - Inasmuch as .*exu.*
  ## 5    text1 32040 32040              words on what I call  Sexual       Selection . This depends , .*exu.*
  ## 6    text1 32082 32082             few or no offspring .  Sexual       selection is , therefore , .*exu.*
  ## 7    text1 32157 32157     chance of leaving offspring .  Sexual selection by always allowing the .*exu.*
  ## 8    text1 32330 32330         be given through means of  sexual          selection , as the mane .*exu.*
  ## 9    text1 32628 32628   having been chiefly modified by  sexual      selection , acting when the .*exu.*
  ## 10   text1 32726 32726        have been mainly caused by  sexual            selection ; that is , .*exu.*

  kwic_nonet <- kwic(x = origin, pattern = "\\bnonet.*", valuetype = "regex") %>%
  as.data.frame() %>%
  nrow()

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  kwic_ption <- kwic(x = origin, pattern = "ption\\b", valuetype = "regex")

  ## Warning: 'kwic.character()' is deprecated. Use 'tokens()' first.

  # inspect
  kwic_ption %>%
  as.data.frame() %>%
  head(5)

  ##   docname from   to                          pre    keyword                        post  pattern
  ## 1   text1 1605 1605    extended . Effects of its   adoption     on the study of Natural ption\\b
  ## 2   text1 2641 2641          see them ; but this assumption           seems to me to be ption\\b
  ## 3   text1 3926 3926         or at the instant of conception   . Geoffroy St . Hilaire's ption\\b
  ## 4   text1 3990 3990          prior to the act of conception   . Several reasons make me ption\\b
  ## 5   text1 4233 4233 under confinement , with the  exception of the plantigrades or bear ption\\b

`

Piping concordances

Quite often, we only want to retrieve patterns if they occur in a certain context. For instance, we might be interested in instances of selection but only if the preceding word is natural. Such conditional concordances could be extracted using regular expressions but they are easier to retrieve by piping. Piping is done using the %>% function from the dplyr package and the piping sequence can be translated as and then. We can then filter those concordances that contain natural using the filter function from the dplyr package. Note the the $ stands for the end of a string so that natural$ means that natural is the last element in the string that is preceding the keyword.

kwic_pipe <- kwic(x = origin, pattern = "selection") %>%
  dplyr::filter(stringr::str_detect(pre, "natural$|NATURAL$"))

First 10 concordances for instances of selection that are preceeded by natural.

Piping is a very useful helper function and it is very frequently used in R - not only in the context of text processing but in all data science related domains.

Arranging concordances and adding frequency information

When inspecting concordances, it is useful to re-order the concordances so that they do not appear in the order that they appeared in the text or texts but by the context. To reorder concordances, we can use the arrange function from the dplyr package which takes the column according to which we want to re-arrange the data as it main argument.

In the example below, we extract all instances of natural and then arrange the instances according to the content of the post column in alphabetical.

kwic_ordered <- kwic(x = origin, pattern = "natural") %>%
  dplyr::arrange(post)

First 10 re-ordered concordances for instances of natural.

Arranging concordances according to alphabetical properties may, however, not be the most useful option. A more useful option may be to arrange concordances according to the frequency of co-occurring terms or collocates. In order to do this, we need to extract the co-occurring words and calculate their frequency. We can do this by combining the mutate, group_by, n() functions from the dplyr package with the str_remove_all function from the stringr package. Then, we arrange the concordances by the frequency of the collocates in descending order (that is why we put a - in the arrange function). In order to do this, we need to

1. create a new variable or column which represents the word that co-occurs with, or, as in the example below, immediately follows the search term. In the example below, we use the mutate function to create a new column called post_word. We then use the str_remove_all function to remove everything except for the word that immediately follows the search term (we simply remove everything and including a white space).

2. group the data by the word that immediately follows the search term.

3. create a new column called post_word_freq which represents the frequencies of all the words that immediately follow the search term.

4. arrange the concordances by the frequency of the collocates in descending order.

kwic_ordered_coll <- kwic(x = origin, pattern = "natural") %>%
  dplyr::mutate(post_word = str_remove_all(pre, " .*")) %>%
  dplyr::group_by(post_word) %>%
  dplyr::mutate(post_word_freq = n()) %>%
  dplyr::arrange(-post_word_freq)

First 10 re-ordered concordances for instances of natural.

We add more columns according to which we could arrange the concordance following the same schema. For example, we could add another column that represented the frequency of words that immediately preceded the search term and then arrange according to this column.

Concordances from transcriptions

As many analyses use transcripts as their primary data and because transcripts have features that require additional processing, we will now perform concordancing based on on transcripts. As a first step, we load five example transcripts that represent the first five files from the Irish component of the International Corpus of English.

# define corpus files
files <- paste("https://slcladal.github.io/data/ICEIrelandSample/S1A-00", 1:5, ".txt", sep = "")
# load corpus files
transcripts <- sapply(files, function(x){
  x <- readLines(x)
})

First 10 utterances in the sample transcripts.

The first ten lines shown above let us know that, after the header (<S1A-001 Riding>) and the symbol which indicates the start of the transcript (<I>), each utterance is preceded by a sequence which indicates the section, file, and speaker (e.g. <S1A-001$A>). The first utterance is thus uttered by speaker A in file 001 of section S1A. In addition, there are several sequences that provide meta-linguistic information which indicate the beginning of a speech unit (<#>), pauses (<,>), and laughter (<&> laughter </&>).

To perform the concordancing, we need to change the format of the transcripts because the kwic function only works on character, corpus, tokens object- in their present form, the transcripts represent a list which contains vectors of strings. To change the format, we collapse the individual utterances into a single character vector for each transcript.

transcripts_collapsed <- sapply(files, function(x){
  x <- readLines(x)
  x <- paste0(x, collapse = " ")
  x <- str_squish(x)
})

First 500 characters of the collapsed sample transcripts.

We can now extract the concordances.

kwic_trans <- kwic(x = transcripts_collapsed, pattern = phrase("you know"))

First 10 concordances for you know in three example transcripts.

The results show that each non-alphanumeric character is counted as a single word which reduces the context of the keyword substantially. Also, the docname column contains the full path to the data which make it hard to parse the content of the table. To address the first issue, we remove symbols by adding remove_symbols = T and remove punctuation by adding remove_punct = T. In addition, we clean the docname column and extract only the file name.

kwic_trans <- quanteda::kwic(x = transcripts_collapsed, pattern = phrase("you know"),
                   remove_symbols = T, remove_punct = T)
# clean docnames
kwic_trans$docname <- kwic_trans$docname %>%
  str_replace_all(".*/([A-Z][0-9][A-Z]-[0-9]{1,3}).txt", "\\1") 

First 10 concordances for you know in three example transcripts.

We could also extend the context window and merge the symbols that the kwic function has separated.

Extending the context can also be used to identify the speaker that has uttered the search pattern that we are interested in. We will do just that as this is a common task in linguistics analyses.

To extract speakers, we need to follow these steps:

1. Create normal concordances of the pattern that we are interested in.

2. Generate concordances of the pattern that we are interested in with a substantially enlarged context window size.

3. Extract the speakers from the enlarged context window size.

4. Add the speakers to the normal concordances using the left-join function from the dplyr package.

kwic_normal <- quanteda::kwic(transcripts_collapsed, phrase("you know")) %>%
  as.data.frame()
kwic_long <- quanteda::kwic(transcripts_collapsed, phrase("you know"), window = 500) %>%
  as.data.frame() %>%
  dplyr::mutate(pre = stringr::str_remove_all(pre, ".*\\$")) %>%
  dplyr::mutate(pre = stringr::str_remove_all(pre, "\\>.*"),
                speaker = stringr::str_squish(pre)) %>%
  dplyr::select(docname, speaker)
# add speaker to normal kwic
kwic_combined <- dplyr::left_join(kwic_normal, kwic_long) %>%
  dplyr::mutate(docname = stringr::str_replace_all(docname, ".*/([A-Z][0-9][A-Z]-[0-9]{1,3}).txt", "\\1")) %>%
  dplyr::select(-to, -from, -pattern)

First 10 concordances for you know in three example transcripts with speakers that uttered them..

The resulting table shows that we have successfully extracted the speakers (identified by the letters in the speaker column) and cleaned the file names (in the docnames column).

Customizing concordances

As R represents a fully-fledged programming environment, we can, of course, also write our own, customized concordance function. The code below shows how you could go about doing so. Note, however, that this function only works if you enter more than a single file.

mykwic <- function(txts, pattern, context) {
  # activate packages
  require(stringr)
  require(plyr)
  # list files
  conc <- sapply(txts, function(x) {
    # determine length of text
    lngth <- as.vector(unlist(nchar(x)))
    # determine position of hits
    idx <- str_locate_all(x, pattern)
    idx <- idx[[1]]
    ifelse(nrow(idx) >= 1, idx <- idx, return("No hits found"))
    # define start position of hit
    token.start <- idx[,1]
    # define end position of hit
    token.end <- idx[,2]
    # define start position of preceding context
    pre.start <- ifelse(token.start-context < 1, 1, token.start-context)
    # define end position of preceding context
    pre.end <- token.start-1
    # define start position of subsequent context
    post.start <- token.end+1
    # define end position of subsequent context
    post.end <- ifelse(token.end+context > lngth, lngth, token.end+context)
    # extract the texts defined by the positions
    PreceedingContext <- substring(x, pre.start, pre.end)
    Token <- substring(x, token.start, token.end)
    SubsequentContext <- substring(x, post.start, post.end)
    conc <- cbind(PreceedingContext, Token, SubsequentContext)
    # return concordance
    return(conc)
    })
  concdf <- ldply(conc, data.frame)
  colnames(concdf)[1]<- "File"
  return(concdf)
}

We can now try if this function works by searching for the sequence you know in the transcripts that we have loaded earlier. One difference between the kwic function provided by the quanteda package and the customized concordance function used here is that the kwic function uses the number of words to define the context window, while the mykwic function uses the number of characters or symbols instead (which is why we use a notably higher number to define the context window).

myconcordances <- mykwic(transcripts_collapsed, "you know", 50)

First 10 concordances for you know extracted using the mykwic function.

As this concordance function only works for more than one text, we split the text of Darwin’s On the Origin of Species into chapters and assign each section a name.

# read in text
origin_split <- origin %>%
  stringr::str_squish() %>%
  stringr::str_split("[CHAPTER]{7,7} [XVI]{1,7}\\. ") %>%
  unlist()
origin_split <- origin_split[which(nchar(origin_split) > 2000)]
# add names
names(origin_split) <- paste0("text", 1:length(origin_split))
# inspect data
nchar(origin_split)

##  text1  text2  text3  text4  text5  text6  text7  text8  text9 text10 text11 text12 text13 text14 text15 
##  17465  69701  29396  35636  94170  73401  66349  69085  61085  58518  62094  67855  51300  87340  86574

Now that we have named elements, we can search for the pattern natural selection. We also need to clean the concordance as some sections do not contain any instances of the search pattern. To clean the data, we select only the columns File, PreceedingContext, Token, and SubsequentContext and then remove all rows where information is missing.

natsel_conc <- mykwic(origin_split, "natural selection", 50) %>%
  dplyr::select(File, PreceedingContext, Token, SubsequentContext) %>%
  na.omit()

First 10 concordances for you know extracted using the mykwic function.

You can go ahead and modify the customized concordance function to suit your needs.