R for reproducible scientific analysis

Subsetting data

R has many powerful subset operators and mastering them will allow you to easily perform complex operations on any kind of dataset.

There are six different ways we can subset any kind of object, and three different subsetting operators for the different data structures.

Let’s start with the workhorse of R: atomic vectors.

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
x

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

So now that we’ve created a dummy vector to play with, how do we get at its contents?

Accessing elements using their indices

To extract elements of a vector we can give their corresponding index, starting from one:

x[1]

  a 
5.4 

x[4]

  d 
4.8 

It may look different, but the square brackets operator is a function. For atomic vectors (and matrices), it means “get me the nth element”.

We can ask for multiple elements at once:

x[c(1, 3)]

  a   c 
5.4 7.1 

Or slices of the vector:

x[1:4]

  a   b   c   d 
5.4 6.2 7.1 4.8 

the : operator reates a sequence of numbers from the left element to the right.

1:4

[1] 1 2 3 4

c(1, 2, 3, 4)

[1] 1 2 3 4

We can ask for the same element multiple times:

x[c(1,1,3)]

  a   a   c 
5.4 5.4 7.1 

If we ask for a number outside of the vector, R will return missing values:

x[6]

<NA> 
  NA 

This is a vector of length one containing an NA, whose name is also NA.

If we ask for the 0th element, we get an empty vector:

x[0]

named numeric(0)

Skipping and removing elements

If we use a negative number as the index of a vector, R will return every element except for the one specified:

x[-2]

  a   c   d   e 
5.4 7.1 4.8 7.5 

We can skip multiple elements:

x[c(-1, -5)]  # or x[-c(1,5)]

  b   c   d 
6.2 7.1 4.8 

To remove elements from a vector, we need to assign the results back into the variable:

x <- x[-4]
x

  a   b   c   e 
5.4 6.2 7.1 7.5 

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

  b   c   d 
6.2 7.1 4.8 

Subsetting by name

We can extract elements by using their name, instead of index:

x[c("a", "c")]

  a   c 
5.4 7.1 

This is usually a much more reliable way to subset objects: the position of various elements can often change when chaining together subsetting operations, but the names will always remain the same!

Unfortunately we can’t skip or remove elements so easily.

To skip (or remove) a single named element:

x[-which(names(x) == "a")]

  b   c   d   e 
6.2 7.1 4.8 7.5 

The which function returns the indices of all TRUE elements of its argument. Remember that expressions evaluate before being passed to functions. Let’s break this down so that its clearer what’s happening.

First this happens:

names(x) == "a"

[1]  TRUE FALSE FALSE FALSE FALSE

The condition operator is applied to every name of the vector x. Only the first name is “a” so that element is TRUE.

which then converts this to an index:

which(names(x) == "a")

[1] 1

Only the first element is TRUE, so which returns 1. Now that we have indices the skipping works because we have a negative index!

Skipping multiple named indices is similar, but uses a different comparison operator:

x[-which(names(x) %in% c("a", "c"))]

  b   d   e 
6.2 4.8 7.5 

The %in% goes through each element of its left argument, in this case the names of x, and asks, “Does this element occur in the second argument?”.

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

x[-which(names(x) == "g")]

x <- 1:3
x

[1] 1 2 3

names(x) <- c('a', 'a', 'a')  
x

a a a 
1 2 3 

x['a']  # only returns first value

a 
1 

x[which(names(x) == 'a')]  # returns all three values

a a a 
1 2 3 

So why can’t we use == like before? That’s an excellent question.

Let’s take a look at the comparison component of this code:

names(x) == c('a', 'c')

Warning in names(x) == c("a", "c"): longer object length is not a multiple
of shorter object length

[1]  TRUE FALSE  TRUE

Obviously “c” is in the names of x, so why didn’t this work? == works slightly differently than %in%. It will compare each element of its left argument to the corresponding element of its right argument.

Here’s a mock illustration:

c("a", "b", "c", "e")  # names of x
   |    |    |    |    # The elements == is comparing
c("a", "c")

When one vector is shorter than the other, it gets recycled:

c("a", "b", "c", "e")  # names of x
   |    |    |    |    # The elements == is comparing
c("a", "c", "a", "c")

In this case R simply repeats c("a", "c") twice. If the longer vector length isn’t a multiple of the shorter vector length, then R will also print out a warning message:

names(x) == c('a', 'c', 'e')

[1]  TRUE FALSE FALSE

This difference between == and %in% is important to remember, because it can introduce hard to find and subtle bugs!

Subsetting through other logical operations

We can also more simply subset through logical operations:

x[c(TRUE, TRUE, FALSE, FALSE)]

a a 
1 2 

Note that in this case, the logical vector is also recycled to the length of the vector we’re subsetting!

x[c(TRUE, FALSE)]

a a 
1 3 

Since comparison operators evaluate to logical vectors, we can also use them to succinctly subset vectors:

x[x > 7]

named integer(0)

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

Handling special values

At some point you will encounter functions in R which cannot handle missing, infinite, or undefined data.

There are a number of special functions you can use to filter out this data:

• is.na will return all positions in a vector, matrix, or data.frame containing NA.
• likewise, is.nan, and is.infinite will do the same for NaN and Inf.
• is.finite will return all positions in a vector, matrix, or data.frame that do not contain NA, NaN or Inf.
• na.omit will filter out all missing values from a vector

Factor subsetting

Now that we’ve explored the different ways to subset vectors, how do we subset the other data structures?

Factor subsetting works the same way as vector subsetting.

f <- factor(c("a", "a", "b", "c", "c", "d"))
f[f == "a"]

[1] a a
Levels: a b c d

f[f %in% c("b", "c")]

[1] b c c
Levels: a b c d

f[1:3]

[1] a a b
Levels: a b c d

An important note is that skipping elements will not remove the level even if no more of that category exists in the factor:

f[-3]

[1] a a c c d
Levels: a b c d

Matrix subsetting

Matrices are also subsetted using the [ function. In this case it takes two arguments: the first applying to the rows, the second to its columns:

set.seed(1)
m <- matrix(rnorm(6*4), ncol=4, nrow=6)
m[3:4, c(3,1)]

            [,1]       [,2]
[1,]  1.12493092 -0.8356286
[2,] -0.04493361  1.5952808

You can leave the first or second arguments blank to retrieve all the rows or columns respectively:

m[, c(3,4)]

            [,1]        [,2]
[1,] -0.62124058  0.82122120
[2,] -2.21469989  0.59390132
[3,]  1.12493092  0.91897737
[4,] -0.04493361  0.78213630
[5,] -0.01619026  0.07456498
[6,]  0.94383621 -1.98935170

If we only access one row or column, R will automatically convert the result to a vector:

m[3,]

[1] -0.8356286  0.5757814  1.1249309  0.9189774

If you want to keep the output as a matrix, you need to specify a third argument; drop = FALSE:

m[3, , drop=FALSE]

           [,1]      [,2]     [,3]      [,4]
[1,] -0.8356286 0.5757814 1.124931 0.9189774

Unlike vectors, if we try to access a row or column outside of the matrix, R will throw an error:

m[, c(3,6)]

Error in m[, c(3, 6)]: subscript out of bounds

Because matrices are vectors, we can also subset using only one argument:

m[5]

[1] 0.3295078

This usually isn’t useful, and often confusing to read. However it is useful to note that matrices are laid out in column-major format by default. That is the elements of the vector are arranged column-wise:

matrix(1:6, nrow=2, ncol=3)

     [,1] [,2] [,3]
[1,]    1    3    5
[2,]    2    4    6

If you wish to populate the matrix by row, use byrow=TRUE:

matrix(1:6, nrow=2, ncol=3, byrow=TRUE)

     [,1] [,2] [,3]
[1,]    1    2    3
[2,]    4    5    6

Matrices can also be subsetted using their rownames and column names instead of their row and column indices.

m <- matrix(1:18, nrow=3, ncol=6)
print(m)

     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    1    4    7   10   13   16
[2,]    2    5    8   11   14   17
[3,]    3    6    9   12   15   18

List subsetting

Now we’ll introduce some new subsetting operators. There are three functions used to subset lists. [, as we’ve seen for atomic vectors and matrices, as well as [[ and $.

Using [ will always return a list. If you want to subset a list, but not extract an element, then you will likely use [.

xlist <- list(a = "Software Carpentry", b = 1:10, data = head(iris))
xlist[1]

$a
[1] "Software Carpentry"

This returns a list with one element.

We can subset elements of a list exactly the same was as atomic vectors using [. Comparison operations however won’t work as they’re not recursive, they will try to condition on the data structures in each element of the list, not the individual elements within those data structures.

xlist[1:2]

$a
[1] "Software Carpentry"

$b
 [1]  1  2  3  4  5  6  7  8  9 10

To extract individual elements of a list, you need to use the double-square bracket function: [[.

xlist[[1]]

[1] "Software Carpentry"

Notice that now the result is a vector, not a list.

You can’t extract more than one element at once:

xlist[[1:2]]

Error in xlist[[1:2]]: subscript out of bounds

Nor use it to skip elements:

xlist[[-1]]

Error in xlist[[-1]]: attempt to select more than one element in get1index <real>

But you can use names to both subset and extract elements:

xlist[["a"]]

[1] "Software Carpentry"

The $ function is a shorthand way for extracting elements by name:

xlist$data

  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2  setosa
2          4.9         3.0          1.4         0.2  setosa
3          4.7         3.2          1.3         0.2  setosa
4          4.6         3.1          1.5         0.2  setosa
5          5.0         3.6          1.4         0.2  setosa
6          5.4         3.9          1.7         0.4  setosa

xlist <- list(a = "Software Carpentry", b = 1:10, data = head(iris))

mod <- aov(pop ~ lifeExp, data=gapminder)

Data frames

Remember the data frames are lists underneath the hood, so similar rules apply. However they are also two dimensional objects:

[ with one argument will act the same was as for lists, where each list element corresponds to a column. The resulting object will be a data frame:

head(gapminder[3])

       pop
1  8425333
2  9240934
3 10267083
4 11537966
5 13079460
6 14880372

Similarly, [[ will act to extract a single column:

head(gapminder[["lifeExp"]])

[1] 28.801 30.332 31.997 34.020 36.088 38.438

And $ provides a convenient shorthand to extract columns by name:

head(gapminder$year)

[1] 1952 1957 1962 1967 1972 1977

With two arguments, [ behaves the same way as for matrices:

gapminder[1:3,]

      country year      pop continent lifeExp gdpPercap
1 Afghanistan 1952  8425333      Asia  28.801  779.4453
2 Afghanistan 1957  9240934      Asia  30.332  820.8530
3 Afghanistan 1962 10267083      Asia  31.997  853.1007

If we subset a single row, the result will be a data frame (because the elements are mixed types):

gapminder[3,]

      country year      pop continent lifeExp gdpPercap
3 Afghanistan 1962 10267083      Asia  31.997  853.1007

But for a single column the result will be a vector (this can be changed with the third argument, drop = FALSE).

gapminder[gapminder$year = 1957,]

gapminder[,-1:4]

gapminder[gapminder$lifeExp > 80]

gapminder[1, 4, 5]

gapminder[gapminder$year == 2002 | 2007,]

Challenge solutions

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

  b   c   d 
6.2 7.1 4.8 

x[2:4] 
x[-c(1,5)]
x[c("b", "c", "d")]
x[c(2,3,4)]

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

x[-which(names(x) == "g")]

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

x_subset <- x[x<7 & x>4]
print(x_subset)

  a   b   d 
5.4 6.2 4.8 

x <- c(5.4, 6.2, 7.1, 4.8, 7.5)
names(x) <- c('a', 'b', 'c', 'd', 'e')
print(x)

  a   b   c   d   e 
5.4 6.2 7.1 4.8 7.5 

x_subset <- x[x<7 & x>4]
print(x_subset)

  a   b   d 
5.4 6.2 4.8 

m <- matrix(1:18, nrow=3, ncol=6)
print(m)

     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    1    4    7   10   13   16
[2,]    2    5    8   11   14   17
[3,]    3    6    9   12   15   18

xlist <- list(a = "Software Carpentry", b = 1:10, data = head(iris))

xlist$b[2]
xlist[[2]][2]
xlist[["b"]][2]

mod <- aov(pop ~ lifeExp, data=gapminder)

attributes(mod) ## `df.residual` is one of the names of `mod`
mod$df.residual

# gapminder[gapminder$year = 1957,]
gapminder[gapminder$year == 1957,]

# gapminder[,-1:4]
gapminder[,-c(1:4)]

# gapminder[gapminder$lifeExp > 80]
gapminder[gapminder$lifeExp > 80,]

# gapminder[1, 4, 5]
gapminder[1, c(4, 5)]

# gapminder[gapminder$year == 2002 | 2007,]
gapminder[gapminder$year == 2002 | gapminder$year == 2007,]
gapminder[gapminder$year %in% c(2002, 2007),]

gapminder_small <- gapminder[c(1:9, 19:23),]