How to get podcast ratings and reviews data using R

Overview

As part of some work I’m doing at the moment with my BCMCR colleague Simon Barber, I recently set about trying out find if there was a way of getting access to the reviews and ratings posted to the Apple store about podcasts. I discovered that this information is relatively easy to get hold of, and along the way found out that getting the episode data for a podcast is also possible. Once I had the process working well, I then spent a bit of time developing it into a replicable workflow that other people could use. This post will walk you through that process.

If you are someone who makes podcasts, this should help you produce a database of your podcast episodes, an archive of listener reviews and ratings, and some quick visualisations.

In a follow up post, I will demonstrate how you can run some analysis of the data you’ve gathered and combine all elements together into a report that can be rendered as a PDF or HTML document. In this first post, however, we’ll concentrate on gathering and organising the data.

All the code required is provided below, along with explanations of what the various elements do. The code was working correctly as of 14/05/20, but please do let me know if you encounter errors. I can be contacted via email at craig.hamilton@bcu.ac.uk.

Sodajerker on Songwriting

The podcast I will be looking at in this post is Sodajerker on Songwriting. This show is made by Simon and his co-host, Brian O’Connor and - as the name suggests - it focusses on the art and craft of songwriting. Since 2011, Simon and Brian have interviewed some of the worlds most successful songwriters, including Sir Paul McCartney, Nile Rodgers, Alicia Keys, Roseanne Cash, Beck, Paul Simon, and many, many more, across 160+ episodes. It’s a great show - if you are a fan of pop music then you will almost certainly enjoy Sodajerker.

For more information on Sodajerker, visit their website or follow them on Twitter.

Ok, let’s get on with gathering the data….

1. Getting started

library(lubridate)
library(readxl)
library(hms)
library(rvest)
library(stringr)
library(tibble)
library(dplyr)
library(tidyverse)
library(ggplot2)

podcast_name <- 'Name of Podcast' 
podcase_file_name <- 'Name_of_podcast'

URL <- 'https://[Podcast RSS address]/rss' 

pod_id <- #Podcast ID without quote marks, e.g. 123456789

apple <- read_excel('apple_store_codes.xlsx', sheet = 1)

head(apple)

## # A tibble: 6 × 2
##   code  country            
##   <chr> <chr>              
## 1 al    Albania            
## 2 dz    Algeria            
## 3 ao    Angola             
## 4 ai    Anguilla           
## 5 ag    Antigua and Barbuda
## 6 ar    Argentina

2. Get Episode Data

By following this excellent tutorial written by Robert Muwanga and using the code he supplied in his post, the following will call on the URL we created above to produce a new dataframe called episode_data

css_descriptors <- c('title', 'pubDate', 'itunes\\:summary', 'itunes\\:duration') # XML tags of interest
col_names <- c('title', 'pubdate','summary', 'duration') # Initial column names for tibble

# Load XML feed and extract items nodes
podcast_feed <- read_xml(URL)
items <- xml_nodes(podcast_feed, 'item')

# Extracts from an item node the content defined by the css_descriptor
extract_element <- function(item, css_descriptor) { 
  element <- xml_node(item, css_descriptor) %>% xml_text
  element 
}

episode_data <- sapply(css_descriptors, function(x) { 
  extract_element(items, x)}
) %>% as_tibble()

names(episode_data) <- col_names # Set new column names

episode_data$pubdate<- dmy_hms(episode_data$pubdate) 

less_than_hour <- episode_data %>%
  filter(nchar(episode_data$duration) < 6)
more_than_hour <- episode_data %>%
  filter(nchar(episode_data$duration) > 6)
less_than_hour$duration <- paste0("00:", less_than_hour$duration)
episode_data <- rbind(less_than_hour, more_than_hour)
rm(less_than_hour, more_than_hour)
episode_data$duration_clean <- as_hms(episode_data$duration)

#add podcast name
episode_data$podcast_name <- podcast_name

#order by date
episode_data <- episode_data[order(episode_data$pubdate , decreasing = FALSE), ]

#add ids
episode_data$id <- 1:nrow(episode_data)

str(episodes_data)

## tibble [168 × 6] (S3: tbl_df/tbl/data.frame)
##  $ title         : chr [1:168] "Episode 1 - Billy Steinberg" "Episode 2 - Todd Rundgren" "Episode 3 - Sacha Skarbek" "Episode 4 - Jimmy Webb" ...
##  $ pubdate       : POSIXct[1:168], format: "2011-11-09 22:23:00" "2011-11-25 11:36:00" ...
##  $ summary       : chr [1:168] "In the first episode of Sodajerker On Songwriting, co-hosts Simon Barber and Brian O'Connor introduce the podca"| __truncated__ "In the second episode of Sodajerker On Songwriting, Simon and Brian talk with writer/artist/producer Todd Rundg"| __truncated__ "In the third episode of Sodajerker On Songwriting, Simon and Brian talk to Grammy nominated and two-time Ivor N"| __truncated__ "Simon and Brian talk to legendary American songwriter Jimmy Webb about the writing of classic songs like 'By Th"| __truncated__ ...
##  $ duration      : chr [1:168] "00:47:30" "00:25:13" "00:44:19" "00:57:44" ...
##  $ duration_clean: 'hms' num [1:168] 00:47:30 00:25:13 00:44:19 00:57:44 ...
##   ..- attr(*, "units")= chr "secs"
##  $ podcast_name  : chr [1:168] "Sodajerker On Songwriting" "Sodajerker On Songwriting" "Sodajerker On Songwriting" "Sodajerker On Songwriting" ...

episode_data %>% 
  ggplot() +
  aes(pubdate, duration_clean) +
  geom_jitter(aes(colour = "red")) + 
  ggtitle(paste(podcast_name, ": Episodes - Release date and episode duration", sep = "")) +
  theme_minimal() +
  xlab("") +
  ylab("") +
  theme(legend.position="none")

#Episode data
write_rds(episode_data, paste(podcast_file_name, '_episode_data.rds', sep = ""))
write.csv(episode_data, paste(podcast_file_name, '_episode_data.csv', sep = ""))

3. Get Apple ratings and reviews.

For this part of the script I used the itunesr packaged developed by amrrs. At the time of writing this package has been removed from the CRAN repository. Until it is reinstated you will need to use devtools to install it from amrrs’ Github page.

devtools::install_github("amrrs/itunesr")
library(itunesr)

The function we will use from the package is getReviews. This requires 3 inputs: the id of a podcast; the country code of the Apple store you want to get reviews and ratings from; and a page number.

We can use the pod_id variable created earlier for the first element in order to test this function out by retriving the first page of reviews for the US store. If no reviews are returned, try a different country code - ‘gb’ is the code for the UK store, for instance.

reviews <- getReviews(pod_id,'us',1) #needs ID, country and page number
reviews$country <- "us" #adds country variable

There are two unknowns as this stage. 1) which stores have reviews, and 2) if there are reviews, how many pages of reviews are there?

This was the trickiest part of the process for me to work out, but (thanks to some help from Stackoverflow!) the code below will deal with those two issues by running through the following process:

• First it will iterate through each country code row in the apple dataset and check for reviews. If no reviews are returned for that country, the script will move on to the next country code, and so on until it reaches the end of the apple dataset.

• If reviews are returned for a given country code they will be written to an object called reviews_test and then added to the reviews dataframe we created in the example step above.

• Then, if the number of reviews returned one page 1 for a given country is 49 - the maximum number for a page - we known there is a second page of reviews. The script will then go and retrieve data from page 2 for that country.

• If page 2 has fewer then 49 rows, we know that this is the last page of reviews for that country. At that point the script will break and move on to the next country code. If, however, there are 49 reviews on page 2, we know that there is a third page and the script will move on to that. This will repeat until reaching a page with fewer than 49 reviews.

• Finally, and because we will by now have retrieved page 1 from the US (or GB) store twice, the last line removes duplicates from the reviews dataframe.

for (i in 1:nrow(apple)) {
  try({
    page = 1
    x <- apple$code[[i]]
    reviews_test <- getReviews(pod_id, x, page) 
    reviews_test$country <- x
    reviews <- rbind(reviews, reviews_test)
    if (nrow(reviews_test) == 49)
      repeat {
        try({
          Sys.sleep(2) #This introduces a short delay and should help avoid request denial errors.
          page = page + 1
          reviews_test <- getReviews(pod_id, x , page) 
          reviews_test$country <- x
          reviews <- rbind(reviews, reviews_test)
          if (nrow(reviews_test) < 49) break
        }, silent = T) #Remove , silent = T to monitor progress
      }
  }, silent = T) #Remove , silent = T to monitor progress
}

reviews <- reviews[!duplicated(reviews$Review), ]

All being well - and assuming the podcast you are looking at has been reviewed by at least one Apple user - you should now have a dataframe of reviews that looks like this.

str(reviews)

## 'data.frame':    207 obs. of  8 variables:
##  $ Title      : chr  "Well done" "Brilliant In Every Way" "Pleasing the Publishers" "Well done!" ...
##  $ Author_URL : chr  "https://itunes.apple.com/us/reviews/id801900430" "https://itunes.apple.com/us/reviews/id39986619" "https://itunes.apple.com/us/reviews/id139306401" "https://itunes.apple.com/us/reviews/id1083165409" ...
##  $ Author_Name: chr  "Orin NJ" "robert9861" "JQMusic1" "pdxpodcast" ...
##  $ App_Version: chr  "" "" "" "" ...
##  $ Rating     : num  5 5 5 5 5 5 5 5 5 5 ...
##  $ Review     : chr  "Great interviews with great artists e.g. Tom Bailey" "I, and you, will never encounter a more brilliant podcast about the music we love, than that offered by the two"| __truncated__ "I am so grateful to you both. As an aspiring songwriter, I have soaked in every bit of information from these p"| __truncated__ "Ad free, nicely produced & well researched.  A show for music lovers by music lovers!" ...
##  $ Date       : POSIXct, format: "2020-03-22 09:15:10" "2020-03-16 17:48:34" ...
##  $ country    : chr  "us" "us" "us" "us" ...

reviews %>% 
  ggplot() +
  aes(Date, nchar(Review)) +
  geom_jitter(aes(colour = country), size = 3) + 
  ggtitle(paste(podcast_name, "Reviews in the Apple Store", sep = "") +
  theme_minimal() +
  xlab("") +
  ylab("Length of review") +
  theme(legend.position="top") +
  scale_color_discrete(name = "Country of Reviewer")

write.csv(reviews, paste(podcase_file_name, "_apple_reviews.csv")
write_rds(reviews, paste(podcase_file_name, "_apple_reviews.rds")