Practical Introduction to Web Scraping in R — NBA Players

Understanding the basics of web scraping, when and why it is useful, and how to implement it in R

Data does not always come in a neat tabular format that we can readily use to analyze and derive insights from. Sometimes we may find ourselves in situations where we need to extract data from alternative sources, for example, the internet.

Some companies even use web scraping to gain an advantage over their competitors by accessing data that most people overlook. When used right, web scraping enables us to obtain data from any website and transform it into a usable form in order to supplement an analysis or project.

Web scraping is the method of extracting data from websites in an automated way. It saves the trouble of having to manually download or copy data but instead automates the entire process from start to finish.

Some interesting use cases of web scraping include analyzing customer or product reviews, getting real-time ticket prices for flights or hotels, or even aggregating open job postings.

There are many existing packages in common programming languages that are built for the purpose of parsing HTML documents. For instance, Beautiful Soup in Python is widely used for web scraping.

However, for today’s exercise, we will be using rvest, a package in R used for harvesting data from web pages. Furthermore, we will learn how to apply web scraping techniques to obtain information about current players in the National Basketball Association (NBA) from the ESPN website.

Player profile

An important thing to keep in mind when performing web scraping is to check for consistency across the web pages that you intend on scraping. Specifically, the only way that we can automate scraping across multiple pages is if there is a pattern that occurs in how the data is structured.

For example, let’s take a closer look at the roster page for the Boston Celtics and the New York Knicks.

As you can see, both teams have their roster set up in a table format which contains information about each player’s name, position, age, height, weight, college, and annual salary. In addition, you might also notice the end of the URL changes according to the name of the team: ../bos/boston-celtics for the Boston Celtics and ../ny/new-york-knicks for the New York Knicks. This will come in handy later.

Before we look at how to scrape data for each NBA team, let’s only focus on one team for now. Once we succeed, we can easily write a for loop to repeat the whole process for the other teams.

Traditionally, in order to web scrape, you would ideally have some basic knowledge of HTML and CSS, which are the two fundamental technologies for building and designing web pages.

HTML is used to create content and provide structure to a web page. CSS, on the other hand, is used for design, layout, and styling to make the web page appear more slick and presentable.

We are going slightly cheat around this process via a Chrome extension called SelectorGadget, which you can add to your browser and immediately start using. It helps to generate the CSS selector of an element on a web page by simply highlighting the element that you are interested in.

Here, we are going to highlight the elements that contain each player’s name, position, age, height, weight, college, and salary for the Boston Celtics and subsequently store them in their respective variables. Then, we will collate them together to create a data frame.

Player regular season statistics

Let’s take our scraping one step further, shall we?

You may have noticed the hyperlink on each player’s name that when clicked on will bring you to a separate web page containing the individual player’s performance in the most recent NBA regular season.

For example, let’s look at Jayson Tatum and Derrick Rose.

As part of this exercise, suppose we would like to combine this information with our original data frame from earlier on.

To do this, we will have to write a function that fetches the season statistics for each player on the team roster.

Automate for all NBA teams

Now that we understand how the process works for the Boston Celtics, we can repeat it for all 30 NBA teams.

We simply need to use a for loop to change the end of the URL name for each team. The whole process took my computer close to 9 minutes to complete. Not bad.

After some further data cleaning, like changing player height and weight to centimetres and kilograms respectively (I’m not the biggest fan of the imperial system), data types, and renaming values, we get a final data frame that looks like the following.

Bonus: exploratory data analysis

This section of the blog post is more so for my own entertainment than it is about web scraping. Effectively, I have used some basic data visualizations to gather insights from the data we have prepared about current NBA players.

While it is not the most groundbreaking analysis, I have nevertheless included my comments in the caption for each of the charts below.

Thank you for reading, and I hope that you took away some new knowledge about the basics of web scraping in R. I also encourage you to take a look at the workbook for this exercise on my GitHub here. It contains all the code for this project from start to end.

If you found any value from this article and are not yet a Medium member, it would mean a lot to me as well as the other writers on this platform if you sign up for membership using the link below. It encourages us to continue putting out high-quality and informative content just like this one — thank you in advance!

Don’t know what to read next? Here are some suggestions.