Mastering the Tidyverse (Data Carpentry)
The tidyverse is essential for any statistician or data scientist who deals with data on a day-to-day basis. By focusing on small key tasks, the tidyverse suite of packages removes the pain of data manipulation. The tidyverse allows you to
- Import data from databases and data sources with ease
- Remove the pain of data cleaning
- Start understanding that data by transforming it, visualising it with imagery and modelling it
- Communicate your findings throughout your organisation securely and simply with apps, documents or plots
- Make business decisions based on accurate data
This training course covers key aspects of the tidyverse, including dplyr, lubridate, tidyr and tibbles.
Belfast, UK | September 2, 2019
Edinburgh, UK | October 25, 2019
The core tidyverse data structure is a tibble; this is a modern take on the data frame.
The course commences with a brief introduction to this structure.
- What are they?
- How do they differ from data frames?
- Convert a data frame to a tibble and back
- Why tibbles – motivation for the other tidyverse packages and sets the theme for the day
dplyr: the workhorse of the tidyverse
Before the first coffee break, we’ll tackle the dplyr package. This package forms the foundation of the tidyverse by providing a standardised data manipulation grammar.
- What is
- The grammar of tidyverse functions
summarise()– it may be that a review of boolean algebra is necessary at this point for subsetting
- The pipe operator
%>%and chaining functions into a workflow
- Some other useful dplyr functions
Your data should be tidy. An obvious statement, except what do we mean by tidy? This section will elucidate what we mean by tidy data and how to make it part of our workflow.
- Tidy data
- What is tidy data?
gather()for reshaping data
unite()for splitting data into one column or the reverse
- dealing with missing values
- Joins for dealing with data split across multiple data frames
In order to manipulate data, we need to be able to load data into R. We’ll cover the key packages and provide advice as required.
- Data storage: practical advice for managing data
- Tidyverse packages
readxlfor dealing with .csv and .xls/.xlsx files
- Database connections
- Non-tidyverse packages
- Not all data sets can be loaded using tidyverse packages
foreignpackage for reading data from other statistical systems (SAS, SPSS, Minitab)
We’ll finish the day by looking at common difficulties that may crop up in a data scientist’s day
- Dates/times with the
- String manipulation with
- First steps in regular expressions