Lecture 3 | Basics of Data Analysis I
Max Pellert
IS 616: Large Scale Data Analysis and
Visualization
Aim
These course units are intended as a supplement to your actual work
with data
It wants to teach you some tricks that are often not taught
🔨🧰🪛
Some Caveats
Don’t expect a full-fledged course that answers it all for you
That also doesn’t fit the subject matter
Data science is more like dentistry than particle physics
But, the aim is to bring everybody to the same level to be able to
actually do visualizations (while at the same time also providing
content that very likely also the more advanced student also haven’t
heard yet)
It should convey some of the (softer) skills that you actually need
often
“It is often said that 80% of data analysis is spent on the process
of cleaning and preparing the data (Dasu and Johnson 2003).”
Wickham, 2014
What makes a data set tidy?
“each variable is a column”
“each observation is a row”
“each type of observational unit is a table” (also called data frame
or data table)
“data tidying: structuring datasets to facilitate analysis”
It provides a “philosophy of data”
What makes a data set untidy?
Generally, data sets can be constructed in all bizarre ways
imaginable
![]()
Create and use tidy data also in the interest of
reproducibility and open science (think of git too!)
library(data.table)
DT = as.data.table(iris)
head(iris)
## Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1 5.1 3.5 1.4 0.2 setosa
## 2 4.9 3.0 1.4 0.2 setosa
## 3 4.7 3.2 1.3 0.2 setosa
## 4 4.6 3.1 1.5 0.2 setosa
## 5 5.0 3.6 1.4 0.2 setosa
## 6 5.4 3.9 1.7 0.4 setosa
# FROM[WHERE, SELECT, GROUP BY]
# DT [i, j, by]
DT[Petal.Width > 1.0, mean(Petal.Length), by = Species]
## Species V1
## 1: versicolor 4.362791
## 2: virginica 5.552000
# Species V1
#1: versicolor 4.362791
#2: virginica 5.552000
