Campbellsville University The Regression Line Analysis Questions Check attached Documents and ans below questionsDo you think the regression line sufficien

Campbellsville University The Regression Line Analysis Questions Check attached Documents and ans below questionsDo you think the regression line sufficiently captures the relationship between the two variables? What might you do differently? In the Iris slide example, how would you characterize the relationship between sepal width and sepal length? Did you notice the use of color in the Iris slide? Was it effective? Why or why not?2 pages, No plagarisam, 2 or 3 refererences Review of Basic Data Analytic Methods Using R
Copyright © 2014 EMC Corporation. All Rights Reserved.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
1
Module 3: Basic Data Analytic Methods Using R
1
Review of Basic Data Analytic Methods Using R
Analyzing and Exploring the Data
During this lesson the following topics are covered:
• Why visualize?
• Examining a single variable
• Examining pairs of variables
• Indications of dirty data.
• Data exploration vs. presentation
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
2
The topics for this lesson are listed.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
2
Why Visualize?
Summary statistics give us some sense of the data:
 Mean vs. Median.
 Standard deviation.
 Quartiles, Min/Max.
 Correlations between variables.
summary(data)
x
Min.
:-3.05439
1st Qu.:-0.61055
Median : 0.04666
Mean
:-0.01105
3rd Qu.: 0.56067
Max.
: 2.60614
y
Min.
:-3.50179
1st Qu.:-0.75968
Median : 0.07340
Mean
: 0.09383
3rd Qu.: 0.88114
Max.
: 4.28693
Visualization gives us
a more holistic sense
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
3
In the previous lesson, we saw how to examine data in R, including how to generate the
descriptive statistics: averages, data ranges, and quartiles (which are included in the summary()
report).
We also saw how to compute correlations between pairs of variables of interest. These
statistics do give us a sense of a data: an idea of its magnitude and range, and some obvious
dirty data (missing values, values with obviously wrong magnitude or sign).
Visualization, however, gives us a succinct, more holistic view of the data that we may not be
able to get from the numbers and summaries alone. It is an important facet of the initial data
exploration. Visualization helps you assess data cleanliness, and also gives you an idea of
potentially important relationships in the data before going on to build your models.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
3
Anscombe’s Quartet
4 data sets, characterized by the following. Are they the same, or are they different?
Property
Values
Mean of x in each case
9
Exact variance of x in each
case
11
Exact mean of y in each
case
7.5 (to 2 d.p)
Variance of Y in each case
4.13 (to 2 d.p)
Correlations between x
and y in each case
Linear regression line in
each case
0.816
Y = 3.00 +
0.500x (to 2
d.p and 3 d.p
resp.)
Copyright © 2014 EMC Corporation. All Rights Reserved.
i
X
y
10.00
8.04
8.00
6.95
13.00
7.58
9.00
8.81
11.00
8.33
14.00
9.96
6.00
7.24
4.00
4.26
12.00
10.84
7.00
4.82
5.00
5.68
ii
x
y
10.00
8.00
13.00
9.00
11.00
14.00
6.00
4.00
12.00
7.00
9.14
8.14
8.74
8.77
9.26
8.10
6.13
3.10
9.13
7.26
5.00
4.74
iii
x
y
10.00
8.00
13.00
9.00
11.00
14.00
6.00
4.00
12.00
7.00
7.46
6.77
12.74
7.11
7.81
8.84
6.08
5.39
8.15
6.42
5.00
5.73
iv
x
y
8.00
6.58
8.00
5.76
8.00
7.71
8.00
8.84
8.00
8.47
8.00
7.04
8.00
5.25
19.00 12.50
8.00
5.56
8.00
7.91
8.00
6.89
Module 3: Basic Data Analytic Methods Using R
4
Anscombe’s Quartet is a synthesized example by the statistician F. J. Anscombe. Look at the
properties and values of these four data sets. Based on standard statistical measures of mean,
variance, and correlation (our descriptive statistics), these data sets are identical. Or are they?
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
4
Moral: Visualize Before Analyzing!
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
5
However, if we visualize each data set using a scatterplot and a regression line superimposed
over each plot, the datasets appear quite different. Dataset 1 is the best candidate for a
regression line, although there is a lot of variation. Dataset 2 is definitely non-linear. Dataset 3
is a close match, but over predicts at higher value of x and has an extreme outlier. And Dataset
4 isn’t captured at all by a simple regression line.
Assuming we have datasets represented by data frames s1, s2, s3, and s4, we can generate
these plots in R by using the following code:
R-Code
plot(s1)
plot(lm(s1$y ~ s1$x))
…
(Yes, a loop is possible but requires more advanced data manipulation: for information, consult
the R “eval” function if interested). We also must take care to overwrite the preceding graph
in each instance.
Code to produce these graphs is included in the script AnscombePlot.R. Note that the dataset
for these plots are included in the standard R distribution. Type data() for a list of dataset
included in the base distribution. data(name) will make that dataset available in your
workspace.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
5
Visualizing Your Data
• Examining the distribution of a single variable
• Analyzing the relationship between two variables
• Establishing multiple pair wise relationships between variables
• Analyzing a single variable over time
• Data exploration versus data presentation
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
6
In a previous lesson, we’ve looked at how you can characterize your data by using traditional
statistics. But we also showed how datasets could appear identical when using descriptive
statistics, and yet look completely different when visualizing the data via a plot.
Using visual representations of data is the hallmark of exploratory data analysis: letting the data
speak to us rather than necessarily imposing an interpretation on the data a priori. In the rest
of this lesson, we are going to examine ways of displaying data so that we can better
understand the underlying distributions of a single variable or the relationships between two or
more variables.
Although data visualization is a powerful tool, the results we obtain may not be suitable when
it comes time for us to “tell a story” about the data. Our last slide will discuss what kind of
presentations are most effective.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
6
Examining the Distribution of a Single Variable
Graphing a single variable
• plot(sort(.)) – for low volume data
• hist(.) – a histogram
• plot(density(.)) – densityplot
 A “continuous histogram“
• Example
 Frequency table of household
income
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
7
R has multiple functions available to examine a single variable. Some of them are listed above.
See the R documentation for each of these. Some other useful functions are barplot() and
dotplot().
The example included is a frequency table of household income. We can certainly see a
concentration of households in the leftmost portion of the graph.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
7
Examining the Distribution of a Single Variable
Graphing a single variable
• plot(sort(.)) – for low volume data
• hist(.) – a histogram
• plot(density(.)) – densityplot
 A “continuous histogram“
• Example
 Frequency table of household
income
 rug() plot emphasizes
distribution
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
8
R has multiple functions available to examine a single variable. Some of them are listed above.
See the R documentation for each of these. Some other useful functions are barplot(),
dotplot() and stem().
The example included is a frequency table of log10 of household income. We can certainly see
a concentration of households in the rightmost portion of the graph. The rug() function creates
a 1-dimensional density plot as well: notice how it emphasizes the area under the curve.
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
8
What are we looking for?
A sense of the data range
• If it’s very wide, or very skewed, try computing the
log
Outliers, anomalies
• Possibly evidence of dirty data
Shape of the Distribution
• Unimodal? Bimodal?
• Skewed to left or right?
• Approximately normal? Approximately lognormal?
Example – Distribution of purchase size ($)
• Range from 0 to > $10K, right skewed
• Typical of monetary data
• Plotting log of data gives better sense of distribution
• Two purchasing distributions


~ $55
~ $2900
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R
9
When viewing the variables during the data exploration phase, you are looking for a sense of the data
range, and whether the values are strongly concentrated in a certain range. If the data is very skewed,
viewing the log of the data (if it’s all positive) can help you detect structure that you might otherwise
miss in a regularly scaled graph.
This is your chance to look for obvious signs of dirty data (outliers or unlikely looking values). See if
the data is unimodel or multimodal: that gives you an idea of how many distinct populations (with
distinct behavior patterns) might be mixed into your overall population. Knowing if the data is
approximately normal (or can be transformed to approximately normal – for example, by taking the log)
is important, since many modeling techniquest assume that the data is approximately normal in
distribution.
For our example, we can look at the densityplot of purchase sizes (in $ US) of customers at our online
retail site. The range here is extremely wide – from around $1 US to over $10,000 US. Extreme ranges
like this are typical of monetary data, like income, customer value, tax liabilities, bank account sizes, etc.
(In fact, all of this kind of data is often assumed to be distributed lognormally – that is, its log is a normal
distribution).
The data range makes it really hard for us to see much detail, so we take the log of it, and then density
plot it. Now we can see that there are (at least) two distinct population in our customer base: One
population that makes small to medium size purchases (median purchase size about $55 US) and one
that makes larger purchases (median purchase size about $2900 US). Can you see those two
populations in the top graph?
The plots shown were made using the lattice package. If the data is in the vector purchase_size, then
the lattice plot is:
library(lattice)
densityplot(purchase_size) # top plot
# bottom plot as log10 is actually
# easier to read, but this plot is in natural log
densityplot(log(purchase_size)
(The commands were actually more complicated than that, but these commands give the basic
equivalent)
Copyright © 2014 EMC Corporation. All rights reserved.
Module 3: Basic Data Analytic Methods Using R
9
Evidence of Dirty Data
Missing
values?
Mis-entered
data?
Inherited
accounts?
Copyright © 2014 EMC Corporation. All Rights Reserved.
Module 3: Basic Data Analytic Methods Using R 10
Here’s an example of how dirty data might manifest itself in your visualizations. We are looking
at the age distribution of account holders at our bank. Mean age is about 40, approximately
normally distributed with a standard deviation of about 15 years or so, which makes sense.
We see a few accounts with accountholder age