Monroe College Chapter 5 Association Rules for Buying Data Questions Use lecture notes and read carefully chapter 5, use the enclosed lab to sharpen your s

Monroe College Chapter 5 Association Rules for Buying Data Questions Use lecture notes and read carefully chapter 5, use the enclosed lab to sharpen your skills in WEKA and learn how to do end of the chapter problems on pg 124 problems 8-19. 5
Classification
Chapter Objectives
To comprehend the concept, types and working of classification
To identify the major differences between classification and regression problems
To become familiar about the working of classification
To introduce the decision tree classification system with concepts of information gain and
Gini Index
To understand the workings of the Naïve Bayes method
5.1 Introduction to Classification
Nowadays databases are used for making intelligent decisions. Two forms of data analysis namely
classification and regression are used for predicting future trends by analyzing existing data.
Classification models predict discrete value or class, while Regression models predict a continuous
value. For example, a classification model can be built to predict whether India will win a cricket
match or not, while regression can be used to predict the runs that will be scored by India in a
forthcoming cricket match.
Classification is a classical method which is used by machine learning researchers and statisticians
for predicting the outcome of unknown samples. It is used for categorization of objects (or things)
into given discrete number of classes. Classification problems can be of two types, either binary
or multiclass. In binary classification the target attribute can only have two possible values. For
example, a tumor is either cancerous or not, a team will either win or lose, a sentiment of a sentence
is either positive or negative and so on. In multiclass classification, the target attribute can have
more than two values. For example, a tumor can be of type 1, type 2 or type 3 cancer; the sentiment
of a sentence can be happy, sad, angry or of love; news stories can be classified as weather, finance,
entertainment or sports news.
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
66 Data Mining and Data Warehousing
Some examples of business situations where the classification technique is applied are:
• To analyze the credit history of bank customers to identify if it would be risky or safe to grant
them loans.
• To analyze the purchase history of a shopping mall’s customers to predict whether they will
buy a certain product or not.
In first example, the system will predict a discrete value representing either risky or safe, while
in second example, the system will predict yes or no.
Some more examples to distinguish the concept of regression from classification are:
• To predict how much a given customer will spend during a sale.
• To predict the salary-package of a student that he/she may get during his/her placement.
In these two examples, there is a prediction of continuous numeric value. Therefore, both are
regression problems.
In this chapter, we will discuss the basic approaches to perform the classification of data.
5.2 Types of Classification
Classification is defined as two types. These are:
• Posteriori classif ication
• Priori classif ication
5.2.1
Posteriori classification
The word ‘Posteriori’ means something derived by reasoning from the observed facts. It is a supervised
machine learning approach, where the target classes are already known, i.e., training data is already
labeled with actual answers.
5.2.2
Priori classification
The word ‘Priori’ means something derived by reasoning from self-evident propositions. It is an
unsupervised machine learning approach, where the target classes are not given. The question is
‘Is it possible to make predictions, if labeled data is not available?’
The answer is yes. If data is not labeled, then we can use clustering (unsupervised technique) to
divide unlabeled data into clusters. Then these clusters can be assigned some names or labels and
can be further used to apply classification to make predictions based on this dataset. Thus, although
data is not labeled, we can still make predictions based on data by first applying clustering followed
by classification. This approach is known as Priori classification.
In this chapter, we will learn posteriori classification, a supervised machine learning approach
to predict the outcome when training dataset is labeled.
5.3 Input and Output Attributes
In any machine learning approach the input data is very important. Data contains two types of
attributes, namely, input attributes and output attributes. The class attribute that represents the
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
Classification 67
output of all other attributes is known as an output attribute or dependent attribute, while all other
attributes are known as input attributes or independent attributes. For example, the dataset for iris
plant’s flowers given in Figure 5.1 has Sepal length, Sepal width, Petal length and Petal width as
input attributes and species as an output attribute.
Figure 5.1
Input and output attributes
The attributes can be of different types. The attributes having numbers are called numerical
attributes while attributes whose domain is not numerical are known as nominal or categorical
attributes. Here, input attributes are of numerical type while species, i.e., the output attribute is of
nominal type (as shown in Figure 5.1).
During classification, it is important to have a database of sufficient size for training the model
accurately. The next section covers how a classification model is built.
5.4 Working of Classification
Classification is a two-step process. The first step is training the model and the second step is testing
the model for accuracy. In the first step, a classifier is built based on the training data obtained by
analyzing database tuples and their associated class labels. By analyzing training data, the system
learns and creates some rules for prediction. In the second step, these prediction rules are tested on
some unknown instances, i.e., test data. In this step, rules are used to make the predictions about
the output attribute or class. In this step, the predictive accuracy of the classifier is calculated. The
system performs in an iterative manner to improve its accuracy, i.e., if accuracy is not good on test
data, the system will reframe its prediction rules until it gets optimized accuracy on test data as
shown in Figure 5.2.
The test data is randomly selected from the full dataset. The tuples of the test data are independent
of the training tuples. This means that the system has not been exposed to testing data during the
training phase. The accuracy of a classifier on a given test data is defined as the percentage of test
data tuples that are correctly classified by the classifier. The associated class label of each test tuple is
compared with the class prediction made by the classifier for that particular tuple. If the accuracy of
the classifier is satisfactory then it can be used to classify future data tuples with unknown class labels.
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
68 Data Mining and Data Warehousing
Figure 5.2
Training and testing of the classifier
For example, by analyzing the data of previous loan applications as shown in Figure 5.3, the
classification rules obtained can be used to approve or reject the new or future loan applicants as
shown in Figure 5.4.
Figure 5.3
Building a classifier to approve or reject loan applications
Figure 5.4
Predicting the type of customer based on trained classifier
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
Classification 69
The same process of training and testing the classifier has also been illustrated in Figure 5.5.
Figure 5.5
Training and testing of the classifier
5.5 Guidelines for Size and Quality of the Training Dataset
There should be a balance between the number of training samples and independent attributes. It has
been observed that generally, the number of training samples required is likely to be relatively small
if the number of independent or input attributes is small and similarly, number of training samples
required is likely to be relatively large if the number of independent or input attributes is large. The
quality of the classifier depends upon the quality of the training data. If there are two or more than
two classes, then sufficient training data should be available belonging to each of these classes.
Researchers have developed a number of classifiers that include: Decision Tree, Naïve Bayes,
Support Vector Machine and Neural Networks. In this chapter, two important classification methods,
namely, Decision Tree and Naive Bayes are discussed in detail, as these are widely used by scientists
and organizations for classification of data.
5.6 Introduction to the Decision Tree Classifier
In the decision tree classifier, predictions are made by using multiple ‘if…then…’ conditions which
are similar to the control statements in different programming languages, that you might have learnt.
The decision tree structure consists of a root node, branches and leaf nodes. Each internal node
represents a condition on some input attribute, each branch specifies the outcome of the condition
and each leaf node holds a class label. The root node is the topmost node in the tree.
The decision tree shown in Figure 5.6 represents a classifier tasked for predicting whether a
customer will buy a laptop or not. Here, each internal node denotes a condition on the input attributes
and each leaf node denotes the predicted outcome (class). By traversing the decision tree, one can
analyze that if a customer is middle aged then he will probably buy a laptop, if a customer is young
and a student then he will probably not buy a laptop. If a customer is a senior citizen and has an
excellent credit rating then he can probably buy a laptop. The system makes these predictions with
a certain level of probability.
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
70 Data Mining and Data Warehousing
Figure 5.6
Decision tree to predict whether a customer will buy a laptop or not
Decision trees can easily be converted to classification rules in the form of if-then statements.
Decision tree based classification is very similar to a ‘20 questions game’. In this game, one player
writes something on a page and other player has to find what was written by asking at most 20
questions; the answers to which can only be yes or no. Here, each node of the decision tree denotes
a choice between numbers of alternatives and the choices are binary. Each leaf node specifies a
decision or prediction. The training process that produces this tree is known as induction.
5.6.1
Building decision tree
J. Ross Quinlan, a researcher in machine learning, developed a decision tree algorithm known as
ID3 (Iterative Dichotomiser) during the late 1970s and early 1980s. Quinlan later proposed C4.5
(a successor of ID3), which became a benchmark to which newer supervised learning algorithms are
often compared. Decision tree is a common machine learning technique which has been implemented
in many machine learning tools like Weka, R, Matlab as well as some programming languages such
as Python, Java, etc.
These algorithms are based on the concept of Information Gain and Gini Index. So, let us first
understand the role of information gain in building the decision tree.
5.6.2
Concept of information theory
Decision tree algorithm works on the basis of information theory. It has been observed that
information is directly related with uncertainty. If there is uncertainty then there is information and
if there is no uncertainty then there is no information. For example, if a coin is biased having a head
on both sides, then the result of tossing it does not give any information but if a coin is unbiased
having a head and a tail then the result of the toss provides some information.
Usually the newspaper carries the news that provides maximum information. For example,
consider the case of an India-UAE world cup cricket match. It appears certain that India will beat
UAE, so this news will not appear on front page as main headlines, but if UAE beats India in a
world cup cricket match then this news being very unexpected (uncertain) will appear on the first
page as headlines.
Let us consider another example, if in your university or college, there is holiday on Sunday then a
notice regarding the same will not carry any information (because it is certain) but if some particular
Sunday becomes a working day then it will be information and henceforth becomes a news.
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
Classification 71
From these examples we can observe that information is related to the probability of occurence
of an event. Another important question to consider is, whether the probability of occurrence of an
event is more. Then, the information gain will be more frequent or less frequent?
It is certain from above examples that ‘more certain’ events such as India defeating UAE in
cricket or Sunday being a holiday carry very little information. But if UAE beats India or Sunday
is working, then even though the probability of these events is lesser than the previous event, it will
carry more information. Hence, less probability means more information.
5.6.3
Defining information in terms of probability
Information theory was developed by Claude Shannon. Information theory defines entropy which
is average amount of information given by a source of data. Entropy is measured as follows.
entropy (p1, p2, …, pn) = – p1 log(p1) – p2 log(p2) – … – pn log(pn)
Therefore, the total information for an event is calculated by the following equation:
I = ∑ ( − pi log pi )
i
In this, information is defined as -pi log pi where pi is the probability of some event. Since,
probability pi is always less than 1, log pi is always negative; thus negating log pi we get the overall
information gain (–pi log pi) as positive.
It is important to remember that the logarithm of any number greater than 1 is always positive
and the logarithm of any number smaller than 1 is always negative. Logarithm of 1 is always zero,
no matter what the base of logarithm is. In case of log with base 2, following are some examples.
log2(2) = 1
log2(2n) = n
log2(1/2) = -1
log2(1/2n) = -n
Let us calculate the information for the event of throwing a coin. It has two possible values, i.e.,
head (p1) or tail (p2). In case of unbiased coin, the probability of head and tail is 0.5 respectively.
Thus, the information is
I = -0.5 log(0.5) – 0.5 log(0.5)
= – (0.5) * (-1) – (0.5) * (-1)
[As, log2(0.5) = -1]
= 0.5 + 0.5=1
The result is 1.0 (using log base 2) and it is the maximum information that we can have for an
event with two possible outcomes. This is also known as entropy.
But if the coin is biased and has heads on both the sides, then probability for head is 1 while the
probability of tails will be 0. Thus, total information in tossing this coin will be as follows.
I = -1 log(1) – 0 log(0) = 0
[As, log2(1) = 0]
You can clearly observe that tossing of biased coin carries no information while tossing of unbiased
coin carries information of 1.
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
72 Data Mining and Data Warehousing
Suppose, an unbiased dice is thrown which has six possible outcomes with equal probability,
then the information is given by:
I = 6(-1/6) log(1/6) = 2.585
[the probability of each possible outcome is 1/6 and there are in
total six possible outcomes from 1 to 6]
But, if dice is biased such that there is a 50% chance of getting a 6, then the information content
of rolling the die would be lower as given below.
I = 5(-0.1) log(0.1) – 0.5 log(0.5) = 2.16
[One event has a probability of 0.5 while 5 other events has probability of 0.5,
which makes 0.5/5 = 0.1 as the probability of each of remaining 5 events.]
And if the dice is further biased such that there is a 75% chance of getting a 6, then the information
content of rolling the die would be further low as given below.
I = 5(-0.05) log(0.05) – 0.75 log(0.75) = 1.39
[One event has a probability of 0.75 while 5 other events has probability of 0.25,
which makes 0.25/5 = 0.05 as probability of each of remaining 5 events.]
We can observe that as the certainty of an event goes up, the total information goes down.
Information plays a key role in selecting the root node or attribute for building a decision tree.
In other words, selection of a split attribute plays an important role. Split attribute is an attribute
that reduces the uncertainty by largest amount, and is always accredited as a root node. So, the
attribute must distribute the objects such that each attribute value results in objects that have as
little uncertainty as possible. Ideally, each attribute value should provide us with objects that belong
to only one class and therefore have zero information.
5.6.4
Information gain
Information gain specifies the amount of information that is gained by knowing the value of the
attribute. It measures the ‘goodness’ of an input attribute for predicting the target attribute. The
attribute with the highest information gain is selected as the next split attribute.
Mathematically, it is defined as the entropy of the distribution before the split minus the entropy
of the distribution after split.
Information gain = (Entropy of distribution before the split)
– (Entropy of distribution after the split)
The largest information gain is equivalent to the smallest entropy or minimum information. It
means that if the result of an event is certain, i.e., the probability of an event is 1 then information
provided by it is zero while the information gain will be the largest, thus it should be selected as a
split attribute.
Assume that there are two classes, P and N, and let the set of training data S (with a total number
of records s) contain p records of class P and n records of class N. The amount of information is
defined as
I = – (p/s)log(p/s) – (n/s)log(n/s)
Downloaded from https://www.cambridge.org/core. University of Sussex Library, on 01 May 2019 at 23:39:50, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/9781108635592.006
Classification 73
Obviously if p = n, i.e., the probability is equally distributed then I is equal to 1 and if p = s or n =
0, i.e., training data S contains all the elements of a single class only, then I = 0. Therefore if there
was an attribute for which all the records had the same value (for example, consider the attribute
gender, when all people are male), using this attribute would lead to no information gain that is, no
reduction in uncertainty. On the other hand, if an attribute divides the t…
Purchase answer to see full
attachment