A tutorial on machine learning using Python to implement Bayesian classifier from zero

Naive Bayesian algorithm is simple and efficient, and it is one of the first methods to be considered in dealing with classification problems.

In this tutorial, you will learn the principles of the naive Bayesian algorithm and the progressive implementation of the Python version.

Update: See the following article on naive Bayesian usage tips "Better Naive bayes:12 tips to Get the most from the Naive Bayes algorithm"
Naive Bayesian classifier, Matt Buck retains part of the copyright
About Naive Bayes

Naive Bayesian algorithm is an intuitive method that uses each attribute to predict the probability of belonging to a class. You can use this supervised learning method to model A predictive modeling problem in probabilistic mode.

Given a class, naive Bayes assumes that each attribute's probability of attribution to this class is independent of all remaining attributes, simplifying the calculation of probabilities. This strong assumption produces a fast and effective method.

Given a property value, the probability of its belonging to a class is called the conditional probability. For a given class value, multiply the conditional probabilities of each attribute to get the probability that a data sample belongs to a class.

We can make predictions by calculating the probability of the sample being attributed to each class, and then choosing the class with the highest probability.

Generally, we use categorical data to describe naive Bayes, because it is easier to describe and calculate by ratios. A more useful algorithm that fits our purpose needs to support numeric attributes, while assuming that each numeric attribute obeys a normal distribution (distributed on a bell-shaped curve), which is a strong assumption, but still gives a robust result.
Predicting the onset of diabetes

The test question used in this article is "The Pima Indians Diabetes Problem".

The problem includes 768 medical observation details for the patients with Pima Indian, and records the instantaneous measurements described, taken from the age of the patient, the number of pregnancies and blood tests. All patients are women over 21 years old (including 21 years old), all attributes are numeric, and the units of the properties are different.

Each record belongs to a class, which indicates whether the patient is infected with diabetes within 5 years, as measured by time. If it is, then 1, otherwise 0.

This standard data set has been studied many times in the machine learning literature, with a good prediction accuracy of 70%-76%.

Here is a sample from the Pima-indians.data.csv file to find out what data we are going to use.

Note: Download the file and save it as a. csv extension (for example: pima-indians-diabetes.data.csv). View a description of all the attributes in the file.

6,148,72,35,0,33.6,0.627,50,11,85,66,29,0,26.6,0.351,31,08,183,64,0,0,23.3,0.672,32,11,89,66,23,94,28.1,0.167,21,00,137,4 0,35,168,43.1,2.288,33,1

Naive Bayesian Algorithm Tutorial

The tutorials are divided into the following steps:

1. Process data: load data from a CSV file and then divide it into training and test sets.

2. Extract Data features: extract the attribute characteristics of the training dataset so that we can calculate probabilities and make predictions.

3. Single prediction: use the characteristics of a dataset to generate a single forecast.

4. Multiple predictions: generate predictions based on a given test data set and a trained dataset that has extracted features.

5. Evaluation accuracy: Evaluate the predictive accuracy of the test data set as the predictive correctness rate.

6. Merge code: use all code to present a complete, independent implementation of the naïve Bayesian algorithm.

1. Processing data

Load the data file first. Data in CSV format does not have a header row and any quotation marks. We can open the file using the Open function in the CSV module and read the row data using the reader function.

We also need to convert the properties loaded in the string type into a number that we can use. The following is the Loadcsv () function used to load the horse Indian dataset (Pima Indians dataset).

Import csvdef loadcsv (filename):  lines = csv.reader (open (filename, "RB"))  DataSet = list (lines) for  i in Range (len (DataSet)):    dataset[i] = [Float (x) for x in Dataset[i]]  return dataset

We can test this function by loading the Pima Indians data set and then printing out the number of data samples.

filename = ' pima-indians-diabetes.data.csv ' DataSet = loadcsv (filename) print (' Loaded data file {0} with {1} rows '). Format (FileName, Len (DataSet))

Run the test and you will see the following results:

Loaded data file iris.data.csv with rows

Next, we divide the data into training datasets for naive Bayesian predictions, and test datasets to evaluate the accuracy of the model. We need to randomly divide the dataset into a training set containing 67% and a test set containing 33% (this is the usual ratio of the test algorithm on this dataset).

Here is the Splitdataset () function, which divides the data set at a given scale.

Import Randomdef Splitdataset (DataSet, Splitratio):  trainsize = Int (len (DataSet) * splitratio)  trainset = []  copy = List (DataSet) while  Len (trainset) < trainsize:    index = random.randrange (len (copy))    Trainset.append (Copy.pop (index))  return [trainset, copy]

We can define a dataset with 5 samples to test, first it is divided into training data sets and test data sets, and then print out to see which dataset each data sample eventually falls into.

DataSet = [[1], [2], [3], [4], [5]]splitratio = 0.67train, test = Splitdataset (DataSet, Splitratio) print (' Split {0} rows I Nto train with {1} and test with {2} '). Format (len (DataSet), train, test)

Run the test and you will see the following results:

Split 5 rows into train with [[4], [3], [5]] and test with [[1], [2]]

Extracting Data features

The naive Bayesian model contains the characteristics of the data in the training data set, and then uses this data feature to make predictions.

The characteristics of the training data collected contain the mean and standard deviation of each attribute relative to each class. For example, if there are 2 classes and 7 numeric attributes, then we need the mean and standard deviation of the combination of each attribute (7) and Class (2), which is 14 attribute characteristics.

These characteristics are used when calculating and predicting the probability of a particular attribute being attributed to each class.

We divide the acquisition of data features into the following sub-tasks:

Divide data by Category
Calculate mean value
Calculate standard deviation
Extracting data set features
Extract attribute characteristics by category

Divide data by Category

First, the samples in the training data set are divided by category, and then the statistics for each class are computed. We can create a category to the map that belongs to the sample list of this category and classify the samples from the entire dataset into the corresponding list.

The following separatebyclass () function can accomplish this task:

def separatebyclass (DataSet):  separated = {} for  I in range (len (DataSet)):    vector = dataset[i]    if ( Vector[-1] Not in separated):      separated[vector[-1]] = []    separated[vector[-1]].append (vector)  return Separated

As you can see, the function assumes that the last attribute in the sample (-1) is a category value, and returns a category value to the map of the Data sample list.

We can use some sample data to test the following:

DataSet = [[1,20,1], [2,21,0], [3,22,1]]separated = Separatebyclass (DataSet) print (' separated instances: {0} '). Format ( Separated)

Run the test and you will see the following results:

Separated instances: {0: [[2, 21, 0]], 1: [[1, 20, 1], [3, 22, 1]}

Calculate mean value

We need to calculate the mean value of each property in each class. The mean value is the midpoint or concentration trend of the data, and we use it as the median of the Gaussian distribution when calculating the probability.

We also need to calculate the standard deviation for each attribute in each class. The standard deviation describes the deviation of the data dispersion, and in calculating the probability, we use it to characterize the distribution of the desired dispersion of each attribute in the Gaussian distributions.

The standard deviation is the square root of the variance. Variance is the average of the squared deviations of each attribute value from the mean. Notice how we use the N-1 method (translator note: see unbiased estimates), which is the number of attribute values minus 1 when calculating variance.

Import mathdef mean (numbers):  return sum (numbers)/float (len (numbers)) def stdev (numbers):  avg = mean (numbers)  variance = SUM ([Pow (x-avg,2) for x in numbers])/float (len (Numbers)-1)  return math.sqrt (variance)

The function is tested by calculating the mean of the 5 numbers from 1 to 5.

numbers = [1,2,3,4,5]print (' Summary of {0}: Mean={1}, Stdev={2} '). Format (numbers, mean (numbers), STDEV (numbers))

Run the test and you will see the following results:

Summary of [1, 2, 3, 4, 5]: mean=3.0, stdev=1.58113883008

Extracting features from a dataset

Now we can extract the dataset features. For a given sample list (corresponding to a class), we can calculate the mean and standard deviation for each attribute.

The ZIP function groups data samples into lists by attributes, and then calculates the mean and standard deviation for each property.

Def summarize (DataSet):  summaries = [(mean (attribute), Stdev (attribute)) for attribute in Zip (*dataset)]  del SUMMARIES[-1]  return summaries

We can use some test data to test the summarize () function, and the test data shows a significant difference between the mean and standard deviation of the first and second data attributes.

DataSet = [[1,20,0], [2,21,1], [3,22,0]]summary = Summarize (DataSet) print (' Attribute summaries: {0} '). Format (Summary)

Run the test and you will see the following results:

Attribute summaries: [(2.0, 1.0), (21.0, 1.0)]

Extract attribute characteristics by category

Merging the code, we first divide the training data set by category and then calculate a summary of each attribute.

def summarizebyclass (DataSet):  separated = Separatebyclass (DataSet)  summaries = {}  for Classvalue, Instances in Separated.iteritems ():    Summaries[classvalue] = summarize (instances)  return summaries

Use a small test data set to test the Summarizebyclass () function.

DataSet = [[[1,20,1], [2,21,0], [3,22,1], [4,22,0]]summary = Summarizebyclass (DataSet) print (' Summary by class value: {0} ') . Format (Summary)

Run the test and you will see the following results:

Summary by Class value:{0: [(3.0, 1.4142135623730951), (21.5, 0.7071067811865476)],1: [(2.0, 1.4142135623730951), (21.0, 1.4142135623730951)]}

Forecast

We can now use the summaries obtained from the training data to make predictions. Making predictions involves calculating the probability that they belong to each class for a given sample of the data, and then selecting the class with the maximum probability as the result of the prediction.

We can divide this section into the following tasks:

Calculate Gaussian probability density function
Calculate the probability of the corresponding class
Single forecast
Evaluation accuracy

Calculate Gaussian probability density function

Given the mean and standard deviation of a known attribute from the training data, we can use the Gaussian function to evaluate the probability of a given property value.

Given the attribute characteristics of each property and class value, the conditional probability of the given property value can be obtained under the condition of the class value.

For the Gaussian probability density function, you can view the reference literature. In short, we need to incorporate the known details into the Gaussian function (attribute value, mean, standard deviation) and get the likelihood that the attribute value belongs to a class (the translator notes: the possibility).

In the Calculateprobability () function, we first calculate the exponential part and then calculate the backbone of the equation. This can be well organized into 2 rows.

Import Mathdef calculateprobability (x, Mean, STDEV):  exponent = math.exp (-(Math.pow (x-mean,2)/(2*math.pow (Stdev), 2)))  return (1/(MATH.SQRT (2*MATH.PI) * stdev)) * exponent

Use some simple data to test the following:

x = 71.5mean = 73stdev = 6.2probability = calculateprobability (x, mean, Stdev) print (' Probability of belonging to this Clas S: {0} '). Format (probability)

Run the test and you will see the following results:

Probability of belonging to this class:0.0624896575937

Calculate the probability of the owning class

Now that we can calculate the probability that an attribute belongs to a class, combine the probabilities of all the attributes in a data sample, and finally get the probability that the entire data sample belongs to a class.

Using multiplication to combine probabilities, in the following calculclassprobilities () function, given a data sample, the probability that it belongs to each category can be obtained by multiplying its attribute probabilities. The result is a class-value-to-probability mapping.

def calculateclassprobabilities (summaries, inputvector):  probabilities = {}  for Classvalue, classsummaries in Summaries.iteritems ():    Probabilities[classvalue] = 1    for i in range (len (classsummaries)):      mean, Stdev = Classsummaries[i]      x = inputvector[i]      probabilities[classvalue] *= calculateprobability (x, mean, STDEV)  return probabilities

Test the Calculateclassprobabilities () function.

summaries = {0:[(1, 0.5)], 1:[(5.0)]}inputvector = [1.1, '? '] probabilities = calculateclassprobabilities (summaries, inputvector) print (' probabilities for each class: {0} '). Format ( Probabilities)

Run the test and you will see the following results:

Probabilities for each class: {0:0.7820853879509118, 1:6.298736258150442e-05}

Single forecast

Now that we can calculate the probability that a data sample belongs to each class, we can find the maximum probability value and return the associated class.

The following predict () function can accomplish the above tasks.

def predict (summaries, inputvector):  probabilities = calculateclassprobabilities (summaries, inputvector)  Bestlabel, Bestprob = none, 1 for  classvalue, probability in Probabilities.iteritems ():    if Bestlabel is None or P Robability > Bestprob:      bestprob = probability      Bestlabel = classvalue  return Bestlabel

The test predict () function is as follows:

summaries = {' A ': [(1, 0.5)], ' B ': [(c, 5.0)]}inputvector = [1.1, '? '] result = predict (summaries, inputvector) print (' prediction: {0} '). Format (Result)

Run the test and you will get the following result:

Prediction:a

Multiple predictions

Finally, we can evaluate the model accuracy by predicting each data sample in the test data set. The GetPredictions () function can implement this function and return a list of predictions for each test sample.

def getpredictions (summaries, testset):  predictions = [] for  I in range (len (testset)):    result = Predict ( Summaries, testset[i])    predictions.append (Result)  return predictions

The test getpredictions () function is as follows.

summaries = {' A ': [(1, 0.5)], ' B ': [(c, 5.0)]}testset = [[1.1, '? '], [19.1, '? ']] predictions = getpredictions (summaries, testset) print (' predictions: {0} '). Format (predictions)

Run the test and you will see the following results:

Predictions: [' A ', ' B ']

Calculation accuracy

The predicted values are compared with the category values in the test data set, and you can calculate a precision that is between the 0%~100% accuracy and the classification. The Getaccuracy () function calculates this exact rate.

def getaccuracy (Testset, predictions):  correct = 0 for  x in range (len (testset)):    if testset[x][-1] = = PREDICTIONS[X]:      correct + = 1  return (Correct/float (Len (testset))) * 100.0

We can use the following simple code to test the Getaccuracy () function.

Testset = [[1,1,1, ' a '], [2,2,2, ' a '], [3,3,3, ' b ']]predictions = [' A ', ' a ', ' a ']accuracy = Getaccuracy (Testset, predictions ) Print (' accuracy: {0} '). Format (accuracy)

Run the test and you will get the following result:

accuracy:66.6666666667

Merge code

Finally, we need to make the code coherent.

Here is the full code for the progressive implementation of the naive Bayesian python version.

# Example of Naive Bayes implemented from Scratch on Pythonimport csvimport randomimport math def loadcsv (filename): line  s = csv.reader (open (filename, "RB")) DataSet = list (lines) for I in Range (len (DataSet)): dataset[i] = [Float (x) for X In Dataset[i]] return DataSet def splitdataset (DataSet, splitratio): trainsize = Int (len (DataSet) * Splitratio) TrainS ET = [] copy = List (DataSet) while Len (trainset) < Trainsize:index = Random.randrange (len (copy)) Trainset.appe    nd (Copy.pop (index)) return [trainset, copy] def separatebyclass (DataSet): separated = {} for I in range (len (DataSet)): Vector = Dataset[i] if (vector[-1] not in separated): separated[vector[-1]] = [] Separated[vector[-1]].appen D (Vector) return separated def mean (numbers): return sum (Numbers)/float (len (numbers)) def stdev (numbers): avg = Mean (nu mbers) Variance = SUM ([Pow (x-avg,2) for x in numbers])/float (len (Numbers)-1) return math.sqrt (variance) def summarize (da Taset): summaries = [(MEan (attribute), Stdev (attribute)) for attribute in Zip (*dataset)] del Summaries[-1] return summaries def Summarizebyclas    S (DataSet): separated = Separatebyclass (DataSet) summaries = {} for Classvalue, instances in Separated.iteritems (): Summaries[classvalue] = summarize (instances) return summaries def calculateprobability (x, Mean, STDEV): exponent = Math. Exp (-(Math.pow (x-mean,2)/(2*math.pow (stdev,2))) return (1/(MATH.SQRT (2*MATH.PI) * stdev)) * Exponent def CALCULATECLA    Ssprobabilities (summaries, inputvector): probabilities = {} for Classvalue, classsummaries in Summaries.iteritems (): Probabilities[classvalue] = 1 for i in range (len (classsummaries)): mean, Stdev = classsummaries[i] x = INPUTV Ector[i] Probabilities[classvalue] *= calculateprobability (x, mean, Stdev) return probabilities def predict (Summarie  S, inputvector): probabilities = calculateclassprobabilities (summaries, inputvector) Bestlabel, BestProb = None, 1 for Classvalue, probabilityIn Probabilities.iteritems (): If Bestlabel is None or probability > bestprob:bestprob = probability bestl Abel = classvalue return Bestlabel def getpredictions (summaries, testset): predictions = [] for i in range (len (testset) ): result = predict (summaries, testset[i]) predictions.append (result) return predictions def getaccuracy (Testset, p  redictions): correct = 0 for I in range (len (testset)): if testset[i][-1] = = Predictions[i]: correct + = 1 return (Correct/float (Len (testset))) * 100.0 def main (): filename = ' pima-indians-diabetes.data.csv ' splitratio = 0.67 datase t = loadcsv (filename) trainingset, Testset = Splitdataset (DataSet, Splitratio) print (' Split {0} rows into Train={1} and test={2} rows '). Format (len (DataSet), Len (Trainingset), Len (Testset)) # Prepare model summaries = Summarizebyclass (train Ingset) # test Model predictions = getpredictions (summaries, testset) accuracy = getaccuracy (testset, predictions) pri NT (' accuracy: {0}% '). Format (accuracy) main () 

Run the sample to get the following output:

Split 768 rows into train=514 and test=254 rowsaccuracy:76.3779527559%

