Naive Bayes

Machine Learning

P (B ∣ E) = P (B) \times \frac{P ( E ∣ B )}{P ( E )}

E: Evidence
B: Belief
P(B): Prior probability
P(E): Marginal probability
P(E|B): Likelihood
P(B|E): Posterior probability

If we have $n$ beliefs

P (B_{i} ∣ E) = \frac{P ( B _{i} ) P ( E ∣ B _{i} )}{\sum _{j = 1}^{n} P ( E ∣ B _{j} ) P ( B _{j} )}

If we have $d$ evidences, i.e. $E = {e_{1}, e_{2}, e_{3}, \dots, e_{d}}$

P (B_{i} ∣ E) = \frac{P ( B _{i} ) P (( e _{1} , e _{2} , e _{3} , \dots , e _{d} ) ∣ B _{i} )}{\sum _{j = 1}^{n} P (( e _{1} , e _{2} , e _{3} , \dots , e _{d} ) ∣ B _{j} ) P ( B _{j} )}

Assume each evidence makes an independent and equal contribution to the belief, therefore

P (B_{i} ∣ E) = \frac{P ( B _{i} ) P ( e _{1} ∣ B _{i} ) P ( e _{2} ∣ B _{i} ) P ( e _{3} ∣ B _{i} ) \dots P ( e _{d} ∣ B _{i} )}{\sum _{j = 1}^{n} P ( e _{1} ∣ B _{j} ) P ( e _{2} ∣ B _{j} ) P ( e _{3} ∣ B _{j} ) \dots P ( e _{d} ∣ B _{j} ) P ( B _{j} )}

B_{NB} = argmax_{B_{i}} P (B_{i}) (P (e_{1} ∣ B_{i}) P (e_{2} ∣ B_{i}) \dots P (e_{d} ∣ B_{i}))

P (e_{j} = v_{j} ∣ B_{i} = c_{i}) = \frac{( Count of e _{j} = v _{j} AND B _{i} = c _{i} ) + α}{(Count of B _{i} = c _{i} ) + n \times α}

$n$ is the number of possible values of $e_{j}$

f (x) = \frac{1}{2 π σ} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

B_{NB} = argmax_{B_{i}} (lo g P (B_{i}) + n = 1 \sum d lo g P (e_{n} ∣ B_{i}))

Assumed features are independent
Precision low when dataset is small
Doing regression will provide biased results (probability instead of classification)
Zero Frequency Problem and α-Laplace Smoothing