Neural Network

Traditional Neural Network

$$\begin{array}{rl}& w_i=w_i+\alpha (d-y)x_i\\ & b=b+\alpha (d-y)\end{array}$$

$\alpha$	Learning rate
$d$	Desired output
$y$	Output

With one neuron, we can only solve linear separable problems.
With multi-layer perceptron (MLP), we can solve linear separable problems and non-linear separable problems. It is also an universal approximator, which can model all types of function.

Recurrent Neural Network

• Capture the dependency in data (e.g. temporal data)
• Input $x_t$, internal state variable $s_t$, output $y_t$

Types

• Basic RNN
  • Only 1 activation function
  • Too simple
  • Not easy to converge (take long time to train)
• Traditional LSTM
  • Forget, input, output feature
  • Greater power of capturing the properties in the data
  • Remember longer sequences
• GRU
  • Forget, input, output feature
  • Simpler than LSTM
    • Training time shorter
    • Require fewer data points to capture the properties of the data