🧠 Neural Network From Scratch

       

A comprehensive implementation of various neural network architectures built from scratch in Python, using only NumPy for computational operations. This project aims to provide educational implementations of fundamental neural network algorithms with a clean, builder-pattern API.

📋 Table of Contents

• Project Structure
• Installation
• Implemented Algorithms
  • Feedforward Neural Networks
  • Perceptron
  • MADALINE
• Examples
• Extensibility
• Visualization
• Limitations
• Contributing
• License

📁 Project Structure

nn-from-scratch/
├── src/
│   ├── neural/             # Feedforward Neural Networks
│   │   ├── core/
│   │   │   ├── activations.py
│   │   │   ├── costs.py
│   │   │   ├── initializers.py
│   │   │   └── layers.py
│   │   ├── network.py
│   │   ├── factory.py
│   │   └── builder.py
│   ├── perceptron/         # Perceptron Networks
│   │   ├── core/
│   │   │   └── activations.py
│   │   ├── network.py
│   │   ├── factory.py
│   │   └── builder.py
│   └── madaline/           # MADALINE Networks
│       ├── core/
│       │   └── trainer.py
│       ├── network.py
│       ├── factory.py
│       └── builder.py
└── README.md         # Project Documentation

---

🚀 Installation

Installation Steps

1. Clone the repository:

   git clone https://github.com/your-username/nn-from-scratch.git
   cd nn-from-scratch

2. Install dependencies:

   pip install -r requirements.txt

   Note: The project uses numpy, scikit-learn, and plotly for numerical computations and visualizations.

---

🧮 Implemented Algorithms

1. Feedforward Neural Networks

Implementation of the Multi-layer neural networks with backpropagation training.

✨ Main Features

• Customizable Neural Network Architecture: Add layers with different activation functions and weight initializers.
• Cost Functions: Support for MSE, MAE, and Huber loss functions.
• Activation Functions: Includes Sigmoid, ReLU, Leaky ReLU, and Linear activations.
• Weight Initializers: Uniform and Nguyen-Widrow initializers.
• Training Features: Early stopping based on error thresholds and patience.
• Visualization Tools: Plot loss curves, decision boundaries, and confusion matrices.
• Use Cases: Supports binary classification, multi-class classification, and regression tasks.
• Builder pattern API: Convenient for intuitive network construction

📝 Usage

Code Examples

1.1 Building the Model

Use the NeuralNetworkFactory to create a neural network:

from src.neural.factory import NeuralNetworkFactory

model = (
   NeuralNetworkFactory.create(input_size=2)
   .add_layer(n_units=20, activation="sigmoid", initializer="nguyen_widrow")
   .add_layer(n_units=20, activation="sigmoid", initializer="nguyen_widrow")
   .add_layer(n_units=1, activation="sigmoid")
   .compile(learning_rate=0.2, cost_function="mse")
   .build()
)

1.2 Training the Model

Train the model using the train method:

inputs = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
targets = np.array([[0], [1], [1], [0]])

model.train(inputs, targets, epochs=5000)

1.3 Visualizing Results

Plot the loss curve:

from src.core.visualizers import Visualizer

Visualizer.plot_loss(model.cost_history).show()

Visualize the decision boundary (for 2D inputs):

Visualizer.plot_decision_boundary(model, inputs, targets).show()

1.4 Making Predictions

Predict outputs for new inputs:

predictions = model.predict(inputs)
print(predictions)

2. Perceptron

Implementation of the classic Perceptron algorithm for binary classification.

✨ Main Features

• Single-neuron model for linear separation problems
• Threshold and Step activation functions
• Configurable learning rate
• Simple and efficient implementation

📝 Example Usage

import numpy as np
from src.components.visualizers import Visualizer
from src.perceptron.factory import PerceptronFactory

# AND logic gate problem
inputs = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
targets = np.array([[-1], [-1], [-1], [1]])

# Create and configure the perceptron
model = (
   PerceptronFactory.create(input_size=2)
   .compile(learning_rate=0.2, activation_function="step")
   .build()
)

# Train the model
model.train(inputs, targets, epochs=100)

# Visualize decision boundary
Visualizer.plot_decision_boundary(model, inputs, targets).show()

3. MADALINE (Many ADALINE)

Implementation of MADALINE (Many Adaptive Linear Elements), one of the earliest multi-layer neural networks.

✨ Main Features

• Multiple adaptive linear neurons in the hidden layer
• Three training strategies:
  • Standard: Updates all units
  • Rule One: Selective update for positive targets
  • Rule Two: Network majority voting-based updates
• Threshold or Step activation functions
• Advanced training rules for non-linearly separable problems

📝 Example Usage

import numpy as np
from src.madaline.factory import MadalineFactory
from src.components.visualizers import Visualizer

# XOR problem (using -1/1 encoding)
inputs = np.array([[1, 1], [1, -1], [-1, 1], [-1, -1]])
targets = np.array([[-1], [1], [1], [-1]])

# Create and configure the MADALINE network
model = (
   MadalineFactory.create(input_size=2, hidden_size=2)
   .compile(learning_rate=0.5, activation_function="threshold", trainer_strategy="rule_1")
   .build()
)

# Train the model
model.train(inputs, targets, epochs=100)

# Visualize decision boundary
Visualizer.plot_decision_boundary(model, inputs, targets).show()

---

🔍 Examples

Multi-class Classification

The notebook also demonstrates multi-class classification using synthetic datasets like blobs and the IRIS dataset. You can see the results in the dev/experimental_results.ipynb notebook.

---

🧩 Extensibility

The framework is designed to be modular. You can easily add:

• New Activation Functions: Extend the Activation class in src/neural/core/activations.py.
• New Cost Functions: Extend the CostFunction class in src/neural/core/costs.py.
• New Initializers: Extend the Initializer class in src/neural/core/initializers.py.
• New Algorithms: Implement a new algorithm by creating a folder in src/ with the following structure:

  src/your_algorithm/
  ├── core/           # Algorithm-specific components
  │   └── ...
  ├── network.py      # Main implementation
  ├── factory.py      # Factory pattern implementation
  └── builder.py      # Builder pattern implementation
  

  Then integrate it with the existing visualization and evaluation tools.

---

📊 Visualization

The src/components/visualizers.py module provides tools for visualizing:

Loss Curves	Decision Boundaries	Confusion Matrices

---

⚠️ Limitations

• The framework is not optimized for large-scale datasets or deep networks.
• Training is performed sequentially, which may be slow for large datasets.

---

🤝 Contributing

Contributions are welcome! Here are some ways you can contribute:

1. Add new neural network architectures
2. Implement additional activation functions
3. Add more cost functions or optimization algorithms
4. Improve documentation or add more examples
5. Fix bugs or optimize existing code

Please follow these steps for contributing:

1. Fork the repository
2. Create a new branch (git checkout -b feature-branch)
3. Make your changes
4. Test your changes
5. Commit your changes (git commit -m 'Add new feature')
6. Push to the branch (git push origin feature-branch)
7. Create a Pull Request

---

📝 License

This project is licensed under the MIT License. See the LICENSE file for details.