mmDiffBERT - Multi-lingual Diffusion Text Generation

A research project exploring diffusion-based text generation using different transformer models as alternatives to traditional autoregressive language models.

Project Structure

mmDiffBERT/
├── pyproject.toml              # Project dependencies and configuration
├── README.md                   # This file
├── wiki-mmBERT/                # mmBERT-based diffusion implementation
│   ├── finetune.py            # Training script for mmBERT
│   ├── inference.py           # mmBERT diffusion inference
│   ├── compare.py             # mmBERT vs GPT-2 comparison
│   ├── gpt2_inference.py      # GPT-2 baseline
│   ├── README.md              # mmBERT-specific documentation
│   └── model_weights/  # Trained mmBERT model
└── wiki-roberta/              # RoBERTa-based diffusion implementation (legacy)
    ├── finetune.py            # Training script for RoBERTa
    ├── inference.py           # RoBERTa diffusion inference
    ├── compare.py             # RoBERTa vs GPT-2 comparison
    ├── gpt2_inference.py      # GPT-2 baseline
    ├── README.md              # RoBERTa-specific documentation
    └── model_weights/  # Trained RoBERTa model

Overview

This project explores diffusion-based text generation as an alternative to traditional autoregressive language models like GPT-2. Instead of generating text left-to-right one token at a time, this approach uses:

1. Fixed prefix (first 16 tokens)
2. Mask tokens for remaining positions
3. Iterative denoising over multiple steps
4. Progressive unmasking until fully denoised

Key Features

• Bidirectional Generation: Unlike autoregressive models, can attend to full sequence context
• Iterative Denoising: Gradually reveals text over configurable number of steps
• Prefix Control: First 16 tokens remain fixed, providing stable context
• Visual Animations: Step-by-step matplotlib animations showing generation process
• Comparative Analysis: Side-by-side comparison with GPT-2 baseline
• Multi-Model Support: Support for different transformer models (mmBERT, RoBERTa)

Installation

This project uses uv for package management.

# Install dependencies
uv sync

Requirements:

• Python >= 3.11
• PyTorch 2.7.0+
• Transformers 4.52.4
• Datasets 3.6.0
• Matplotlib 3.10.3
• Accelerate 1.7.0

Usage

mmBERT Diffusion (Recommended)

cd wiki-mmBERT

# Basic text generation
python inference.py "Your prompt here"

# Fine-tuning
python finetune.py

# Side-by-side comparison with GPT-2
python compare.py "Your prompt here"

RoBERTa Diffusion (Legacy)

cd wiki-roberta

# Basic text generation
python inference.py "Your prompt here"

# Fine-tuning
python finetune.py

# Side-by-side comparison with GPT-2
python compare.py "Your prompt here"

Model-Specific Documentation

• mmBERT Implementation: See wiki-mmBERT/README.md for detailed mmBERT-specific documentation
• RoBERTa Implementation: See wiki-roberta/README.md for detailed RoBERTa-specific documentation

Research Applications

This project is designed for research into:

• Alternative text generation paradigms
• Bidirectional context utilization
• Iterative refinement approaches
• Model comparison and evaluation

Citation

This project is based on the original RoBERTaDiffusion implementation:

@misc{robertadiffusion2024,
  title={RoBERTa Diffusion Text Generation},
  author={Nathan Barry},
  year={2024},
  url={https://github.com/nathan-barry/RoBERTaDiffusion},
  note={A research project exploring fine-tuning BERT-style models for text generation}
}

Original Repository: nathan-barry/RoBERTaDiffusion

This mmDiffBERT project extends the original work by:

• Supporting multiple transformer models (mmBERT, RoBERTa)
• Adding Hindi language support with Wikipedia and Sangraha datasets
• With the primary objective of checking the syntax and compatiability.