diff --git a/examples/nlp_and_llms/cpu-small-nlp/README.md b/examples/nlp_and_llms/cpu-small-nlp/README.md
new file mode 100644
index 00000000..91421224
--- /dev/null
+++ b/examples/nlp_and_llms/cpu-small-nlp/README.md
@@ -0,0 +1,65 @@
+# Small Transformer Inference (CPU Baseline)
+
+This template implements a **high-efficiency CPU inference** workflow for Natural Language Processing (NLP). It uses **DistilBERT**, a smaller, faster version of BERT, and demonstrates how to further optimize it using **Dynamic Quantization** to achieve production-grade performance without GPUs.
+
+**Infrastructure:** [Saturn Cloud](https://saturncloud.io/)
+**Resource:** Jupyter Notebook
+**Hardware:** CPU
+**Tech Stack:** PyTorch, Hugging Face Transformers, Scikit-Learn
+
+---
+
+## ๐ Overview
+
+Deploying massive Large Language Models (LLMs) often requires expensive GPUs. However, for specific enterprise tasks like **Sentiment Analysis** or **Named Entity Recognition (NER)**, smaller "distilled" transformers running on standard CPUs are often sufficient, faster, and significantly cheaper.
+
+This template provides a **CPU-optimized baseline**:
+1. **Sentiment Analysis:** Using `distilbert-base-uncased`.
+2. **Named Entity Recognition (NER):** Using `distilbert-base-cased`.
+3. **Optimization:** Applies PyTorch **Dynamic Quantization** to boost inference speed by ~2x and reduce memory usage by ~40%.
+
+---
+
+## ๐ Quick Start
+
+### 1. Workflow
+
+1. Open **`small_transformer_cpu.ipynb`** in the Jupyter interface.
+2. **Run All Cells**:
+* **Install:** Sets up `transformers` and `torch` in the current environment.
+* **Download:** Fetches the public DistilBERT model (no login required).
+* **Benchmark (FP32):** Measures the baseline latency of the standard 32-bit floating point model.
+* **Quantize (INT8):** Converts the model weights to 8-bit integers on the fly.
+* **Compare:** Validates the speedup (typically **1.5x - 2.0x faster**).
+
+---
+
+## ๐ง Architecture: "Distill & Quantize"
+
+We use a two-step optimization strategy to ensure the model runs efficiently on commodity hardware.
+
+### 1. Distillation
+
+We use **DistilBERT**, which acts as a student model trained to mimic the behavior of the larger BERT model.
+
+* **40% fewer parameters** than BERT.
+* **60% faster** inference.
+* **97% retained accuracy** on standard benchmarks.
+
+### 2. Dynamic Quantization
+
+Standard models store weights as 32-bit floating point numbers (FP32). This template uses **Dynamic Quantization** to convert the linear layer weights to **8-bit integers (INT8)**.
+
+* **Size Reduction:** The model file shrinks by ~40% (e.g., 255MB โ 130MB).
+* **Speedup:** CPUs can process 8-bit integer math significantly faster than 32-bit float math, resulting in lower latency per request.
+
+---
+
+## ๐ Conclusion
+
+This template proves that you don't always need a GPU for NLP. For targeted tasks, a quantized DistilBERT on a modern CPU can handle hundreds of requests per second with minimal cost.
+
+To scale this solutionโfor example, processing millions of documents or deploying this as a serverless APIโconsider moving this workload to a [Saturn Cloud](https://saturncloud.io/) CPU cluster.
+
+```
+
diff --git a/examples/nlp_and_llms/cpu-small-nlp/setup.sh b/examples/nlp_and_llms/cpu-small-nlp/setup.sh
new file mode 100755
index 00000000..be732959
--- /dev/null
+++ b/examples/nlp_and_llms/cpu-small-nlp/setup.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+set -e
+
+GREEN='\033[0;32m'
+NC='\033[0m'
+
+echo -e "${GREEN}๐ Starting Small Transformer Setup...${NC}"
+
+# 1. Robust Python Detection
+if command -v python3 &> /dev/null; then
+ PY_CMD="python3"
+elif command -v python &> /dev/null; then
+ PY_CMD="python"
+else
+ echo "โ Error: Could not find 'python3' or 'python' in your PATH."
+ exit 1
+fi
+
+# 2. Create Virtual Environment
+echo "๐ฆ Creating Virtual Environment 'venv'..."
+$PY_CMD -m venv venv
+
+# 3. Install Dependencies
+echo "โฌ๏ธ Installing libraries..."
+. venv/bin/activate
+pip install --upgrade pip
+# Core stack: PyTorch (CPU), Transformers (Hugging Face), Scikit-Learn (Metrics)
+pip install torch transformers scikit-learn numpy pandas
+
+echo -e "${GREEN}โ
Environment Ready!${NC}"
+echo "-------------------------------------------------------"
+echo "To generate the notebook:"
+echo " $PY_CMD generate_notebook.py"
+echo "-------------------------------------------------------"
\ No newline at end of file
diff --git a/examples/nlp_and_llms/cpu-small-nlp/small_transformer_cpu.ipynb b/examples/nlp_and_llms/cpu-small-nlp/small_transformer_cpu.ipynb
new file mode 100644
index 00000000..2bdaa87e
--- /dev/null
+++ b/examples/nlp_and_llms/cpu-small-nlp/small_transformer_cpu.ipynb
@@ -0,0 +1,449 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# โก Small Transformer Inference (CPU Baseline)\n",
+ "\n",
+ "This notebook demonstrates how to achieve **high-performance inference** on a CPU using DistilBERT and **Dynamic Quantization**.\n",
+ "\n",
+ "**Tasks:**\n",
+ "1. **Sentiment Analysis**: Classifying text as Positive/Negative.\n",
+ "2. **NER**: Extracting entities (Names, Locations) from text.\n",
+ "3. **Optimization**: Quantizing the model to `int8`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# 1. Install Dependencies\n",
+ "%pip install torch transformers numpy pandas\n",
+ "\n",
+ "import torch\n",
+ "import time\n",
+ "import os\n",
+ "import pandas as pd\n",
+ "from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline\n",
+ "\n",
+ "# ๐ง CPU Optimization: Control Threads\n",
+ "# Setting this to the number of physical cores is usually best for latency.\n",
+ "torch.set_num_threads(os.cpu_count())\n",
+ "print(f\"โ
Threads set to: {torch.get_num_threads()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1. Sentiment Analysis Baseline (FP32)\n",
+ "We load `distilbert-base-uncased-finetuned-sst-2-english`. It is a standard baseline for sentiment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "โฌ๏ธ Downloading distilbert-base-uncased-finetuned-sst-2-english...\n"
+ ]
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "e7d674d4cb61417784587c5d248c1735",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Loading weights: 0%| | 0/104 [00:00\n",
+ "\n",
+ "\n",
+ " \n",
+ " \n",
+ " | \n",
+ " word | \n",
+ " entity_group | \n",
+ " score | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " Apple Inc | \n",
+ " ORG | \n",
+ " 0.999508 | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " Abuja | \n",
+ " LOC | \n",
+ " 0.998583 | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " Nigeria | \n",
+ " LOC | \n",
+ " 0.999648 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ ""
+ ],
+ "text/plain": [
+ " word entity_group score\n",
+ "0 Apple Inc ORG 0.999508\n",
+ "1 Abuja LOC 0.998583\n",
+ "2 Nigeria LOC 0.999648"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "ner_pipe = pipeline(\"ner\", model=\"dslim/bert-base-NER\", aggregation_strategy=\"simple\")\n",
+ "\n",
+ "text = \"Apple Inc. is planning to open a new store in Abuja, Nigeria next month.\"\n",
+ "entities = ner_pipe(text)\n",
+ "\n",
+ "pd.DataFrame(entities)[['word', 'entity_group', 'score']]"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "cpu-plotly-env",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.13.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/README.md b/examples/nlp_and_llms/nvidia-deepspeed/README.md
new file mode 100644
index 00000000..ea0fb3b3
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-deepspeed/README.md
@@ -0,0 +1,97 @@
+# DeepSpeed ZeRO-3 Training
+
+
+
+This template provides a robust environment for training large-scale Transformer models (like GPT-2 Large) using **DeepSpeed ZeRO-Stage 3**. By partitioning model parameters, gradients, and optimizer states across multiple GPUs, this setup overcomes the memory limitations of a single device.
+
+For more information on the underlying platform, visit the [Saturn Cloud Documentation](https://saturncloud.io/docs/).
+## ๐ Project Structure
+
+* **`setup_saturn.sh`**: Environment initialization script to install DeepSpeed and dependencies.
+* **`src/train_transformers.py`**: Main training script using Hugging Face `Trainer` and DeepSpeed.
+* **`ds_config_zero3.json`**: Configuration file for ZeRO-3 sharding and CPU offloading.
+* **`run_job.sh`**: Distributed training launcher script.
+* **`test_inference.py`**: Optimized generation script using DeepSpeed Inference kernels.
+
+---
+
+## ๐ Complete Procedure
+
+### 1. Environment Setup
+
+Before running any code, you must initialize the virtual environment to install the necessary DeepSpeed and Torch libraries. Refer to the [Saturn Cloud Docs for Setup Guide](https://saturncloud.io/docs/) for advanced configuration.
+
+```bash
+chmod +x setup_saturn.sh
+./setup_saturn.sh
+
+```
+
+### 2. Hardware Preparation
+
+To prevent filesystem errors during kernel compilation on Saturn Cloud's distributed architecture, create the Triton autotune directory:
+
+```bash
+mkdir -p /root/.triton/autotune
+
+```
+
+### 3. Training Execution
+
+Launch the training process across your GPUs using the provided job script:
+
+```bash
+./run_job.sh
+
+```
+
+* **The "Silent Phase"**: Note that ZeRO-3 requires a period of "silence" (usually 2-5 minutes for GPT-2) while it shards the model parameters before the first step appears.
+* **Automatic Consolidation**: The script is configured to automatically gather sharded 16-bit weights into a single `model.safetensors` file upon saving.
+
+### 4. Inference Testing
+
+After training completes and a checkpoint folder (e.g., `checkpoint-65`) is created, run the optimized inference test.
+
+**Update `test_inference.py`:**
+Ensure the `model_path` variable matches your checkpoint folder:
+
+```python
+model_path = "./checkpoints/checkpoint-65"
+
+```
+
+**Launch Inference:**
+Ensure the virtual environment is activated and then run the python test script:
+
+```bash
+python test_inference.py
+
+```
+
+---
+
+## ๐ ๏ธ Key Configurations
+
+### ZeRO-3 Optimization (`ds_config_zero3.json`)
+
+* **`stage3_gather_16bit_weights_on_model_save`**: Set to `true` to ensure your checkpoints are saved in a standard format for easy testing.
+* **`overlap_comm`**: Set to `false` in this template to maximize stability and prevent deadlocks on virtualized interconnects.
+
+### Training Stability (`src/train_transformers.py`)
+
+* **NCCL Flags**: The script forces `NCCL_P2P_DISABLE=1` to ensure reliable communication on cloud-based GPU clusters.
+* **Data Collator**: Uses `DataCollatorForLanguageModeling` to handle padding and ensure uniform tensor shapes during training, preventing "ValueError" crashes.
+
+---
+
+## ๐ Scaling Guide
+
+To scale from the verified test to a production-level run:
+
+1. **Model**: Change `model_id` to `"gpt2-large"` in `src/train_transformers.py`.
+2. **Dataset**: Remove the `[:1%]` slice to train on the full dataset.
+3. **Sequence Length**: Increase `max_length` to `512` or `1024` in the `tokenize_function`.
+
+For more community support, visit the [Saturn Cloud Community Slack](https://www.google.com/search?q=https://saturncloud.io/community/).
+
+---
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/ds_config_zero3.json b/examples/nlp_and_llms/nvidia-deepspeed/ds_config_zero3.json
new file mode 100644
index 00000000..5b1cfd29
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-deepspeed/ds_config_zero3.json
@@ -0,0 +1,20 @@
+{
+ "fp16": {
+ "enabled": "auto"
+ },
+ "zero_optimization": {
+ "stage": 3,
+ "overlap_comm": false,
+ "contiguous_gradients": true,
+ "sub_group_size": 1e6,
+ "reduce_bucket_size": 1e6,
+ "stage3_prefetch_bucket_size": 1e6,
+ "stage3_param_persistence_threshold": 1e5,
+ "stage3_max_live_parameters": 1e8,
+ "stage3_max_reuse_distance": 1e8,
+ "stage3_gather_16bit_weights_on_model_save": true
+ },
+ "train_micro_batch_size_per_gpu": "auto",
+ "gradient_accumulation_steps": "auto",
+ "steps_per_print": 10
+}
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/memory-sharding.png b/examples/nlp_and_llms/nvidia-deepspeed/memory-sharding.png
new file mode 100644
index 00000000..474b4caa
Binary files /dev/null and b/examples/nlp_and_llms/nvidia-deepspeed/memory-sharding.png differ
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/run_job.sh b/examples/nlp_and_llms/nvidia-deepspeed/run_job.sh
new file mode 100644
index 00000000..1a3a153b
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-deepspeed/run_job.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+source virt-env/bin/activate
+# Automatically uses all detected GPUs for ZeRO-3 sharding
+deepspeed src/train_transformers.py
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/setup_saturn.sh b/examples/nlp_and_llms/nvidia-deepspeed/setup_saturn.sh
new file mode 100644
index 00000000..d1883933
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-deepspeed/setup_saturn.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+# 1. Update system and install virtual environment tools
+apt-get update && apt-get install -y python3-venv python3-pip ninja-build
+
+# 2. Create and activate the virtual environment
+python3 -m venv virt-env
+source virt-env/bin/activate
+
+# 3. Install core dependencies
+# We recommend installing deepspeed from source for best hardware matching
+pip install --upgrade pip
+pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
+pip install "transformers[deepspeed]>=4.31.0" datasets accelerate tqdm
+
+# 4. Optional: Install DeepSpeed with optimized ops
+# This step builds the C++/CUDA extensions required for high performance
+DS_BUILD_OPS=1 pip install deepspeed
+
+# 5. Pre-cache dataset to prevent network timeouts during training
+echo "๐ฆ Pre-caching WikiText-2 dataset..."
+python3 -c "from datasets import load_dataset; load_dataset('wikitext', 'wikitext-2-raw-v1', cache_dir='./data')"
+
+echo "โ
Saturn Cloud Environment Setup Complete."
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/src/train_transformerrs.py b/examples/nlp_and_llms/nvidia-deepspeed/src/train_transformerrs.py
new file mode 100644
index 00000000..b2111e3a
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-deepspeed/src/train_transformerrs.py
@@ -0,0 +1,71 @@
+import os
+import datetime
+import torch
+import torch.distributed as dist
+from transformers import (
+ AutoModelForCausalLM,
+ AutoTokenizer,
+ TrainingArguments,
+ Trainer,
+ DataCollatorForLanguageModeling
+)
+from datasets import load_dataset
+
+# Force NCCL stability on cloud instances
+os.environ["NCCL_P2P_DISABLE"] = "1"
+os.environ["NCCL_IB_DISABLE"] = "1"
+
+def main():
+ if not dist.is_initialized():
+ dist.init_process_group(backend="nccl", timeout=datetime.timedelta(minutes=10))
+
+ model_id = "gpt2"
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
+ tokenizer.pad_token = tokenizer.eos_token
+
+ # 1. Load tiny dataset
+ dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train[:1%]")
+
+ # 2. Tokenize function with padding and truncation
+ def tokenize_function(examples):
+ return tokenizer(
+ examples["text"],
+ truncation=True,
+ max_length=128, # Keep small for fast test
+ padding="max_length"
+ )
+
+ # 3. Prepare data (Filter out empty rows to avoid errors)
+ dataset = dataset.filter(lambda x: len(x["text"]) > 5)
+ tokenized_ds = dataset.map(tokenize_function, batched=True, remove_columns=dataset.column_names)
+
+ # 4. Data Collator
+ data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
+
+ training_args = TrainingArguments(
+ output_dir="./checkpoints",
+ per_device_train_batch_size=2,
+ num_train_epochs=1,
+ deepspeed="ds_config_zero3.json",
+ fp16=True,
+ logging_steps=1,
+ report_to="none"
+ )
+
+ # 5. Load Model
+ model = AutoModelForCausalLM.from_pretrained(model_id)
+ model.config.use_cache = False
+
+ # 6. Initialize Trainer
+ trainer = Trainer(
+ model=model,
+ args=training_args,
+ train_dataset=tokenized_ds,
+ data_collator=data_collator # ADDED THIS
+ )
+
+ print("๐ Launching Final Verified ZeRO-3 Engine...")
+ trainer.train()
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-deepspeed/test_inference.py b/examples/nlp_and_llms/nvidia-deepspeed/test_inference.py
new file mode 100644
index 00000000..4cb7caec
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-deepspeed/test_inference.py
@@ -0,0 +1,27 @@
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+
+def test_generation():
+
+ model_path = "./checkpoints/checkpoint-65"
+
+ print(f"๐ฆ Loading model from {model_path}...")
+ tokenizer = AutoTokenizer.from_pretrained("gpt2")
+ model = AutoModelForCausalLM.from_pretrained(model_path)
+
+ # Move to GPU if available
+ device = 0 if torch.cuda.is_available() else -1
+ generator = pipeline("text-generation", model=model, tokenizer=tokenizer, device=device)
+
+ # Test prompt from the WikiText domain
+ prompt = "The phenomenon of distributed computing allows"
+
+ print("๐ฎ Generating...")
+ output = generator(prompt, max_length=50, num_return_sequences=1, truncation=True)
+
+ print("\n--- GENERATED TEXT ---")
+ print(output[0]['generated_text'])
+ print("----------------------")
+
+if __name__ == "__main__":
+ test_generation()
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-embeddings-api/FastAPI_Embeddings_Service.ipynb b/examples/nlp_and_llms/nvidia-embeddings-api/FastAPI_Embeddings_Service.ipynb
new file mode 100644
index 00000000..2a85703b
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-embeddings-api/FastAPI_Embeddings_Service.ipynb
@@ -0,0 +1,342 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "47a1df64",
+ "metadata": {},
+ "source": [
+ "# โก FastAPI Embeddings Service (FAISS + Transformers)\n",
+ "A Jupyter notebook template demonstrating how to build a lightweight embeddings and semantic search API using FastAPI, FAISS, and Transformers โ all running interactively within Saturn Cloud."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa72bf72",
+ "metadata": {},
+ "source": [
+ "## ๐ง Overview\n",
+ "This notebook walks you through building a FastAPI-based Embeddings Service that:\n",
+ "- Generates text embeddings using a Transformer model\n",
+ "- Stores them in a FAISS index for similarity search\n",
+ "- Exposes both embedding and search endpoints via FastAPI\n",
+ "\n",
+ "Youโll be able to:\n",
+ "- Add texts to the API dynamically\n",
+ "- Perform semantic similarity queries\n",
+ "- Test everything live inside a notebook\n",
+ "\n",
+ "This is perfect for quickly prototyping or demonstrating retrieval-based workflows on [Saturn Cloud](https://saturncloud.io/)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c13ddc05",
+ "metadata": {},
+ "source": [
+ "## โ๏ธ 1. Install Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a9029b37",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!pip install torch transformers sentence-transformers faiss-cpu fastapi uvicorn[standard] pydantic requests numpy"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ff652f07",
+ "metadata": {},
+ "source": [
+ "## ๐งฉ 2. Load Embedding Model and Initialize FAISS"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0c2641fe",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sentence_transformers import SentenceTransformer\n",
+ "import faiss\n",
+ "import numpy as np\n",
+ "\n",
+ "print('๐ง Loading embedding model...')\n",
+ "model = SentenceTransformer('all-MiniLM-L6-v2', device='cpu')\n",
+ "embedding_dim = model.get_sentence_embedding_dimension()\n",
+ "print(f'โ
Model loaded โ Embedding dimension: {embedding_dim}')\n",
+ "\n",
+ "# Initialize FAISS (L2 distance)\n",
+ "index = faiss.IndexFlatL2(embedding_dim)\n",
+ "texts = []"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7b298c16",
+ "metadata": {},
+ "source": [
+ "## ๐ง 3. Define Core Embedding and Search Functions"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ad6d70e3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def add_text(text: str):\n",
+ " vector = model.encode([text])[0]\n",
+ " index.add(np.array([vector]).astype('float32'))\n",
+ " texts.append(text)\n",
+ " return {'message': 'Text added successfully.', 'total_texts': len(texts)}\n",
+ "\n",
+ "def search_texts(query: str, top_k: int = 3):\n",
+ " if len(texts) == 0:\n",
+ " return {'error': 'No texts in index. Please add some first.'}\n",
+ "\n",
+ " query_vector = model.encode([query])[0]\n",
+ " D, I = index.search(np.array([query_vector]).astype('float32'), top_k)\n",
+ " results = [{'text': texts[i], 'distance': float(D[0][j])} for j, i in enumerate(I[0])]\n",
+ " return {'query': query, 'results': results}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "779df430",
+ "metadata": {},
+ "source": [
+ "## โก 4. Create the FastAPI Application"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2797f8d8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from fastapi import FastAPI\n",
+ "from pydantic import BaseModel\n",
+ "\n",
+ "app = FastAPI(title='FastAPI Embeddings Service (Notebook Edition)')\n",
+ "\n",
+ "class TextIn(BaseModel):\n",
+ " text: str\n",
+ "\n",
+ "class SearchQuery(BaseModel):\n",
+ " query: str\n",
+ " top_k: int = 3\n",
+ "\n",
+ "@app.post('/add_text')\n",
+ "def add_text_endpoint(item: TextIn):\n",
+ " return add_text(item.text)\n",
+ "\n",
+ "@app.post('/search')\n",
+ "def search_endpoint(query: SearchQuery):\n",
+ " return search_texts(query.query, query.top_k)\n",
+ "\n",
+ "@app.get('/healthz')\n",
+ "def healthz():\n",
+ " return {'status': 'ok', 'count': len(texts)}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ff37917c",
+ "metadata": {},
+ "source": [
+ "## ๐ 5. Run the API Server Inside Jupyter"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "45918e70",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import threading, time, requests, uvicorn\n",
+ "\n",
+ "PORT = 8002\n",
+ "_uvicorn_server = None\n",
+ "_uvicorn_thread = None\n",
+ "\n",
+ "def start_api_in_thread(host='0.0.0.0', port=PORT, log_level='info'):\n",
+ " global _uvicorn_server, _uvicorn_thread\n",
+ " if _uvicorn_thread and _uvicorn_thread.is_alive():\n",
+ " print(f'โน๏ธ Server already running at http://127.0.0.1:{port}')\n",
+ " return _uvicorn_thread\n",
+ "\n",
+ " config = uvicorn.Config(app, host=host, port=port, log_level=log_level)\n",
+ " _uvicorn_server = uvicorn.Server(config)\n",
+ "\n",
+ " def _run():\n",
+ " _uvicorn_server.run()\n",
+ "\n",
+ " _uvicorn_thread = threading.Thread(target=_run, daemon=True)\n",
+ " _uvicorn_thread.start()\n",
+ "\n",
+ " for _ in range(30):\n",
+ " try:\n",
+ " time.sleep(0.1)\n",
+ " r = requests.get(f'http://127.0.0.1:{port}/healthz', timeout=0.25)\n",
+ " if r.status_code == 200:\n",
+ " print(f'๐ FastAPI running at http://127.0.0.1:{port} (thread: {_uvicorn_thread.name})')\n",
+ " return _uvicorn_thread\n",
+ " except Exception:\n",
+ " pass\n",
+ " print('โ ๏ธ Server thread started but not reachable yet.')\n",
+ " return _uvicorn_thread\n",
+ "\n",
+ "def stop_api(join_timeout=5):\n",
+ " global _uvicorn_server, _uvicorn_thread\n",
+ " if _uvicorn_server is None or _uvicorn_thread is None:\n",
+ " print('โน๏ธ No server is currently running.')\n",
+ " return\n",
+ " _uvicorn_server.should_exit = True\n",
+ " _uvicorn_thread.join(timeout=join_timeout)\n",
+ " print('๐ Server stopped.')\n",
+ " _uvicorn_server = None\n",
+ " _uvicorn_thread = None"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1ea47f60",
+ "metadata": {},
+ "source": [
+ "## โถ๏ธ 6. Start the FastAPI Service"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a8b07b0a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "start_api_in_thread()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f45ac59a",
+ "metadata": {},
+ "source": [
+ "## ๐งช 7a. Test the API"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1d38b25c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import requests\n",
+ "requests.post('http://127.0.0.1:8002/add_text', json={'text': 'The quick brown fox jumps over the lazy dog.'}).json()\n",
+ "requests.post('http://127.0.0.1:8002/search', json={'query': 'A fast brown animal jumps over a sleepy dog', 'top_k': 3}).json()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "370e5271",
+ "metadata": {},
+ "source": [
+ "## ๐งช 7b. More Test the API (using more text)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "48f556a6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import requests\n",
+ "\n",
+ "BASE_URL = \"http://127.0.0.1:8002\"\n",
+ "\n",
+ "# --- Add multiple sample texts ---\n",
+ "samples = [\n",
+ " \"Artificial intelligence enables machines to learn from experience.\",\n",
+ " \"Machine learning is a subset of artificial intelligence focused on pattern recognition.\",\n",
+ " \"FastAPI is a modern, high-performance web framework for building APIs with Python.\",\n",
+ " \"FAISS is a library for efficient similarity search and clustering of dense vectors.\",\n",
+ " \"Deep learning models often require GPUs for accelerated computation.\",\n",
+ " \"Natural language processing helps computers understand human language.\"\n",
+ "]\n",
+ "\n",
+ "for text in samples:\n",
+ " res = requests.post(f\"{BASE_URL}/add_text\", json={\"text\": text})\n",
+ " print(f\"๐ Added: {text[:50]}... -> {res.status_code}\")\n",
+ "\n",
+ "# --- Example 1: Semantic similarity query ---\n",
+ "query_1 = \"What library is used for fast vector similarity search?\"\n",
+ "result_1 = requests.post(f\"{BASE_URL}/search\", json={\"query\": query_1, \"top_k\": 3}).json()\n",
+ "print(f\"\\n๐ Query: {query_1}\")\n",
+ "print(result_1)\n",
+ "\n",
+ "# --- Example 2: Conceptual link query ---\n",
+ "query_2 = \"How do computers understand human speech?\"\n",
+ "result_2 = requests.post(f\"{BASE_URL}/search\", json={\"query\": query_2, \"top_k\": 3}).json()\n",
+ "print(f\"\\n๐ Query: {query_2}\")\n",
+ "print(result_2)\n",
+ "\n",
+ "# --- Example 3: Broader topic search ---\n",
+ "query_3 = \"Explain how AI learns from data\"\n",
+ "result_3 = requests.post(f\"{BASE_URL}/search\", json={\"query\": query_3, \"top_k\": 3}).json()\n",
+ "print(f\"\\n๐ Query: {query_3}\")\n",
+ "print(result_3)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ee19c2aa",
+ "metadata": {},
+ "source": [
+ "## โน๏ธ 8. Stop the API"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ddbf0f70",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "stop_api()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "12146134",
+ "metadata": {},
+ "source": [
+ "## ๐ **Conclusion**\n",
+ "\n",
+ "Youโve built a lightweight **FastAPI Embeddings Service** that generates and searches text embeddings using **Transformers** and **FAISS** โ all within **Saturn Cloud**.\n",
+ "\n",
+ "This template serves as a quick starting point for developing AI-powered APIs and retrieval systems.\n",
+ "You can extend it to support larger datasets, custom models, or integrate it into RAG pipelines directly in your Saturn workspace.\n",
+ "\n",
+ "**Built with โค๏ธ using**\n",
+ "๐ค **Transformers**โ|โ๐งฎ **FAISS**โ|โโก **FastAPI**โ|โโ๏ธ **[Saturn Cloud](https://saturncloud.io/)**\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/nlp_and_llms/nvidia-embeddings-api/README.md b/examples/nlp_and_llms/nvidia-embeddings-api/README.md
new file mode 100644
index 00000000..4fbb64a9
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-embeddings-api/README.md
@@ -0,0 +1,121 @@
+# โก **FastAPI Embeddings Service (FAISS + Transformers)**
+
+*A Jupyter notebook example template demonstrating how to build a lightweight embeddings and semantic search API using FastAPI, FAISS, and Transformers โ all running interactively within [Saturn Cloud](https://saturncloud.io/).*
+
+---
+This is perfect for quickly prototyping or demonstrating **retrieval-based workflows** on **[Saturn Cloud](https://saturncloud.io/)**.
+
+---
+
+## โ๏ธ **1. Install Dependencies**
+
+> First, install all required Python libraries.
+> These packages handle embedding generation, FAISS indexing, and API serving.
+
+```python
+!pip install torch transformers sentence-transformers faiss-cpu fastapi uvicorn[standard] pydantic requests numpy
+```
+---
+
+## ๐งฉ **2. Load Embedding Model and Initialize FAISS**
+
+> We load a pre-trained SentenceTransformer model (`all-MiniLM-L6-v2`)
+> and initialize a FAISS index to store embeddings in memory.
+
+> FAISS (Facebook AI Similarity Search) provides an efficient vector index for fast nearest-neighbor queries.
+
+---
+
+## ๐ง **3. Define Core Embedding and Search Functions**
+
+> These helper functions form the โmachineโ behind the API:
+>
+> * `add_text()`: Encodes a new text and stores it in FAISS
+> * `search_texts()`: Finds similar texts to a given query
+
+---
+
+## โก **4. Create the FastAPI Application**
+
+> Now we wrap the embedding and search logic into a FastAPI service.
+> It exposes three main endpoints:
+>
+> * `/add_text`: Add and embed new text
+> * `/search`: Retrieve similar texts
+> * `/healthz`: API health check
+
+---
+
+## ๐ **5. Run the API Server Inside Jupyter**
+
+> Since Jupyter runs its own event loop, we launch Uvicorn in a **background thread**.
+
+---
+
+## โถ๏ธ **6. Start the FastAPI Service**
+
+> Launch the service.
+> Once it starts, open your browser to **[http://127.0.0.1:8002/docs](http://127.0.0.1:8002/docs)** to explore the Swagger UI.
+
+```python
+start_api_in_thread()
+```
+
+---
+
+## ๐งช **7. Test the API**
+
+> We can interact with the service directly from the notebook using HTTP requests.
+> Try adding a text and then searching for semantically similar content.
+
+### โ Add Text
+
+```python
+requests.post("http://127.0.0.1:8002/add_text",
+ json={"text": "The quick brown fox jumps over the lazy dog."}).json()
+```
+
+### ๐ Search Texts
+
+```python
+requests.post("http://127.0.0.1:8002/search",
+ json={"query": "A fast brown animal jumps over a sleepy dog", "top_k": 3}).json()
+```
+
+---
+
+## โน๏ธ **8. Stop the API**
+
+> When youโre done testing, stop the running FastAPI service gracefully.
+
+```python
+stop_api()
+```
+
+---
+
+## โ๏ธ **9. Run This Template on Saturn Cloud**
+
+This notebook is designed for **[Saturn Cloud](https://saturncloud.io/)** โ it runs entirely inside Jupyter, without needing an external process.
+
+**To deploy:**
+
+1. Create a new **Jupyter Server** resource on Saturn Cloud.
+2. Upload this notebook and install dependencies.
+3. Run the cells sequentially.
+4. Open **Port 8002** in your Saturn environment to access the running API.
+5. Use `/add_text` and `/search` to interact with your live embeddings service.
+
+๐ Learn more at: [https://saturncloud.io/docs](https://saturncloud.io/docs)
+
+---
+
+## ๐ **Credits**
+
+Built with โค๏ธ using:
+
+* ๐ค **Transformers**
+* ๐งฎ **FAISS**
+* โก **FastAPI**
+* ๐ง **SentenceTransformers**
+* โ๏ธ **[Saturn Cloud](https://saturncloud.io/)**
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-fsdp/README.md b/examples/nlp_and_llms/nvidia-fsdp/README.md
new file mode 100644
index 00000000..ad3b3ed7
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-fsdp/README.md
@@ -0,0 +1,84 @@
+# ๐ GPT-2 FSDP Training & Inference Template
+
+This repository provides a production-ready setup for fine-tuning GPT-2 on the WikiText-103 dataset using **Fully Sharded Data Parallel (FSDP)**. It is designed for researchers who need to validate models quickly without running exhaustive training cycles.
+
+### โจ Key Features
+
+* **FSDP Optimization**: Efficiently shards model parameters across available GPUs.
+* **Automated Lifecycle**: Training automatically saves checkpoints every 100 steps and terminates after 3 saves to manage cloud costs.
+* **Dynamic Disk Management**: Automatically cleans up old checkpoints to prevent storage exhaustion.
+* **Robust Inference**: A dedicated validation script that identifies and loads the latest healthy checkpoint.
+
+---
+
+## ๐ ๏ธ 1. Setup and Environment
+
+Initialize your environment once per Saturn Cloud resource to install dependencies and cache the dataset.
+
+### Procedural Setup
+
+1. **Permissions**: Make the utility scripts executable.
+```bash
+chmod +x setup_saturn.sh run_job.sh
+
+```
+
+
+2. **Environment Build**: This script creates the `virt-env` and pre-downloads the WikiText shards to the local `./data` folder.
+```bash
+./setup_saturn.sh
+
+```
+
+
+
+---
+
+## ๐ 2. Running the Model
+
+### A. Distributed Training
+
+Launch the training process using the job runner. It automatically handles distributed initialization.
+
+```bash
+./run_job.sh
+
+```
+
+* **What to Expect**: The console will log Loss and VRAM metrics every 10 steps.
+* **Checkpointing**: Every 100 steps, a `.bin` file is saved to the `checkpoints/` directory.
+* **Auto-Termination**: The script exits gracefully after saving 3 checkpoints.
+
+### B. Validation Inference
+
+You can test the model as soon as the first checkpoint is saved.
+
+```bash
+source virt-env/bin/activate
+python test_inference.py
+
+```
+
+**Expected Result**:
+The script will load the latest `.bin` file and generate a text completion.
+
+> **Output example**:
+> `๐ Attempting to load: checkpoints/gpt2_wikitext_epoch0_step100.bin`
+> `โ
Successfully loaded: checkpoints/gpt2_wikitext_epoch0_step100.bin`
+> `--- Result ---`
+> `The history of WikiText is a collection of high-quality articles used for benchmarking language models...`
+
+---
+
+## ๐ Project Structure
+
+* **`checkpoints/`**: Auto-managed directory for model weights.
+* **`src/train_fsdp.py`**: Core logic for sharding and training.
+* **`test_inference.py`**: Script for model weight validation.
+* **`data/`**: Local cache for the WikiText-103 dataset.
+
+---
+
+## ๐ Powered by Saturn Cloud
+
+This template is built to run seamlessly on the **[Saturn Cloud Platform](https://saturncloud.io/)**. For advanced scaling, multi-node configurations, or persistent storage volumes, refer to the **[Saturn Cloud Documentation](https://saturncloud.io/docs/)**.
diff --git a/examples/nlp_and_llms/nvidia-fsdp/requirements.txt b/examples/nlp_and_llms/nvidia-fsdp/requirements.txt
new file mode 100644
index 00000000..3082157d
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-fsdp/requirements.txt
@@ -0,0 +1,6 @@
+torch>=2.0.0
+transformers>=4.31.0
+datasets>=2.12.0
+pyarrow
+tqdm
+accelerate
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-fsdp/run_job.sh b/examples/nlp_and_llms/nvidia-fsdp/run_job.sh
new file mode 100644
index 00000000..c0c20e5f
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-fsdp/run_job.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+# 1. Activate the isolated environment
+source virt-env/bin/activate
+
+# 2. Set defaults for RunPod environment
+# If SATURN_GPUS_PER_NODE is empty, use the number of GPUs detected by nvidia-smi
+DETECTOR_GPUS=$(nvidia-smi -L | wc -l)
+NUM_GPUS=${SATURN_GPUS_PER_NODE:-$DETECTOR_GPUS}
+
+# If SATURN variables are missing, default to a standalone single node
+NODE_RANK=${SATURN_NODE_RANK:-0}
+MASTER_ADDR=${SATURN_MASTER_ADDR:-"127.0.0.1"}
+NUM_NODES=${SATURN_NUM_NODES:-1}
+
+echo "๐ Starting Distributed Training on RunPod..."
+echo "๐ GPUs detected/used: $NUM_GPUS"
+echo "๐ Node Rank: $NODE_RANK"
+
+# 3. Launch with torchrun
+# Use --standalone for single-node RunPod testing
+torchrun \
+ --nproc_per_node=$NUM_GPUS \
+ --nnodes=$NUM_NODES \
+ --node_rank=$NODE_RANK \
+ --master_addr=$MASTER_ADDR \
+ --master_port=12355 \
+ src/train_fsdp.py
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-fsdp/setup_saturn.sh b/examples/nlp_and_llms/nvidia-fsdp/setup_saturn.sh
new file mode 100644
index 00000000..e5d92b41
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-fsdp/setup_saturn.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+# 1. Update system packages
+apt update && apt install -y python3-venv
+
+# 2. Create the virtual environment
+echo "๐ Creating virtual environment..."
+python3 -m venv virt-env
+
+# 3. Activate the environment
+source virt-env/bin/activate
+
+# 4. Install high-performance libraries
+echo "๐ Installing dependencies..."
+pip install --upgrade pip
+pip install -r requirements.txt
+
+echo "โ
Environment ready."
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-fsdp/src/train_fsdp.py b/examples/nlp_and_llms/nvidia-fsdp/src/train_fsdp.py
new file mode 100644
index 00000000..1accaaac
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-fsdp/src/train_fsdp.py
@@ -0,0 +1,146 @@
+import os
+import sys
+import time
+import functools
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+from datasets import load_dataset
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from torch.utils.data import DataLoader
+from torch.utils.data.distributed import DistributedSampler
+
+# FSDP Specific Imports
+from torch.distributed.fsdp import (
+ FullyShardedDataParallel as FSDP,
+ MixedPrecision,
+ ShardingStrategy,
+ CPUOffload,
+ StateDictType,
+ FullStateDictConfig,
+)
+from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
+from transformers.models.gpt2.modeling_gpt2 import GPT2Block
+from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (
+ checkpoint_wrapper,
+ CheckpointImpl,
+ apply_activation_checkpointing,
+)
+
+
+
+def setup():
+ """Initializes the distributed process group for NCCL."""
+ dist.init_process_group("nccl")
+
+def cleanup():
+ """Cleans up the distributed process group."""
+ dist.destroy_process_group()
+
+def log_stats(epoch, batch_idx, loss, start_time):
+ """Logs training progress, throughput, and GPU memory efficiency."""
+ if dist.get_rank() == 0:
+ # 1. Calculate Throughput (Samples per Second)
+ elapsed = time.time() - start_time
+ samples_per_sec = (batch_idx + 1) * 4 * dist.get_world_size() / elapsed
+
+ # 2. Get Memory Metrics in GB
+ peak_mem = torch.cuda.max_memory_allocated() / (1024 ** 3)
+ reserved_mem = torch.cuda.memory_reserved() / (1024 ** 3)
+
+ print(f"| Epoch: {epoch} | Batch: {batch_idx} | Loss: {loss:.4f} |")
+ print(f"| Speed: {samples_per_sec:.2f} samples/sec | Peak Mem: {peak_mem:.2f} GB | Reserved: {reserved_mem:.2f} GB |")
+ print("-" * 60)
+
+def train_one_epoch(model, dataloader, optimizer, epoch):
+ model.train()
+ start_time = time.time()
+
+ # --- CONFIGURATION FOR QUICK TESTING ---
+ save_every_n_steps = 100 # Save a checkpoint every 100 steps
+ max_total_steps = 300 # Stop training completely after 300 steps
+ # ----------------------------------------
+
+ for batch_idx, batch in enumerate(dataloader):
+ if batch_idx >= max_total_steps:
+ if dist.get_rank() == 0:
+ print(f"๐ Reached max steps ({max_total_steps}). Terminating training.")
+ return # Exit the function and stop training
+
+ input_ids = batch["input_ids"].to(torch.cuda.current_device())
+ optimizer.zero_grad()
+ loss = model(input_ids, labels=input_ids).loss
+ loss.backward()
+ optimizer.step()
+
+ if batch_idx % 10 == 0:
+ log_stats(epoch, batch_idx, loss.item(), start_time)
+
+ # Save Checkpoint
+ if batch_idx > 0 and batch_idx % save_every_n_steps == 0:
+ if dist.get_rank() == 0:
+ os.makedirs("checkpoints", exist_ok=True)
+ checkpoint_path = f"checkpoints/gpt2_step_{batch_idx}.bin"
+ # Standard FSDP state dict gathering
+ with FSDP.state_dict_type(model, StateDictType.FULL_STATE_DICT):
+ state_dict = model.state_dict()
+ torch.save(state_dict, checkpoint_path)
+ print(f"๐พ Checkpoint saved at {checkpoint_path}")
+
+
+def main():
+ setup()
+ local_rank = int(os.environ["LOCAL_RANK"])
+ torch.cuda.set_device(local_rank)
+
+ # 1. Load Data: WikiText-103
+ dataset = load_dataset("wikitext", "wikitext-103-v1", split="train")
+ tokenizer = AutoTokenizer.from_pretrained("gpt2")
+ tokenizer.pad_token = tokenizer.eos_token
+
+ def tokenize_function(examples):
+ return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=512)
+
+ tokenized_dataset = dataset.map(tokenize_function, batched=True, remove_columns=["text"])
+ tokenized_dataset.set_format("torch")
+
+ sampler = DistributedSampler(tokenized_dataset, num_replicas=dist.get_world_size(), rank=dist.get_rank())
+ dataloader = DataLoader(tokenized_dataset, batch_size=4, sampler=sampler)
+
+ # 2. Configure FSDP Policies
+ # Use BF16 Mixed Precision for Ampere (A100/H100) efficiency
+ mp_policy = MixedPrecision(param_dtype=torch.bfloat16, reduce_dtype=torch.bfloat16)
+
+ gpt2_auto_wrap_policy = functools.partial(
+ transformer_auto_wrap_policy,
+ transformer_layer_cls={GPT2Block},
+ )
+
+ # 3. Initialize and Wrap Model
+ model = AutoModelForCausalLM.from_pretrained("gpt2").to(local_rank)
+
+ model = FSDP(
+ model,
+ auto_wrap_policy=gpt2_auto_wrap_policy,
+ mixed_precision=mp_policy,
+ sharding_strategy=ShardingStrategy.FULL_SHARD, # Max memory efficiency
+ device_id=local_rank
+ )
+
+ # 4. Apply Activation Checkpointing to each Transformer Block
+ non_reentrant_wrapper = functools.partial(checkpoint_wrapper, offload_to_cpu=False)
+ for module in model.modules():
+ if isinstance(module, GPT2Block):
+ module = non_reentrant_wrapper(module)
+
+ # 5. Initialize Optimizer AFTER FSDP Wrapping
+ optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
+
+ # 6. Execute Training
+ for epoch in range(1):
+ train_one_epoch(model, dataloader, optimizer, epoch)
+
+ cleanup()
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-fsdp/test_inference.py b/examples/nlp_and_llms/nvidia-fsdp/test_inference.py
new file mode 100644
index 00000000..4ea6b7ec
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-fsdp/test_inference.py
@@ -0,0 +1,25 @@
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+def run_inference(checkpoint_path, prompt="The history of WikiText is"):
+ print(f"Loading checkpoint: {checkpoint_path}")
+
+ # 1. Initialize Model and Tokenizer
+ tokenizer = AutoTokenizer.from_pretrained("gpt2")
+ model = AutoModelForCausalLM.from_pretrained("gpt2")
+
+ # 2. Load trained weights
+ # Use map_location='cpu' if testing on a machine without GPU
+ state_dict = torch.load(checkpoint_path, map_location='cpu')
+ model.load_state_dict(state_dict)
+
+ # 3. Generate Text
+ inputs = tokenizer(prompt, return_tensors="pt")
+ outputs = model.generate(**inputs, max_length=200, do_sample=True, top_p=0.95)
+
+ print("\n--- GENERATED TEXT ---")
+ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+
+if __name__ == "__main__":
+ # Point this to your latest .bin file in the checkpoints folder
+ run_inference("checkpoints/gpt2_step_200.bin")
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-langgraph/README.md b/examples/nlp_and_llms/nvidia-langgraph/README.md
new file mode 100644
index 00000000..82d37738
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-langgraph/README.md
@@ -0,0 +1,56 @@
+# ๐ง LangGraph Agent Sandbox
+
+This sample notebook demonstrates how to build a **local multi-agent coding assistant** using **LangGraph**, **Transformers**, and **LangChain Sandbox** โ fully compatible with [Saturn Cloud](https://saturncloud.io/).
+
+The system allows you to:
+- Generate clean Python code from natural-language prompts.
+- Check the code for syntax and structure validity.
+- Execute the generated code safely in an isolated sandbox.
+- Interactively explore different code generation tasks โ all locally, with **no API keys required**.
+
+---
+
+## โ๏ธ What Youโll Learn
+- How to design an **agentic workflow** using LangGraph.
+- How to use **local transformer models (Phi-3 Mini)** for reasoning.
+- How to integrate a **safe execution sandbox** (with local fallback).
+- How to run multi-stage LLM pipelines entirely within Saturn Cloud.
+
+---
+
+## ๐งฉ Notebook Structure
+
+| Stage | Description |
+|--------|--------------|
+| **1. Install Dependencies** | Installs LangGraph, LangChain, and related libraries. |
+| **2. Load Model & Sandbox** | Loads the local Hugging Face model and initializes a secure sandbox (with fallback). |
+| **3. Define Workflow Agents** | Builds LangGraph nodes for Code Generation, Syntax Checking, and Execution. |
+| **4. Batch Testing** | Runs several example coding tasks through the full pipeline. |
+| **5. Interactive Mode** | Launches an interactive terminal to test your own code prompts. |
+
+---
+
+## ๐ How to Run on Saturn Cloud
+
+1. **Open this template** in your Saturn Cloud environment.
+2. Run all cells sequentially from top to bottom.
+3. The local model (`Phi-3-mini`) will load automatically.
+4. Explore the pre-loaded test prompts or use the **interactive assistant** in Stage 5.
+5. All runs execute securely inside your Saturn Cloud instance โ no external API calls.
+
+---
+
+## โ๏ธ About Saturn Cloud
+
+[Saturn Cloud](https://saturncloud.io/) provides powerful GPU-accelerated Jupyter environments that make it easy to run, scale, and share AI and data-science projects.
+This template is part of Saturn Cloudโs **open-source educational catalog**, showcasing safe, local AI workflows.
+
+---
+
+### ๐ง Built With
+
+- ๐ค **Transformers**
+- ๐งฎ **LangGraph**
+- โก **LangChain Sandbox**
+- โ๏ธ **Saturn Cloud**
+
diff --git a/examples/nlp_and_llms/nvidia-langgraph/langGraph_agent_sandbox.ipynb b/examples/nlp_and_llms/nvidia-langgraph/langGraph_agent_sandbox.ipynb
new file mode 100644
index 00000000..3ab15999
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-langgraph/langGraph_agent_sandbox.ipynb
@@ -0,0 +1,256 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "cdb861f5",
+ "metadata": {},
+ "source": [
+ "\n",
+ "# ๐ง LangGraph Agent Sandbox\n",
+ "\n",
+ "This template demonstrates how to build a **local multi-agent code assistant** using **LangGraph** and **Transformers** โ fully compatible with **[Saturn Cloud](https://saturncloud.io/)**.\n",
+ "\n",
+ "It uses:\n",
+ "- ๐งฉ **LangGraph** for workflow control \n",
+ "- ๐ค **Transformers (Phi-3-mini)** for local code generation \n",
+ "- ๐งฎ **LangChain Sandbox** for safe isolated execution (with fallback mode) \n",
+ "\n",
+ "Run this notebook on a Saturn Cloud Jupyter Server to explore autonomous LLM agents that generate, validate, and execute code locally.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "47a75517",
+ "metadata": {},
+ "source": [
+ "## ๐งฉ Stage 1 โ Install Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e83bbd47",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!pip install langchain langchain-community langchain-openai langchain-sandbox"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b6164d9f",
+ "metadata": {},
+ "source": [
+ "## ๐ง Stage 2 โ Load Local Model and Sandbox"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2ce8e67d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "import os\n",
+ "from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline\n",
+ "from langchain_community.llms import HuggingFacePipeline\n",
+ "from langchain_sandbox import PyodideSandbox\n",
+ "\n",
+ "model_id = \"microsoft/Phi-3-mini-4k-instruct\"\n",
+ "print(f\"๐ง Loading model: {model_id} ...\")\n",
+ "\n",
+ "try:\n",
+ " tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
+ " model = AutoModelForCausalLM.from_pretrained(model_id)\n",
+ " pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=256)\n",
+ " llm = HuggingFacePipeline(pipeline=pipe)\n",
+ " print(\"โ
Local LLM ready (using Hugging Face Transformers).\")\n",
+ "except Exception as e:\n",
+ " print(f\"โ ๏ธ Model load failed: {e}\")\n",
+ " llm = None\n",
+ "\n",
+ "try:\n",
+ " sandbox = PyodideSandbox()\n",
+ " print(\"๐งช Pyodide Sandbox ready for isolated execution.\")\n",
+ "except Exception as e:\n",
+ " print(f\"โ ๏ธ Sandbox initialization failed: {e}\")\n",
+ " print(\"โก๏ธ Using lightweight local sandbox emulator (safe fallback).\")\n",
+ "\n",
+ " class LocalSandbox:\n",
+ " def run(self, code):\n",
+ " try:\n",
+ " exec_locals = {}\n",
+ " exec(code, {}, exec_locals)\n",
+ " return {\"output_text\": str(exec_locals)}\n",
+ " except Exception as err:\n",
+ " return {\"output_text\": f\"Error: {err}\"}\n",
+ "\n",
+ " sandbox = LocalSandbox()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "12677323",
+ "metadata": {},
+ "source": [
+ "## โ๏ธ Stage 3 โ Define Workflow Agents"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "46377a9e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "from langgraph.graph import StateGraph\n",
+ "from typing import TypedDict, Dict\n",
+ "import re, io, sys, contextlib\n",
+ "\n",
+ "class CodeState(TypedDict):\n",
+ " prompt: str\n",
+ " code: str\n",
+ " check: str\n",
+ " output: str\n",
+ "\n",
+ "class CodeGenerator:\n",
+ " def run(self, state: CodeState) -> Dict:\n",
+ " query = f\"Write clean, well-commented Python 3 code for: {state['prompt']}\"\n",
+ " response = llm.invoke(query)\n",
+ " generated = response if isinstance(response, str) else getattr(response, \"content\", str(response))\n",
+ " print(\"๐ง Generated code:\")\n",
+ " print(generated[:300], \"...\" if len(generated) > 300 else \"\")\n",
+ " return {\"code\": generated}\n",
+ "\n",
+ "class SyntaxChecker:\n",
+ " def run(self, state: CodeState) -> Dict:\n",
+ " code = state.get(\"code\", \"\")\n",
+ " issues = []\n",
+ " if not re.search(r\"def\\s+\\w+\\(\", code):\n",
+ " issues.append(\"โ No function definition detected.\")\n",
+ " if code.count(\"(\") != code.count(\")\"):\n",
+ " issues.append(\"โ ๏ธ Unbalanced parentheses.\")\n",
+ " msg = \"โ
Syntax check passed.\" if not issues else \" | \".join(issues)\n",
+ " print(f\"๐งฎ Syntax Check : {msg}\")\n",
+ " return {\"check\": msg}\n",
+ "\n",
+ "class SandboxExecutor:\n",
+ " def __init__(self, sandbox_instance):\n",
+ " self.sandbox = sandbox_instance\n",
+ "\n",
+ " def run(self, state: CodeState) -> Dict:\n",
+ " code = state.get(\"code\", \"\")\n",
+ " if not code:\n",
+ " return {\"output\": \"No code detected.\"}\n",
+ " code = re.sub(r\"```python|```\", \"\", code).strip()\n",
+ " buffer = io.StringIO()\n",
+ " try:\n",
+ " with contextlib.redirect_stdout(buffer):\n",
+ " exec_locals = {}\n",
+ " exec(code, {}, exec_locals)\n",
+ " return {\"output\": buffer.getvalue().strip() or \"โ
Executed successfully.\"}\n",
+ " except Exception as e:\n",
+ " return {\"output\": f\"Error: {e}\"}\n",
+ "\n",
+ "def build_workflow():\n",
+ " g = StateGraph(CodeState)\n",
+ " g.add_node(\"generate_code\", CodeGenerator().run)\n",
+ " g.add_node(\"check_syntax\", SyntaxChecker().run)\n",
+ " g.add_node(\"execute_code\", SandboxExecutor(sandbox).run)\n",
+ " g.set_entry_point(\"generate_code\")\n",
+ " g.add_edge(\"generate_code\", \"check_syntax\")\n",
+ " g.add_edge(\"check_syntax\", \"execute_code\")\n",
+ " print(\"โ
LangGraph workflow built.\")\n",
+ " return g.compile()\n",
+ "\n",
+ "workflow = build_workflow()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "44849559",
+ "metadata": {},
+ "source": [
+ "## ๐งช Stage 4 โ Batch Testing"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e5b25c7d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "examples = [\n",
+ " \"Write a Python function to check if a number is prime.\",\n",
+ " \"Create a Python script that sorts a list of strings alphabetically.\",\n",
+ " \"Generate a function to calculate factorial using recursion.\"\n",
+ "]\n",
+ "for i, prompt in enumerate(examples, start=1):\n",
+ " print(f\"\\n๐ง Test {i}: {prompt}\")\n",
+ " print(\"=\" * 80)\n",
+ " result = workflow.invoke({\"prompt\": prompt, \"code\": \"\", \"check\": \"\", \"output\": \"\"})\n",
+ " print(result)\n",
+ " print(\"=\" * 80)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0160c826",
+ "metadata": {},
+ "source": [
+ "## ๐ฌ Stage 5 โ Interactive Assistant"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "93c1be83",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "from rich.console import Console\n",
+ "from rich.panel import Panel\n",
+ "\n",
+ "console = Console()\n",
+ "def run_interactive():\n",
+ " console.print(Panel.fit(\"๐ฌ Type a coding task (or 'exit' to quit)\", style=\"bold cyan\"))\n",
+ " while True:\n",
+ " prompt = input(\"\\n๐ง Enter task: \").strip()\n",
+ " if prompt.lower() in [\"exit\", \"quit\", \"q\"]:\n",
+ " console.print(\"\\n๐ Exiting interactive mode.\\n\", style=\"bold yellow\")\n",
+ " break\n",
+ " state = {\"prompt\": prompt, \"code\": \"\", \"check\": \"\", \"output\": \"\"}\n",
+ " result = workflow.invoke(state)\n",
+ " console.print(result)\n",
+ "run_interactive()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "692377c3",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## ๐ Conclusion\n",
+ "\n",
+ "Youโve built a **LangGraph Agent Sandbox** โ a self-contained multi-agent system that generates, checks, and executes Python code using local models. \n",
+ "All this runs **directly inside Saturn Cloud**, without external APIs.\n",
+ "\n",
+ "**Built with โค๏ธ using:** \n",
+ "๐ค Transformers | ๐งฎ LangGraph | โก LangChain | โ๏ธ [Saturn Cloud](https://saturncloud.io/)\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/nlp_and_llms/nvidia-lora/README.md b/examples/nlp_and_llms/nvidia-lora/README.md
new file mode 100644
index 00000000..38b24c71
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-lora/README.md
@@ -0,0 +1,12 @@
+# LoRA Fine-Tuning (PEFT + Transformers)
+
+
+
+This template illustrates how **LoRA fine-tuning** can significantly reduce resource requirements while maintaining strong model performance.
+By running it on **Saturn Cloud**, you benefit from a GPU-optimized, scalable environment that simplifies the entire fine-tuning workflow โ from experimentation to production deployment.
+
+Learn more:
+
+* ๐ [Saturn Cloud Documentation](https://saturncloud.io/docs/)
+* ๐ [Saturn Cloud Templates Gallery](https://saturncloud.io/resources/templates/)
+* ๐ [PEFT Library (Hugging Face)](https://huggingface.co/docs/peft/index)
diff --git a/examples/nlp_and_llms/nvidia-lora/nvidia_lora.ipynb b/examples/nlp_and_llms/nvidia-lora/nvidia_lora.ipynb
new file mode 100644
index 00000000..4ac2ecd0
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-lora/nvidia_lora.ipynb
@@ -0,0 +1 @@
+{"cells":[{"cell_type":"markdown","id":"0c21f79d","metadata":{"id":"0c21f79d"},"source":["# LoRA Fine-Tuning\n","\n","\n","\n","This example template demonstrates **parameter-efficient fine-tuning (PEFT)** using **LoRA (Low-Rank Adaptation)** with the FLAN-T5 model on a free public dataset (SAMSum) for summarization.\n","\n","This provides a lightweight, GPU-friendly workflow that runs fully offline โ no API keys required. The notebook guides you through each step: loading data, applying LoRA adapters, fine-tuning, evaluating, and saving your model for reuse.\n","\n","On [Saturn Cloud](https://saturncloud.io), you can scale from a single NVIDIA GPU to multi-GPU clusters, enabling distributed inference for larger models or higher throughput workloads โ all within a managed, GPU-ready environment."]},{"cell_type":"markdown","id":"572d0e23-b689-4be9-999b-a5da2f670d90","metadata":{"id":"572d0e23-b689-4be9-999b-a5da2f670d90"},"source":["## Install dependencies"]},{"cell_type":"code","execution_count":10,"id":"982862db-82e2-4c70-9221-3ed04c03aad3","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"982862db-82e2-4c70-9221-3ed04c03aad3","executionInfo":{"status":"ok","timestamp":1761300519635,"user_tz":-60,"elapsed":444023,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"3779bc45-2105-4f23-ab71-65ad97e06f29"},"outputs":[{"output_type":"stream","name":"stdout","text":["\u001b[1m\u001b[33mwarning\u001b[39m\u001b[0m\u001b[1m:\u001b[0m \u001b[1mThe `--system` flag has no effect, a system Python interpreter is always used in `uv venv`\u001b[0m\n","Using CPython 3.12.12 interpreter at: \u001b[36m/usr/bin/python3\u001b[39m\n","Creating virtual environment at: \u001b[36mlora-env\u001b[39m\n","\u001b[33m?\u001b[0m \u001b[1mA virtual environment already exists at `lora-env`. Do you want to replace it?\u001b[0m \u001b[38;5;8m[y/n]\u001b[0m \u001b[38;5;8mโบ\u001b[0m \u001b[36myes\u001b[0m\n","\n","\u001b[0J\u001b[32mโ\u001b[0m \u001b[1mA virtual environment already exists at `lora-env`. Do you want to replace it?\u001b[0m \u001b[38;5;8mยท\u001b[0m \u001b[36myes\u001b[0m\n","\u001b[?25hActivate with: \u001b[32msource lora-env/bin/activate\u001b[39m\n","0.00s - Debugger warning: It seems that frozen modules are being used, which may\n","0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off\n","0.00s - to python to disable frozen modules.\n","0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.\n","Installed kernelspec lora-env in /root/.local/share/jupyter/kernels/lora-env\n"]}],"source":["# Step 1: Install UV (fast, modern package manager)\n","!pip install -q uv\n","# Step 2: Create a clean environment with Python 3.12\n","!uv venv lora-env -p 3.12\n","\n","# Step 3: Activate and install all required libraries inside it\n","!source lora-env/bin/activate && uv pip install -q torch transformers datasets peft accelerate evaluate bitsandbytes jedi\n","\n","# Step 4: Add the environment as a selectable Jupyter kernel\n","!source lora-env/bin/activate && pip install -q ipykernel\n","!python -m ipykernel install --user --name=lora-env --display-name \"LoRA Fine-Tune Env\"\n","\n","# (Optional fallback for environments without bitsandbytes)\n","try:\n"," import bitsandbytes\n","except Exception:\n"," print(\"โ ๏ธ bitsandbytes not available โ skipping GPU quantisation support.\")\n","\n","\n","!pip install -q --upgrade \\\n"," sentencepiece \\\n"," protobuf \\\n"," tqdm"]},{"cell_type":"markdown","id":"c12336a1-ae67-4f40-8bcc-df3b5ce9c404","metadata":{"id":"c12336a1-ae67-4f40-8bcc-df3b5ce9c404"},"source":["Download and prepares the GovReport Summarization dataset from `Hugging Face (ccdv/govreport-summarization)`. The dataset contains long government reports paired with their human-written summaries, making it suitable for text summarization tasks."]},{"cell_type":"code","execution_count":11,"id":"3b6f4321-71f6-4358-bce8-7665b0c3e560","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"3b6f4321-71f6-4358-bce8-7665b0c3e560","executionInfo":{"status":"ok","timestamp":1761300520639,"user_tz":-60,"elapsed":978,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"c9ff928c-0bff-4a54-ec94-3b4301bf0b45"},"outputs":[{"output_type":"stream","name":"stdout","text":["โณ Downloading dataset: ccdv/govreport-summarization\n","โ
Dataset ready (govreport-summarization)\n"]}],"source":["from datasets import load_dataset, Dataset\n","import pandas as pd\n","\n","print(\"โณ Downloading dataset: ccdv/govreport-summarization\")\n","ds = load_dataset(\"ccdv/govreport-summarization\")\n","train_ds = ds[\"train\"].select(range(1000))\n","eval_ds = ds[\"validation\"].select(range(200))\n","TEXT_COL, TARGET_COL = \"report\", \"summary\"\n","print(\"โ
Dataset ready (govreport-summarization)\")"]},{"cell_type":"markdown","id":"0dd28e48-64a4-4133-8310-e9aed982e595","metadata":{"id":"0dd28e48-64a4-4133-8310-e9aed982e595"},"source":["Loads the **FLAN-T5-Small model** and its tokenizer from Hugging Face. The tokenizer converts text into numerical tokens the model can understand, while the model itself (a sequence-to-sequence language model) performs tasks such as summarization or text generation."]},{"cell_type":"code","execution_count":12,"id":"1b080fd4-c153-4657-8642-bdb858a3f5e9","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"1b080fd4-c153-4657-8642-bdb858a3f5e9","executionInfo":{"status":"ok","timestamp":1761300521822,"user_tz":-60,"elapsed":1174,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"bef9cc71-31d4-41fb-eaca-6d63312e5379"},"outputs":[{"output_type":"stream","name":"stdout","text":["โณ Loading model: google/flan-t5-small\n","โ
Model and tokenizer loaded successfully!\n","Tokenizer vocab size: 32100\n"]}],"source":["from transformers import AutoTokenizer, AutoModelForSeq2SeqLM\n","\n","model_name = \"google/flan-t5-small\"\n","print(f\"โณ Loading model: {model_name}\")\n","\n","tokenizer = AutoTokenizer.from_pretrained(model_name)\n","model = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n","\n","print(\"โ
Model and tokenizer loaded successfully!\")\n","print(\"Tokenizer vocab size:\", len(tokenizer))\n"]},{"cell_type":"markdown","source":["Adding LoRA (Low-Rank Adaptation) adapter to the base model using PEFT (Parameter-Efficient Fine-Tuning). Instead of updating all model parameters, LoRA inserts lightweight adapter layers that learn task-specific updatesโmaking fine-tuning faster and more memory-efficient."],"metadata":{"id":"KhKaRIjZom1R"},"id":"KhKaRIjZom1R"},{"cell_type":"code","execution_count":13,"id":"d5f3740d-76c3-4f76-92ee-c61dcbed3144","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"d5f3740d-76c3-4f76-92ee-c61dcbed3144","executionInfo":{"status":"ok","timestamp":1761300521858,"user_tz":-60,"elapsed":19,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"5ff20b33-214e-4a15-9338-3a7aac5fdd31"},"outputs":[{"output_type":"stream","name":"stdout","text":["โ
LoRA adapter added successfully!\n","trainable params: 688,128 || all params: 77,649,280 || trainable%: 0.8862\n"]}],"source":["from peft import LoraConfig, get_peft_model\n","\n","# LoRA configuration\n","lora_config = LoraConfig(\n"," r=16, # rank\n"," lora_alpha=32, # scaling factor\n"," lora_dropout=0.05, # dropout for regularisation\n"," bias=\"none\",\n"," task_type=\"SEQ_2_SEQ_LM\" # T5-style sequence-to-sequence\n",")\n","\n","# Apply adapter to model\n","model = get_peft_model(model, lora_config)\n","\n","# Print summary\n","print(\"โ
LoRA adapter added successfully!\")\n","model.print_trainable_parameters()\n"]},{"cell_type":"markdown","source":["Prepare the text data for training by converting it into numerical tokens that the model can process."],"metadata":{"id":"dLitVvFvo5vn"},"id":"dLitVvFvo5vn"},{"cell_type":"code","execution_count":14,"id":"93fa696d-a672-4ad3-8343-73f8ebc71c7a","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":121,"referenced_widgets":["9db3a5ac0dd84249a2b236b96c58aad8","2c821f95cbf94e6f972651544b51bacf","69e89bf8eace41aa850498fd3fd61f99","3aaca7366ecb47d8b4ac27b6301aa91b","48ba285de8364e65a380add6e08e4d69","29dfb08a2a1d43b3878cb8a98b285b09","4edaefbb46844f8ba1583f63c20f9ccf","168534f6a2f3457b8dfa29da5aa15d6a","3e54568d0ae94350a1a461a6b1cc3423","bd8316fe2cc24289bf8d39ab6f065e43","d802453c7a484c89897a30b8ddde157b"]},"id":"93fa696d-a672-4ad3-8343-73f8ebc71c7a","executionInfo":{"status":"ok","timestamp":1761300535524,"user_tz":-60,"elapsed":13642,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"e9c09e29-4170-4e06-e229-5b7b5f002740"},"outputs":[{"output_type":"display_data","data":{"text/plain":["Map: 0%| | 0/200 [00:00"],"text/html":["\n"," \n"," \n","
\n"," [500/500 01:36, Epoch 1/1]\n","
\n"," \n"," \n"," | Step | \n"," Training Loss | \n","
\n"," \n"," \n"," \n"," | 25 | \n"," 0.000000 | \n","
\n"," \n"," | 50 | \n"," 0.000000 | \n","
\n"," \n"," | 75 | \n"," 0.000000 | \n","
\n"," \n"," | 100 | \n"," 0.000000 | \n","
\n"," \n"," | 125 | \n"," 0.000000 | \n","
\n"," \n"," | 150 | \n"," 0.000000 | \n","
\n"," \n"," | 175 | \n"," 0.000000 | \n","
\n"," \n"," | 200 | \n"," 0.000000 | \n","
\n"," \n"," | 225 | \n"," 0.000000 | \n","
\n"," \n"," | 250 | \n"," 0.000000 | \n","
\n"," \n"," | 275 | \n"," 0.000000 | \n","
\n"," \n"," | 300 | \n"," 0.000000 | \n","
\n"," \n"," | 325 | \n"," 0.000000 | \n","
\n"," \n"," | 350 | \n"," 0.000000 | \n","
\n"," \n"," | 375 | \n"," 0.000000 | \n","
\n"," \n"," | 400 | \n"," 0.000000 | \n","
\n"," \n"," | 425 | \n"," 0.000000 | \n","
\n"," \n"," | 450 | \n"," 0.000000 | \n","
\n"," \n"," | 475 | \n"," 0.000000 | \n","
\n"," \n"," | 500 | \n"," 0.000000 | \n","
\n"," \n","
"]},"metadata":{}},{"output_type":"stream","name":"stdout","text":["โ
Training complete!\n"]}],"source":["import torch\n","from transformers import DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer\n","\n","# Prepare data collator\n","data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)\n","\n","# Define training arguments\n","args = Seq2SeqTrainingArguments(\n"," output_dir=\"outputs-lora\",\n"," per_device_train_batch_size=2,\n"," per_device_eval_batch_size=2,\n"," learning_rate=2e-4,\n"," num_train_epochs=1,\n"," save_strategy=\"epoch\",\n"," logging_steps=25,\n"," predict_with_generate=True,\n"," fp16=torch.cuda.is_available(), # Use mixed precision if GPU supports it\n"," report_to=[], # disables online tracking (no API needed)\n",")\n","\n","# Initialise trainer\n","trainer = Seq2SeqTrainer(\n"," model=model,\n"," args=args,\n"," train_dataset=train_tok,\n"," eval_dataset=eval_tok,\n"," tokenizer=tokenizer,\n"," data_collator=data_collator,\n",")\n","\n","print(\"๐ Starting fine-tuningโฆ\")\n","trainer.train()\n","print(\"โ
Training complete!\")"]},{"cell_type":"markdown","id":"cb3261ba-fd89-42f8-8cbc-b9391b859ee6","metadata":{"id":"cb3261ba-fd89-42f8-8cbc-b9391b859ee6"},"source":["Let's test the fine-tuned model to verify that it can generate meaningful summaries. It performs a full inference pass using the model and tokenizer."]},{"cell_type":"code","execution_count":16,"id":"f86f32e1-49c1-426e-b013-3156cb6d6e4f","metadata":{"jp-MarkdownHeadingCollapsed":true,"colab":{"base_uri":"https://localhost:8080/"},"id":"f86f32e1-49c1-426e-b013-3156cb6d6e4f","executionInfo":{"status":"ok","timestamp":1761300634308,"user_tz":-60,"elapsed":233,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"057ca513-731d-438a-a6d3-c41225bfa966"},"outputs":[{"output_type":"stream","name":"stdout","text":["\n","๐ง Fine-tuned Model Output:\n","\n","Bob and Alice discuss the museum's history.\n"]}],"source":["test_input = \"Write a brief summary: Alice and Bob discussed weekend plans. Bob suggested hiking, but Alice preferred visiting the museum.\"\n","\n","# Tokenise and move to model device\n","inputs = tokenizer(test_input, return_tensors=\"pt\", truncation=True, padding=True).to(model.device)\n","\n","# Generate output\n","outputs = model.generate(**inputs, max_new_tokens=80)\n","\n","# Decode and display\n","print(\"\\n๐ง Fine-tuned Model Output:\\n\")\n","print(tokenizer.decode(outputs[0], skip_special_tokens=True))\n"]},{"cell_type":"markdown","id":"3ee4b6cb-1684-49ca-9cc2-74609bf610bd","metadata":{"id":"3ee4b6cb-1684-49ca-9cc2-74609bf610bd"},"source":["This allows interactively test the fine-tuned model with your own custom input."]},{"cell_type":"code","execution_count":17,"id":"3bad36a0-89b4-484d-953c-7371d83cfff6","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"3bad36a0-89b4-484d-953c-7371d83cfff6","executionInfo":{"status":"ok","timestamp":1761300740710,"user_tz":-60,"elapsed":106374,"user":{"displayName":"Durojaye Olusegun","userId":"09188621512197003284"}},"outputId":"cee233ae-58d2-42ac-90a9-3e49430bc355"},"outputs":[{"output_type":"stream","name":"stdout","text":["๐ฌ Try your own prompt!\n","\n","Enter a text or paragraph you'd like the model to summarise: what is it doing \n","\n","๐งฉ Model Output:\n","\n","It is doing it doing it doing it\n"]}],"source":["print(\"๐ฌ Try your own prompt!\")\n","\n","user_prompt = input(\"\\nEnter a text or paragraph you'd like the model to summarise: \")\n","\n","# Tokenise user prompt\n","inputs = tokenizer(user_prompt, return_tensors=\"pt\", truncation=True, padding=True).to(model.device)\n","\n","# Generate output\n","outputs = model.generate(**inputs, max_new_tokens=80)\n","\n","# Decode and print\n","print(\"\\n๐งฉ Model Output:\\n\")\n","print(tokenizer.decode(outputs[0], skip_special_tokens=True))\n"]},{"cell_type":"markdown","id":"a0a3c84e-2d27-46ad-9356-95e2ef9a598b","metadata":{"id":"a0a3c84e-2d27-46ad-9356-95e2ef9a598b"},"source":["In this template, you fine-tuned **Googleโs FLAN-T5-Small** model using **LoRA (Low-Rank Adaptation)** with the **PEFT** library โ a modern, lightweight approach to large language model adaptation.\n","\n","Running this workflow on **Saturn Cloud** makes it both **scalable and cost-effective**. Saturn Cloudโs managed infrastructure allows you to:\n","\n","* Start with a **single NVIDIA GPU** for experimentation and scale up to multi-GPU clusters for larger models.\n","* Collaborate across teams easily through shared Jupyter environments.\n","* Integrate this fine-tuning workflow into production pipelines for enterprise-ready deployment.\n","\n","By using this template, you now have a complete, ready-to-run foundation for **adapter-based fine-tuning** in Saturn Cloud โ ideal for tasks like summarisation, translation, or instruction-following with minimal resource use.\n","\n","To continue exploring, check out:\n","\n","* [Saturn Cloud Documentation](https://saturncloud.io/docs/) โ for advanced configuration and GPU scaling.\n","* [Saturn Cloud Templates](https://saturncloud.io/resources/templates/) โ for more examples of ML, LLM, and data science workflows."]}],"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.13.7"},"colab":{"provenance":[],"gpuType":"T4"},"accelerator":"GPU","widgets":{"application/vnd.jupyter.widget-state+json":{"9db3a5ac0dd84249a2b236b96c58aad8":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_2c821f95cbf94e6f972651544b51bacf","IPY_MODEL_69e89bf8eace41aa850498fd3fd61f99","IPY_MODEL_3aaca7366ecb47d8b4ac27b6301aa91b"],"layout":"IPY_MODEL_48ba285de8364e65a380add6e08e4d69"}},"2c821f95cbf94e6f972651544b51bacf":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_29dfb08a2a1d43b3878cb8a98b285b09","placeholder":"โ","style":"IPY_MODEL_4edaefbb46844f8ba1583f63c20f9ccf","value":"Map:โ100%"}},"69e89bf8eace41aa850498fd3fd61f99":{"model_module":"@jupyter-widgets/controls","model_name":"FloatProgressModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_168534f6a2f3457b8dfa29da5aa15d6a","max":200,"min":0,"orientation":"horizontal","style":"IPY_MODEL_3e54568d0ae94350a1a461a6b1cc3423","value":200}},"3aaca7366ecb47d8b4ac27b6301aa91b":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_bd8316fe2cc24289bf8d39ab6f065e43","placeholder":"โ","style":"IPY_MODEL_d802453c7a484c89897a30b8ddde157b","value":"โ200/200โ[00:13<00:00,โ15.11โexamples/s]"}},"48ba285de8364e65a380add6e08e4d69":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"29dfb08a2a1d43b3878cb8a98b285b09":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"4edaefbb46844f8ba1583f63c20f9ccf":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"168534f6a2f3457b8dfa29da5aa15d6a":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"3e54568d0ae94350a1a461a6b1cc3423":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"bd8316fe2cc24289bf8d39ab6f065e43":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"d802453c7a484c89897a30b8ddde157b":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}}}}},"nbformat":4,"nbformat_minor":5}
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-nim-tgi/README.md b/examples/nlp_and_llms/nvidia-nim-tgi/README.md
new file mode 100644
index 00000000..1da88fb1
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-nim-tgi/README.md
@@ -0,0 +1,222 @@
+# ๐ NIM / TGI Server โ Drop-In API
+
+**Tech Stack:** NVIDIA NIM + TGI (Text Generation Inference)
+**Built for:** Saturn Cloud Custom Templates
+โก๏ธ [https://saturncloud.io/](https://saturncloud.io/)
+
+---
+
+## ๐ง Overview
+
+This template provides a **plug-and-play inference server** that supports **two interchangeable LLM backends**:
+
+| Backend | Description | Use Case |
+| -------------------- | ------------------------------------------------------------ | --------------------------------------------------------------- |
+| **NVIDIA NIM Cloud** | Fully hosted LLMs on NVIDIA's high-performance GPU cloud | High-accuracy, large models (Qwen 80B, Mistral, Nemotron, etc.) |
+| **Local TGI Server** | Lightweight local model running via HuggingFace Transformers | Fast prototyping, offline usage |
+
+The API exposes **the same unified interface** for both backends, so users can switch engines without changing frontend code.
+
+This is ideal for **Saturn Cloud Data Science workflows**, allowing teams to quickly integrate LLM inference inside their notebooks, pipelines, or applications.
+
+---
+
+# ๐ Project Structure
+
+```
+NIM-TGI-Server/
+โ
+โโโ server.py # Main FastAPI server (unified interface)
+โโโ backend_tgi.py # Local TGI backend (SmolLM)
+โโโ backend_nim.py # NVIDIA cloud backend
+โโโ cli.py # CLI tool (select backend from terminal)
+โโโ requirements.txt
+โโโ README.md # (this file)
+```
+
+---
+
+# โ๏ธ 1. Environment Setup
+
+## **Create and activate a virtual environment**
+
+### Linux / MacOS
+
+```bash
+python -m venv venv
+source venv/bin/activate
+```
+
+### Windows (PowerShell)
+
+```powershell
+python -m venv venv
+venv\Scripts\activate
+```
+
+---
+
+## **Install dependencies**
+
+```bash
+pip install -r requirements.txt
+```
+
+---
+
+# ๐ 2. Getting a NVIDIA NIM API Key
+
+To use the **NIM Cloud backend**, you need an **NVIDIA AI Foundation API Key**.
+
+### Steps:
+
+1. Visit:
+ ๐ [https://build.nvidia.com/explore/discover](https://build.nvidia.com/explore/discover)
+2. Sign in with NVIDIA account
+3. Open your "API Keys" panel
+4. Click **Create New API Key**
+5. Copy the key
+6. **Paste it into `backend_nim.py`**, replacing:
+
+```python
+API_KEY = "nvapi-xxxxxxxxxxxxxxxxxxxx"
+```
+
+โ ๏ธ **Note:**
+This template currently embeds the key directly for simplicity, but in production you should store it in environment variables or a secret manager.
+
+---
+
+# ๐ง 3. Backend Models
+
+## **A. NVIDIA NIM Backend (Cloud)**
+
+* Model used: `qwen/qwen3-next-80b-a3b-instruct`
+* Endpoint: `https://integrate.api.nvidia.com/v1`
+* Requires API Key
+* Supports streaming + large prompts
+
+## **B. Local TGI Backend (Lightweight CPU/GPU)**
+
+* Model: `HuggingFaceTB/SmolLM-1.7B-Instruct`
+* Runs entirely inside Python (no Docker needed)
+* Great for local experimentation
+
+---
+
+# ๐ 4. Running the Server
+
+Start FastAPI server:
+
+```bash
+uvicorn server:app --reload
+```
+
+Youโll see:
+
+```
+INFO: Uvicorn running on http://127.0.0.1:8000
+```
+
+---
+
+# ๐งช 5. Testing the Server
+
+## A. Test Local TGI Model
+
+**POST /chat/local**
+
+### Curl:
+
+```bash
+curl -X POST -F "prompt=Explain machine learning" http://localhost:8000/chat/local
+```
+
+### Expected Response:
+
+```json
+{
+ "backend": "tgi-local",
+ "response": "Machine learning is..."
+}
+```
+
+---
+
+## B. Test NVIDIA NIM Model
+
+**POST /chat/nim**
+
+### Curl:
+
+```bash
+curl -X POST -F "prompt=Write a short poem" http://localhost:8000/chat/nim
+```
+
+### Streaming:
+
+```bash
+curl -N -X POST -F "prompt=Tell me a story" -F "stream=true" http://localhost:8000/chat/nim
+```
+
+---
+
+# ๐ฅ๏ธ 6. Command-Line Interface (CLI)
+
+The template includes a **CLI wrapper**:
+
+### Local TGI:
+
+```bash
+python cli.py --backend local "Explain photosynthesis"
+```
+
+### NVIDIA NIM:
+
+```bash
+python cli.py --backend nim "Write 5 facts about Jupiter"
+```
+
+Streaming output works automatically.
+
+---
+
+# ๐ก 7. Using with Saturn Cloud
+
+This template is designed as a **plug-and-play server component** inside Saturn Cloud:
+
+* Run the server inside a Jupyter workspace
+* Use the API from notebooks or external apps
+* Swap between local inference (TGI) and cloud inference (NIM)
+* Ideal for ML research, RAG systems, agent development, and batch inference jobs
+
+Saturn Cloud provides scalable Jupyter environments with GPUs:
+๐ [https://saturncloud.io/](https://saturncloud.io/)
+
+---
+
+# โ๏ธ 8. Summary
+
+This template provides:
+
+### **โ A drop-in inference server**
+
+Supports both NVIDIA Cloud NIM and local TGI backends.
+
+### **โ Ready to use in Saturn Cloud**
+
+Works inside a GPU instance or CPU instance.
+
+### **โ Unified API**
+
+Same route structure for both engines.
+
+### **โ Full CLI + server support**
+
+### **โ Ideal foundation for:**
+
+* Chatbots
+* RAG pipelines
+* Model comparison apps
+* AI feature development
+* ML/DS experimentation
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-nim-tgi/backend_nim.py b/examples/nlp_and_llms/nvidia-nim-tgi/backend_nim.py
new file mode 100644
index 00000000..8638bfc1
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-nim-tgi/backend_nim.py
@@ -0,0 +1,32 @@
+from openai import OpenAI
+import os
+
+# set it up in the key in the environment first
+## sample free API key: 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
+API_KEY = "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"
+
+if not API_KEY:
+ raise ValueError("โ NVIDIA_API_KEY is not set. Export it first!")
+
+client = OpenAI(
+ base_url="https://integrate.api.nvidia.com/v1",
+ api_key=API_KEY,
+)
+
+def nim_chat(prompt, model="qwen/qwen3-next-80b-a3b-instruct", stream=False):
+ completion = client.chat.completions.create(
+ model=model,
+ messages=[{"role": "user", "content": prompt}],
+ temperature=0.6,
+ top_p=0.7,
+ max_tokens=1024,
+ stream=stream
+ )
+
+ if stream:
+ for chunk in completion:
+ delta = chunk.choices[0].delta
+ if delta and delta.content:
+ yield delta.content
+ else:
+ return completion.choices[0].message["content"]
diff --git a/examples/nlp_and_llms/nvidia-nim-tgi/backend_tgi.py b/examples/nlp_and_llms/nvidia-nim-tgi/backend_tgi.py
new file mode 100644
index 00000000..8c8b8c0a
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-nim-tgi/backend_tgi.py
@@ -0,0 +1,24 @@
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MODEL_ID = "HuggingFaceTB/SmolLM-1.7B-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(
+ MODEL_ID,
+ device_map="auto",
+ torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
+)
+
+def tgi_chat(prompt, max_tokens=256, temperature=0.7):
+ formatted_prompt = f"User: {prompt}\nAssistant:"
+ inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)
+
+ outputs = model.generate(
+ **inputs,
+ max_new_tokens=max_tokens,
+ temperature=temperature
+ )
+
+ text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+ return text.split("Assistant:")[-1].strip()
diff --git a/examples/nlp_and_llms/nvidia-nim-tgi/cli.py b/examples/nlp_and_llms/nvidia-nim-tgi/cli.py
new file mode 100644
index 00000000..a54ab10c
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-nim-tgi/cli.py
@@ -0,0 +1,19 @@
+import argparse
+from backend_tgi import tgi_chat
+from backend_nim import nim_chat
+
+parser = argparse.ArgumentParser(description="NIM/TGI CLI")
+parser.add_argument("--backend", choices=["local", "nim"], required=True)
+parser.add_argument("prompt", type=str)
+
+args = parser.parse_args()
+
+if args.backend == "local":
+ print("\n๐ข Local TGI Response:")
+ print(tgi_chat(args.prompt))
+
+else:
+ print("\n๐ข NVIDIA NIM Response:")
+ for chunk in nim_chat(args.prompt, stream=True):
+ print(chunk, end="", flush=True)
+ print("\n")
diff --git a/examples/nlp_and_llms/nvidia-nim-tgi/requirements.txt b/examples/nlp_and_llms/nvidia-nim-tgi/requirements.txt
new file mode 100644
index 00000000..e4211a4c
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-nim-tgi/requirements.txt
@@ -0,0 +1,5 @@
+fastapi
+uvicorn
+transformers
+torch
+openai
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-nim-tgi/server.py b/examples/nlp_and_llms/nvidia-nim-tgi/server.py
new file mode 100644
index 00000000..abb6fdb5
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-nim-tgi/server.py
@@ -0,0 +1,26 @@
+from fastapi import FastAPI, Form
+from fastapi.responses import StreamingResponse, JSONResponse
+from backend_tgi import tgi_chat
+from backend_nim import nim_chat
+
+app = FastAPI(title="NIM / TGI Drop-in API Server")
+
+@app.post("/chat/local")
+def chat_local(prompt: str = Form(...)):
+ response = tgi_chat(prompt)
+ return {"backend": "tgi-local", "response": response}
+
+
+@app.post("/chat/nim")
+def chat_nim(prompt: str = Form(...), stream: bool = False):
+ if stream:
+ generator = nim_chat(prompt, stream=True)
+ return StreamingResponse(generator, media_type="text/event-stream")
+
+ response = nim_chat(prompt, stream=False)
+ return {"backend": "nvidia-nim", "response": response}
+
+
+@app.get("/")
+def root():
+ return {"message": "NIM/TGI Server Running", "endpoints": ["/chat/local", "/chat/nim"]}
diff --git a/examples/nlp_and_llms/nvidia-rag-mini/README.md b/examples/nlp_and_llms/nvidia-rag-mini/README.md
new file mode 100644
index 00000000..aa22e6d3
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-mini/README.md
@@ -0,0 +1,222 @@
+# ๐ง RAG Mini Docs Q&A
+
+A lightweight **Retrieval-Augmented Generation (RAG)** system that lets you drop `.txt` files into a folder and ask natural-language questions about them.
+
+This template combines:
+
+* **SentenceTransformers** for document embeddings
+* **ChromaDB** for vector storage & retrieval
+* **๐ค Transformers (FLAN-T5)** for answer generation
+* **FastAPI** for serving an interactive Q&A API
+
+Designed for fast prototyping and educational use on **[Saturn Cloud](https://saturncloud.io/)**.
+
+---
+
+## ๐ 1. Get Started โ Understand the Folder Layout
+
+Before you start coding, review the project structure below.
+Each file serves a clear role; ensure youโre working from the correct one.
+
+```
+NVIDIA_RAG-MINI/
+โโ data/ # Folder for your .txt documents
+โ โโ saturndoc.txt # Sample document included for testing
+โโ rag_machine.py # Core logic: embeddings, Chroma, QA engine
+โโ rag-api.py # REST API built with FastAPI
+โโ requirements.txt
+```
+
+๐ **Action:** Create or upload `.txt` files into the `data/` folder before running the template.
+A sample file named **`saturndoc.txt`** is already included โ you can use it immediately to test model training and query responses.
+
+---
+
+## ๐งฉ 2. Set Up the Environment
+
+To run this project, youโll need Python โฅ 3.10.
+If youโre using **Saturn Cloud**, create a new environment and install dependencies from `requirements.txt`.
+
+### โ๏ธ Step-by-step
+
+```bash
+# (optional) create a fresh virtual environment
+python -m venv rag-env
+source rag-env/bin/activate # or .\rag-env\Scripts\activate on Windows
+
+# install dependencies
+pip install -r requirements.txt
+```
+
+### ๐ฆ requirements.txt
+
+```text
+torch>=2.2.0
+transformers>=4.44.0
+sentence-transformers>=3.0.0
+chromadb>=0.5.0
+fastapi>=0.115.0
+uvicorn[standard]>=0.30.0
+pydantic>=2.7.0
+tqdm>=4.66.0
+```
+
+๐ **Action:** Run the install command inside your active environment before executing any Python file.
+
+---
+
+## โ๏ธ 3. Configure Models and Paths
+
+All configuration happens inside **`rag_machine.py`**.
+Defaults are already suitable for most cases:
+
+```python
+CHROMA_DIR = "rag_chroma_store" # Persistent database for embeddings
+DATA_DIR = Path("data") # Directory containing your .txt files
+EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+LLM_MODEL = "google/flan-t5-base"
+```
+
+๐ **Action:**
+If you want faster inference, you can change `LLM_MODEL` to `google/flan-t5-small`.
+If you have a GPU, keep `flan-t5-base` or try `flan-t5-large`.
+
+---
+
+## ๐ป 4. Run in CLI Mode โ Test the RAG Machine
+
+Use this mode for quick experimentation.
+The script loads models, indexes your `.txt` files, and opens an interactive prompt.
+
+```bash
+python rag_machine.py
+```
+
+Youโll see output similar to:
+
+```
+๐ง Starting RAG Machine (Transformers + Chroma)...
+โป๏ธ Reindexing documents...
+๐ Indexing 5 documents...
+โ
Indexed 5 documents successfully.
+๐ Current collection size: 5 documents
+โ Enter your question (or 'exit'):
+```
+
+๐ **Action:**
+Type a question like
+`What is this project about?`
+and the model will respond based on your documents.
+
+> You can use the included **`saturndoc.txt`** file for your first run โ itโs already in the `data/` folder and serves as a ready-made example for testing and model training.
+
+---
+
+## ๐ 5. Run as an API โ Serve Questions via HTTP
+
+Now, letโs turn your RAG engine into a service.
+Start the FastAPI server with Uvicorn:
+
+```bash
+uvicorn rag-api:app --reload
+```
+
+Once running, open your browser at [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs)
+to explore the built-in Swagger interface.
+
+### ๐งญ Endpoints
+
+| Endpoint | Method | Description |
+| ---------------------- | ------ | -------------------------------------------------- |
+| `/query` | POST | Submit a question and get an answer |
+| `/reload` *(optional)* | POST | Reindex `.txt` files without restarting the server |
+
+### Example Query
+
+```bash
+curl -X POST "http://127.0.0.1:8000/query" \
+ -H "Content-Type: application/json" \
+ -d "{\"query\": \"What does the onboarding doc say?\"}"
+```
+
+Response:
+
+```json
+{
+ "result": "The onboarding doc explains the project setup and data structure."
+}
+```
+
+๐ **Action:** Use `/query` to test, and `/reload` whenever you add new `.txt` files.
+
+---
+
+## ๐ 6. How It Works (Conceptually)
+
+1. **Document Loading** โ Reads all `.txt` files from `data/`.
+2. **Embedding Generation** โ Converts text into dense vectors using SentenceTransformers.
+3. **Vector Storage** โ Saves these embeddings persistently in **ChromaDB** (`rag_chroma_store/`).
+4. **Retrieval** โ Finds the most relevant text chunks for your query.
+5. **LLM Answering** โ Passes retrieved context + query into **FLAN-T5** to generate the final answer.
+
+๐ **Action:** Skim through `rag_machine.py` to see how each step is implementedโyou can easily swap models or add chunking later.
+
+---
+
+## ๐ 7. Reindex vs Reuse
+
+* **`reindex=True`** โ Clears and rebuilds embeddings from scratch
+* **`reindex=False`** โ Loads existing persistent store (faster)
+
+```python
+index_documents(reindex=True) # rebuild everything
+index_documents(reindex=False) # reuse old vectors
+```
+
+๐ **Action:**
+Use reindexing only after you add or update text files in `data/`.
+The included **`saturndoc.txt`** is already indexed by default when you run the script for the first time โ so you can test immediately without adding new documents.
+
+---
+
+## ๐งฉ 8. Best Practices
+
+* Keep each text file focused on one topic for cleaner retrieval.
+* For long documents, consider manually splitting them into sections.
+* If using CPU only, choose smaller models for faster inference.
+* Delete the `rag_chroma_store/` folder to fully reset the database.
+
+---
+
+## ๐ฐ๏ธ 9. Deploying on Saturn Cloud
+
+You can easily host this on **Saturn Cloud**:
+
+1. Create a new Jupyter or VS Code resource.
+2. Upload this project folder.
+3. Install requirements:
+
+ ```bash
+ pip install -r requirements.txt
+ ```
+4. Run `python rag_machine.py` to test indexing.
+5. Launch the API:
+
+ ```bash
+ uvicorn rag-api:app --host 0.0.0.0 --port 8000
+ ```
+6. Expose port **8000** in your Saturn environment to access it externally.
+
+๐ Learn more about Saturn Cloud and GPU-accelerated workflows at **[https://saturncloud.io](https://saturncloud.io)**
+
+---
+
+## ๐ Credits
+
+Built with โค๏ธ using:
+
+* ๐ค **Transformers**
+* ๐ง **SentenceTransformers**
+* ๐พ **ChromaDB**
+* โก **FastAPI**
+* and hosted proudly on **[Saturn Cloud](https://saturncloud.io/)**
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-rag-mini/data/saturndoc.txt b/examples/nlp_and_llms/nvidia-rag-mini/data/saturndoc.txt
new file mode 100644
index 00000000..f9375715
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-mini/data/saturndoc.txt
@@ -0,0 +1,5 @@
+Saturn Cloud provides a scalable cloud platform for data science and machine learning.
+It supports Jupyter environments, Dask clusters, and GPU-powered instances.
+Users can collaborate on notebooks, deploy APIs, and run scheduled jobs.
+You can also fine-tune large language models and deploy them with minimal effort.
+Saturn Cloud offers integrations with Hugging Face, AWS, and GitHub.
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-rag-mini/rag-api.py b/examples/nlp_and_llms/nvidia-rag-mini/rag-api.py
new file mode 100644
index 00000000..aa072e79
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-mini/rag-api.py
@@ -0,0 +1,17 @@
+from fastapi import FastAPI
+from pydantic import BaseModel
+from rag_machine import query_docs, index_documents
+
+app = FastAPI(title="RAG Mini Docs Q&A")
+
+class QueryRequest(BaseModel):
+ query: str
+
+@app.on_event("startup")
+def startup_event():
+ index_documents(reindex=False)
+
+@app.post("/query")
+def query(req: QueryRequest):
+ answer = query_docs(req.query)
+ return {"result": answer}
diff --git a/examples/nlp_and_llms/nvidia-rag-mini/rag_machine.py b/examples/nlp_and_llms/nvidia-rag-mini/rag_machine.py
new file mode 100644
index 00000000..5721867a
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-mini/rag_machine.py
@@ -0,0 +1,114 @@
+# rag_machine.py
+from pathlib import Path
+import os
+import torch
+from sentence_transformers import SentenceTransformer
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+import chromadb
+
+# --------------------------
+# ๐ง Configuration
+# --------------------------
+CHROMA_DIR = "rag_chroma_store"
+DATA_DIR = Path("data")
+EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+LLM_MODEL = "google/flan-t5-base"
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+DATA_DIR.mkdir(exist_ok=True)
+Path(CHROMA_DIR).mkdir(exist_ok=True)
+
+# --------------------------
+# โ๏ธ Initialize Components
+# --------------------------
+print("๐ Loading models...")
+embedder = SentenceTransformer(EMBED_MODEL)
+tokenizer = AutoTokenizer.from_pretrained(LLM_MODEL)
+llm = AutoModelForSeq2SeqLM.from_pretrained(LLM_MODEL)
+
+client = chromadb.PersistentClient(path=CHROMA_DIR)
+collection = client.get_or_create_collection("rag_docs")
+
+# --------------------------
+# ๐ Document Loader
+# --------------------------
+def load_all_documents(data_dir: Path):
+ docs = []
+ for file in data_dir.glob("*.txt"):
+ with open(file, "r", encoding="utf-8") as f:
+ text = f.read().strip()
+ if text:
+ docs.append({"file": file.name, "text": text})
+ print(f"๐ Loaded: {file.name}")
+ return docs
+
+# --------------------------
+# ๐ข Index Documents
+# --------------------------
+def index_documents(reindex: bool = False):
+ """Rebuild or load existing document embeddings."""
+ if reindex:
+ print("โป๏ธ Reindexing documents...")
+ try:
+ collection.reset()
+ print("๐งน Cleared existing collection.")
+ except AttributeError:
+ ids = collection.get()["ids"]
+ if ids:
+ collection.delete(ids=ids)
+ print("๐งน Deleted existing documents manually.")
+
+ docs = load_all_documents(DATA_DIR)
+ for i, d in enumerate(docs):
+ emb = embedder.encode(d["text"])
+ collection.add(
+ ids=[str(i)],
+ documents=[d["text"]],
+ embeddings=[emb.tolist()],
+ metadatas=[{"source": d["file"]}],
+ )
+ print("โ
Documents reindexed and stored in Chroma.")
+ else:
+ print("๐ฆ Using existing Chroma store.")
+
+
+# --------------------------
+# ๐ Query System
+# --------------------------
+def query_docs(question: str, top_k: int = 3):
+ """Retrieve top-k relevant docs and generate an answer."""
+ print(f"\n๐ Question: {question}")
+
+ # Embed the query and search
+ q_emb = embedder.encode(question).tolist()
+ results = collection.query(query_embeddings=[q_emb], n_results=top_k)
+
+ if not results["documents"]:
+ return "No relevant documents found."
+
+ context = "\n".join(results["documents"][0])
+ prompt = f"Answer based on the following context:\n{context}\n\nQuestion: {question}"
+
+ inputs = tokenizer(prompt, return_tensors="pt", truncation=True)
+ outputs = llm.generate(**inputs, max_length=512)
+ answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+ return answer
+
+# --------------------------
+# ๐งช CLI Test Mode
+# --------------------------
+if __name__ == "__main__":
+ print("๐ง Starting RAG Machine (Transformers + Chroma)...")
+ index_documents(reindex=True)
+
+ while True:
+ q = input("\nโ Enter your question (or 'exit'): ").strip()
+ if q.lower() == "exit":
+ break
+ try:
+ ans = query_docs(q)
+ print(f"\n๐ฌ {ans}\n")
+ except Exception as e:
+ print(f"โ ๏ธ Error: {e}")
diff --git a/examples/nlp_and_llms/nvidia-rag-mini/requirements.txt b/examples/nlp_and_llms/nvidia-rag-mini/requirements.txt
new file mode 100644
index 00000000..624f58dd
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-mini/requirements.txt
@@ -0,0 +1,7 @@
+torch>=2.2.0
+transformers>=4.44.0
+sentence-transformers>=3.0.0
+chromadb>=0.5.0
+fastapi>=0.115.0
+uvicorn[standard]>=0.30.0
+pydantic>=2.7.0
diff --git a/examples/nlp_and_llms/nvidia-rag-serve-api/README.md b/examples/nlp_and_llms/nvidia-rag-serve-api/README.md
new file mode 100644
index 00000000..911fabd4
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-serve-api/README.md
@@ -0,0 +1,105 @@
+# ๐ Ray Serve LLM API โ Qwen 1.5B (vLLM)
+
+This template shows how to deploy a **Qwen2.5-1.5B-Instruct LLM** using:
+
+* **Ray Serve**
+* **vLLM**
+* **OpenAI-compatible API format**
+
+You get a local inference server running at:
+
+```
+http://127.0.0.1:8000/v1/chat/completions
+```
+
+This template is designed for **Saturn Cloud custom templates** so users can plug-and-play LLM inference environments with GPU acceleration.
+
+๐ **Back to Saturn Cloud โ [https://saturncloud.io](https://saturncloud.io)**
+
+---
+
+## ๐ Features
+
+* Fully OpenAI-compatible API endpoint
+* Deploys Qwen 1.5B using vLLM (fast inference)
+* Simple Ray Serve deployment
+* Example client request included
+* Clean and minimal code structure
+* Works inside Jupyter or full terminal environment
+
+---
+
+## ๐ฆ Requirements
+
+The notebook installs everything automatically:
+
+```
+torch
+transformers
+ray[serve, llm]
+fastapi
+uvicorn
+requests
+huggingface_hub
+```
+
+GPU recommended for optimal performance.
+
+---
+
+## ๐ Project Structure
+
+```
+ray-serve-llm/
+โ
+โโโ serve_llm.py # Ray Serve deployment definition
+โโโ start_server.py # Ray launcher (if using outside notebook)
+โโโ test_client.py # Example API client test
+โโโ ray_serve_llm_template.ipynb # Full Jupyter notebook template (generated)
+```
+
+---
+
+## โถ๏ธ How It Works
+
+### 1. Write your Ray Serve deployment file
+
+Defines:
+
+* Model ID (`Qwen2.5-1.5B-Instruct`)
+* Engine config
+* Autoscaling
+* OpenAI-compatible app
+
+### 2. Start Ray and deploy the model
+
+Ray Serve loads the model via vLLM and exposes the API.
+
+### 3. Send a test request
+
+JSON API format identical to OpenAI:
+
+```python
+payload = {
+ "model": "qwen-1.5b",
+ "messages": [{"role": "user", "content": "Explain API design."}]
+}
+```
+
+### 4. Extract the assistant text
+
+```python
+res = out.json()["choices"][0]["message"]["content"]
+```
+
+---
+
+## ๐ Conclusion
+
+This template provides a clean, reproducible Ray Serve LLM deployment that works both in Jupyter and full terminal mode.
+You can adapt it to larger models, scale it across nodes, or wrap it inside FastAPI.
+
+๐ **Back to Saturn Cloud โ [https://saturncloud.io](https://saturncloud.io)**
+
+---
+
diff --git a/examples/nlp_and_llms/nvidia-rag-serve-api/ray_serve_llm.ipynb b/examples/nlp_and_llms/nvidia-rag-serve-api/ray_serve_llm.ipynb
new file mode 100644
index 00000000..fe2de35a
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-rag-serve-api/ray_serve_llm.ipynb
@@ -0,0 +1,188 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "d1a1950f",
+ "metadata": {},
+ "source": [
+ "# ๐ Ray Serve LLM API\n",
+ "\n",
+ "This template demonstrates how to deploy **Models** using **Ray Serve + vLLM** and expose it through an **OpenAI-compatible API**.\n",
+ "\n",
+ "This a custom template on **Saturn Cloud custom templates** so users can plug-and-play LLM inference environments with GPU acceleration.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d06caaab",
+ "metadata": {},
+ "source": [
+ "## ๐ฆ Install required libraries\n",
+ "Install all the requireed library for the template"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "78b5ec11",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install required libraries\n",
+ "!pip install torch transformers fastapi uvicorn \"ray[serve, llm]\" requests huggingface_hub\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ec4c2ac1",
+ "metadata": {},
+ "source": [
+ "## ๐งฉ Create Ray Serve Deployment File\n",
+ "\n",
+ "his writes a file called **`serve_llm.py`** which:\n",
+ "\n",
+ "* Configures the model (Qwen2.5-1.5B-Instruct)\n",
+ "* Creates a Ray Serve LLMConfig\n",
+ "* Builds an OpenAI-compatible API using Ray's `build_openai_app`\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bc3b43ec",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%writefile serve_llm.py\n",
+ "from ray.serve.llm import LLMConfig, build_openai_app\n",
+ "\n",
+ "MODEL_ID = \"Qwen/Qwen2.5-1.5B-Instruct\"\n",
+ "MODEL_ALIAS = \"qwen-1.5b\"\n",
+ "\n",
+ "engine_kwargs = dict(\n",
+ " tensor_parallel_size=1,\n",
+ " max_model_len=4096,\n",
+ ")\n",
+ "\n",
+ "deployment_config = dict(\n",
+ " autoscaling_config=dict(\n",
+ " min_replicas=1,\n",
+ " max_replicas=1,\n",
+ " )\n",
+ ")\n",
+ "\n",
+ "llm_config = LLMConfig(\n",
+ " model_loading_config=dict(\n",
+ " model_id=MODEL_ALIAS,\n",
+ " model_source=MODEL_ID,\n",
+ " ),\n",
+ " engine_kwargs=engine_kwargs,\n",
+ " deployment_config=deployment_config,\n",
+ ")\n",
+ "\n",
+ "app = build_openai_app({\"llm_configs\": [llm_config]})"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8f3464f5",
+ "metadata": {},
+ "source": [
+ "## โถ๏ธ Start Ray Serve and Deploy the Model\n",
+ "\n",
+ "This will:\n",
+ "\n",
+ "* Initialize Ray\n",
+ "* Start Ray Serve\n",
+ "* Deploy the Qwen model as an API at:\n",
+ " **[http://127.0.0.1:8000/v1/chat/completions](http://127.0.0.1:8000/v1/chat/completions)**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1e011e24",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import ray\n",
+ "from serve_llm import app\n",
+ "from ray import serve\n",
+ "\n",
+ "ray.init(ignore_reinit_error=True)\n",
+ "\n",
+ "serve.start(detached=False)\n",
+ "serve.run(app)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3700cc7d",
+ "metadata": {},
+ "source": [
+ "## ๐ฌ Test the API\n",
+ "\n",
+ "Sends a real chat request to your Ray Serve LLM deployment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "eb912c3a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import requests\n",
+ "\n",
+ "payload = {\n",
+ " \"model\": \"qwen-1.5b\",\n",
+ " \"messages\": [{\"role\": \"user\", \"content\": \"Explain API design.\"}]\n",
+ "}\n",
+ "\n",
+ "out = requests.post(\"http://127.0.0.1:8000/v1/chat/completions\", json=payload)\n",
+ "print(out.json())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "78c72539",
+ "metadata": {},
+ "source": [
+ "## โจ Extract Only the Model \n",
+ "\n",
+ "This grabs the generated text only (no metadata)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e440e110",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "res = out.json()[\"choices\"][0][\"message\"][\"content\"]\n",
+ "print(res)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "17f4ac64",
+ "metadata": {},
+ "source": [
+ "## ๐ **Conclusion**\n",
+ "\n",
+ "You now have a fully running **Ray Serve LLM API** using Qwen2.5-1.5B-Instruct, powered by **vLLM** and exposed through an **OpenAI-compatible endpoint**.\n",
+ "This template can be extended to larger models, added to pipelines, or used inside production-grade ML workloads within Saturn Cloud.\n",
+ "\n",
+ "๐ **Back to Saturn Cloud โ [https://saturncloud.io](https://saturncloud.io)**"
+ ]
+ }
+ ],
+ "metadata": {
+ "language_info": {
+ "name": "python"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/nlp_and_llms/nvidia-vector-db/.env b/examples/nlp_and_llms/nvidia-vector-db/.env
new file mode 100644
index 00000000..1622d39c
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/.env
@@ -0,0 +1,3 @@
+ZILLIZ_URI="https://in03-e969f44404493f8.serverless.aws-eu-central-1.cloud.zilliz.com"
+ZILLIZ_TOKEN="a71de8fc4a75f5cb758d0fcf2b92fb2ebc1f851d7e776247d440e887cd355d7b575649f63a514fb7b78fdeac6f3b416e2ef11150"
+PG_CONNECTION="postgresql://neondb_owner:npg_ymHkZNUVr2I7@ep-lingering-silence-ah4wmlqw-pooler.c-3.us-east-1.aws.neon.tech/neondb?sslmode=require&channel_binding=require"
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-vector-db/README.md b/examples/nlp_and_llms/nvidia-vector-db/README.md
new file mode 100644
index 00000000..0d97129f
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/README.md
@@ -0,0 +1,233 @@
+
+# ๐ **Vector DB Menu (FAISS โข Zilliz Milvus โข Neon PGVector)**
+
+> A unified FastAPI search service that lets you test and compare **FAISS (local)**, **Milvus (Zilliz Cloud free tier)**, and **PostgreSQL with PGVector (Neon free tier)** using a common API.
+
+๐ **Built for the Saturn Cloud AI Community**
+๐ [https://saturncloud.io/](https://saturncloud.io/)
+
+---
+
+## ๐ง Overview
+
+This project loads a public dataset (State of the Union speeches), embeds it with `sentence-transformers/all-MiniLM-L6-v2`, stores vectors in **three different databases**, and exposes a **FastAPI endpoint** to query them interchangeably.
+
+### โ
Whatโs included:
+
+* FAISS (local in-memory vector search)
+* Milvus (via **Zilliz Cloud free tier**)
+* PostgreSQL + PGVector (via **Neon free tier**)
+* FastAPI for querying all 3 backends
+* CLI & Browser UI testing
+* Modular, deploy-ready architecture
+
+---
+
+## โ ๏ธ Free-Tier Credentials Notice
+
+This repo includes **working test credentials** for quick validation.
+However, because they are **free-tier**, they may:
+
+โ ๏ธ expire at any time
+โ ๏ธ be rate-limited
+โ ๏ธ be deleted automatically
+
+โ
You are **strongly encouraged to create your own accounts** using the setup guide below.
+
+---
+
+---
+
+# ๐ ๏ธ **1. Project Setup**
+
+### Clone Repository
+
+```sh
+git clone https://github.com/your-repo/nvidia-vector-db.git
+cd nvidia-vector-db
+```
+
+---
+
+### Create and Activate Virtual Environment
+
+#### Windows (PowerShell)
+
+```sh
+python -m venv vectordb-env
+vectordb-env\Scripts\activate
+```
+
+#### macOS / Linux
+
+```sh
+python3 -m venv vectordb-env
+source vectordb-env/bin/activate
+```
+
+---
+
+### Install Dependencies
+
+```sh
+pip install -r requirements.txt
+```
+
+---
+
+# โ๏ธ **2. Create Neon (PostgreSQL + PGVector) Free Account**
+
+1. Visit: [https://neon.tech/](https://neon.tech/)
+2. Click **Sign Up** (free tier)
+3. Create a new project
+4. Go to **Dashboard โ Connection Details**
+5. Copy the connection string:
+
+ ```
+ postgresql://:@.neon.tech/?sslmode=require
+ ```
+6. Edit it to SQLAlchemy format for this project:
+
+ ```
+ postgresql+psycopg2://:@.neon.tech/?sslmode=require
+ ```
+
+---
+
+# โ๏ธ **3. Create Zilliz Cloud (Milvus) Free Account**
+
+1. Visit: [https://cloud.zilliz.com/signup](https://cloud.zilliz.com/signup)
+2. Create account (Free tier)
+3. Create a new **Serverless cluster**
+4. Go to **API Keys**
+5. Copy:
+
+ * `Public Endpoint (URI)`
+ * `API Key (Token)`
+
+Example:
+
+```
+ZILLIZ_URI=https://in03-xxxx.serverless.aws-eu-central-1.cloud.zilliz.com
+ZILLIZ_TOKEN=xxxxxxxxxxxxxxxxxxxx
+```
+
+---
+
+# ๐งฉ **4. Configure Environment Variables**
+
+Create a `.env` file in the project root:
+
+```
+PG_CONNECTION=postgresql+psycopg2://your-user:your-pass@your-host.neon.tech/your-db?sslmode=require
+
+ZILLIZ_URI=https://your-zilliz-endpoint.serverless.aws-xyz.cloud.zilliz.com
+ZILLIZ_TOKEN=your-zilliz-api-key
+```
+
+> โ ๏ธ You may test with the current free credentials included in the code, but replace them when creating yours.
+
+---
+
+# ๐งฑ **5. Load Dataset & Build Vector Stores**
+
+Run:
+
+```sh
+python data_loader.py
+```
+
+Expected output:
+
+```
+โ
Split into X chunks
+๐ Loading FAISS...
+๐ Connecting to Zilliz Cloud...
+๐ Connecting to PGVector (Neon)...
+โ
All vector stores ready!
+```
+
+---
+
+# ๐ **6. Start the FastAPI Server**
+
+```sh
+uvicorn app:app --reload
+```
+
+Server should start at:
+
+```
+http://127.0.0.1:8000
+```
+
+Swagger UI (API testing interface):
+
+```
+http://127.0.0.1:8000/docs
+```
+
+---
+
+# ๐งช **7. Test the API**
+
+## โ
Browser UI (Swagger)
+
+1. Open: `http://127.0.0.1:8000/docs`
+2. Go to **POST /search**
+3. Test queries like:
+
+```json
+{
+ "db": "faiss",
+ "query": "Who talked about peace?",
+ "k": 3
+}
+```
+
+Try other DBs:
+
+```json
+{ "db": "milvus", "query": "war economy", "k": 3 }
+{ "db": "pgvector", "query": "mars mission", "k": 3 }
+```
+
+---
+
+## โ
CLI Testing with `curl`
+
+```sh
+curl -X POST "http://127.0.0.1:8000/search" \
+ -H "Content-Type: application/json" \
+ -d '{"db":"faiss", "query":"state of the economy", "k":2}'
+```
+
+```sh
+curl -X POST "http://127.0.0.1:8000/search" \
+ -H "Content-Type: application/json" \
+ -d '{"db":"milvus", "query":"foreign policy", "k":2}'
+```
+
+```sh
+curl -X POST "http://127.0.0.1:8000/search" \
+ -H "Content-Type: application/json" \
+ -d '{"db":"pgvector", "query":"education reform", "k":2}'
+```
+
+---
+
+# ๐งฌ Supported Vector Backends
+
+| Backend | Type | Notes |
+| ----------------- | -------------- | ------------------------------------ |
+| **FAISS** | Local | Fastest, no cloud, resets on restart |
+| **Zilliz Milvus** | Cloud | Free tier, scalable, best for prod |
+| **Neon PGVector** | Cloud Postgres | SQL + vectors, persistent, queryable |
+
+
+# ๐ About Saturn Cloud
+
+If you're experimenting with **GPU workloads, LLM inference, vector search, or MLOps**, check out the best community platform for AI builders:
+
+๐ **[https://saturncloud.io/](https://saturncloud.io/)**
+
diff --git a/examples/nlp_and_llms/nvidia-vector-db/app.py b/examples/nlp_and_llms/nvidia-vector-db/app.py
new file mode 100644
index 00000000..78b9df69
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/app.py
@@ -0,0 +1,19 @@
+from fastapi import FastAPI
+from pydantic import BaseModel
+from vectordb import search
+
+app = FastAPI(title="RAG Vector DB Compare API")
+
+class QueryRequest(BaseModel):
+ db: str # faiss | milvus | pgvector
+ query: str
+ k: int = 3
+
+@app.post("/search")
+def search_vectors(req: QueryRequest):
+ results = search(req.db.lower(), req.query, req.k)
+ return {
+ "db": req.db,
+ "query": req.query,
+ "results": [r.page_content[:400] for r in results]
+ }
diff --git a/examples/nlp_and_llms/nvidia-vector-db/data_loader.py b/examples/nlp_and_llms/nvidia-vector-db/data_loader.py
new file mode 100644
index 00000000..f4043694
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/data_loader.py
@@ -0,0 +1,14 @@
+from datasets import load_dataset
+from langchain_core.documents import Document
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+
+def load_and_chunk():
+ print("๐ฅ Loading dataset...")
+ ds = load_dataset("jsulz/state-of-the-union-addresses")
+
+ texts = [row["speech_html"] for row in ds["train"]]
+ docs = [Document(page_content=t) for t in texts]
+
+ splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
+ return splitter.split_documents(docs)
diff --git a/examples/nlp_and_llms/nvidia-vector-db/embed.py b/examples/nlp_and_llms/nvidia-vector-db/embed.py
new file mode 100644
index 00000000..ef3d3f13
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/embed.py
@@ -0,0 +1,4 @@
+from langchain_huggingface import HuggingFaceEmbeddings
+
+def get_embeddings():
+ return HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
diff --git a/examples/nlp_and_llms/nvidia-vector-db/requirements.txt b/examples/nlp_and_llms/nvidia-vector-db/requirements.txt
new file mode 100644
index 00000000..75eb6e47
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/requirements.txt
@@ -0,0 +1,15 @@
+fastapi
+uvicorn
+datasets
+langchain
+langchain-core
+langchain-huggingface
+langchain-community
+langchain-postgres
+sentence-transformers
+langchain-milvus
+pymilvus
+faiss-cpu
+psycopg2-binary
+python-dotenv
+sqlalchemy
diff --git a/examples/nlp_and_llms/nvidia-vector-db/vectordb.py b/examples/nlp_and_llms/nvidia-vector-db/vectordb.py
new file mode 100644
index 00000000..47ccf520
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vector-db/vectordb.py
@@ -0,0 +1,59 @@
+import os
+from langchain_community.vectorstores import FAISS
+from langchain_postgres.vectorstores import PGVector
+from langchain_milvus import Milvus
+from pymilvus import connections
+from sqlalchemy import create_engine
+
+
+from data_loader import load_and_chunk
+from embed import get_embeddings
+
+# Load data + embeddings
+docs = load_and_chunk()
+embeddings = get_embeddings()
+
+# ---------- FAISS ----------
+print("๐ Loading FAISS...")
+faiss_db = FAISS.from_documents(docs, embeddings)
+
+# ---------- Milvus (Zilliz) ----------
+print("๐ Connecting to Zilliz Cloud...")
+connections.connect(
+ alias="default",
+ uri=os.getenv("ZILLIZ_URI"),
+ token=os.getenv("ZILLIZ_TOKEN")
+)
+milvus_db = Milvus.from_documents(
+ docs,
+ embeddings,
+ collection_name="state_union_collection",
+ connection_args={"alias": "default"},
+)
+
+# ---------- PGVector ----------
+print("๐ Connecting to PGVector...")
+NEON_CONN = os.getenv("PG_CONNECTION") # must contain full neon URL
+print("๐ Connecting to Neon PGVector...")
+engine = create_engine(NEON_CONN)
+
+pg_db = PGVector.from_documents(
+ docs,
+ embeddings,
+ connection=engine,
+ collection_name="state_union_pg"
+)
+
+print("โ
PGVector (Neon) loaded!")
+
+print("โ
All vector DBs ready!")
+
+# Generic search method
+def search(db: str, query: str, k: int = 3):
+ if db == "faiss":
+ return faiss_db.similarity_search(query, k)
+ if db == "milvus":
+ return milvus_db.similarity_search(query, k)
+ if db == "pgvector":
+ return pg_db.similarity_search(query, k)
+ return []
diff --git a/examples/nlp_and_llms/nvidia-vllm-7b/README.md b/examples/nlp_and_llms/nvidia-vllm-7b/README.md
new file mode 100644
index 00000000..503f7a73
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vllm-7b/README.md
@@ -0,0 +1,69 @@
+# ๐ง LLM Inference with vLLM 7B
+
+**Saturn Cloud | GPU-Optimised Template**
+
+Run and serve large language models (LLMs) efficiently using **vLLM**, a high-performance inference and serving engine designed for speed and scalability.
+This Saturn Cloud template demonstrates how to deploy **7B-class models** such as *Mistral*, *Llama*, or *Gemma* for text generation and interactive inference.
+
+---
+
+## ๐ Overview
+
+**vLLM** delivers lightning-fast text generation through techniques such as **PagedAttention**, **continuous batching**, and **quantisation**.
+On **Saturn Cloud**, this notebook enables you to:
+
+* Deploy and test 7B-class LLMs for inference and serving.
+* Scale seamlessly from a single GPU to **multi-GPU clusters**.
+* Experiment interactively or integrate models into larger data-science pipelines.
+
+> โ๏ธ Fully compatible with Saturn Cloudโs managed GPU environments and ready for immediate use.
+
+---
+
+## ๐งฉ Features
+
+* **Pre-configured vLLM environment** for fast setup.
+* **Support for NVIDIA GPUs** (A10G, A100) and multi-GPU scaling.
+* **Quick-start workflow**: load, run, and test model prompts.
+* **Local API-style inference** via vLLMโs serving engine.
+* **Interactive prompt input** for experimentation.
+
+---
+
+## ๐ Requirements
+
+* **Saturn Cloud account** with GPU instance access.
+* Python โฅ 3.12
+* Compatible with **CUDA 12.0+** and **Transformers โฅ 4.40**
+
+All dependencies are pre-installed when running the notebook on Saturn Cloud.
+
+---
+
+## ๐ก Usage
+
+1. **Open the template** in Saturn Cloud.
+2. **Select a GPU instance** (A10G or A100 recommended).
+3. **Run the notebook cells sequentially** to:
+
+ * Install dependencies
+ * Configure vLLM settings
+ * Load and test your model
+ * Input prompts interactively to generate text
+
+> For production, vLLM can also serve models as an **OpenAI-compatible API** using the `vllm serve` command.
+
+---
+
+## ๐งญ Learn More
+
+* [Saturn Cloud Documentation](https://saturncloud.io/docs/?utm_source=github&utm_medium=template)
+* [Saturn Cloud Templates](https://saturncloud.io/templates/?utm_source=github&utm_medium=template)
+* [vLLM Official Docs](https://docs.vllm.ai/en/latest/?utm_source=saturn&utm_medium=template)
+
+---
+
+## ๐ Conclusion
+
+This template provides a ready-to-run setup for **LLM inference with vLLM 7B on Saturn Cloud**, combining high performance, scalability, and ease of use.
+Adapt it for experimentation, prototyping, or production-grade LLM deployments in your Saturn Cloud workspace.
diff --git a/examples/nlp_and_llms/nvidia-vllm-7b/nvidia_vllm_7b.ipynb b/examples/nlp_and_llms/nvidia-vllm-7b/nvidia_vllm_7b.ipynb
new file mode 100644
index 00000000..9c9dac42
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vllm-7b/nvidia_vllm_7b.ipynb
@@ -0,0 +1,255 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Es_w2TvemoO3"
+ },
+ "source": [
+ "# LLM Inference vLLM 7B\n",
+ "\n",
+ " \n",
+ "\n",
+ "**vLLM** is a high-performance inference and serving engine for large language models, optimised for speed and scalability. It delivers efficient text generation through innovations such as **PagedAttention**,** continuous batching**, and support for **quantisation**.\n",
+ "\n",
+ "This is a template demonstrates on how to run **7B-class models** (e.g. Mistral, Llama, Gemma) on Saturn Cloud.\n",
+ "\n",
+ "On [Saturn Cloud](https://saturncloud.io), you can scale from a single NVIDIA GPU to multi-GPU clusters, enabling distributed inference for larger models or higher throughput workloads โ all within a managed, GPU-ready environment."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "1hhl8dEPmoO5"
+ },
+ "source": [
+ "## 1. Install dependencies\n",
+ "\n",
+ "\n",
+ "We install **vLLM** and **Transformers**. A recent NVIDIA CUDA runtime is recommended for best performance."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "xDTiLAdfmoO6"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -q jedi\n",
+ "!pip install -q vllm transformers\n",
+ "!pip install uv\n",
+ "!uv venv vllm-env -p 3.12\n",
+ "!source vllm-env/bin/activate && uv pip install vllm\n",
+ "!source vllm-env/bin/activate && pip install ipykernel\n",
+ "!python -m ipykernel install --user --name=vllm-env --display-name \"vLLM Env\"\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ehqOzc4hmoO8"
+ },
+ "source": [
+ "## 2. Environment check\n",
+ "\n",
+ "Verify the GPU is visible and print library versions. Confirm the environment is GPU-enabled."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "_A7AYnJmmoO9"
+ },
+ "outputs": [],
+ "source": [
+ "import torch, platform\n",
+ "import vllm, transformers\n",
+ "\n",
+ "cuda_ok = torch.cuda.is_available()\n",
+ "print(f\"โ
CUDA available: {cuda_ok}\")\n",
+ "if cuda_ok:\n",
+ " print(\"๐ง GPU:\", torch.cuda.get_device_name(0))\n",
+ "print(\"๐งฉ torch:\", torch.__version__)\n",
+ "print(\"๐งฉ vllm:\", vllm.__version__)\n",
+ "print(\"๐งฉ transformers:\", transformers.__version__)\n",
+ "print(\"๐ python:\", platform.python_version())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Qpk7TkAhmoO-"
+ },
+ "source": [
+ "## 3. Select model and vLLM settings\n",
+ "\n",
+ "Choose a **7B** model from Hugging Face. The defaults below work with common, openly available options. If a model is gated, select a different one."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "Vujk0jtwmoO-"
+ },
+ "outputs": [],
+ "source": [
+ "# ๐ง Model & runtime config (edit these as needed)\n",
+ "MODEL_ID = \"mistralai/Mistral-7B-Instruct-v0.2\" # e.g., \"meta-llama/Llama-2-7b-chat-hf\", \"google/gemma-7b\"\n",
+ "DTYPE = \"auto\" # \"auto\", \"float16\", \"bfloat16\", \"float32\"\n",
+ "TENSOR_PARALLEL = 1 # single GPU = 1\n",
+ "GPU_MEMORY_UTIL = 0.90 # 0.6โ0.95 depending on VRAM\n",
+ "MAX_MODEL_LEN = 8192 # context length (depends on model)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "gMjiJkoTmoPA"
+ },
+ "source": [
+ "## 4. Basic model inference\n",
+ "\n",
+ "Load the model with **vLLM** and generate text for one or more prompts using **SamplingParams** (temperature, top_p, max_tokens, etc.)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "D7IXT5FWmoPB"
+ },
+ "outputs": [],
+ "source": [
+ "from vllm import LLM, SamplingParams\n",
+ "\n",
+ "print(\"โณ Loading model (this may download weights on first run)...\")\n",
+ "llm = LLM(\n",
+ " model=MODEL_ID,\n",
+ " dtype=DTYPE,\n",
+ " tensor_parallel_size=TENSOR_PARALLEL,\n",
+ " gpu_memory_utilization=GPU_MEMORY_UTIL,\n",
+ " max_model_len=MAX_MODEL_LEN,\n",
+ ")\n",
+ "print(\"โ
Model loaded!\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 5. Sample prompts\n",
+ "\n",
+ "Use the customise Let's test the model using sample prompts."
+ ],
+ "metadata": {
+ "id": "yaaCIaOfDILx"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Example prompts\n",
+ "prompts = [\n",
+ " \"You are a helpful assistant. Summarise why efficient attention helps LLM inference.\",\n",
+ " \"List three creative uses of a 7B model for education.\",\n",
+ "]\n",
+ "\n",
+ "# Sampling parameters\n",
+ "sampling = SamplingParams(\n",
+ " temperature=0.7,\n",
+ " top_p=0.9,\n",
+ " max_tokens=256,\n",
+ ")\n",
+ "\n",
+ "# Generate\n",
+ "outputs = llm.generate(prompts, sampling)\n",
+ "for out in outputs:\n",
+ " print(\"\\n---\")\n",
+ " print(\"Prompt:\", out.prompt)\n",
+ " print(\"Completion:\", out.outputs[0].text.strip())\n"
+ ],
+ "metadata": {
+ "id": "1s_ALheCCwfP"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## 6. User Custom Prompt Testing\n",
+ "\n",
+ "You can enter your prompt to test the model's chat capabilities here."
+ ],
+ "metadata": {
+ "id": "kaSLGm0_GL62"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Helper function for quick generation\n",
+ "def generate_text(prompt, temperature=0.7, top_p=0.9, max_tokens=256):\n",
+ " params = SamplingParams(temperature=temperature, top_p=top_p, max_tokens=max_tokens)\n",
+ " result = llm.generate([prompt], params)[0].outputs[0].text\n",
+ " return result.strip()\n",
+ "\n",
+ "print(\"\\nQuick test:\")\n",
+ "new_Prompt = input(\"Enter a prompt: \")\n",
+ "print(generate_text(new_Prompt))\n",
+ "\n",
+ "\n",
+ "# print(generate_text(\"Explain what continuous batching means in vLLM.\"))"
+ ],
+ "metadata": {
+ "id": "AI9CELj5Ej5g"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "yJSF_-4FmoPD"
+ },
+ "source": [
+ "## 7. Conclusion\n",
+ "\n",
+ "You have successfully deployed and run a 7B-class Large Language Model using vLLM on Saturn Cloud. This template demonstrates how to perform high-speed inference, interact with your model via prompts, and scale seamlessly across single or multiple GPUs.\n",
+ "\n",
+ "\n",
+ "By using [Saturn Cloudโs GPU infrastructure](https://saturncloud.io/docs/user-guide/how-to/resources/), you can easily extend this workflow for larger models, API serving, or integrated data science pipelines โ all within a managed, scalable environment designed for production-grade AI workloads. Visit [saturn cloud](https://saturncloud.io/) to easily deploy this model."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.7",
+ "mimetype": "text/x-python",
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "pygments_lexer": "ipython3",
+ "nbconvert_exporter": "python",
+ "file_extension": ".py"
+ },
+ "colab": {
+ "provenance": [],
+ "gpuType": "A100"
+ },
+ "accelerator": "GPU"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-vllm-tp/README.md b/examples/nlp_and_llms/nvidia-vllm-tp/README.md
new file mode 100644
index 00000000..56413701
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vllm-tp/README.md
@@ -0,0 +1,164 @@
+# **vLLM Server for Meta-Llama-3-70B-Instruct**
+
+This template provides a **production-ready deployment environment** on **Saturn Cloud** to serve the **Meta-Llama-3-70B-Instruct** model using the high-performance **vLLM** inference engine and a FastAPI web API.
+
+โก๏ธ **Saturn Cloud lets you quickly launch multi-GPU machines to run large-scale models like Llama-3 70B. Learn more:**
+[https://saturncloud.io](https://saturncloud.io)
+
+---
+
+## **๐ Overview**
+
+The **Meta-Llama-3-70B-Instruct** model is a powerful open-source LLM from Meta AI.
+This template demonstrates how to deploy it efficiently using:
+
+* **Model:** `meta-llama/Meta-Llama-3-70B-Instruct`
+* **Inference Engine:** vLLM
+* **API Interface:** FastAPI (OpenAI-compatible)
+* **Precision:** bfloat16
+* **Parallelism:** Tensor Parallelism across 4 GPUs
+* **Use Cases:** Chatbots, RAG systems, model serving backends, enterprise AI apps
+
+vLLM provides optimized **PagedAttention**, **continuous batching**, and multi-GPU scalingโresulting in **significantly faster inference** compared to HuggingFace Transformers.
+
+---
+
+## **๐ป Requirements & Setup**
+
+Running a 70B parameter model requires substantial hardware and proper authentication.
+
+---
+
+### **1. Hardware Requirements**
+
+To run Llama-3 70B with vLLM, you need:
+
+| Component | Minimum Requirement |
+| ---------------------- | -------------------------------------------------- |
+| **GPUs** | 4ร GPUs (A40 48GB, RTX 3090/4090, or 2ร A100 80GB) |
+| **VRAM** | ~140GB total (bfloat16 precision) |
+| **Disk Space** | **150GB+** to store model weights |
+| **Tensor Parallelism** | `tensor_parallel_size = 4` |
+
+This template is suited for Saturn Cloud multi-GPU instances.
+
+---
+
+### **2. Hugging Face Authentication (Required)**
+
+Llama-3 models are **license-restricted** ("gated").
+You must authenticate before downloading.
+
+#### **Steps:**
+
+1. **Accept the License**
+ Visit:
+ [https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)
+
+2. **Log in via CLI**
+
+ ```bash
+ hf auth login
+ ```
+
+ Paste your HF access token.
+
+3. **Token in Script**
+ A placeholder token is included for testing, but **you must use your own token** in production.
+
+
+---
+
+### **3. Environment Setup**
+
+#### Create & activate your Python environment:
+
+```bash
+python3 -m venv env
+source env/bin/activate
+```
+
+#### Install dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+All required libraries (vLLM, FastAPI, Uvicorn, HF Hub support) are included inside `requirements.txt`.
+
+---
+
+## **๐ Running the Model**
+
+### **Step 1 โ Start the API Server**
+
+Launch the vLLM FastAPI server:
+
+```bash
+python start_server.py
+```
+
+You will see logs for:
+
+* GPU detection
+* Model download progress
+* Tensor parallel initialization
+* Engine warm-up
+
+When ready:
+
+```
+INFO: Uvicorn running on http://0.0.0.0:8000
+```
+
+Your vLLM server is now live and accepting OpenAI-style requests.
+
+---
+
+### **Step 2 โ Test Using the Client Script**
+
+In a separate terminal window:
+
+```bash
+source env/bin/activate
+python test_client.py
+```
+
+You will receive a JSON response similar to:
+
+```json
+{
+ "choices": [
+ {
+ "message": {
+ "content": "Tensor parallelism is a technique that..."
+ }
+ }
+ ]
+}
+```
+
+This confirms the vLLM server is functioning correctly.
+
+---
+
+## **๐ Notes for Saturn Cloud Users**
+
+This template is ideal for running on **Saturn Cloud GPU clusters**, which provide:
+
+* Multi-GPU instances compatible with vLLM
+* Prebuilt CUDA, NCCL, Python environments
+* Fast storage needed for models of this size
+* Ability to schedule long-running inference servers
+
+โก๏ธ Learn more or launch GPU resources: [https://saturncloud.io](https://saturncloud.io)
+
+---
+
+## **๐ Conclusion**
+
+This template demonstrates how to deploy **Meta-Llama-3-70B-Instruct** efficiently using the **vLLM inference engine** with **tensor parallelism** across multiple GPUs.
+It provides a fast and scalable foundation for real-world applications such as chat systems, RAG pipelines, or large-scale AI services.
+
+By combining vLLMโs optimizations with infrastructure from **Saturn Cloud**, you get a robust, production-grade environment for serving massive open-source LLMs.
+
diff --git a/examples/nlp_and_llms/nvidia-vllm-tp/requirements.txt b/examples/nlp_and_llms/nvidia-vllm-tp/requirements.txt
new file mode 100644
index 00000000..20093290
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vllm-tp/requirements.txt
@@ -0,0 +1,7 @@
+vllm
+torch
+huggingface_hub
+fastapi
+uvicorn
+requests
+hf_transfer
\ No newline at end of file
diff --git a/examples/nlp_and_llms/nvidia-vllm-tp/start_server.py b/examples/nlp_and_llms/nvidia-vllm-tp/start_server.py
new file mode 100644
index 00000000..739fd5bb
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vllm-tp/start_server.py
@@ -0,0 +1,69 @@
+from vllm import LLM, SamplingParams
+from fastapi import FastAPI
+from pydantic import BaseModel
+import uvicorn
+import os
+
+# -----------------------------
+# โ๏ธ Model Setup
+# -----------------------------
+# Use the Llama 3 model ID
+MODEL_ID = "meta-llama/Meta-Llama-3-70B-Instruct"
+
+# ---- Tensor Parallelism ----
+TENSOR_PARALLEL = 4
+
+# -----------------------------
+# ๐ Initialize vLLM
+# -----------------------------
+print(f"๐ Loading model {MODEL_ID} using vLLM tensor parallelism...")
+llm = LLM(
+ model=MODEL_ID,
+ tensor_parallel_size=TENSOR_PARALLEL,
+ gpu_memory_utilization=0.95, # High utilization as recommended
+ dtype="bfloat16", # Use bfloat16 for Ampere GPUs (A40/3090/etc)
+ enforce_eager=True, # Fixes AsyncEngineDead issues
+ max_model_len=8128, # Matches the context length in the guide
+)
+
+sampling = SamplingParams(
+ temperature=0.7,
+ top_p=0.9,
+ max_tokens=512
+)
+
+# -----------------------------
+# ๐ FastAPI (OpenAI-style API)
+# -----------------------------
+app = FastAPI(title="vLLM Tensor Parallel Server")
+
+
+class ChatRequest(BaseModel):
+ model: str
+ messages: list
+
+
+
+@app.post("/v1/chat/completions")
+async def chat(req: ChatRequest):
+ user_msg = req.messages[-1]["content"]
+
+ outputs = llm.generate([user_msg], sampling)
+ # Access the first element of the list before accessing attributes
+ text = outputs[0].outputs[0].text
+
+ return {
+ "id": "tensorpar-chat",
+ "object": "chat.completion",
+ "model": req.model,
+ "choices": [
+ {"index": 0, "message": {"role": "assistant", "content": text}}
+ ]
+ }
+
+
+# -----------------------------
+# โถ๏ธ Run the Server
+# -----------------------------
+if __name__ == "__main__":
+ uvicorn.run(app, host="0.0.0.0", port=8000)
diff --git a/examples/nlp_and_llms/nvidia-vllm-tp/test_client.py b/examples/nlp_and_llms/nvidia-vllm-tp/test_client.py
new file mode 100644
index 00000000..832890d2
--- /dev/null
+++ b/examples/nlp_and_llms/nvidia-vllm-tp/test_client.py
@@ -0,0 +1,24 @@
+import requests
+import json # Import json module
+
+payload = {
+
+ "model": "meta-llama/Meta-Llama-3-70B-Instruct",
+ "messages": [
+ {"role": "user", "content": "Explain tensor parallelism simply."}
+ ]
+}
+
+# Use the correct internal address
+url = "http://127.0.0.1:8000/v1/chat/completions"
+
+res = requests.post(url, json=payload)
+
+# Check if the request was successful before parsing JSON
+if res.status_code == 200:
+ data = res.json()
+ print("RAW:", json.dumps(data, indent=2))
+ print("\nASSISTANT:", data["choices"][0]["message"]["content"])
+else:
+ print(f"Request failed with status code {res.status_code}")
+ print("Response text:", res.text)