lora-finetune

# LoRA Fine-Tuning (Apple Silicon) Train custom LoRA adapters for Stable Diffusion 1.5 on Mac hardware. Tested on M4 24GB — produces 3.1MB weight files in ~15 minutes at 500 steps. ## Hardware Requirements | Config | Model | Resolution | VRAM | |---|---|---|---| | M4 24GB | SD 1.5 | 512×512 | ✅ Works | | M4 24GB | SDXL | 512×512 | ⚠️ Tight, may OOM | | M4 24GB | FLUX.1-schnell | Any | ❌ OOMs | | M4 Pro 48GB | SDXL | 1024×1024 | ✅ Estimated | ## Training Pipeline 1. **Prepare dataset:** 15-25 images in consistent style, 512×512, with text captions 2. **Train LoRA:** 500 steps, learning rate 1e-4, rank 4 3. **Evaluate:** Generate test images, compare base vs LoRA vs reference (Gemini/DALL-E) 4. **Score:** LLM-as-judge rates each on style consistency, quality, prompt adherence ## Quick Start ```bash # Prepare training images in a folder ls training_data/ # image_001.png image_001.txt image_002.png image_002.txt ... # Train (see scripts/train_lora.py for full options) python3 scripts/train_lora.py \ --data_dir ./training_data \ --output_dir ./lora_weights \ --steps 500 \ --lr 1e-4 \ --rank 4 ``` ## Evaluation with LLM-as-Judge ```python # Compare base model vs LoRA vs commercial (Gemini/DALL-E) # Pixtral Large scores each image 1-10 on: # - Style consistency with training data # - Image quality and coherence # - Prompt adherence # Our results: Base 6.8 → LoRA 9.0 → Gemini 9.5 # Lesson: Gemini wins without training, but LoRA closes the gap significantly ``` ## Key Lessons - **float32 required on MPS** — float16 silently produces NaN on Apple Silicon for SD pipelines - **mflux is faster than PyTorch MPS for FLUX** (~105s vs ~90min) but doesn't support LoRA training - **SD 1.5 is the ceiling for 24GB** — FLUX LoRA OOMs even with gradient checkpointing - **15-25 images is the sweet spot** — fewer undertrain, more doesn't help proportionally - **Gemini (Imagen 4.0) beats fine-tuned SD 1.5** with zero training — use commercial APIs for production, LoRA for experimentation and offline use ## Files - `scripts/train_lora.py` — Training script with Apple Silicon MPS support - `scripts/compare_models.py` — LLM-as-judge evaluation comparing base vs LoRA vs reference