feat: Add inference support for MiniT2I model.

[KenForever1]

Summary

Add inference support for MiniT2I in stable-diffusion.cpp.

This PR adds a MiniT2I diffusion runner, T5/flan-t5 text conditioning integration, model detection/loading support, and MiniT2I-specific sampling flow. It also caches step-invariant positional embeddings/RoPE tensors and removes an unused conditioning branch after validating output consistency.

Changes

• Add MiniT2I model type detection and loading path.
• Add MiniT2I::MiniT2IRunner implementation for MMJiT-style diffusion inference.
• Add MiniT2I conditioner path using google/flan-t5-large.
• Add MiniT2I sampling path with conditional/unconditional forward and CFG update.
• Add backend support needed by MiniT2I graph execution.
• Cache MiniT2I positional embeddings, text RoPE, and vision/joint RoPE in runner-level backend buffers.
• Remove unused t_vec + pooled_text conditioning branch that is not consumed by the current MiniT2I graph.

Commits

• b9493fa Add MiniT2I inference support
• 8de8f95 Optimize MiniT2I position cache
• dfb6ca2 Remove unused MiniT2I conditioning branch

Models Used

MiniT2I diffusion model:

• Model: MiniT2I/minit2i-b-16
• Weight: transformer/diffusion_pytorch_model.safetensors

Text encoder:

• Model: google/flan-t5-large
• Weight: model.safetensors

Test Commands

Mac Metal test:

cd stable-diffusion.cpp

./build/bin/sd-cli \
  --backend metal \
  --model MiniT2I/MiniT2I/minit2i-b-16/transformer/diffusion_pytorch_model.safetensors \
  --t5xxl google/flan-t5-large/model.safetensors \
  --prompt "a cat" \
  --steps 100 \
  --cfg-scale 6 \
  --width 512 \
  --height 512 \
  --seed 42 \
  --sampling-method euler \
  --rng cpu \
  --output /private/tmp/minit2i_metal.png \
  --threads 8

CUDA with diffusion flash attention:

cd stable-diffusion.cpp

./build-cuda/bin/sd-cli \
  --backend cuda \
  --diffusion-fa \
  --model MiniT2I/MiniT2I/minit2i-b-16/transformer/diffusion_pytorch_model.safetensors \
  --t5xxl google/flan-t5-large/model.safetensors \
  --prompt "a cat" \
  --steps 100 \
  --cfg-scale 6 \
  --width 512 \
  --height 512 \
  --seed 42 \
  --sampling-method euler \
  --rng cpu \
  --output /tmp/minit2i_cuda_diffusion_fa.png \
  --threads 8

Validation Notes

• MiniT2I generation succeeds on CUDA and Metal.
• Position/RoPE cache optimization preserves model batch semantics.
• Removing the unused conditioning branch produced identical output in local validation.
• CUDA --diffusion-fa works with MiniT2I and reduces stable diffusion forward time significantly.

[leejet]

Please follow the existing model docs and examples to add documentation and examples for MiniT2I.

[KenForever1]

Please follow the existing model docs and examples to add documentation and examples for MiniT2I.

Thanks for the review! Addressed all four points: simplified model detection, dropped resolve_prefix in favor of --diffusion-model, added docs/examples, and moved MiniT2I onto the generic sampling flow. Output stays consistent with the original implementation. PTAL.

[wbruna]

You should also rebase it on top of latest master, to make tests and integration easier.

[KenForever1]

You should also rebase it on top of latest master, to make tests and integration easier.

completed.

[leejet]

Thank you for your contribution.

[CMay]

There is a B model and an L model and this only seems to support the B model, yet the examples are using the recommended guidance scale of the L model.

No matter which cfg-scale I use, 100 steps for the prompt "a cat" gives me some strange result.

What's the correct way to run this?

[GreenShadows]

"[INFO ] stable-diffusion.cpp:4619 - generating image: 1/1 - seed 42
D:\a\stable-diffusion.cpp\stable-diffusion.cpp\ggml\src\ggml.c:3243: GGML_ASSERT(ggml_can_mul_mat(a, b)) failed"

I think you need to convert it to use it in SD.cpp. SeFi Image has a similar problem.

Hmm. On the HF-hosted space, it actually works with fewer steps.

[CMay]

@GreenShadows Think I'm having text encoder issues. Every prompt produces the same image whether using the safetensors directly or converted gguf.

[GreenShadows]

@GreenShadows Think I'm having text encoder issues. Every prompt produces the same image whether using the safetensors directly or converted gguf.

It's best to open an issue thread for better visibility.