#14862 auto-enables the comfy-kitchen Triton backend whenever torch.version.hip
is set and Triton >= 3.7. The INT8 matmul kernels compile tl.dot to matrix-core
instructions (WMMA on RDNA3+/gfx11xx-gfx12xx, MFMA on CDNA/gfx9xx); RDNA1/RDNA2
(gfx10xx) have neither, so the auto-enabled INT8 path hangs the GPU there
(reported on RDNA2 + triton-windows 3.7.1: native and custom-node INT8 freeze
until reset).
Gate the automatic ROCm default on GPU architecture as well as Triton version so
RDNA1/RDNA2 stay on the working eager fallback. Add --disable-triton-backend as
an explicit override; --enable-triton-backend still force-enables on any arch.
On AMD/ROCm the CUDA backend is unavailable, so Triton is the only accelerated
comfy-kitchen backend. It was disabled by default (opt-in --enable-triton-backend),
leaving AMD on the slow eager path. Enable it by default when torch.version.hip is
set AND Triton is >= 3.7 -- older Triton lacks libdevice.rint on the HIP backend and
hard-crashes the INT8 path, so on Triton < 3.7 it stays disabled with a log line.
NVIDIA behavior is unchanged; the explicit --enable-triton-backend flag still works
as an override.
Fixes#14861
mixed_precision_ops.Linear.forward only quantized activations that were 2D, or
3D (reshaped to 2D). Inputs with rank >= 4 (e.g. Anima's MLP activations, which
are not reshaped to 3D the way the attention path is) fell through the
`input_reshaped.ndim == 2` guard and reached scaled_mm as bf16, silently
dispatching a bf16 kernel instead of FP8. Since MLP is roughly half the compute,
the FP8 speedup was far below expectation.
Generalize the existing 3D->2D reshape to any rank >= 3 (flatten the leading
dims, keep the contraction dim) and reshape the output back to the original
leading dims. 2D and 3D inputs are handled exactly as before; only rank >= 4
inputs change (now quantized instead of skipped). This matches the rank-agnostic
handling already used by the training path (flatten(0, -2) / unflatten).
Fixes#14595.
linear_dtype in comfy_quant metadata can be used to set if the int4 op does
the matrix multiplication in int8 or int4, the default is int4 on GPUs that
support it with fallback to int8 for GPUs that don't.
On Apple Silicon vram_state is VRAMState.SHARED (unified memory), but
text_encoder_device() only returned the GPU for HIGH_VRAM/NORMAL_VRAM,
so text encoders ran on the CPU. For LM-style encoders like ACE-Step
1.5 the text encode stage dominates generation time on Mac.
This re-lands #12809, which was reverted in #13070 because it broke
quantized text encoders on Mac: the MPS backend cannot cast float8
dtypes (pytorch/pytorch#132624), and the existing supports_cast()
fallback in CLIP.__init__ only inspects declared model dtypes, which
never reflect fp8 weights behind comfy_quant quantization metadata
(QuantizedTensor reports the compute dtype, not the fp8 storage dtype).
To keep quantized text encoders working, CLIP.__init__ now decides the
device before any weights are allocated: it checks supports_cast() on
the resolved dtype, and when the load device cannot cast fp8 it scans
the incoming state dict for fp8 tensors or comfy_quant markers and
keeps those text encoders on the offload device. Devices that can cast
fp8 (cuda etc.) skip the scan entirely. The pre-existing shift-back
path now also updates the current device so the load log stays
accurate.
Adds MPS unit tests: fp16 placement on the GPU, fp8 fallback via
declared dtype, secondary dtype (dtype_llama style), state-dict fp8
weights, and comfy_quant markers, plus a canary that fails when a
torch release adds fp8 casts on MPS so the fallback can be relaxed.
Add a --enable-asset-hashing CLI flag (action=store_true, default False)
and plumb it into the two asset-seeder call sites in main.py that
previously hardcoded compute_hashes=True (the startup scan and the
post-job output enqueue). Local runs now skip blake3 hashing unless the
user opts in, avoiding the startup/per-output cost on large models
directories while keeping hashing available for asset-portability
features.
Co-authored-by: Alexis Rolland <alexisrolland@hotmail.com>
* main: implement --vram-headroom
Implement --vram-headroom for dynamic vram as a hybrid debug/diagnostic
option that can be used for people who still report shared VRAM spills.
They can trial and error the setting to maintain a bit more headroom to
avoid shared VRAM spills.
* main: implement --reserve-vram
Implement --reserve-vram as extra headroom on the simple method which
is semantically as close as possible to the stated functionality and
formet behaviour of non-dynamic VRAM.
Add this option for users who know they have so much ram they want
to pin everything or have a pagefile that outruns their disk speed.
The removes the RAM pressure caps completely and pins behind the
primary model load forcing all models to be permanently comitted
to RAM.
Some custom nodes .to weights completely out of load context which
can wreak havoc if its for a model that is not active. Detect this
condition and just let it fall-through to the non-dynamic loader
straight up.
Some custom nodes try to set this true globally. It messes with dynamic
VRAM with one-off spikes that can OOM but this is also very high risk
for windows where such allocations might get serviced by shared memory
fallback.
Trump it.
Chris Lundquist
comfyanonymous
彼彼
liminfei-amd
John Pollock
Alexander Piskun
j2gg0s
Silver
Chris Lundquist
Matt Miller
Jukka Seppänen
Barish Ozbay
rattus
Jedrzej Kosinski
Kohaku-Blueleaf
Talmaj