#14862 auto-enables the comfy-kitchen Triton backend whenever torch.version.hip
is set and Triton >= 3.7. The INT8 matmul kernels compile tl.dot to matrix-core
instructions (WMMA on RDNA3+/gfx11xx-gfx12xx, MFMA on CDNA/gfx9xx); RDNA1/RDNA2
(gfx10xx) have neither, so the auto-enabled INT8 path hangs the GPU there
(reported on RDNA2 + triton-windows 3.7.1: native and custom-node INT8 freeze
until reset).
Gate the automatic ROCm default on GPU architecture as well as Triton version so
RDNA1/RDNA2 stay on the working eager fallback. Add --disable-triton-backend as
an explicit override; --enable-triton-backend still force-enables on any arch.
On AMD/ROCm the CUDA backend is unavailable, so Triton is the only accelerated
comfy-kitchen backend. It was disabled by default (opt-in --enable-triton-backend),
leaving AMD on the slow eager path. Enable it by default when torch.version.hip is
set AND Triton is >= 3.7 -- older Triton lacks libdevice.rint on the HIP backend and
hard-crashes the INT8 path, so on Triton < 3.7 it stays disabled with a log line.
NVIDIA behavior is unchanged; the explicit --enable-triton-backend flag still works
as an override.
Fixes#14861
mixed_precision_ops.Linear.forward only quantized activations that were 2D, or
3D (reshaped to 2D). Inputs with rank >= 4 (e.g. Anima's MLP activations, which
are not reshaped to 3D the way the attention path is) fell through the
`input_reshaped.ndim == 2` guard and reached scaled_mm as bf16, silently
dispatching a bf16 kernel instead of FP8. Since MLP is roughly half the compute,
the FP8 speedup was far below expectation.
Generalize the existing 3D->2D reshape to any rank >= 3 (flatten the leading
dims, keep the contraction dim) and reshape the output back to the original
leading dims. 2D and 3D inputs are handled exactly as before; only rank >= 4
inputs change (now quantized instead of skipped). This matches the rank-agnostic
handling already used by the training path (flatten(0, -2) / unflatten).
Fixes#14595.
linear_dtype in comfy_quant metadata can be used to set if the int4 op does
the matrix multiplication in int8 or int4, the default is int4 on GPUs that
support it with fallback to int8 for GPUs that don't.
JoyImage differs from the standard process_qwen2vl_images only in using bicubic interpolation and post-resize clamping.
I add interpolation and clamp parameters to the shared helper function, allowing JoyImage to directly reuse process_qwen2vl_images without a duplicated implementation.
(note for reviewer: this modifies qwen_vl.py. If you feel this approach is not appropriate, we can discuss alternative implementations.)
Some cleanup of the JoyImage transformer:
- FP32LayerNorm/JoyImageModulate: drop constructor params that were never
used (dtype/device on the param-free FP32LayerNorm, operations on
JoyImageModulate).
- modulate_table: init with torch.empty instead of torch.zeros since it is
loaded from the state dict, and cast at use with comfy.ops.cast_to_input.
- Replace the FeedForward nn.Dropout(0.0) no-op with nn.Identity, keeping the
ModuleList slot so state-dict keys are unchanged.
- Drop the always-true vec.shape guard (time_proj_dim is always hidden_size*6).
- Remove the redundant per-reference .to(device,dtype) in JoyImage._apply_model;
the extra_conds cast loop already moves ref_latents to the compute dtype/device.
- Drop the redundant .to(xq.device) in the RoPE apply; the freqs are built on
the latent device.
Move JoyImage CFG guidance rescale to a JoyImageGuidanceRescale node that clones the model
and calls set_model_sampler_cfg_function, following the RenormCFG
(nodes_lumina2.py) precedent for model-specific guidance nodes.
JoyImage text_encoder now detects as QWEN3VL_8B and is selected by clip_type == JOYIMAGE.
Also copy transformer_options in JoyImage._apply_model like the base
method, so the per-block bookkeeping written during forward no longer
leaks into the sampler's shared dict.
JoyImageEditPlus is the multi-image (1-6 reference images) variant of
JoyImageEdit, trained from the same base. Its diffusers transformer shares
byte-identical weight structure with the single-image variant (894 keys, zero
rename) but injects references differently: instead of the single-image
slot-stack (stack refs + noise into a 6D tensor and rotate on the frame dim,
which forces all items to share resolution), each reference is independently
patchified and concatenated on the sequence dim with per-image temporal-offset
3D RoPE, allowing references at different resolutions.
Since the single-image port is not yet upstream, this unifies both variants
onto the Plus-style forward rather than keeping two paths; single-image is now
the ref=1 special case. Verified numerically: at ref=1 with equal resolution
the new path's RoPE is bit-identical to the old slot-stack layout, and the
transformer output matches the diffusers Plus reference (fp32, incl. the
different-resolution case).
ComfyUI runs cond/uncond in one forward with a shared reference configuration,
so the diffusers Plus batched RoPE, padding attention_mask, and dedicated
attention processor are unnecessary here: the unified forward reuses the
existing unbatched _apply_rotary_emb and JoyImageAttention. Confirmed
equivalent to the diffusers batched+mask path for a single sample.
- comfy/ldm/joyimage/model.py: forward takes ref_latents and builds
components=[target, ref0, ...]; per-component patchify + temporal-offset
RoPE; output keeps only the target segment. Old single-grid RoPE removed.
- comfy/model_base.py: JoyImage drops the slot-stack / frame-rotation /
shape-equality path in _apply_model, passing ref_latents straight to the
transformer. Guidance-rescale and the reference_latents requirement are kept.
- comfy/text_encoders/joyimage.py: the image template emits one vision block
per reference (N = image count); N=1 is byte-for-byte the old template.
- comfy_extras/nodes_joyimage.py: add TextEncodeJoyImageEditPlus with optional
image1..image6 inputs, each bucket-resized and VAE-encoded into the
reference_latents list.
Detection, supported_models, and sd.py need no changes: the identical weight
structure routes both variants through image_model="joyimage".
Add a --enable-asset-hashing CLI flag (action=store_true, default False)
and plumb it into the two asset-seeder call sites in main.py that
previously hardcoded compute_hashes=True (the startup scan and the
post-job output enqueue). Local runs now skip blake3 hashing unless the
user opts in, avoiding the startup/per-output cost on large models
directories while keeping hashing available for asset-portability
features.
Co-authored-by: Alexis Rolland <alexisrolland@hotmail.com>
Upstream merged native Qwen3-VL support (#14298), adding
comfy/text_encoders/qwen3vl.py plus helpers in qwen_vl.py / llama.py /
qwen35.py. The JoyImage port previously shipped its own duplicate
Qwen3-VL implementation (comfy/text_encoders/qwen3_vl.py); that
duplication is now removed and the JoyImage text encoder rides on the
upstream stack.
- Delete comfy/text_encoders/qwen3_vl.py.
- Rewrite comfy/text_encoders/joyimage.py to subclass upstream
comfy.text_encoders.qwen3vl. The JoyImage checkpoint is a stock
qwen3vl_8b, so only JoyImage-specific behavior is overridden:
* Qwen3VL8B_JoyImage.forward builds the 3D MRoPE position ids and
injects deepstack visual features on the conditioning path. Upstream
Qwen3VL only does this inside generate() via build_image_inputs;
SDClipModel.forward never passes those kwargs. The JoyImage node
feeds an image through the encoder (clip.tokenize(prompt, images=[..])),
so the override reuses build_image_inputs to reproduce the multimodal
conditioning that Llama2_.forward already accepts kwargs for.
* preprocess_embed keeps JoyImage's bicubic+clamp image preprocessing
(process_qwen3vl_image) instead of upstream's bilinear path, to
preserve validated DiT numerics.
* JoyImageTokenizer keeps the JoyImage system-prompt templates,
suppresses the Qwen3 <think> block, and raises on image-placeholder
count mismatch.
* JoyImageTEModel keeps the drop_idx=34 system-prompt strip and the
pre-final-norm layer tap (layer="hidden", layer_idx=-1).
- sd.py QWEN3VL_8B_JOYIMAGE branch: apply the same state-dict prefix
remap the sibling QWEN3VL branch uses (model.language_model.->model.,
model.visual.->visual., lm_head.->model.lm_head.) so the checkpoint
loads into the upstream Qwen3VL namespace, then use the module-level
llama_detect. Detection ordering is preserved: the JoyImage
discriminator is checked before the generic Qwen3-VL deepstack key.
No changes to llama.py / qwen3vl.py / qwen_vl.py / qwen35.py.
JoyImageEdit is an image-edit diffusion transformer from JD (jd-opensource),
Apache 2.0. This adds native ComfyUI support so it loads and runs like other
edit models (load checkpoint -> TextEncode + ReferenceLatent -> KSampler ->
VAEDecode), with no diffusers dependency.
Architecture:
- Transformer (comfy/ldm/joyimage/model.py): dual-stream (img/txt) DiT with a
Conv3d patch embed (patch_size [1,2,2]), Wan-style learnable modulation,
and 3D RoPE (rope_dim_list [16,56,56]). All attention goes through
comfy.ldm.modules.attention.optimized_attention.
- Text encoder (comfy/text_encoders/{qwen3_vl,joyimage}.py): a reusable
Qwen3-VL multimodal stack (vision tower + LM) in qwen3_vl.py, plus a thin
JoyImage-specific layer (prompt templates, drop_idx, tokenizer, te() factory)
in joyimage.py that depends on it. text_dim 4096.
- VAE: reuses the existing Wan 2.1 latent format (AutoencoderKLWan), no new
latent format.
- Edit conditioning: reuses the reference_latents mechanism. Reference and
noise latents are stacked on a new n-slot dimension and rotated at the model
boundary (model_base.JoyImage), so the transformer stays 5D-in/5D-out.
Guidance-rescale is built into the CFG path.
Model wiring:
- model_base.JoyImage uses ModelType.FLOW with sampling_settings
multiplier=1000 (the time embedding is trained on t in [0,1000]) and
shift=1.5; FLOW's linear time_snr_shift matches the diffusers
FlowMatchEuler sigma schedule.
- model_detection sniffs the transformer state-dict (double_blocks.*,
condition_embedder.*, 5D img_in Conv3d) to route image_model="joyimage".
- supported_models.JoyImage and the CLIPLoader "joyimage" type register it.
User-facing node TextEncodeJoyImageEdit (comfy_extras/nodes_joyimage.py)
bucket-resizes the input image to the nearest 1024-base bucket, encodes the
prompt with the image, and emits both the conditioning and the bucketed image
so the same pixels feed VAEEncode and the negative encode (JoyImage requires
noise and reference latents to share spatial dims).
* main: implement --vram-headroom
Implement --vram-headroom for dynamic vram as a hybrid debug/diagnostic
option that can be used for people who still report shared VRAM spills.
They can trial and error the setting to maintain a bit more headroom to
avoid shared VRAM spills.
* main: implement --reserve-vram
Implement --reserve-vram as extra headroom on the simple method which
is semantically as close as possible to the stated functionality and
formet behaviour of non-dynamic VRAM.
comfyanonymous
comfyanonymous
liminfei-amd
John Pollock
Alexander Piskun
j2gg0s
Silver
huangfeice
kijai
Matt Miller
Barish Ozbay
rattus