5f04ae7ad5
Rework merge_network_on_save to handle dequantization on merging and saving for much more efficient full finetuning.
2026-06-25 13:19:38 -06:00
c8cd78b1a4
Allow nested transformer block names for quantization, lora targeting, quantizing
2026-06-13 14:33:16 -06:00
5e84bf0d0b
Fixed issue with hidream-01 that could cause a weird nan state. Took forever to track down as it was 1 in 10 starts.
2026-05-28 12:48:24 -06:00
fbac1cb7f5
Dont force flash attention on hidream 01. Causes random issues and is slower.
2026-05-24 16:05:22 -06:00
e03c6e4dc9
Fix potential inconsistency with different attention mentods in hidream01
2026-05-13 09:08:11 -06:00
fc4d6ebf39
Add support for fine-tuning Hidream O1 ( #831 )
...
* Initial support for hidream. Lora keys likely need work
* Fix saving for hidream-o1
* Remove dependence on flash attention for hidream o1
* Fix gradient checkpointing for hidream o1
* A lot of fixes for hidream. Handle loading and saving as comfy model.
* Omit layers not used in comfy. Fix issue with lora loading keys in comfy
* Version bumpo
2026-05-12 11:15:16 -06:00
cefa2ca5fe
Added initial support for Hidream E1 training
2025-07-27 15:12:56 -06:00
add83df5cc
Fixed issue with training hidream when batch size is larger than 1
2025-04-21 17:26:29 +00:00
77001ee77f
Upodate model tag on loras
2025-04-19 10:41:27 -06:00
0f99fce004
Adjust hidream lora names to work with comfy
2025-04-16 09:24:23 -06:00
524bd2edfc
Make flash attn optional. Handle larger batch sizes.
2025-04-14 14:34:46 +00:00
3a5ea2c742
Remove some moe stuff for finetuning. Drastically reduces vram usage
2025-04-14 00:57:34 +00:00
f80cf99f40
Hidream is training, but has a memory leak
2025-04-13 23:28:18 +00:00
594e166ca3
Initial support for hidream. Still a WIP
2025-04-13 13:50:11 -06:00
Jaret Burkett