Commit Graph

1406 Commits

Author SHA1 Message Date
Jaret Burkett 72623ed3d6 When doing auto frame count. Ensure the time is not squeezed or expanded to fit tempooral spacing. trime the few extra frames. Also fixed frame counts of buckets. 2026-08-09 12:31:02 -06:00
Jaret Burkett 682b27c6ee Reworked freeing memory manager for removing text encoder completly when not needed. 2026-08-09 07:08:24 -06:00
Jaret Burkett 3a28c4b1b7 Keep alive cron server to prevent failed connections 2026-08-08 20:27:01 -06:00
Jaret Burkett 8c1a4082fd Allow images to work with auto frame count, and include images in video datasets if they exist. 2026-08-08 20:26:28 -06:00
Jaret Burkett 6d8afa5684 Version bump 2026-08-08 18:58:11 -06:00
Jaret Burkett (Ostris) c596d4ab27
Merge pull request #1002 from whatsthisaithing/codex/fix-convrot-offload-stream-lifetime
Fix offload buffer stream lifetime
2026-08-08 18:55:18 -06:00
Fitzy d184c6c622 Fix offload buffer stream lifetime 2026-08-08 19:59:23 -04:00
Jaret Burkett f4e9130547 Fix race condition that can corrupt grads under certain conditions. 2026-08-07 14:52:47 -06:00
Jaret Burkett 817f3dcbcb Fix finite check on inverted masked prior 2026-08-07 09:49:12 -06:00
Jaret Burkett 685ce37a8d Adjust sample timestep sigmas to be model evals for h3 for 1 extra step 2026-08-06 19:54:04 -06:00
Jaret Burkett 9171d5ec1d Update training adapter path 2026-08-06 11:21:28 -06:00
Jaret Burkett 71625d1207 Add the alpha version of the MiniMax H3 trianing adapter and set it as the new default training method. 2026-08-06 11:11:31 -06:00
Jaret Burkett b904b99705 varsion bump 2026-08-06 09:25:35 -06:00
Jaret Burkett b811636ae4 Check for finite vs isnan on loss before backpropigating. to catch infinity overflows 2026-08-06 09:18:56 -06:00
Jaret Burkett edacd406b3 Handle minimax loading of non pruned model 2026-08-06 09:18:00 -06:00
Jaret Burkett 7309db4d74 Fix layer offloaded adaln projection layer move 2026-08-06 08:10:20 -06:00
Jaret Burkett 139a38f5bd Upcast adaln pruned layers to fp32 on h3 to prevent overflow. 2026-08-06 07:51:15 -06:00
Jaret Burkett 1e1418b22c Apply sigma to contrastive guidance to balance loss better. Prevent noise grads on images/non audio datasets. Prep for training adapters on MiniMax H3 2026-08-05 13:27:12 -06:00
Jaret Burkett 9065951da3 Version bump 2026-08-04 22:14:54 -06:00
Jaret Burkett 3afa270ab5 Fix audio losses for DOP and other preservation losses 2026-08-04 22:08:48 -06:00
Jaret Burkett 0f9094db95 Fix audio loss when doing do_guidance_loss 2026-08-04 21:43:31 -06:00
Jaret Burkett d870e9b68a Fix backwards compatability for older versions of triton 2026-08-04 21:07:57 -06:00
Jaret Burkett a8d67ecd90 Version Bump 2026-08-04 16:28:04 -06:00
Jaret Burkett 9fc1f208df Remove adaln_proj from the network modules for minimax_h3 2026-08-04 16:19:46 -06:00
Jaret Burkett 183433ae8e Make MiniMax H3 default to using contrastive guidance loss to prevent distillation breakdown 2026-08-04 16:13:45 -06:00
Jaret Burkett 00a93e3830 Add dataset flag to cache the raw tensors 2026-08-04 15:32:48 -06:00
Jaret Burkett 8a0bcf1ffe Fix issue loading H3 test encoder on older GPUS 2026-08-04 15:30:29 -06:00
Jaret Burkett dc29ae1187 Version Bump 2026-08-04 09:27:46 -06:00
Jaret Burkett d20a17c10e Limit max tokens to 512. Allow override with model kwargs. 2026-08-04 07:55:21 -06:00
Jaret Burkett 602306da77 Add gradient checkpointing to vae 2026-08-04 07:54:51 -06:00
Jaret Burkett 18f5810d6c Adjust default alpha for h3 2026-08-03 20:15:28 -06:00
Jaret Burkett a9a04547e9 Dont move encoders on and off device when caching until the first instance of needing to process a cache item. 2026-08-03 15:54:30 -06:00
Jaret Burkett 41676bb258 Queue up videos with multiple threads when caching latents so the VAE is not waiting on videos to process 2026-08-03 15:41:36 -06:00
Jaret Burkett 546eb7daff Look for existing models in folders recursivly 2026-08-03 15:39:55 -06:00
Jaret Burkett d3a3f70a2a Speed up quantization processing on H3 2026-08-03 15:21:39 -06:00
Jaret Burkett 88ac27fc8f Handle images with MiniMax H3. 2026-08-03 11:58:44 -06:00
Jaret Burkett bf739ff966 Fix issue with layer offloading with MinMax H3 2026-08-03 11:32:36 -06:00
Jaret Burkett 9d614a51fb Hard fail if a step in the docker build fails. 2026-08-03 10:38:31 -06:00
Jaret Burkett 8502a845b1 Add support for MiniMax H3 T2V and I2V training 2026-08-03 10:17:39 -06:00
Jaret Burkett 73cab2acf5 Reworked merge in out of loras with convrot weights for better roundtrip accuracy. 2026-08-02 08:29:17 -06:00
Jaret Burkett a6f6b6b896 Fixed experiment opt name match 2026-08-01 17:43:36 -06:00
Jaret Burkett 6b95282097 Recover from issue when a video model first fram may not have been cached properly 2026-08-01 12:00:33 -06:00
Jaret Burkett 5baa495585 DFE optimizations 2026-08-01 10:06:16 -06:00
Jaret Burkett fc78b07332 Attach the ema to the base model so it can be used on specific models 2026-08-01 10:05:42 -06:00
Jaret Burkett c68e58083f Add code for automagic experiment 2026-08-01 10:04:46 -06:00
Jaret Burkett 497014bf5d Set i2v to default to false when omitted from the config 2026-07-31 19:11:58 -06:00
Jaret Burkett 038f24e8c3 Revert torchao back to older version 2026-07-31 16:34:06 -06:00
Jaret Burkett 6e7bc81241 Allow random noise shift with video latents. 2026-07-31 05:58:08 -06:00
Jaret Burkett ddc69745fe Update docker build image with newer dependencies 2026-07-30 14:20:31 -06:00
Jaret Burkett (Ostris) 2cab330392
Merge pull request #986 from ostris/dev
Add AI Toolkit Manager script that auto installs/runs/and updates AI Toolkit.
2026-07-30 11:55:13 -06:00