89dd041b97
Added ability to pair samples with a closer noise with optimal_noise_pairing_samples
2025-01-21 18:30:10 -07:00
29122b1a54
Added code to handle diffusion feature extraction loss
2025-01-21 14:21:34 -07:00
fadb2f3a76
Allow quantizing the te independently on flux. added lognorm_blend timestep schedule
2025-01-18 18:02:31 -07:00
4723f23c0d
Added ability to split up flux across gpus (experimental). Changed the way timestep scheduling works to prep for more specific schedules.
2024-12-31 07:06:55 -07:00
8ef07a9c36
Added training for an experimental decoratgor embedding. Allow for turning off guidance embedding on flux (for unreleased model). Various bug fixes and modifications
2024-12-15 08:59:27 -07:00
92ce93140e
Adjustments to defaults for automagic
2024-11-29 10:28:06 -07:00
f213996aa5
Fixed saving and displaying for automagic
2024-11-29 08:00:22 -07:00
cbe31eaf0a
Initial work on a auto adjusting optimizer
2024-11-29 04:48:58 -07:00
67c2e44edb
Added support for training flux redux adapters
2024-11-21 20:01:52 -07:00
96d418bb95
Added support for full finetuning flux with randomized param activation. Examples coming soon
2024-11-21 13:05:32 -07:00
894374b2e9
Various bug fixes and optimizations for quantized training. Added untested custom adam8bit optimizer. Did some work on LoRM (dont use)
2024-11-20 09:16:55 -07:00
6509ba4484
Fix seed generation to make it deterministic so it is consistant from gpu to gpu
2024-11-15 12:11:13 -07:00
025ee3dd3d
Added ability for adafactor to fully fine tune quantized model.
2024-10-30 16:38:07 -06:00
58f9d01c2b
Added adafactor implementation that handles stochastic rounding of update and accumulation
2024-10-30 05:25:57 -06:00
e72b59a8e9
Added experimental 8bit version of prodigy with stochastic rounding and stochastic gradient accumulation. Still testing.
2024-10-29 14:28:28 -06:00
4aa19b5c1d
Only quantize flux T5 is also quantizing model. Load TE from original name and path if fine tuning.
2024-10-29 14:25:31 -06:00
4747716867
Fixed issue with adapters not providing gradients with new grad activator
2024-10-29 14:22:10 -06:00
22cd40d7b9
Improvements for full tuning flux. Added debugging launch config for vscode
2024-10-29 04:54:08 -06:00
3400882a80
Added preliminary support for SD3.5-large lora training
2024-10-22 12:21:36 -06:00
9f94c7b61e
Added experimental param multiplier to the ema module
2024-10-22 09:25:52 -06:00
bedb8197a2
Fixed issue with sizes for some images being loaded sideways resulting in squished images.
2024-10-20 11:51:29 -06:00
e3ebd73610
Add a projection layer on vision direct when doing image embeds
2024-10-20 10:48:23 -06:00
0640cdf569
Handle errors in loading size database
2024-10-20 07:04:19 -06:00
ce759ebd8c
Normalize the image embeddings on vd adapter forward
2024-10-12 15:09:48 +00:00
628a7923a3
Remove norm on image embeds on custom adapter
2024-10-12 00:43:18 +00:00
3922981996
Added some additional experimental things to the vision direct encoder
2024-10-10 19:42:26 +00:00
ab22674980
Allow for a default caption file in the folder. Minor bug fixes.
2024-10-10 07:31:33 -06:00
9452929300
Apply a mask to the embeds for SD if using T5 encoder
2024-10-04 10:55:20 -06:00
a800c9d19e
Add a method to have an inference only lora
2024-10-04 10:06:53 -06:00
28e6f00790
Fixed bug in returning clip image embed to actually return it
2024-10-03 10:49:09 -06:00
67e0aca750
Added ability to load clip pairs randomly from folder. Other small bug fixes
2024-10-03 10:03:49 -06:00
f05224970f
Added Vision Languate Adapter usage for pixtral vd adapter
2024-09-29 19:39:56 -06:00
b4f64de4c2
Quick patch to scope xformer imports until a better solution
2024-09-28 15:36:42 -06:00
e4c82803e1
Handle random resizing for pixtral input on direct vision adapter
2024-09-28 14:53:38 -06:00
69aa92bce5
Added support for AdEMAMix8bit
2024-09-28 14:33:51 -06:00
a508caad1d
Change pixtral to crop based on number of pixels instead of largest dimension
2024-09-28 13:05:26 -06:00
58537fc92b
Added initial direct vision pixtral support
2024-09-28 10:47:51 -06:00
86b5938cf3
Fixed the webp bug finally.
2024-09-25 13:56:00 -06:00
6b4034122f
REmove layers from direct vision resampler
2024-09-24 15:08:29 -06:00
10817696fb
Fixed issue where direct vision was not passing additional params from resampler when it is added
2024-09-24 10:34:11 -06:00
037ce11740
Always return vision encoder in state dict
2024-09-24 07:43:17 -06:00
04424fe2d6
Added config setting to set the timestep type
2024-09-24 06:53:59 -06:00
40a8ff5731
Load local hugging face packages for assistant adapter
2024-09-23 10:37:12 -06:00
2776221497
Added option to cache empty prompt or trigger and unload text encoders while training
2024-09-21 20:54:09 -06:00
f85ad452c6
Added initial support for pixtral vision as a vision encoder.
2024-09-21 15:21:14 -06:00
79b4e04b80
Feat: Wandb logging ( #95 )
...
* wandb logging
* fix: start logging before train loop
* chore: add wandb dir to gitignore
* fix: wrap wandb functions
* fix: forget to send last samples
* chore: use valid type
* chore: use None when not type-checking
* chore: resolved complicated logic
* fix: follow log_every
---------
Co-authored-by: Plat <github@p1at.dev>
Co-authored-by: Jaret Burkett <jaretburkett@gmail.com>
2024-09-19 20:01:01 -06:00
951e223481
Added support to disable single transformers in vision direct adapter
2024-09-11 08:54:51 -06:00
fc34a69bec
Ignore guidance embed when full tuning flux. adjust block scaler to decat to 1.0. Add MLP resampler for reducing vision adapter tokens
2024-09-09 16:24:46 -06:00
279ee65177
Remove block scaler
2024-09-06 08:28:17 -06:00
3a1f464132
Added support for training vision direct weight adapters
2024-09-05 10:11:44 -06:00
121a760c19
Added proper grad accumulation
2024-09-03 07:24:18 -06:00
e5fadddd45
Added ability to do prompt attn masking for flux
2024-09-02 17:29:36 -06:00
d44d4eb61a
Added a new experimental linear weighing technique
2024-09-02 09:22:13 -06:00
7d9ab22405
Rework ip adapter and vision direct adapters to apply to the single transformer blocks even though they are not cross attn.
2024-09-01 10:40:42 -06:00
40f5c59da0
Fixes for training ilora on flux
2024-08-31 16:55:26 -06:00
3e71a99df0
Check for contains only against clean name for lora, not the adjusted one
2024-08-31 07:44:13 -06:00
60232def91
Made peleminary arch for flux ip adapter training
2024-08-28 08:55:39 -06:00
3843e0d148
Added support for vision direct adapter for flux
2024-08-26 16:27:28 -06:00
e127c079da
fix: print out the path where the image encode failed ( #107 )
2024-08-22 21:34:35 -06:00
34db804c76
Modal cloud training support, fixed typo in toolkit/scheduler.py, Schnell training support for Colab, issue #92 , issue #114 ( #115 )
...
* issue #76 , load_checkpoint_and_dispatch() 'force_hooks'
https://github.com/ostris/ai-toolkit/issues/76
* RunPod cloud config
https://github.com/ostris/ai-toolkit/issues/90
* change 2x A40 to 1x A40 and price per hour
referring to https://github.com/ostris/ai-toolkit/issues/90#issuecomment-2294894929
* include missed FLUX.1-schnell setup guide in last commit
* huggingface-cli login required auth
* #92 peft, #114 colab, schnell training in colab
* modal cloud - run_modal.py and .yaml configs
* run_modal.py mount path example
* modal_examples renamed to modal
* Training in Modal README.md setup guide
* rename run command in title for consistency
2024-08-22 21:25:44 -06:00
4d35a29c97
Add `push_to_hub` to the trainer ( #109 )
...
* add push_to_hub
* fix indentation
* indent again
* model_config
* allow samples to not exist
* repo creation fix
* dont show empty [] if widget doesnt exist
* dont submit the config and optimizer
* Unsafe to have tokens saved in the yaml file
* make sure to catch only the latest samples
* change name to slug
* formatting
* formatting
---------
Co-authored-by: multimodalart <joaopaulo.passos+multimodal@gmail.com>
2024-08-22 21:18:56 -06:00
338c77d677
Fixed breaking change with diffusers. Allow flowmatch on normal stable diffusion models.
2024-08-22 14:36:22 -06:00
a939cf3730
WIP - adding support for flux DoRA and ip adapter training
2024-08-22 04:36:39 -06:00
c45887192a
Unload interum weights when doing multi lora fuse
2024-08-18 09:35:10 -06:00
13a965a26c
Fixed bad key naming on lora fuse I just pushed
2024-08-18 09:33:31 -06:00
f944eeaa4d
Fuse flux schnell assistant adapter in pieces when doing lowvram to drastically speed ip up from minutes to seconds.
2024-08-18 09:09:11 -06:00
81899310f8
Added support for training on flux schnell. Added example config and instructions for training on flux schnell
2024-08-17 06:58:39 -06:00
f9179540d2
Flush after sampling
2024-08-16 17:29:42 -06:00
452e0e286d
For lora assisted training, merge in before quantizing then sample with schnell at -1 weight. Almost doubles training speed with lora adapter.
2024-08-16 17:28:44 -06:00
0355662e8e
Added support for polarity guidance for flow matching models
2024-08-15 12:22:00 -06:00
9001e5c933
Change flux latent spact if so it will not use old cache
2024-08-14 11:27:40 -06:00
7fed4ea761
fixed huge flux training bug. Added ability to use an assistatn lora
2024-08-14 10:14:13 -06:00
00bd3d54a3
Actually use the save dtype from the config file.
2024-08-13 17:08:27 -06:00
f7cf2f866f
Make 100% sure lora alpha matches for flux
2024-08-13 14:24:03 -06:00
418f5f7e8c
Added new experimental time step weighing that should solve a lot of issues with distribution. Updated example. Removed a warning
2024-08-13 12:02:11 -06:00
9ee1ef2a0a
Added experimental modified sigma sqrt weight mapping for linear timestep scheduling for flowmatching
2024-08-12 17:03:09 -06:00
599fafe01f
Allow user to have the full flux checkpoint local
2024-08-12 09:57:16 -06:00
af108bb964
Bug fix with dataloader. Added a flag to completly disable sampling
2024-08-12 09:19:40 -06:00
a6aa4b2c7d
Added ability to set timesteps to linear for flowmatching schedule
2024-08-11 13:06:08 -06:00
7f0ecdb377
Merge branch 'main' of github.com:ostris/ai-toolkit
2024-08-11 11:10:45 -06:00
fbed8568fb
Actually use the correct timestep sampling instead of calculating it and moving on lol. Tested a few with it and it seems to work better.
2024-08-11 11:10:37 -06:00
6d31c6db73
Added a fix for windows dataloader
2024-08-11 10:48:24 -06:00
6490a326e5
Fixed issue for vaes without a shift
2024-08-11 10:30:55 -06:00
ec1ea7aa0e
Added support for training on primary gpu with low_vram flag. Updated example script to remove creepy horse sample at that seed
2024-08-11 09:54:30 -06:00
2308ef2868
Added flux training instructions
2024-08-10 14:10:02 -06:00
b3e03295ad
Reworked flux pred. Again
2024-08-08 13:06:34 -06:00
e69a520616
Reworked timestep distribution on flowmatch sampler when training.
2024-08-08 06:01:45 -06:00
acafe9984f
Adjustments to loading of flux. Added a feedback to ema
2024-08-07 13:17:26 -06:00
653fe60f16
Updates to flow matching algo
2024-08-07 15:04:17 +00:00
c2424087d6
8 bit training working on flux
2024-08-06 11:53:27 -06:00
272c8608c2
Make a CFG version of flux pipeline
2024-08-05 16:35:53 -06:00
187663ab55
Use peft format for flux loras so they are compatible with diffusers. allow loading an assistant lora
2024-08-05 14:34:37 -06:00
edb7e827ee
Adjusted flow matching so target noise multiplier works properly with it.
2024-08-05 11:40:05 -06:00
0ea27011d5
Bug fix
2024-08-04 11:07:19 -06:00
f321de7bdb
Setup to retrain guidance embedding for flux. Use defualt timestep distribution for flux
2024-08-04 10:37:23 -06:00
9beea1c268
Flux training should work now... maybe
2024-08-03 09:17:34 -06:00
369aa143bc
Only train a few blocks on flux (for now)
2024-08-03 07:02:27 -06:00
87ba867fdc
Added flux training. Still a WIP. Wont train right without rectified flow working right
2024-08-02 15:00:30 -06:00
03613c523f
Bugfixes and cleanup
2024-08-01 11:45:12 -06:00
47744373f2
Change img multiplier math
2024-07-30 11:33:41 -06:00
8f0f467c20
Switch back to old ilora
2024-07-29 07:22:05 -06:00
e81e19fd0f
Added target_norm_std which is a game changer
2024-07-28 16:08:33 -06:00
0bc4d555c7
A lot of pixart sigma training tweaks
2024-07-28 11:23:18 -06:00
80aa2dbb80
New image generation img2img. various tweaks and fixes
2024-07-24 04:13:41 -06:00
6e92922c14
Add a mergable linear to the mid of ilora
2024-07-20 21:17:53 -06:00
c51235c486
Fixed misnamed var
2024-07-20 23:00:20 +00:00
c2c4e8cf34
Added ability to target parts of lora for ilora
2024-07-20 22:45:52 +00:00
4c249cf607
Added ilora2
2024-07-20 16:40:57 -06:00
c2d5f712a3
Reworked ilora arch
2024-07-20 15:35:59 -06:00
22d2f6e28f
Fixed issue with grad scaling
2024-07-20 08:21:57 -06:00
a2301cf28c
Amall bug fixes
2024-07-18 10:39:55 -06:00
11e426fdf1
Various features and fixes. Too much brain fog to do a proper description
2024-07-18 07:34:14 -06:00
58dffd43a8
Added caching to image sizes so we dont do it every time.
2024-07-15 19:07:41 -06:00
e4558dff4b
Partial implementation for training auraflow.
2024-07-12 12:11:38 -06:00
c062b7716c
Varous bug fixes
2024-07-10 15:20:04 -06:00
93e5df1d59
Merge branch 'main' of github.com:ostris/ai-toolkit
2024-07-07 07:56:56 -06:00
045e4a6e15
Save entire pixart model again
2024-07-07 07:56:48 -06:00
76f225a467
Fixed issue with TE adapter caption projection
2024-07-06 19:09:58 +00:00
cab8a1c7b8
WIP to add the caption_proj weight to pixart sigma TE adapter
2024-07-06 13:00:21 -06:00
acb06d6ff3
Bug fixes
2024-07-03 10:56:34 -06:00
bb57623a35
Merge branch 'main' of github.com:ostris/ai-toolkit
2024-06-29 15:53:40 -06:00
3072d20f17
Add ability to include conv_in and conv_out to full train when doing a lora
2024-06-29 14:54:50 -06:00
f6b21f47bb
Increased the number of heads for ip adapters.
2024-06-28 16:09:52 +00:00
603ceca3ca
added ema
2024-06-28 10:03:26 -06:00
657fd09f25
Added more control over Sigma sizes
2024-06-26 08:57:53 -06:00
8407c4deea
Merge branch 'main' of github.com:ostris/ai-toolkit
2024-06-23 14:47:43 -06:00
64f2b085b7
Minor fixes
2024-06-23 14:47:40 -06:00
7165f2d25a
Work to omprove pixart training
2024-06-23 20:46:48 +00:00
5d47244c57
Added support for pixart sigma loras
2024-06-16 11:56:30 -06:00
ada722c9e4
Fixed issue with heads not being added
2024-06-15 08:34:33 -06:00
696f73c30d
Removed variant
2024-06-14 17:09:47 -06:00
e3410413b9
Rework head on ilora
2024-06-14 16:21:26 -06:00
37cebd9458
WIP Ilora
2024-06-14 09:31:01 -06:00
bd10d2d668
Some work on sd3 training. Not working
2024-06-13 12:19:16 -06:00
cb5d28cba9
Added working ilora trainer
2024-06-12 09:33:45 -06:00
3f3636b788
Bug fixes and little improvements here and there.
2024-06-08 06:24:20 -06:00
833c833f28
WIP on SAFE encoder. Work on fp16 training improvements. Various other tweaks and improvements
2024-05-27 10:50:24 -06:00
68b7e159bc
Bug Fixes
2024-05-17 08:41:20 -06:00
5a45c709cd
Work on ipadapters and custom adapters
2024-05-13 06:37:54 -06:00
10e1ecf1e8
Added single value adapter training
2024-04-28 06:04:47 -06:00
b96913d73c
Improvements to dataloader
2024-04-27 09:28:28 -06:00
5da3613e0b
Bug fixes and minor features
2024-04-25 06:14:31 -06:00
5a70b7f38d
Added pixart sigma support, but it wont work until i address breaking changes with lora code in diffusers so it can be upgraded.
2024-04-20 10:46:56 -06:00
377b81ee3e
Adjustments to guidance
2024-04-19 15:00:35 -06:00
2d0a1be59d
Bug fixes
2024-04-16 03:48:13 -06:00
7284aab7c0
Added specialized scaler training to ip adapters
2024-04-05 08:17:09 -06:00
427847ac4c
Small tweaks and fixes for specialized ip adapter training
2024-03-26 11:35:26 -06:00
9c1cc9641e
Added keep tokens to keep so many tokens in a prompt when dropping
2024-03-18 13:18:25 -06:00
016687bda1
Adapter work. Bug fixes. Auto adjust LR when resuming optimizer.
2024-03-17 10:21:47 -06:00
72de68d8aa
WIP on clip vision encoder
2024-03-13 07:24:08 -06:00
d87b49882c
Work on embedding adapters
2024-03-11 15:18:42 -06:00
b01e8d889a
Added stochastic rounding to adafactor. ILora adjustments
2024-03-05 07:07:09 -07:00
1325613583
rework ilora
2024-02-29 07:55:52 -07:00
337945de9a
Added this not that guidance. Added ability to replace prompts.
2024-02-28 20:10:14 -07:00
561914d8e6
Removed old code for fixing multistep sampler that is no longer needed
2024-02-25 11:53:35 -07:00
b0a0f28191
Bug fixes
2024-02-25 08:28:29 -07:00
f965a1299f
Fixed Dora implementation. Still highly experimental
2024-02-24 10:26:01 -07:00
1bd94f0f01
Added early DoRA support, but will change shortly. Dont use right now.
2024-02-23 05:55:41 -07:00
b68c3ef734
Added te aug adapter
2024-02-21 21:30:26 -07:00
49c41e6a5f
Bug fixes. allow for random negative prompts
2024-02-21 04:51:52 -07:00
2478554c95
Bug fixes. Added IP adapter training for Pixart
2024-02-17 10:06:57 -07:00
93b52932c1
Added training for pixart-a
2024-02-13 16:00:04 -07:00
4ec4025cbb
Added adapter modules for text encoders and direct vision
2024-02-12 08:46:18 -07:00
e074058faa
Work on additional image embedding methods. Finalized zipper resampler. It works amazing
2024-02-10 09:00:05 -07:00
a8481c1670
randomly adjust scale of unconditional noise on ip adapters if training with cfg
2024-02-06 03:44:54 -07:00
e18e0cb5f8
Added comparitive loss when training clip encoder. Allow selecting clip layer. on ip adapter. Improvements to prior prediction
2024-02-05 07:40:03 -07:00
177c7130ec
improved correction of pred norm by targeting the prior
2024-02-01 06:31:04 -07:00
1ae1017748
Bug fixes. added ability to use l1 loss. varous other tests and improvements
2024-01-31 06:30:54 -07:00
92b9c71d44
Many bug fixes. Ip adapter bug fixes. Added noise to unconditional, it works better. added an ilora adapter for 1 shotting LoRAs
2024-01-28 08:20:03 -07:00
f17ad8d794
various bug fixes. Created an contextual alpha mask module to calculate alpha mask
2024-01-18 16:34:27 -07:00
86c70a2a1f
Added an experimental clip fusion model that is showing promise for embedding concepts
2024-01-17 13:13:04 -07:00
655533d4c7
More work on custom adapter
2024-01-16 17:41:26 -07:00
eebd3c8212
Initial training script for photomaker training. Needs a little more work.
2024-01-15 18:46:26 -07:00
5276975fb0
Added additional config options for custom plugins I needed
2024-01-15 08:31:09 -07:00
e190fbaeb8
Prepwork for ilora
2024-01-12 06:41:15 -07:00
290393f7ae
Imporvements to ip weight adaptation. Bug fixes. Added masking to direct guidance loss. Allow importing a file for random triggers. Handle bas meta images with improper sizing.
2024-01-11 12:22:16 -07:00
b2a54c8f36
Added siglip support
2024-01-09 20:52:21 -07:00
b767d29b3c
Adjustments to the clip preprocessor. Allow merging in new weights for ip adapters so you can change the arcitecture while maintaining as much data as possible
2024-01-06 11:56:53 -07:00
645b27f97a
Bug fixes with ip adapter training. Made a clip pre processor that can be trained with ip adapter to help augment the clip input to squeeze in more detail from a larget input. moved clip processing to the dataloader for speed.
2024-01-04 12:59:38 -07:00
65c08b09c3
Added ability to do cfg during training. Various bug fixes
2024-01-02 11:29:57 -07:00
afc231efc1
Added reference adapters, many bug fixes, more ip adapter work and customizability
2024-01-01 17:15:53 -07:00
bafacf3b65
Initial commit
2023-12-29 13:07:35 -07:00
0892dec4a5
Fixed some new bugs i added. woops
2023-12-28 14:03:42 -07:00
eeee4a1620
Created a size agnostic feature encoder (SAFE) model to be trained in replace of CLIP for ip adapters. It is mostly conv layers so will hopefully be able to handle facial features better than clip can. Also bug fixes
2023-12-28 12:20:27 -07:00
d11ed7f66c
Big fixes and added method to standardize values in both latent and pixel space before feeding into the network. Target values were determined over huge generated regularization sets.
2023-12-26 06:19:48 -07:00
27ad79053e
Added SDXL support for clip vision embedder trainer
2023-12-24 14:31:29 -07:00
05ae95ca89
Added a clip vision adapter trainer. Only works for sd15 for now
2023-12-24 13:26:04 -07:00
0f8daa5612
Bug fixes, work on maing IP adapters more customizable.
2023-12-24 08:32:39 -07:00
7703e3a15e
Fixes for sdxl ip adapter training. Bug fixes
2023-12-21 11:15:58 -07:00
0f597f453e
Switched ip adapter dataloader to clip_image paths so the control paths can be used for training assistant adapters while training ip adapters
2023-12-20 10:32:24 -07:00
dfb64b5957
Allow ip adapters to be much more variable in their creation
2023-12-20 06:18:33 -07:00
82098e5d6e
Added more functionality for ip adapters
2023-12-19 09:54:56 -07:00
b653906715
Fixed ip adapter training. Works now
2023-12-17 08:22:59 -07:00
13d32423f6
Added a polarity balancer to guidance
2023-12-15 15:19:14 -07:00
39870411d8
More guidance work. Improved LoRA module resolver for unet. Added vega mappings and LoRA training for it. Various other bigfixes and changes
2023-12-15 06:02:10 -07:00
e5177833b2
Targeted guidance work
2023-12-09 19:06:18 -07:00
eaa0fb6253
Tons of bug fixes and improvements to special training. Fixed slider training.
2023-12-09 16:38:10 -07:00
eaec2f5a52
Added guidance mentiods. WIP
2023-12-08 08:39:21 -07:00
bd2bce9b92
Switched to trailing timestep spacing to make timesteps for consistant across schedulers. Honed in on targeted guidance. It is finally perfect. (I think)
2023-11-29 14:32:48 -07:00
7624241032
More fixes for noise schedules and fixed targeted guidance inverted masked prior
2023-11-29 10:13:31 -07:00
Jaret Burkett
Plat
liaoliaojun
martintomov
apolinário