89dd041b97
Added ability to pair samples with a closer noise with optimal_noise_pairing_samples
2025-01-21 18:30:10 -07:00
29122b1a54
Added code to handle diffusion feature extraction loss
2025-01-21 14:21:34 -07:00
fadb2f3a76
Allow quantizing the te independently on flux. added lognorm_blend timestep schedule
2025-01-18 18:02:31 -07:00
4723f23c0d
Added ability to split up flux across gpus (experimental). Changed the way timestep scheduling works to prep for more specific schedules.
2024-12-31 07:06:55 -07:00
8ef07a9c36
Added training for an experimental decoratgor embedding. Allow for turning off guidance embedding on flux (for unreleased model). Various bug fixes and modifications
2024-12-15 08:59:27 -07:00
96d418bb95
Added support for full finetuning flux with randomized param activation. Examples coming soon
2024-11-21 13:05:32 -07:00
4aa19b5c1d
Only quantize flux T5 is also quantizing model. Load TE from original name and path if fine tuning.
2024-10-29 14:25:31 -06:00
22cd40d7b9
Improvements for full tuning flux. Added debugging launch config for vscode
2024-10-29 04:54:08 -06:00
9f94c7b61e
Added experimental param multiplier to the ema module
2024-10-22 09:25:52 -06:00
3922981996
Added some additional experimental things to the vision direct encoder
2024-10-10 19:42:26 +00:00
a800c9d19e
Add a method to have an inference only lora
2024-10-04 10:06:53 -06:00
67e0aca750
Added ability to load clip pairs randomly from folder. Other small bug fixes
2024-10-03 10:03:49 -06:00
e4c82803e1
Handle random resizing for pixtral input on direct vision adapter
2024-09-28 14:53:38 -06:00
58537fc92b
Added initial direct vision pixtral support
2024-09-28 10:47:51 -06:00
04424fe2d6
Added config setting to set the timestep type
2024-09-24 06:53:59 -06:00
2776221497
Added option to cache empty prompt or trigger and unload text encoders while training
2024-09-21 20:54:09 -06:00
79b4e04b80
Feat: Wandb logging ( #95 )
...
* wandb logging
* fix: start logging before train loop
* chore: add wandb dir to gitignore
* fix: wrap wandb functions
* fix: forget to send last samples
* chore: use valid type
* chore: use None when not type-checking
* chore: resolved complicated logic
* fix: follow log_every
---------
Co-authored-by: Plat <github@p1at.dev>
Co-authored-by: Jaret Burkett <jaretburkett@gmail.com>
2024-09-19 20:01:01 -06:00
951e223481
Added support to disable single transformers in vision direct adapter
2024-09-11 08:54:51 -06:00
121a760c19
Added proper grad accumulation
2024-09-03 07:24:18 -06:00
e5fadddd45
Added ability to do prompt attn masking for flux
2024-09-02 17:29:36 -06:00
d44d4eb61a
Added a new experimental linear weighing technique
2024-09-02 09:22:13 -06:00
4d35a29c97
Add `push_to_hub` to the trainer ( #109 )
...
* add push_to_hub
* fix indentation
* indent again
* model_config
* allow samples to not exist
* repo creation fix
* dont show empty [] if widget doesnt exist
* dont submit the config and optimizer
* Unsafe to have tokens saved in the yaml file
* make sure to catch only the latest samples
* change name to slug
* formatting
* formatting
---------
Co-authored-by: multimodalart <joaopaulo.passos+multimodal@gmail.com>
2024-08-22 21:18:56 -06:00
00bd3d54a3
Actually use the save dtype from the config file.
2024-08-13 17:08:27 -06:00
af108bb964
Bug fix with dataloader. Added a flag to completly disable sampling
2024-08-12 09:19:40 -06:00
a6aa4b2c7d
Added ability to set timesteps to linear for flowmatching schedule
2024-08-11 13:06:08 -06:00
ec1ea7aa0e
Added support for training on primary gpu with low_vram flag. Updated example script to remove creepy horse sample at that seed
2024-08-11 09:54:30 -06:00
2308ef2868
Added flux training instructions
2024-08-10 14:10:02 -06:00
acafe9984f
Adjustments to loading of flux. Added a feedback to ema
2024-08-07 13:17:26 -06:00
c2424087d6
8 bit training working on flux
2024-08-06 11:53:27 -06:00
272c8608c2
Make a CFG version of flux pipeline
2024-08-05 16:35:53 -06:00
187663ab55
Use peft format for flux loras so they are compatible with diffusers. allow loading an assistant lora
2024-08-05 14:34:37 -06:00
87ba867fdc
Added flux training. Still a WIP. Wont train right without rectified flow working right
2024-08-02 15:00:30 -06:00
03613c523f
Bugfixes and cleanup
2024-08-01 11:45:12 -06:00
47744373f2
Change img multiplier math
2024-07-30 11:33:41 -06:00
e81e19fd0f
Added target_norm_std which is a game changer
2024-07-28 16:08:33 -06:00
0bc4d555c7
A lot of pixart sigma training tweaks
2024-07-28 11:23:18 -06:00
80aa2dbb80
New image generation img2img. various tweaks and fixes
2024-07-24 04:13:41 -06:00
c2c4e8cf34
Added ability to target parts of lora for ilora
2024-07-20 22:45:52 +00:00
22d2f6e28f
Fixed issue with grad scaling
2024-07-20 08:21:57 -06:00
11e426fdf1
Various features and fixes. Too much brain fog to do a proper description
2024-07-18 07:34:14 -06:00
e4558dff4b
Partial implementation for training auraflow.
2024-07-12 12:11:38 -06:00
c062b7716c
Varous bug fixes
2024-07-10 15:20:04 -06:00
3072d20f17
Add ability to include conv_in and conv_out to full train when doing a lora
2024-06-29 14:54:50 -06:00
603ceca3ca
added ema
2024-06-28 10:03:26 -06:00
8407c4deea
Merge branch 'main' of github.com:ostris/ai-toolkit
2024-06-23 14:47:43 -06:00
64f2b085b7
Minor fixes
2024-06-23 14:47:40 -06:00
7165f2d25a
Work to omprove pixart training
2024-06-23 20:46:48 +00:00
37cebd9458
WIP Ilora
2024-06-14 09:31:01 -06:00
bd10d2d668
Some work on sd3 training. Not working
2024-06-13 12:19:16 -06:00
cb5d28cba9
Added working ilora trainer
2024-06-12 09:33:45 -06:00
3f3636b788
Bug fixes and little improvements here and there.
2024-06-08 06:24:20 -06:00
833c833f28
WIP on SAFE encoder. Work on fp16 training improvements. Various other tweaks and improvements
2024-05-27 10:50:24 -06:00
5a45c709cd
Work on ipadapters and custom adapters
2024-05-13 06:37:54 -06:00
10e1ecf1e8
Added single value adapter training
2024-04-28 06:04:47 -06:00
b96913d73c
Improvements to dataloader
2024-04-27 09:28:28 -06:00
5da3613e0b
Bug fixes and minor features
2024-04-25 06:14:31 -06:00
5a70b7f38d
Added pixart sigma support, but it wont work until i address breaking changes with lora code in diffusers so it can be upgraded.
2024-04-20 10:46:56 -06:00
7284aab7c0
Added specialized scaler training to ip adapters
2024-04-05 08:17:09 -06:00
427847ac4c
Small tweaks and fixes for specialized ip adapter training
2024-03-26 11:35:26 -06:00
9c1cc9641e
Added keep tokens to keep so many tokens in a prompt when dropping
2024-03-18 13:18:25 -06:00
016687bda1
Adapter work. Bug fixes. Auto adjust LR when resuming optimizer.
2024-03-17 10:21:47 -06:00
72de68d8aa
WIP on clip vision encoder
2024-03-13 07:24:08 -06:00
b01e8d889a
Added stochastic rounding to adafactor. ILora adjustments
2024-03-05 07:07:09 -07:00
337945de9a
Added this not that guidance. Added ability to replace prompts.
2024-02-28 20:10:14 -07:00
49c41e6a5f
Bug fixes. allow for random negative prompts
2024-02-21 04:51:52 -07:00
93b52932c1
Added training for pixart-a
2024-02-13 16:00:04 -07:00
4ec4025cbb
Added adapter modules for text encoders and direct vision
2024-02-12 08:46:18 -07:00
e18e0cb5f8
Added comparitive loss when training clip encoder. Allow selecting clip layer. on ip adapter. Improvements to prior prediction
2024-02-05 07:40:03 -07:00
177c7130ec
improved correction of pred norm by targeting the prior
2024-02-01 06:31:04 -07:00
1ae1017748
Bug fixes. added ability to use l1 loss. varous other tests and improvements
2024-01-31 06:30:54 -07:00
92b9c71d44
Many bug fixes. Ip adapter bug fixes. Added noise to unconditional, it works better. added an ilora adapter for 1 shotting LoRAs
2024-01-28 08:20:03 -07:00
f17ad8d794
various bug fixes. Created an contextual alpha mask module to calculate alpha mask
2024-01-18 16:34:27 -07:00
eebd3c8212
Initial training script for photomaker training. Needs a little more work.
2024-01-15 18:46:26 -07:00
5276975fb0
Added additional config options for custom plugins I needed
2024-01-15 08:31:09 -07:00
e190fbaeb8
Prepwork for ilora
2024-01-12 06:41:15 -07:00
290393f7ae
Imporvements to ip weight adaptation. Bug fixes. Added masking to direct guidance loss. Allow importing a file for random triggers. Handle bas meta images with improper sizing.
2024-01-11 12:22:16 -07:00
645b27f97a
Bug fixes with ip adapter training. Made a clip pre processor that can be trained with ip adapter to help augment the clip input to squeeze in more detail from a larget input. moved clip processing to the dataloader for speed.
2024-01-04 12:59:38 -07:00
65c08b09c3
Added ability to do cfg during training. Various bug fixes
2024-01-02 11:29:57 -07:00
afc231efc1
Added reference adapters, many bug fixes, more ip adapter work and customizability
2024-01-01 17:15:53 -07:00
bafacf3b65
Initial commit
2023-12-29 13:07:35 -07:00
eeee4a1620
Created a size agnostic feature encoder (SAFE) model to be trained in replace of CLIP for ip adapters. It is mostly conv layers so will hopefully be able to handle facial features better than clip can. Also bug fixes
2023-12-28 12:20:27 -07:00
d11ed7f66c
Big fixes and added method to standardize values in both latent and pixel space before feeding into the network. Target values were determined over huge generated regularization sets.
2023-12-26 06:19:48 -07:00
05ae95ca89
Added a clip vision adapter trainer. Only works for sd15 for now
2023-12-24 13:26:04 -07:00
0f8daa5612
Bug fixes, work on maing IP adapters more customizable.
2023-12-24 08:32:39 -07:00
7703e3a15e
Fixes for sdxl ip adapter training. Bug fixes
2023-12-21 11:15:58 -07:00
0f597f453e
Switched ip adapter dataloader to clip_image paths so the control paths can be used for training assistant adapters while training ip adapters
2023-12-20 10:32:24 -07:00
dfb64b5957
Allow ip adapters to be much more variable in their creation
2023-12-20 06:18:33 -07:00
b653906715
Fixed ip adapter training. Works now
2023-12-17 08:22:59 -07:00
39870411d8
More guidance work. Improved LoRA module resolver for unet. Added vega mappings and LoRA training for it. Various other bigfixes and changes
2023-12-15 06:02:10 -07:00
eaa0fb6253
Tons of bug fixes and improvements to special training. Fixed slider training.
2023-12-09 16:38:10 -07:00
c2a4b8e058
More timestep fixes
2023-11-28 10:34:08 -07:00
792a5e37e2
Numerous fixes for time sampling. Still not perfect
2023-11-28 07:34:43 -07:00
d7e55b6ad4
Bug fixes, negative prompting during training, hardened catching
2023-11-24 07:25:11 -07:00
4f9cdd916a
added prompt dropout to happen indempendently on each TE
2023-11-14 05:26:51 -07:00
1ee62562a4
diffirential guidance is WORKING (from what I can tell)
2023-11-07 19:24:12 -07:00
dc8448d958
Added a way pass refiner ratio to sample config
2023-11-06 09:22:58 -07:00
93ea955d7c
Added refiner fine tuning. Works, but needs some polish.
2023-11-05 17:15:03 -07:00
8a9e8f708f
Added base for using guidance during training. Still not working right.
2023-11-05 04:03:32 -07:00
d35733ac06
Added support for training ssd-1B. Added support for saving models into diffusers format. We can currently save in safetensors format for ssd-1b, but diffusers cannot load it yet.
2023-11-03 05:01:16 -06:00
ceaf1d9454
Various bug fixes, wip stuff, and tweaks
2023-11-02 18:19:20 -06:00
7d707b2fe6
Added masking to slider training. Something is still weird though
2023-11-01 14:51:29 -06:00
a899ec91c8
Added some split prompting started code, adamw8bit, replacements improving, learnable snr gos. A lot of good stuff.
2023-11-01 06:52:21 -06:00
436a09430e
Added flat snr gamma vs min. Fixes timestep timing
2023-10-29 15:41:55 -06:00
298001439a
Added gradient accumulation finally
2023-10-28 13:14:29 -06:00
6f3e0d5af2
Improved lorm extraction and training
2023-10-28 08:21:59 -06:00
002279cec3
Allow short and long caption combinations like form the new captioning system. Merge the network into the model before inference and reextract when done. Doubles inference speed on locon models during inference. allow splitting a batch into individual components and run them through alone. Basicallt gradient accumulation with single batch size.
2023-10-24 16:02:07 -06:00
34eb563d55
Added dataset tagging and management tools using llava
2023-10-24 10:11:47 -06:00
0e9fc42816
Allow for inverted masked prior
2023-10-21 06:50:17 -06:00
d46112a354
Fixes for aug pipeline
2023-10-20 13:24:07 -06:00
07bf7bd7de
Allow augmentations and targeting different loss types fron the config file
2023-10-18 03:04:57 -06:00
da6302ada8
added a method to apply multipliers to noise and latents prior to combining
2023-10-17 06:09:16 -06:00
a05459afaf
Fixed issue with adapters that only had 1 input channel. Added ability to set the percentage chance of adapter matching
2023-10-15 15:13:35 -06:00
b1a22d0b3e
hardened reading prompts from json
2023-10-15 07:20:33 -06:00
7909b50d24
Added adapter assistance to SD training
2023-10-14 08:44:53 -06:00
38e441a29c
allow flipping for point of interesting autocropping. allow num repeats. Fixed some bugs with new free u
2023-10-12 21:02:47 -06:00
4e3b2c2569
Allow for alpha to be used as a mask
2023-10-10 17:08:59 -06:00
f4c90bb589
Added config to set the min value of a mask
2023-10-09 15:47:54 -06:00
bb1d3793e3
Added ability to add masks to dataloader and sd trainer to adjust weight of image
2023-10-09 11:21:00 -06:00
579650eaf8
Fixed big issue with bucketing dataloader and added random cripping to a point of interest
2023-10-02 18:31:08 -06:00
085787b799
Allow loading auxillery images from dataloader
2023-09-30 07:28:23 -06:00
8509da60cb
Added a way to add a t2i adapter guided slider training for more consitant images
2023-09-28 14:08:56 -06:00
abf7cd221d
allow setting adapter weight in prompts
2023-09-24 06:51:54 -06:00
830e87cb87
Added IP adapter training. Not functioning correctly yet
2023-09-24 02:39:43 -06:00
19255cdc7c
Bugfixes. Added small augmentations to dataloader. Will switch to abluminations soon though. Added ability to adjust step count on start to override what is in the file
2023-09-20 05:30:10 -06:00
0f105690cc
Added some further extendability for plugins
2023-09-19 05:41:44 -06:00
61badf85a7
t2i training working from what I can tell at least
2023-09-17 15:56:43 -06:00
181f237a7b
added flipping x and y for dataset loader
2023-09-17 08:42:54 -06:00
c698837241
Fixes to esrgan trainer. Moved logic for sd prompt embeddings out of diffusers pipeline so I can manipulate it
2023-09-16 17:41:07 -06:00
27f343fc08
Added base setup for training t2i adapters. Currently untested, saw something else shiny i wanted to finish sirst. Added content_or_style to the training config. It defaults to balanced, which is standard uniform time step sampling. If style or content is passed, it will use cubic sampling for timesteps to favor timesteps that are beneficial for training them. for style, favor later timesteps. For content, favor earlier timesteps.
2023-09-16 08:30:38 -06:00
569d7464d5
implemented device placement preset system more places. Vastly improved speed on setting network multiplier and activating network. Fixed timing issues on progress bar
2023-09-14 08:31:54 -06:00
4e945917df
added dropout to LoRA networks
2023-09-13 15:23:07 -06:00
ae70200d3c
Bug fixes, speed improvements, compatability adjustments withdiffusers updates
2023-09-13 07:03:53 -06:00
e8583860ad
Upgraded to dev for t2i on diffusers. Minor migrations to make it work.
2023-09-11 14:46:06 -06:00
34bfeba229
Massive speed increase. Added latent caching both to disk and to memory
2023-09-10 08:54:49 -06:00
3feb663a51
Switched to new bucket system that matched sdxl trained buckets. Fixed requirements. Updated embeddings to work with sdxl. Added method to train lora with an embedding at the trigger. Still testing but works amazingly well from what I can see
2023-09-07 13:06:18 -06:00
a4c3507a62
Added LoCON from LyCORIS
2023-09-04 08:48:07 -06:00
22ed539321
Allow special args for schedulers
2023-09-03 20:38:44 -06:00
2a40937b4f
reworked samplers. Trying to find what is wrong with diffusers sampling is sdxl
2023-09-03 07:56:09 -06:00
836fee47a6
Fixed some mismatched weights by adjusting tolerance. The mismatch ironically made the models better lol
2023-08-29 15:20:03 -06:00
14ff51ceb4
fixed issues with converting and saving models. Cleaned keys. Improved testing for cycle load saving.
2023-08-29 12:31:19 -06:00
714854ee86
Hude rework to move the batch to a DTO to make it far more modular to the future ui
2023-08-29 10:22:19 -06:00
e866c75638
Built base interfaces for a DTO to handle batch infomation transports for the dataloader
2023-08-28 12:43:31 -06:00
9b164a8688
Fixed issue with bucket dataloader corpping in too much. Added normalization capabilities to LoRA modules. Testing effects, but should prevent them from burning and also make them more compatable with stacking many LoRAs
2023-08-27 09:40:01 -06:00
8105c05c12
Added bucketting capabilities to dataloader. Finally have full planned capability. noice
2023-08-26 16:36:32 -06:00
3367ab6b2c
Moved SD batch processing to a shared method and added it for use in slider training. Still testing if it affects quality over sampling
2023-08-26 08:55:00 -06:00
7157c316af
Added support for training lora, dreambooth, and fine tuning. Still need testing and docs
2023-08-23 15:37:00 -06:00
d298240cec
Tied in ant tested TI script
2023-08-23 13:26:28 -06:00
2e6c55c720
WIP creating textual inversion training script
2023-08-22 21:02:38 -06:00
36ba08d3fa
Added a converter back to ldm from diffusers for sdxl. Can finally get to training it properly
2023-08-21 16:22:01 -06:00
e8667f856f
Fix issue with there being an extra . on gene
2023-08-20 15:54:38 -06:00
Jaret Burkett
Plat
apolinário