Jaret Burkett
|
cfdc9033a6
|
Add min and max LR or automagic to prefent runaway edge cases.
|
2026-07-17 14:19:53 -06:00 |
|
Jaret Burkett
|
55ce6570f2
|
Automagic 3 rework. Stable in my testing.
|
2026-06-12 07:52:44 -06:00 |
|
Jaret Burkett
|
01b6a9806b
|
Another complete rework of automagic3. Added a decay to the LR spread to the mean to prevent LRs fighting with eachother
|
2026-06-09 12:08:45 -06:00 |
|
Jaret Burkett
|
acb1548722
|
Updated the comments and doc for Automagic v3
|
2026-06-09 07:05:48 -06:00 |
|
Jaret Burkett
|
a1ac6e8b01
|
Reworked automagic v3 again. Seems more stable. Still testing.
|
2026-06-08 22:03:54 -06:00 |
|
Jaret Burkett
|
5d6887fd98
|
Major updates to automagic3 optimizer. Seems to be functioning more ideally and naturally decays, as it should.
|
2026-06-08 14:09:02 -06:00 |
|
Jaret Burkett
|
e281df70dd
|
Allow automagic3 to run unfused. Add some clipping.
|
2026-06-08 08:59:06 -06:00 |
|
Jaret Burkett
|
1dc9a797cf
|
Added Automagic v3
|
2026-06-07 12:06:42 -06:00 |
|
Jaret Burkett
|
2aa60e4ca5
|
Update default agreement threshold for automagic v2 to be 0.5
|
2026-05-26 07:57:54 -06:00 |
|
Jaret Burkett
|
6bb8acbffc
|
Add agreement_threshold default of 0.6 to automagic 2
|
2026-05-05 19:13:00 -06:00 |
|
Jaret Burkett
|
4260a3c5b6
|
Add optimizer test suite and make minor speed adjustments to Automagicv2
|
2026-05-05 10:02:30 -06:00 |
|
Jaret Burkett
|
aeca7fe404
|
Add Automagic v2 optimizer. It uses significantly less vram and is much more efficient.
|
2026-05-05 09:09:07 -06:00 |
|
Jaret Burkett
|
989ebfaa11
|
Added a basic torch profiler that can be used in config during development to find some obvious issues.
|
2025-06-17 13:03:39 -06:00 |
|
Jaret Burkett
|
2b4c525489
|
Reworked automagic optimizer and did more testing. Starting to really like it. Working well.
|
2025-04-28 08:01:10 -06:00 |
|
Jaret Burkett
|
dd700f70b3
|
Avoid loading state dict for automagic for now until I can sort out some issues
|
2025-02-26 17:03:14 -07:00 |
|
Jaret Burkett
|
216ab164ce
|
Experimental features and bug fixes
|
2025-02-04 13:36:34 -07:00 |
|
Jaret Burkett
|
8ef07a9c36
|
Added training for an experimental decoratgor embedding. Allow for turning off guidance embedding on flux (for unreleased model). Various bug fixes and modifications
|
2024-12-15 08:59:27 -07:00 |
|
Jaret Burkett
|
92ce93140e
|
Adjustments to defaults for automagic
|
2024-11-29 10:28:06 -07:00 |
|
Jaret Burkett
|
f213996aa5
|
Fixed saving and displaying for automagic
|
2024-11-29 08:00:22 -07:00 |
|
Jaret Burkett
|
cbe31eaf0a
|
Initial work on a auto adjusting optimizer
|
2024-11-29 04:48:58 -07:00 |
|
Jaret Burkett
|
96d418bb95
|
Added support for full finetuning flux with randomized param activation. Examples coming soon
|
2024-11-21 13:05:32 -07:00 |
|
Jaret Burkett
|
894374b2e9
|
Various bug fixes and optimizations for quantized training. Added untested custom adam8bit optimizer. Did some work on LoRM (dont use)
|
2024-11-20 09:16:55 -07:00 |
|
Jaret Burkett
|
025ee3dd3d
|
Added ability for adafactor to fully fine tune quantized model.
|
2024-10-30 16:38:07 -06:00 |
|
Jaret Burkett
|
58f9d01c2b
|
Added adafactor implementation that handles stochastic rounding of update and accumulation
|
2024-10-30 05:25:57 -06:00 |
|
Jaret Burkett
|
e72b59a8e9
|
Added experimental 8bit version of prodigy with stochastic rounding and stochastic gradient accumulation. Still testing.
|
2024-10-29 14:28:28 -06:00 |