Description
A list of potential learning rate schedulers for model optimization.
A learning rate scheduler strategy can be empty (in which case no learning rate scheduling is applied), or can be configured through three parameters: name
, interval
and kwargs
:
name
: The name of the learning rate scheduler strategy. Valid learning rate scheduler strategies are:constant_with_warmup
: Uses a constant learning rate preceded by a warmup period which increases the learning rate from 0 tobase_lr
. Number of warmup steps can be specified throughkwargs
viawarmup_ratio_or_steps
.linear_with_warmup
: Decays the learning rate linearly frombase_lr
to0
, preceded by a warmup period which increases the learning rate from 0 tobase_lr
. Number of warmup steps can be specified throughkwargs
viawarmup_ratio_or_steps
.
exponential
: Decays the learning rate bygamma
.gamma
can be specified throughkwargs
.cosine_with_warmup
: Adjusts the learning rate betweenbase_lr
and0
following a cosine function, preceded by a warmup period which increases the learning rate from 0 tobase_lr
. Number of warmup steps can be specified throughkwargs
viawarmup_ratio_or_steps
.cosine_with_warmup_restarts
: Adjusts the learning rate betweenbase_lr
and0
following a cosine function, with several hard restarts. Preceded by a warmup period which increases the learning rate from 0 tobase_lr
. Number of hard restarts can be configured throughkwargs
vianum_cycles
(3
by default). Number of warmup steps can be specified throughkwargs
viawarmup_ratio_or_steps
.
interval
: Specifies whether learning rate scheduling is applied per optimization step (step
) or per epoch (epoch
).kwargs
: Additional arguments depending on the chosen LR scheduler strategy. See above for detailed information.
By default, a selected set of learning rate scheduling strategies is applied.
Supported Task Types
- All
Example
lr_scheduler:
-
- name: cosine_with_warumup_restarts
interval: step
kwargs:
warmup_ratio_or_steps: 0.2
- name: exponential
interval: epoch
kwargs:
gamma: 0.9
Default Values
run_mode | Default Value |
---|---|
FAST | Not included |
NORMAL | Not included |
BEST | Not included |
Updated 5 months ago