Adam optimizer

E182821

The Adam optimizer is a popular stochastic gradient descent method in machine learning that adaptively adjusts learning rates for each parameter using estimates of first and second moments of gradients.

All labels observed (4)

Label Occurrences
Adam optimizer canonical 8
Adam optimization algorithm 3
AdamW 3

How this entity was disambiguated

Statements (56)

Predicate Object
instanceOf adaptive learning rate method ⓘ
optimization algorithm ⓘ
stochastic gradient descent variant ⓘ
baseAlgorithm stochastic gradient descent ⓘ
category adaptive gradient method ⓘ
commonlyUsedIn computer vision models ⓘ
deep learning ⓘ
natural language processing models ⓘ
neural network training ⓘ
reinforcement learning ⓘ
defaultBeta1 0.9 ⓘ
defaultBeta2 0.999 ⓘ
defaultEpsilon 1e-8 ⓘ
defaultLearningRate 0.001 ⓘ
fullName Adaptive Moment Estimation ⓘ
gradientRequirement first-order gradients ⓘ
hasVariant AMSGrad ⓘ
linked to: AdaGrad

AdamW ⓘ
linked to: Adam optimizer
hyperparameter beta1 ⓘ
beta2 ⓘ
epsilon ⓘ
learning rate ⓘ
weight decay (in some implementations) ⓘ
implementedIn Chainer ⓘ
JAX ⓘ
Keras ⓘ
MXNet ⓘ
PyTorch ⓘ
TensorFlow ⓘ
fastai ⓘ
inspiredBy AdaGrad ⓘ
RMSProp ⓘ
introducedBy Diederik P. Kingma ⓘ
Jimmy Ba ⓘ
introducedInPaper Adam: A Method for Stochastic Optimization ⓘ
maintains per-parameter first moment vector ⓘ
per-parameter second moment vector ⓘ
optimizationTarget minimization of loss function ⓘ
parameterUpdateDependsOn current gradient ⓘ
first moment estimate ⓘ
second moment estimate ⓘ
performsBiasCorrection true ⓘ
publicationYear 2014 ⓘ
relatedTo AdaDelta ⓘ
AdaGrad ⓘ
RMSProp ⓘ
strength fast convergence in practice ⓘ
suitableFor large-scale problems ⓘ
non-stationary objectives ⓘ
sparse gradients ⓘ
updateRuleType adaptive learning rate ⓘ
uses exponentially decaying averages of past gradients ⓘ
exponentially decaying averages of past squared gradients ⓘ
usesFirstMomentEstimate gradient mean ⓘ
usesSecondMomentEstimate uncentered gradient variance ⓘ
weakness can generalize worse than SGD with momentum in some settings ⓘ

How these facts were elicited

Referenced by (15)

Full triples — surface form annotated when it differs from this entity's canonical label.

Jimmy Ba → knownFor → Adam optimizer ⓘ
Dueling DQN → usesOptimizationMethod → Adam optimizer ⓘ
RMSProp → oftenComparedWith → Adam optimizer ⓘ
Adam optimizer → hasVariant → AdamW ⓘ
linked to: Adam optimizer
Adam: A Method for Stochastic Optimization → influenced → AdamW optimizer ⓘ
linked to: Adam optimizer
Diederik P. Kingma → notableWork → Adam optimization algorithm ⓘ
linked to: Adam optimizer
Diederik P. Kingma → coDeveloperOf → Adam optimization algorithm ⓘ
linked to: Adam optimizer
Diederik P. Kingma → knownFor → Adam optimizer ⓘ
Adam → hasVariant → AdamW ⓘ
linked to: Adam optimizer
Adam → commonVariant → AdamW ⓘ
linked to: Adam optimizer
Jimmy Lei Ba → knownFor → Adam optimization algorithm ⓘ
linked to: Adam optimizer
VQ-VAE → usesOptimizationMethod → Adam optimizer ⓘ
recurrent neural networks → trainedWith → Adam optimizer ⓘ