Adam

E701496

Adam is a popular stochastic optimization algorithm widely used to train deep learning models by adaptively adjusting learning rates for each parameter.

All labels observed (1)

Label Occurrences
Adam canonical 2

How this entity was disambiguated

Statements (50)

Predicate Object
instanceOf gradient-based optimization algorithm ⓘ
optimization algorithm ⓘ
stochastic optimization method ⓘ
advantage computationally efficient ⓘ
invariant to diagonal rescaling of gradients ⓘ
requires little hyperparameter tuning ⓘ
works well with sparse gradients ⓘ
basedOn adaptive learning rate methods ⓘ
momentum methods ⓘ
stochastic gradient descent ⓘ
category first-order optimization algorithm ⓘ
combinesIdeaOf RMSProp ⓘ
momentum ⓘ
commonlyUsedIn computer vision models ⓘ
natural language processing models ⓘ
reinforcement learning ⓘ
defaultBeta1 0.9 ⓘ
defaultBeta2 0.999 ⓘ
defaultEpsilon 1e-8 ⓘ
defaultLearningRate 0.001 ⓘ
field deep learning ⓘ
machine learning ⓘ
fullName Adaptive Moment Estimation ⓘ
hasHyperparameter beta1 ⓘ
beta2 ⓘ
epsilon ⓘ
learning rate ⓘ
hasVariant AMSGrad ⓘ
AdaBound ⓘ
AdamW ⓘ
linked to: Adam optimizer
implementedIn JAX ⓘ
Keras ⓘ
PyTorch ⓘ
TensorFlow ⓘ
introducedBy Diederik P. Kingma ⓘ
Jimmy Ba ⓘ
introducedInPaper Adam: A Method for Stochastic Optimization ⓘ
limitation can converge to different minima than SGD ⓘ
may generalize worse than SGD with momentum in some settings ⓘ
maintains exponential moving average of gradients ⓘ
exponential moving average of squared gradients ⓘ
property adaptive learning rate per parameter ⓘ
bias-corrected first moment estimates ⓘ
bias-corrected second moment estimates ⓘ
publicationVenue International Conference on Learning Representations ⓘ
linked to: ICLR
publicationYear 2014 ⓘ
usedFor minimizing loss functions ⓘ
neural network training ⓘ
stochastic optimization ⓘ
training deep learning models ⓘ

How these facts were elicited

Referenced by (2)

Full triples — surface form annotated when it differs from this entity's canonical label.