Kullback–Leibler divergence

E6392

f-divergence information-theoretic measure relative entropy statistical divergence

Kullback–Leibler divergence is a fundamental information-theoretic measure that quantifies how one probability distribution differs from a reference distribution.

Try in SPARQL Jump to: Surface forms Disambiguation Statements Elicitation Referenced by

AI illustration

All labels observed (7)

Label	Occurrences
Kullback–Leibler divergence canonical	23
Gibbs' inequality	1
KL divergence	1
Kullback	1
Kullback–Leibler distance	1
Kullback–Leibler divergence (locally)	1
Kullback–Leibler divergence inequalities	1

How this entity was disambiguated

This entity first appeared as the object of triple T59003 — resolving that mention is where its identity was fixed. The disambiguator weighed these candidate entities and picked the highlighted one (or “None”, minting a new entity). This is how homonymy is resolved: the same surface form can point to different entities.

NED1 Entity disambiguation (via context triple) gpt-5-mini-2025-08-07

Target entity: Kullback–Leibler divergence
Context triple: [Shannon entropy, relatedConcept, Kullback–Leibler divergence]

A. Shannon entropy
Shannon entropy is a fundamental measure in information theory that quantifies the average uncertainty or information content in a random variable or message source.
B. Feynman–Hellmann theorem
The Feynman–Hellmann theorem is a result in quantum mechanics that relates the derivative of an energy eigenvalue with respect to a parameter in the Hamiltonian to the expectation value of the corresponding derivative of the Hamiltonian.
C. Einstein–Smoluchowski relation
The Einstein–Smoluchowski relation is a fundamental equation in statistical physics that links the diffusion coefficient of particles undergoing Brownian motion to their mobility and thermal energy.
D. Bekenstein–Hawking entropy
Bekenstein–Hawking entropy is the thermodynamic entropy associated with a black hole, proportional to the area of its event horizon and fundamental in linking gravity, quantum theory, and thermodynamics.
E. Feynman–Kac formula
The Feynman–Kac formula is a fundamental result connecting solutions of certain partial differential equations with expectations over stochastic processes, forming a bridge between quantum mechanics, probability theory, and mathematical finance.
F. None of above. chosen
G. Unsure - the case is ambiguous/there is not enough information to decide.

NED2 Entity disambiguation (via description) gpt-5-mini-2025-08-07

Target entity: Kullback–Leibler divergence
Target entity description: Kullback–Leibler divergence is a fundamental information-theoretic measure that quantifies how one probability distribution differs from a reference distribution.

A. Shannon entropy
Shannon entropy is a fundamental measure in information theory that quantifies the average uncertainty or information content in a random variable or message source.
B. Feynman–Hellmann theorem
The Feynman–Hellmann theorem is a result in quantum mechanics that relates the derivative of an energy eigenvalue with respect to a parameter in the Hamiltonian to the expectation value of the corresponding derivative of the Hamiltonian.
C. Einstein–Smoluchowski relation
The Einstein–Smoluchowski relation is a fundamental equation in statistical physics that links the diffusion coefficient of particles undergoing Brownian motion to their mobility and thermal energy.
D. Bekenstein–Hawking entropy
Bekenstein–Hawking entropy is the thermodynamic entropy associated with a black hole, proportional to the area of its event horizon and fundamental in linking gravity, quantum theory, and thermodynamics.
E. Feynman–Kac formula
The Feynman–Kac formula is a fundamental result connecting solutions of certain partial differential equations with expectations over stochastic processes, forming a bridge between quantum mechanics, probability theory, and mathematical finance.
F. None of above. chosen

Statements (51)

Predicate	Object
instanceOf	f-divergence ⓘ information-theoretic measure ⓘ relative entropy ⓘ statistical divergence ⓘ
alsoKnownAs	KL divergence ⓘ linked to: Kullback–Leibler divergence Kullback–Leibler distance ⓘ linked to: Kullback–Leibler divergence relative entropy ⓘ
appearsIn	Kullback and Leibler 1951 paper ⓘ
definedFor	continuous probability distributions ⓘ discrete probability distributions ⓘ
domain	pairs of probability distributions ⓘ
equalsZeroIfAndOnlyIf	two distributions are equal almost everywhere ⓘ
field	information theory ⓘ machine learning ⓘ probability theory ⓘ statistical inference ⓘ statistics ⓘ
hasProperty	additive for independent distributions ⓘ convex in the pair of distributions ⓘ
isMetric	false ⓘ
isNonNegative	true ⓘ
isSymmetric	false ⓘ
minimizedBy	true data-generating distribution in maximum likelihood ⓘ
namedAfter	Richard Leibler ⓘ Solomon Kullback ⓘ
quantifies	difference between probability distributions ⓘ information loss when approximating one distribution with another ⓘ
relatedTo	Bregman divergence ⓘ Jensen–Shannon divergence ⓘ Shannon entropy ⓘ cross-entropy ⓘ mutual information ⓘ
satisfiesTriangleInequality	false ⓘ
specialCaseOf	Csiszár f-divergence ⓘ
takesValuesIn	[0, +∞] ⓘ
usedAs	loss function in classification ⓘ regularizer in probabilistic models ⓘ
usedIn	Bayesian inference ⓘ density estimation ⓘ distributional reinforcement learning ⓘ feature selection ⓘ hypothesis testing ⓘ information geometry ⓘ linked to: Riemannian manifolds information-theoretic clustering ⓘ language modeling ⓘ machine learning model training ⓘ maximum likelihood estimation ⓘ natural gradient descent ⓘ reinforcement learning ⓘ variational autoencoders ⓘ variational inference ⓘ

How these facts were elicited

Referenced by (29)

Full triples — surface form annotated when it differs from this entity's canonical label.

Shannon entropy → relatedConcept → Kullback–Leibler divergence ⓘ

Kullback–Leibler divergence → alsoKnownAs → KL divergence ⓘ

linked to: Kullback–Leibler divergence

Kullback–Leibler divergence → alsoKnownAs → Kullback–Leibler distance ⓘ

linked to: Kullback–Leibler divergence

Solomon Kullback → familyName → Kullback ⓘ

linked to: Kullback–Leibler divergence

Solomon Kullback → knownFor → Kullback–Leibler divergence ⓘ

Solomon Kullback → coDeveloperOf → Kullback–Leibler divergence ⓘ

Solomon Kullback → notableConcept → Kullback–Leibler divergence ⓘ

Solomon Kullback → hasConceptNamedAfter → Kullback–Leibler divergence ⓘ

Rényi divergence → generalizes → Kullback–Leibler divergence ⓘ

Richard Leibler → notableWork → Kullback–Leibler divergence ⓘ

Jensen's inequality → usedFor → Kullback–Leibler divergence inequalities ⓘ

subject linked to: Jensen inequality

linked to: Kullback–Leibler divergence

Jensen's inequality → relatedTo → Gibbs' inequality ⓘ

subject linked to: Jensen inequality

linked to: Kullback–Leibler divergence

information theory → hasCoreConcept → Kullback–Leibler divergence ⓘ

Bhattacharyya distance → relatedTo → Kullback–Leibler divergence ⓘ

Fisher information → relatedTo → Kullback–Leibler divergence ⓘ

Chernoff information → relatedTo → Kullback–Leibler divergence ⓘ

Hellinger distance → relatedTo → Kullback–Leibler divergence ⓘ

Tsallis divergence → generalizes → Kullback–Leibler divergence ⓘ

Pinsker's inequality → relates → Kullback–Leibler divergence ⓘ

subject linked to: Mark Pinsker

Bhattacharyya coefficient → relatedTo → Kullback–Leibler divergence ⓘ

ELBO → relatedTo → Kullback–Leibler divergence ⓘ

Chernoff bound → relatedTo → Kullback–Leibler divergence ⓘ

Jensen–Shannon divergence → basedOn → Kullback–Leibler divergence ⓘ

Jensen–Shannon divergence → isSmoothedVersionOf → Kullback–Leibler divergence ⓘ

Jensen–Shannon divergence → isSymmetrizationOf → Kullback–Leibler divergence ⓘ

Jensen–Shannon divergence → isRobustToSupportMismatchComparedTo → Kullback–Leibler divergence ⓘ

Fisher–Rao metric → relatedTo → Kullback–Leibler divergence (locally) ⓘ

linked to: Kullback–Leibler divergence

Fisher–Rao metric → localApproximationOf → Kullback–Leibler divergence ⓘ

Csiszár f-divergence → generalizes → Kullback–Leibler divergence ⓘ

All labels observed (7)

How this entity was disambiguated Show

Statements (51)

How these facts were elicited Show

Referenced by (29)

How this entity was disambiguated

How these facts were elicited