Academic Journal
View in EDS

An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory.

Bibliographic Details
Title: An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory.
Authors: Truong, Félix1 (AUTHOR) felix.servant7@gmail.com, Giuliani, Alexandre1,2 (AUTHOR)
Source: Entropy. Mar2026, Vol. 28 Issue 3, p301. 21p.
Subject Terms: *Differential entropy, *Scale invariance (Statistical physics), *Uncertainty (Information theory), *K-nearest neighbor classification, *Information theory, *Affine transformations
People: Shannon, Claude Elwood, 1916-2001
Abstract: Shannon's differential entropy for continuous variables suffers from a fundamental limitation: it is not invariant under scale transformations. This makes entropy values dependent on the choice of measurement units rather than reflecting intrinsic properties of distributions. While Jaynes proposed the limiting density of discrete points (LDDP) as a theoretical solution, a concrete method for computing the required invariant measure has been lacking. This paper establishes a rigorous connection between Kullback–Leibler divergence and the invariant measure, providing theoretical proofs of invariance under affine transformations and a practical computational method. We prove that entropy normalized by the median of k-nearest neighbor distances is invariant under affine transformations (Theorems 1 and 2). The non-negativity of the resulting entropy has been validated empirically across all tested distribution families, though a complete theoretical proof remains an open question. This approach extends naturally to multivariate settings, enabling scale-invariant mutual information estimation. We provide open-source implementations in Julia (EntropyInvariant.jl) and Python (entropy_invariant) and demonstrate their advantages over traditional approaches, particularly for variables with disparate scales. [ABSTRACT FROM AUTHOR]
Database: Academic Search Index
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://resolver.ebsco.com/c/fiv2js/result?sid=EBSCO:asx&genre=article&issn=10994300&ISBN=&volume=28&issue=3&date=20260301&spage=301&pages=301-321&title=Entropy&atitle=An%20Invariant%20Measure%20for%20Differential%20Entropy%3A%20From%20Kullback%E2%80%93Leibler%20Divergence%20to%20Scale-Invariant%20Information%20Theory.&aulast=Truong%2C%20F%C3%A9lix&id=DOI:10.3390/e28030301
    Name: Full Text Finder (for New FTF UI) (ns324271)
    Category: fullText
    Text: Full Text Finder
    MouseOverText: Full Text Finder
Header DbId: asx
DbLabel: Academic Search Index
An: 192590273
RelevancyScore: 1411
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 1411.07177734375
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Truong%2C+Félix%22">Truong, Félix</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> felix.servant7@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Giuliani%2C+Alexandre%22">Giuliani, Alexandre</searchLink><relatesTo>1,2</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Entropy%22">Entropy</searchLink>. Mar2026, Vol. 28 Issue 3, p301. 21p.
– Name: Subject
  Label: Subject Terms
  Group: Su
  Data: *<searchLink fieldCode="DE" term="%22Differential+entropy%22">Differential entropy</searchLink><br />*<searchLink fieldCode="DE" term="%22Scale+invariance+%28Statistical+physics%29%22">Scale invariance (Statistical physics)</searchLink><br />*<searchLink fieldCode="DE" term="%22Uncertainty+%28Information+theory%29%22">Uncertainty (Information theory)</searchLink><br />*<searchLink fieldCode="DE" term="%22K-nearest+neighbor+classification%22">K-nearest neighbor classification</searchLink><br />*<searchLink fieldCode="DE" term="%22Information+theory%22">Information theory</searchLink><br />*<searchLink fieldCode="DE" term="%22Affine+transformations%22">Affine transformations</searchLink>
– Name: SubjectPerson
  Label: People
  Group: Su
  Data: <searchLink fieldCode="PE" term="%22Shannon%2C+Claude+Elwood%2C+1916-2001%22">Shannon, Claude Elwood, 1916-2001</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Shannon's differential entropy for continuous variables suffers from a fundamental limitation: it is not invariant under scale transformations. This makes entropy values dependent on the choice of measurement units rather than reflecting intrinsic properties of distributions. While Jaynes proposed the limiting density of discrete points (LDDP) as a theoretical solution, a concrete method for computing the required invariant measure has been lacking. This paper establishes a rigorous connection between Kullback–Leibler divergence and the invariant measure, providing theoretical proofs of invariance under affine transformations and a practical computational method. We prove that entropy normalized by the median of k-nearest neighbor distances is invariant under affine transformations (Theorems 1 and 2). The non-negativity of the resulting entropy has been validated empirically across all tested distribution families, though a complete theoretical proof remains an open question. This approach extends naturally to multivariate settings, enabling scale-invariant mutual information estimation. We provide open-source implementations in Julia (EntropyInvariant.jl) and Python (entropy_invariant) and demonstrate their advantages over traditional approaches, particularly for variables with disparate scales. [ABSTRACT FROM AUTHOR]
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=asx&AN=192590273
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.3390/e28030301
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 21
        StartPage: 301
    Subjects:
      – SubjectFull: Shannon, Claude Elwood, 1916-2001
        Type: general
      – SubjectFull: Differential entropy
        Type: general
      – SubjectFull: Scale invariance (Statistical physics)
        Type: general
      – SubjectFull: Uncertainty (Information theory)
        Type: general
      – SubjectFull: K-nearest neighbor classification
        Type: general
      – SubjectFull: Information theory
        Type: general
      – SubjectFull: Affine transformations
        Type: general
    Titles:
      – TitleFull: An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Truong, Félix
      – PersonEntity:
          Name:
            NameFull: Giuliani, Alexandre
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Text: Mar2026
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 10994300
          Numbering:
            – Type: volume
              Value: 28
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Entropy
              Type: main
ResultId 1