Information Theory Meets Power Laws: Stochastic Processes and Language Models

Discover new theoretical connections between stochastic phenomena and the structure of natural language with this powerful volume!
Information Theory Meets Power Laws: Stochastic Processes and Language Models presents readers with a novel subtype of a probabilistic approach to language, which is based on statistical laws of texts and their analysis by means of information theory. The distinguished author insightfully and rigorously examines the linguistic and mathematical subject matter while eschewing needlessly abstract and superfluous constructions.
The book begins with a less formal treatment of its subjects in the first chapter, introducing its concepts to readers without mathematical training and allowing those unfamiliar with linguistics to learn the book’s motivations. Despite its inherent complexity, Information Theory Meets Power Laws: Stochastic Processes and Language Models is a surprisingly approachable treatment of idealized mathematical models of human language.
The author succeeds in developing some of the theory underlying fundamental stochastic and semantic phenomena, like strong nonergodicity, in a way that has not previously been seriously attempted. In doing so, he covers topics including:

• Zipf’s and Herdan’s laws for natural language
• Power laws for information, repetitions, and correlations
• Markov, finite-state,and Santa Fe processes
• Bayesian and frequentist  interpretations of probability
• Ergodic decomposition, Kolmogorov complexity, and universal coding
• Theorems about facts and words
• Information measures for fields
• Rényi entropies, recurrence times, and subword complexity
• Asymptotically mean stationary processes

Written primarily for mathematics graduate students and professionals interested in information theory or discrete stochastic processes, Information Theory Meets Power Laws: Stochastic Processes and Language Models also belongs on the bookshelves of doctoral students and researchers in artificial intelligence, computational and quantitative linguistics as well as physics of complex systems.