A Mathematical Theory of Communication

SHANNON, Claude Elwood

Published by American Telephone and Telegraph Company, New York, 1948

The Founding Work of Information Theory. First edition, in the two original journal issues as published, of the paper that created information theory and made information, for the first time, a measurable quantity. Claude Shannon defined the entropy of a message source, fixed the maximum rate - the channel capacity - at which information can be sent over any communication link, proved that below that rate transmission can be made as nearly free of error as one likes however noisy the channel, and on his opening pages named the unit of the new quantity the 'bit'. The work reached print here in two installments six months apart in 1948, the form that precedes both the separately issued monograph and the book of the following year. The problem Shannon set himself was deceptively narrow: how to reproduce at one point a message selected at another. His decisive move was to discard meaning. 'These semantic aspects of communication are irrelevant to the engineering problem', he wrote; what matters is only that a message is one selection out of a known set of possibilities, and the size of that set can be counted. From this he built the model every later account of communication has used - an information source and a transmitter at one end, a receiver and a destination at the other, and between them a channel into which noise enters - the schematic he set down as his first figure. Measuring a source's output on a logarithmic scale, he identified the average information per symbol with a quantity formally identical to the entropy of statistical mechanics, a number that grows with a source's unpredictability and falls as its output becomes more redundant; its unit, on the suggestion of his Bell Laboratories colleague John Tukey, he fixed as the binary digit, the bit. The word, in its information-theoretic sense, is printed here for the first time. The paper's central result is the one that startled its first readers. Every channel, Shannon showed, has a definite capacity, a maximum number of bits per second it can carry; and so long as the transmission rate stays below that capacity, a code exists that drives the frequency of errors as low as desired - while above it, reliable communication is impossible. This ran flatly against the engineering common sense of 1948, which assumed that forcing the error rate down must always cost transmission rate. Shannon's proof, moreover, was not constructive: it established that good codes exist without saying how to build them, and the half-century of coding theory that followed was in effect one long campaign to reach the bound he had drawn, closing the last fraction of a decibel only at the century's end. That a result can still be cited as the target six and seven decades after it was stated - the Shannon limit - is the measure of how completely it defined its field. The argument runs in four parts, separating the discrete from the continuous and the noiseless from the noisy, and two theorems carry its weight. The first, on source coding, shows that the output of a source can be compressed to its entropy but no further: the entropy fixes the irreducible number of bits a message requires, and so sets the limit of all data compression. The second, on channel coding, is the capacity result - that information can be driven through a noisy channel almost without error up to the capacity and not beyond. Between them the two theorems mark the boundaries of the possible at either end of a communication system, the least a message can be squeezed to and the most a channel can carry; and Shannon's further demonstration that the two problems can be solved separately, with a simple digital interface between source and channel, is the result that licenses the all-digital design on which every later system rests. The theory grew out of secret work. Through the war Shannon had turned the same probabilistic thinking onto cryptography, and the entropy expression at the centre of the present paper had first appear. Seller Inventory # 6187

Bibliographic Details

Title: A Mathematical Theory of Communication
Publisher: American Telephone and Telegraph Company, New York
Publication Date: 1948
Signed: Signed
Edition: First edition.