THE MEDIA BUSINESS; Random House to Buy British Book Publisher

In a move that will nearly triple the size of its holdings in Britain, Random House announced yesterday that it would acquire Century Hutchinson Ltd., a prominent London-based publisher of fiction and nonfiction. The acquisition will also expand Random House's presence in Australia and New Zealand.

Neither party would disclose the purchase price, but industry officials estimate it at $100 million - slightly more than the combined annual sales of Century Hutchinson and the British publishing houses that Random House acquired two years ago.

Random House, which is owned by Advance Publications Inc., a property of the Newhouse family, is buying the company from an investor group.

''Our strategy has been to remain independent, but we recognized that with all the publishing conglomerates, we had to have an American alliance,'' Anthony Cheetham, managing director of Century Hutchinson, said in a telephone interview yesterday. ''And it also was very attractive to us that Random House is privately owned.'' An American Alliance

Century Hutchinson had been forging an American alliance through Little, Brown & Company, a unit of Time Inc. Under their arrangement, Little, Brown would have been allowed to own as much as 20 percent of the British house. The Little, Brown interest will be purchased by Random House.

Two months ago Simon & Schuster, a unit of Paramount Communciations Inc., offered to buy Century Hutchinson. But Century Hutchinson decided that Random House's extensive publishing operation in Britain, consisting of Chatto & Windus, Jonathan Cape and Bodley Head, was more compatible with its own interests.

Robert L. Bernstein, chairman of Random House, said yesterday that despite the growth of the company's British operations, they were not yet big enough to take advantage of economies of scale like those that Random House has in the United States, where its imprints include Alfred A. Knopf, Ballantine Books, Villard and Vintage.

The acquisition of Century Hutchinson should change that. Among the properties Random House will get are Arrow Books, a mass-market paperback imprint that will give Random House its first mass-market arm outside the United States. A 6 Percent Market Share

Arrow has only a 6 percent share of the paperback market in Britain. ''But with the United Kingdom imprints of Random House feeding into Arrow,'' Mr. Cheetham said, ''we not only will have little trouble doubling that market share, we should go on to become a real rival to Viking Penguin.'' He estimated Viking Penguin's share of the paperback market at 25 percent to 28 percent.

Hutchinson, founded in London in 1887, publishes many literary books. Its spring list includes authors like Anthony Burgess, Kingsley Amis and W. D. Snodgrass, as well as nonfiction authors like Robert Conquest, the historian. Century, founded in 1982, specializes in popular fiction and nonfiction. Its authors include Len Deighton, Andrew Greeley, Joan Collins and Colleen McCullough.