In just over three weeks' time we shall be celebrating the dawn of the new millennium. I think we are on the verge of one of the most hopeful periods the world has ever seen. Indeed, I have felt that way ever since the fall of the Berlin Wall in 1989, surely the defining event of the last part of this century. As Bill Emmott, editor of The Economist, put it in a recent survey:

The fin of this siècle has been an astonishingly positive period, far more positive than most people in the 1980s, and certainly the gloomy 1970s, would have thought possible. Liberty – political, economic and personal – has become a widespread fact for the first time. The threat of war casts its dark shadow over a smaller proportion of the world's population, and fewer people live in constant fear of arbitrary arrest, torture or worse. Too many still do. Nevertheless, Franklin Roosevelt's four freedoms – from fear and from want, and of belief and of expression – are possessed by more people, more securely, than ever before.

To be sure, all is far from the best in the best of all possible worlds. Millions of people still endure appallingly low living standards. There are still many unfree societies. Wars have erupted this decade in Bosnia, Rwanda, Kosovo, East Timor, Chechnya and many other places. Developments in technology bring risks as well as great opportunities. Governments continue to interfere in the lives of citizens in new and unwarranted ways. Human folly has not been eliminated and never will be.

But the bigger picture is surely one of hope. Democracy continues to spread around the globe. Hundreds of millions of people today are free from the yoke of communist totalitarianism under which they laboured just a decade or so ago. International tensions have subsided with the end of the Cold War. Social changes have seen vast improvements in the positions of women and minorities. Developments in computing and the internet are ushering in a new industrial revolution.

Taking a longer-term view reinforces the grounds for hope. The modern world began some 200 years ago with the first industrial revolution, driven by the growth of the rule of law, technology and the abandonment of the mercantilist system. Since then ordinary people worldwide have come to enjoy living standards that were formerly the preserve of a privileged few. Average world per capita income in 1800 in today's money was US$100; by 1900 it was about $500; next year it will be about $5,000 and by the end of the next century some estimates put it to be in excess of $40,000, or higher than the average Western income of today. It could, of course, turn out to be much higher than even that.

The quality of life has improved in many other ways as well. Life expectancy has increased in rich and poor countries alike. Health standards have improved with advances in incomes, housing and medicine. The likelihood of a population explosion appears to be receding; indeed population decline may become a concern in some countries in the next century. The air and water in many advanced countries is becoming cleaner rather dirtier, natural resources are becoming less scarce rather than more scarce, and even any trend towards global warming appears manageable, with scientific estimates of the rate of climate change being steadily downgraded. The economist Julian Simon, who has done more than anyone to debunk the prophecies of environmental doom of the 1960s and '70s, wrote in his last book:

With reasonable surety one can expect that the material conditions of life will continue to get better for most people, in most countries, most of the time, indefinitely.

Along with the spread of democracy in the past 25 years there has been a profound change in the economic climate. The 1950s and '60s were a golden age in the industrial world. Annual growth averaged 3-4 percent in many countries, inflation was negligible and 3 percent was considered a high rate of unemployment. All that changed in the 1970s with the emergence of 'stagflation' – stagnant growth coupled with high inflation and unemployment. There was much pessimism about the future of market economies.

From the mid-1970s, however, governments generally abandoned Keynesian economics and began to reverse earlier trends in policies. Many moved progressively towards policies of monetary and fiscal restraint and economic liberalisation. The result was that by the early 1990s inflation in the industrial countries had generally returned to low levels, and during the decade productivity and growth rates picked up. Unemployment has fallen in a number of countries, with the United States leading the way in returning to a state of close to full employment. The Asian crisis of 1997-98 seems likely to have been just a brief interruption to this trend. Parts of the advanced world look to be regaining the economic vitality of the 1950s and '60s, and the prospects of a return to full employment in many countries over the next decade appear bright.

Many of the economic lessons of the past 25 years have been summed up in a book that has been on the New York Times best seller list for several months called The Lexus and the Olive Tree, written by the Times' chief foreign correspondent, Thomas Friedman. The book is about globalisation, and the theme is that in order to benefit from the new global economy a nation must play by certain rules which he calls the "Golden Straitjacket":

... making the private sector the primary engine of its economic growth, maintaining a low rate of inflation and price stability, shrinking the size of its state bureaucracy, maintaining as close to a balanced budget as possible, if not a surplus, eliminating or lowering tariffs on imported goods, removing restrictions on foreign investment, getting rid of quotas and domestic monopolies, increasing exports, privatizing state-owned industries and utilities, deregulating capital markets, making its currency convertible, opening its industries, stock, and bond markets to direct foreign ownership and investment, deregulating its economy to promote as much domestic competition as possible, eliminating government corruption, subsidies and kickbacks as much as possible, opening its banking and telecommunications systems to private ownership and competition, and allowing its citizens to choose from an array of competing pension options and foreign-run pension and mutual funds.

"As your country puts on the Golden Straitjacket", Friedman writes, "two things tend to happen: your economy grows and your politics shrinks".

Friedman is no intellectual relation to Milton Friedman, Reagan or Thatcher; indeed he is an Al Gore Democrat and he shares some of Gore's wilder ideas on issues such as the environment. He is also conscious of the threats of globalisation to workers in protected industries and the cultural challenges it poses, as we saw last week in the responses of protesters on the streets of Seattle. But the book documents the net gains of trends toward liberalisation that need to be better explained by governments that wish to improve the living standards of their citizens.

Outside the economic arena, changes of thinking appear to be underway that are revising many of the approaches governments have taken to social problems since the 1930s. As always, the United States is the country to watch. Perhaps the major policy change presided over by the Clinton administrations of the 1990s has been the welfare reforms which have so far reduced by around half the number of Americans on welfare benefits. At first controversial, they now enjoy broad popular support. The combination of the strong US economy, very tight labour markets and the new stance on welfare has meant that unskilled and less educated workers have shared in the increase in real earnings, poverty rates have begun to decline, and even the bottom 20 percent of households have had a significant increase in their standard of living.

Similar lessons come from Asia. Singapore's senior minister, Lee Kuan Yew, noted in a speech earlier this year that from the late 1960s he had:

... visited Hong Kong almost every year, to study and to understand why Hong Kong people work with so much more drive and vigour than the people in Singapore, and to learn something from them. They paid for their education, they paid for their medical services; the government owed them nothing and provided only law and order. They made their own housing in little shanty huts on the hillsides. They paid their way, and built modern Hong Kong.

Through Hong Kong watching, I concluded that state welfare and subsidies blunted the individual's drive to succeed. I watched with amazement the ease with which Hong Kong workers adjusted their salaries upwards in boom times and downwards in recessions. I resolved to reverse course on the welfare policies which my party had inherited or copied from British Labour Party policies.

With few exceptions, Asian countries appear resolved not to go down the path of Western countries in creating large welfare states. Countries like New Zealand need to consider whether genuine compassion for the poor consists essentially of increases in welfare benefits or of a radically different strategy.

In other social areas such as health, education and retirement income, there are growing demands by voters worldwide to be able to decide how to spend their money rather than have politicians and bureaucrats make their choices for them. In the United States there is overwhelming support for junking the retirement income policy embodied in social security, the centrepiece of the New Deal. The state education monopoly is under attack as never before – American educational websites report new voucher schemes, charter schools and private non-profit and for-profit initiatives almost on a daily basis. These developments are replicated elsewhere and cannot fail to have an influence on New Zealand.

A common element of all these trends is a move away from a political society to a civil society, in both its commercial and non-commercial aspects. Politics is in decline worldwide. Casting a vote for a candidate or a political party is a terribly crude means of obtaining many of the things we all want in life. This is not, as some would have it, a retreat from community values – rather, it is a retreat from collectivist means. Again, for evidence look at the United States. Reports abound of talented young Americans giving time to help the less privileged, of wealthy business people retiring in their forties to devote their efforts to philanthropy, and even of some of the best college graduates choosing teaching as a career in the new educational establishments. In Gallop polls this year, 88 percent of adult Americans said that religion was either very important or fairly important in their lives, down surprisingly little from the 95 percent who made that claim in 1952.

The next development that will accelerate trends away from political society is the growth of the internet, which has been called the technology of freedom. It will vastly increase the spread of knowledge about better and worse ways of doing things. Financial capital and brainpower will move even faster from unattractive to attractive locations. E-business will further empower consumers and put pressure on firms operating anywhere in the world to deliver better, cheaper services. The internet will greatly limit the ability of governments to tax and regulate. In a 1997 article in Wired magazine, Jon Katz described the digital young, from Silicon Valley entrepreneurs to college students, as being disconnected from conventional political parties and having a nearly universal contempt for government's ability to work: they think it's "wasteful and clueless", not an instrument of positive change or social good. Governments that do not want their countries to become home merely to the unskilled, the immobile and the elderly will have to curb their interventionist impulses if they are to attract and retain mobile resources.

At the same time, lovers of civil society are not haters of governments undertaking their proper functions. The core functions of government are vitally important. Governments serve their citizens well when they protect property rights and enforce contracts, maintain a stable currency, ensure the provision of genuine public goods, keep markets open and provide a social safety net. It is when they move beyond those core roles that their tax, spending and regulatory policies become counterproductive and cause economies and societies to malfunction. A lesson of this century is that government failure is a far bigger problem than market failure. We need government, but we need smaller, better government.

Such ideas appear to be taking root in New Zealand, despite its legacy of "socialism without doctrines", the political history described in Michael Bassett's recent excellent book. The latest Massey University values study found that far more of the public see competition as good rather than harmful; most people favour private ownership rather than government ownership of businesses; more New Zealanders favour individuals looking after themselves, rather than the government doing it for them; and a whopping 92 percent have no confidence or not very much confidence in political parties. The authors state that:

These items, and others as well, do not suggest a picture of a population far to the 'left', yearning for some extreme form of collectivism. It is in fact a picture of a society with a fairly strong preference for the freedom of the individual.

To be sure, there were some counter-observations and seemingly inconsistent responses, but the general thrust of these findings would have been unthinkable only 25 years ago.

Will the recent change of government in New Zealand interrupt the trends I have been describing? Time will tell. The Clinton administration in the United States and the Blair government in the United Kingdom did not fundamentally change the directions of the so-called Reagan and Thatcher revolutions; in most respects they carried them further. New Zealand is now lagging behind many of the current trends and we will lose further ground if we fail to press ahead or, worse, go back on recent achievements. To many New Zealanders, the proposals to restore a state monopoly in accident compensation, for example, must look like a triumph of outdated ideology over logic and experience. New Zealand can ill-afford such diversions, but equally it is unlikely to be able to ignore for long the lessons to which we will be exposed in a world that is not about to stop changing.

One should always temper an optimistic view about the future with caution. A salutary warning was given last year by an octogenarian Ronald Coase, a Nobel laureate in economics. Coase wrote:

Having lived through World War I, World War II, the Great Depression, the emergence of communism in Russia and its spread (with the approval and active support of many intellectuals in the West), the triumph of Nazism in Germany (with the support of the great mass of the German people), the adoption of socialism in Britain, the horrors in countless countries all over the world in the post-war period (of which Bosnia is but a recent example), I find it difficult to ignore the role of stupidity in human affairs.

Thus despite the promising outlook, things could go terribly wrong. Follies also take less dramatic forms. Tasmania is an object lesson for New Zealand of how anti-business policies can turn a state or country into a depopulating economic museum. However, by and large I prefer the conclusion of another octogenarian and Nobel laureate, Milton Friedman, perhaps the most influential practitioner of economic ideas in the last 50 years. Friedman ended his memoirs published jointly last year with his wife Rose with the following words:

Judged by practice, we have been, despite some successes, mostly on the losing side. Judged by ideas, we have been on the winning side. The public in the United States has increasingly recognized that government is not the universal cure for all ills, that government measures taken with good intentions and for good purposes often, if not typically, go astray and do harm instead of good. The growth of government has come to a halt, and seems on the verge of declining as a fraction of the economy. We are in the mainstream of thought, not, as we were fifty years ago, members of a derided minority.

So we close this book full of optimism for the future, in the belief that those ideas will prevail and that our children and grandchildren will live in a country that continues to advance rapidly in material and biological well-being and gives its citizens ever wider freedom to follow their own values and tastes, so long as they do not interfere with the ability of others to do the same.

As for me, I'm off with my family in three weeks' time to see in the new year in Jerusalem. Something happened there 2000 years ago which gave hope to the world in the centuries that followed. It seems like a suitable place on earth to think optimistic thoughts about the new millennium.