MANILA, Monday, Sept. 16— The Philippine Senate voted today to reject a new treaty for the Subic Bay Naval Station and to end an American military presence in the country that has lasted nearly a century. But President Corazon C. Aquino effectively extended the American lease by calling for a national referendum on the base's future.

In a vote of 12 to 11, the Senate adopted a motion to spurn the new treaty, which would allow thousands of American troops to remain at the base for another 10 years. The current treaty on Subic Bay, the largest American military installation in the Philippines, expires today.

"The treaty is defeated," the Senate President, Jovito R. Salonga, announced in a solemn tone after the vote was taken this morning in a show of hands. The decision was greeted with applause and tears in the chamber. Anger Over Compensation

The 11 senators who voted in favor of the new treaty were 5 short of the two-thirds majority needed for ratification.

The vote reflected a view among many senators that the compensation package offered by the United States was too small and, more broadly, that it was time to end an American military presence that began when the United States acquired the Philippines by winning the Spanish-American war.

In voting to spurn the new treaty, Senator Agapito Aquino, brother-in-law of the President and younger brother of her slain husband, Benigno S. Aquino, described the decision as "the dawn of our nation's birth." Ending an 'Adolescence'

"It is a vote for a truly sovereign and independent Philippine nation," he told colleagues in a speech on the Senate floor. "It is a vote to end a political adolescence tied to the purse strings of America -- a crippling dependence."

But Mrs. Aquino's call for a referendum under the terms of the Philippine Constitution has the effect of extending the deadline for an American departure by at least several months, and quite possibly for several years.

Invoking the rallying cry of "people power" that she used with such success in ousting President Ferdinand E. Marcos in 1986, Mrs. Aquino proposed in a televised address Sunday night that "we take the issue directly to the people."

"Once again people power is being called upon," Mrs. Aquino said. "As in 1986, we seek the direct expression of the sovereign will of the Filipino people. Now, as in the past, every one of us must participate in an exercise that gives added substance to the democracy we have established."

The referendum could not be held until late this year or in 1992. Mrs. Aquino's move is expected to set off a series of time-consuming court challenges by lawmakers and others who assert that the Constitution permits voters to overrule a vote by the Senate on a regular law, but not on a treaty. U.S. Stand Called Flexible

While the Bush Administration indicated last week that it would start an immediate and irrevocable withdrawal from Subic if the Philippine Senate vetoed the 10-year treaty, Western diplomats said in recent days that the American position was actually far more flexible, and that American troops would probably remain at the base until after a referendum. Interim arrangements for compensation to the Philippines were unclear.

The base, home to more than 7,000 Americans -- servicemen and women and civilian workers -- is a major refueling and repair station for the United States Seventh Fleet.

While its strategic value has declined with the end of the cold war, Bush Administration officials say that replicating the functions of Subic Bay at installations elsewhere in the Pacific would cost several billion dollars.

Clark Air Base, a sister American installation on the central Philippine island of Luzon, was abandoned this year after it was smothered by volcanic ash released by nearby Mount Pinatubo. Possible Election Issue

Beyond its implications for Philippine-American relations, Mrs. Aquino's call for a referendum on the future of Subic Bay will doubtless have a broad effect on domestic politics.

She may well have created the principal issue in next May's general election, including the vote to pick her successor: whether the United States should be allowed to continue to place troops and weapons, including nuclear weapons, on the soil of a former colony that in many ways has yet to emerge from the American shadow. President Aquino has insisted that she will not seek re-election when her six-year term expires next year.

The new 10-year base agreement rejected today had been hammered out this summer between the Bush Administration and the Manila Government.

. Washington offered $203 million a year in compensation, far less than was originally sought by Manila. Some of the Philippine lawmakers had demanded that the treaty be renegotiated.

Mrs. Aquino, aware of opinion polls showing that an overwhelming majority of Filipinos support the treaty, is refusing to acquiesce in the Senate's rejection of it. The Philippine military also strongly favors the treaty.

At stake are the jobs of tens of thousands of Filipinos who work at the base or in related jobs, as well as hundreds of millions of dollars that the Subic Bay installation injects into the strapped Philippine economy each year.

Acknowledging on Sunday night that Senate rejection of the base agreement was "likely," Mrs. Aquino said that she would invoke a section of the Constitution that gives voters the right to overturn legislative decisions in a national referendum.

Under the Constitution, the signatures of 10 percent of the nation's voters, or about three million Filipinos, are needed on petitions authorizing a referendum.

Many senators disagree with Mrs. Aquino on the question of whether a treaty vote in the Senate, as opposed to the Senate's vote on a regular law, can be overturned by referendum.

The constitutional differences will probably have to be sorted out in the Philippine Supreme Court, a process that could take several months. It could even delay a referendum until after the general election in May, when new senators, including some who may be more receptive to a base treaty with the United States, will be selected. -------------------- U.S. Supports a Referendum

WASHINGTON, Sept. 15 (Special to The New York Times) -- American officials welcomed the effort by President Aquino today to win approval of the base treaty through a referendum.

Richard H. Solomon, the Assistant Secretary of State for East Asian and Pacific Affairs, said in an interview, "We support her efforts."

Exactly when the current agreement ends is, from the American perspective, a complicated question. The Philippines has said it expires on Monday. The United States disagrees, saying the earliest possible date is Sept. 16, 1992.

The Military Bases Agreement was signed in 1947, eight months after the Philippines became independent of the United States. An amendment adopted by both nations in 1966 says it "shall remain in force for a period of 25 years from September 16, 1966, after which, unless extended for a longer period by mutual agreement, it shall become subject to termination upon one year's notice by either government."

The Philippines gave such notice several months ago. State Department officials contend that the notice could not properly be given until Monday.