PAKRAC, Yugoslavia, March 3— The main square of this small Croatian town with a large Serbian population, was, at least for today, the epicenter of Yugoslavia's tensions.

On one side, a young Croatian policeman guarded the Orthodox church normally attended by Serbs. A few yards away, three armored vehicles belonging to Yugoslavia's Federal army were parked in front of the Roman Catholic church, where most local Croats attend.

Though the Federal military, which is dominated by Serbian officers, has recently tried to discourage the exercize of Croatia's police powers, the two forces appeared to be working togther here to maintain the peace.

The arrival of Federal military units and Croation police began after local Serbs took control of the town's police station on Friday. Croatian police arrived Saturday and successfully stormed the building. Federal troops were deployed soon after to prevent what President Bolislav Jovic termed "an escalation of interethnic clashes." Croatian Police Withdraw

The 700 Croatian security police withdrew from the town tonight and the Federal forces are also expected to pull back soon and leave the town in the hands of local authorities. Officials in Pakrac issued a statement tonight urging people to return to work Monday.

Some Serbian residents said that there had been casualties in the exchange of gunfire when the police station was retaken. Croatian officers said, however, no one had been killed or seriously wounded. That account was confirmed by an official at the town's hospital.

The enmity between Serbs and Croats, who make up the largest of Yugoslavia's six republics, has deep roots. About 30 miles from Pakrac, at the Jasenovac concentration camp, tens of thousands of Serbs, gypsies and Jews were killed when it was run by the fascist puppet government of Croatia during World War II.

For the Serbs, the memory of the wartime atrocity is long. Jovan Miljanic, a 70-year-old retired road worker, stood with a group of Serbian men on the outskirts of Pakrac today and declared: "If the Croatian police does not withdraw, there will be victims. The Croatians killed people, stuffed them down wells. People still remember this. They get frightened when they see the same insignia that they saw in the war years." Serbs Oppose Secession

Tensions in Croatia are also being heightened by economic and political concerns. Although they make up only 550,000 of Croatia's 5 million population, Serbs in towns like Pakrac, have long had favorable positions in local government, state-run industries and the police. Last year's victory of a nationalist political party in Croatian elections is disrupting that tradition.

With Croatia's parliament moving towards secession from Yugoslavia, Serbs in Pakrac and elsewhere have been proclaiming allegiance to a separate entity called Krajina, which intends to remain part of Yugoslavia.

For the federal troops and Croatian police in Pakrac, political differences seemed to be of little concern. The soldiers and policemen shared bread and milk with each other this morning.

"Why are all these armored personnel carriers here?" one Croat asked.

"The same reason you are here," the federal officer replied with a laugh. "And I'm telling you it's a good thing we're here."

Map: Map of Yugoslavia showing location of Pakrac.