GENOCIDE Its Political Use in the Twentieth Century. By Leo Kuper. 255 pp. New Haven: Yale University Press. $15.

IN 1943, in the course of his monumental study ''Axis Rule in Occupied Europe,'' the late Raphael Lemkin coined the word genocide - from the Greek genos (race or tribe) and the Latin cide (killing) - to describe the deliberate ''destruction of a nation or of an ethnic group.'' The concept was catching, and in December 1946 the General Assembly of the United Nations adopted a resolution recognizing genocide as a ''crime under international law'' consisting of the ''denial of the right of existence of entire human groups.'' After much debate in the United Nations committees, the General Assembly approved the text of a ''Genocide Convention,'' to become effective upon ratification by 20 member nations. By the end of 1978, the Convention had been ratified by 84 nations. But the United States has not done so, largely because of fears that its enforcement might lead to encroachments on national sovereignty.

Now Leo Kuper, professor emeritus at the University of Southern California, has published a useful study of the subject, with extensive historical detail in an analytical framework. The structure of the book is a bit loose, but it is apparent that the first five of the ten chapters deal with the background and drafting of the Convention, with theories that may explain the genocide phenomenon and with social and ideological ''justifications'' for its perpetration; the next three chapters are devoted to ''The Turkish Genocide Against Armenians'' (about which there is a paucity of recent literature), ''The German Genocide Against Jews'' and ''Related Atrocities,'' including African and Asian episodes and the Soviet slaughters and deportations during the Stalin years.

The author is a specialist in African studies, and his accounts of ''genocidal massacres'' in Rwanda, Burundi, Nigeria and the Sudan are vivid and apparently authentic. For other lands, however, he has relied heavily on secondary sources. The analytical portions are more impressive from a psychological than a legal standpoint, especially in discussing the social and political situations that stimulate genocidal behavior (colonization, decolonization, change of governments, racial and religious ''pluralism'') and the mental processes by which perpetrators of genocide justify their conduct to themselves. These include ''shaping a dehumanized image of the victims'' or analogizing the victimized group to a cancer or bacillus that must be eradicated in order to preserve the dominant social group.

The Genocide Convention declares that genocide is ''a crime against international law,'' and that declaration raises legal problems, in the handling of which Professor Kuper is insecure. For example, he labels as genocide ''the atomic bombing of the Japanese cities of Hiroshima and Nagasaki by the U.S.A. and ... the pattern bombing by the Allies of such cities as Hamburg and Dresden.'' These were wartime actions, and (with the possible exception of Nagasaki and Dresden, where the bombings were arguably unnecessary for any military purpose) it is difficult to perceive the bombings of World War II as violations of the laws of war. But however that question might be resolved, they were certainly not ''genocidal'' within the meaning of the Convention, which limits genocide to ''acts committed with intent to destroy ... a national, ethnical, racial or religious group, as such.'' Berlin, London and Toyko were not bombed because their inhabitants were German, English or Japanese, but because they were enemy strongholds. Accordingly, the killing ceased when the war ended and there was no longer any enemy.

Professor Kuper blames the many genocidal massacres of recent years on the failure of the United Nations and the major powers to enforce the Convention, and he has little difficulty in showing the record to be shabby. But his comparative inattention to genocide as a newly established crime renders his work susceptible to the criticism that it is ''Hamlet'' without Hamlet.

Considering both the course of events since World War II and our own Government's continuing reluctance to ratify the Convention, one would hope that a new book on genocide would include an assessment of the concept's value. Was it a good move to take Lemkin's label and use it as the basis for defining an international crime? How workable are the operational provisions of the Convention? What steps have been taken, by the United Nations or by signatories of the Convention, to give its provisions practical effect?

In such an analysis, it should be noted that, as far as wartime actions against enemy nationals are concerned, the Genocide Convention added virtually nothing to what was already covered (and had been since the Hague Convention of 1899) by the internationally accepted laws of land warfare, which require an occupying power to respect ''family honors and rights, individual lives and private property, as well as religious convictions and liberty'' of the enemy nationals. But the laws of war do not cover, in time of either war or peace, a government's actions against its own nationals (such as Nazi Germany's persecution of German Jews). And at the Nuremberg war crimes trials, the tribunals rebuffed several efforts by the prosecution to bring such ''domestic'' atrocities within the scope of international law as ''crimes against humanity.''

It is primarily in the area of ''domestic'' atrocities that the Genocide Convention seeks to expand pre-existing international penal law. Furthermore, under its Article V, signatory nations are required to enact ''the necessary legislation to give effect to'' the Convention. How many of the 84 signatories have done so, and in what form? In any of those countries, have there been any actual criminal proceedings to enforce the Convention? This book does not tell us.

Article VI of the Convention provides that trials for its violation may be conducted ''by a competent tribunal of the State in the territory of which the act was committed, or any such international penal tribunal as may have jurisdiction.'' Apart from the European Human Rights Court at Strasbourg (with severely limited jurisdiction), no such international tribunal has been established, and courts of the states where such acts are committed are hardly likely to conduct such proceedings against the state itself or state officials.

These factors account in part for the Convention's apparent ineffectiveness, but the dismal record does not, of course, prove that things would have been no worse had the Convention not been adopted. Symbolism has its force, and American failure to ratify the Convention has certainly detracted from its influence.