On reaching Yemen, Ali realized that the people of Yemen were very much under the influence of the Christian priests and Jewish Rabbis, and unless such priests and Rabbis were effectively tackled, there was little likelihood of any success with the common men. The Christian priests and Jewish Rabbis were proud of their religious knowledge. They challenged Ali to a debate. Ali was used to fighting duels with his adversaries in the case of battles, now he had to fight an intellectual duel. The first challenge came from a Christian priest renowned for his learning and piety. A public debate was held. The Christian priest advanced arguments in favor of the perfection of Christianity, and of Jesus Christ being the son of God. It was observed that in such circumstances there could be no question of another faith. Ali smashed these arguments one by one. He quoted from the Christian scriptures wherein Jesus Christ had himself spoken of the advent of a prophet after him. Ali took pains to explain that Islam acknowledged Jesus Christ as a prophet, but did not regard him as a son of God. He maintained that Islam was a perfected form of all previous religions. The debate lasted for several days and at last Ka'ab admitted the superiority of Islam and was converted to Islam. With his conversion many Christians were also converted to Islam.

The Jewish Rabbis felt concerned at the success of Islam. They challenged Ali to a public debate. This debate lasted for several days, and ended in the victory for Islam. Some of the Jewish Rabbis accepted Islam, and following them many Jews accepted Islam. Thereafter Ali spread his men throughout the length and breadth of Yemen to carry the message of Islam to the people. When the people came to know that their priests and rabbis had accepted Islam, they lost the will to resist Islam and accepted the new faith willingly. Within a few months most of the people in Yemen were converted to Islam. These people came to look to Ali as a great hero.