The emperor and empress greeted Obama with handshakes at the entrance of the Imperial Residence in Tokyo before the three had lunch with only an interpreter attending, according to the Imperial Household Agency.
The imperial couple normally host top foreign leaders at the main building of the Imperial Palace. But their residence was chosen for Saturday's lunch with Obama because of tight schedules for both parties, the agency said.
The same arrangement in hosting an incumbent U.S. president was last made in November 1998 when Bill Clinton visited Japan.
In a speech he delivered in Tokyo earlier in the day, Obama said it would be an "extraordinary honor" to meet with the emperor and empress in a year that marks the emperor's 20th anniversary of his ascension to the Chrysanthemum Throne.
Emperor Akihito was enthroned Jan. 7, 1989, following the death of his father Emperor Hirohito, posthumously known as Emperor Showa, and ceremonies commemorating the anniversary were held Thursday.
Obama arrived in Tokyo on Friday afternoon and held talks with Prime Minister Yukio Hatoyama later in the day. He left for Singapore on Saturday afternoon to attend a summit meeting of the Asia-Pacific Economic Cooperation forum after delivering a speech on U.S. policy toward Asia prior to his meeting with the emperor and empress.