Installing Debian GNU/Linux on a p630 LPAR Manuel Lässer Ralf Strauss Florian M. Weps Telekurs Services Ltd. May 20, 2003 The IBM pSeries Model 630 (7028-6C4) is an entry-level server with a 1 to 4-way processor and a 64-bit system architecture. It can be configured into logical partitions (LPAR), which can be booted into AIX or Linux. IBM recommends using Red Hat or SuSE Linux, both of which come in 64-bit versions[1]. We installed Debian GNU/Linux instead. Debian[2] is a Linux distribution developed and maintained by volunteers. It has been released on 11 architectures, among them powerpc, the 32-bit subset of the Power architecture. There is no 64-bit Power distribution available from Debian, but the 32-bit Distribution can be run on top of a 64-bit kernel. 0. Preparations --------------- Some preparations were necessary for cross-compiling the ppc64 kernel: Cross-build versions of the GNU toolchain and compiler had to be obtained. We used an Apple iBook running Debian GNU/Linux "unstable" as a cross-build host, because in this way we only needed one cross-compiler: to build a ppc64 kernel, both a 64-bit compiler (for the kernel code) and a 32-bit compiler (for the bootstrap code) are needed. The source code of the ppc64 kernel is provided on the penguinppc64 site[3], along with some build instructions. The official kernel 2.4.20 source code from kernel.org had to be patched with three patches to yield 2.4.21-pre4-2. One particularly useful resource were the instructions for network-installing Debian on a 44P-170, by Rolf Brudeseth[4]. Some packages had to be present in particular versions for the build to work: flex 2.5.4a-24 from Debian 3.0r1 "woody" automake1.4 from Debian "unstable" The issue with the version of flex in "unstable" was that it broke the toolchain build thoroughly, so we used the version from the "woody" archive (his problem will be fixed eventually). The fact that we used automake1.4 instead of a later version was based on a recommendation in toolchain-source. We used the "root.img" initrd image from the Debian 3.0r1 "woody" archive for powerpc. It contains the installation system. 1. Building the Cross-Binutils ------------------------------ The "toolchain-source" package and all utilities necessary for building Debian packages were installed on the iBook: # apt-get install toolchain-source dpkg-dev file gcc libc6-dev make \ patch perl autoconf dh-make devscripts fakeroot gnupg automake1.4 \ debconf-utils debhelper dpkg-cross html2text po-debconf bzip2 \ autoconf2.13 zlib1g-dev The "flex" package was taken from the "stable" archive and installed: # wget http://ftp.de.debian.org/debian/pool/main/f/flex/flex_2.5.4a-24_powerpc.deb # dpkg -i flex_2.5.4a-24_powerpc.deb There is an online manual for building the cross-toolchain on W. Borgert's page[5]. Since we did not want to work as "root", we added the account used for compiling to the group "src" in order to gain write access to /usr/src. $ cd /usr/src $ EMAIL='fmw@debian.org' tpkg-make powerpc64-linux We changed some setting of the package build process: for compiling a kernel, no C library is necessary, so we were spared some of the complications of a cross-build environment. Furthermore, only statically linked binaries would be generated. And finally, seeing as how the manpage build process was being troublesome, we removed them from the build. The following changes were made to /usr/src/binutils-powerpc64-linux-2.13.90.0.18/debian/rules 14,15c14,18 < --disable-static \ < --target=$(TARGET) --- > --disable-shared \ > --target=$(TARGET) \ > --enable-targets=powerpc64-linux \ > --disable-nls > 71,72c74,75 < mv debian/$(PACKAGE)/usr/man debian/$(PACKAGE)/usr/share/man < cp debian/gasp.1 debian/$(PACKAGE)/usr/share/man/man1/powerpc64-linux-linux-gasp.1 --- > #mv debian/$(PACKAGE)/usr/man debian/$(PACKAGE)/usr/share/man > #cp debian/gasp.1 debian/$(PACKAGE)/usr/share/man/man1/powerpc64-linux-linux-gasp.1 Binutils were build with the following commans (-us and -uc prevent signing of the packages) $ cd /usr/src/binutils-powerpc64-linux-2.13.90.0.18 $ debuild -us -uc $ cd .. $ sudo dpkg -i binutils-powerpc64-linux_2.13.90.0.18-1_powerpc.deb At this point, powerpc64-linux- versions of utilities like ar, as, ld, and nm were installed. 2. Building the Cross-Compiler ------------------------------ The "toolchain-source" package contains code for the compiler as well, and building it is analogous to building binutils. We changed some settings in the "debian/rules" file. 16,19c16 < --infodir=/usr/share/info \ < --mandir=/usr/share/man \ < --with-gxx-include-dir=/usr/$(TARGET)/include/g++ \ < --enable-shared \ --- > --disable-shared \ 22,25c19,21 < --with-system-zlib \ < --enable-long-long \ < --enable-nls \ < --without-included-gettext \ --- > --disable-nls \ > --disable-threads \ > --enable-__cxa_atexit \ The choice of settings was suggested by the penguinppc64 documentation[3]. $ cd /usr/src/gcc-powerpc64-linux-3.2.3.cvs20030221 $ debuild -us -uc $ cd .. $ sudo dpkg -i gcc-powerpc64-linux_3.2.3.cvs20030221-1_powerpc.deb The cross-build environment for the kernel was complete! However, had our build host not been powerpc, it would have been necessary to build "powerpc" binutils and a compiler in addition to the powerpc64 versions just completed, as the kernel-bootstrap is 32-bit powerpc code. 3. Building the Kernel ====================== Kernel sources were obtained form online archives: $ wget -c http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.20.tar.gz $ wget ftp://ftp.kernel.org/pub/linux/kernel/v2.4/testing/patch-2.4.21-pre4.gz $ wget ftp://ftp.linuxppc64.org/pub/people/engebret/linuxppc64-2.4.21-pre4.patch.gz $ wget ftp://ftp.linuxppc64.org/pub/people/engebret/linuxppc64-2.4.21-pre4-2.patch.gz We used the http url for the kernel.org code because our proxy kept dropping the ftp transfer, and did not support partial downloads for ftp. By applying the patches, the official 2.4.20 kernel sources were transformed into ppc64 2.4.21-pre4-2 sources: $ tar xzf linux-2.4.20.tar.gz $ cd linux-2.4.20 $ zcat ../patch-2.4.21-pre4.gz | patch -p1 $ zcat ../linuxppc64-2.4.21-pre4.patch.gz | patch -p1 $ zcat ../linuxppc64-2.4.21-pre4-2.patch.gz | patch -p1 $ cd .. $ mv linux-2.4.20 linuxppc64-2.4.21-pre4-2 The next steps were: to modify the top-level makefile for cross-compilation, to position the ramdisk image so it would be picked up by the build process, to remove the debuggers from the kernel configuration since they did not take kindly to being cross-compiled, and to modify the makefile in arch/ppc64 because it set a compiler parameter ("traceback=full") not understood by our cross tools. Changes to the top-level Makefile: in theory, it should have been sufficient to set environment variables ARCH=ppc64 and CROSS_COMPILE=powerpc64-linux-, but this approach was not successful. 8c8,9 < ARCH := $(shell uname -m | sed -e s/i.86/i386/ -e s/sun4u/sparc64/ -e s/arm.*/arm/ -e s/sa110/arm/) --- > #ARCH := $(shell uname -m | sed -e s/i.86/i386/ -e s/sun4u/sparc64/ -e s/arm.*/arm/ -e s/sa110/arm/) > ARCH := ppc64 22c23 < CROSS_COMPILE = --- > CROSS_COMPILE = powerpc64-linux- Changes to the arch/ppc64 makefile: 25,26c25,26 < -Wno-uninitialized -mminimal-toc -fno-builtin \ < -mtraceback=full --- > -Wno-uninitialized -mminimal-toc -fno-builtin > # -mtraceback=full Moving the Debian installer ramdisk image into position: $ cd arch/ppc64/boot $ wget http://ftp.debian.org/debian/dists/woody/main/disks-powerpc/current/chrp/images-1.44/root.bin $ mv root.bin ramdisk.image.gz Configuring the kernel: there is a default kernel configuration for pSeries in arch/ppc64/configs/pSeries_defconfig. However, the kernel-debuggers are activated in that configuration, leading to errors when cross-compiling. So we removed all debugger options from the configuration, taking care to preserve only "magic sysrq key", because the hvc_console driver unconditionally includes it. (This may be a bug: other driver modules use preprocessor conditionals to protect against inclusion of "linux/sysrq.h" unless CONFIG_MAGIC_SYSRQ is selected). $ cd /usr/src/linuxppc64-2.4.21-pre4-2 $ cp arch/ppc64/configs/pSeries_defconfig .config $ make config Here are the differences between the supplied configuration and ours: 176d175 < # CONFIG_IDE_TASKFILE_IO is not set 231a231 > # CONFIG_BLK_DEV_ATARAID_SII is not set 545a546,550 > # CONFIG_IPMI_HANDLER is not set > # CONFIG_IPMI_PANIC_EVENT is not set > # CONFIG_IPMI_DEVICE_INTERFACE is not set > # CONFIG_IPMI_KCS is not set > # CONFIG_IPMI_WATCHDOG is not set 725,731c730,732 < CONFIG_KDB=y < # CONFIG_KDB_OFF is not set < CONFIG_KALLSYMS=y < CONFIG_PPCDBG=y < CONFIG_DUMP=y < CONFIG_DUMP_COMPRESS_RLE=y < CONFIG_DUMP_COMPRESS_GZIP=y --- > # CONFIG_KDB is not set > # CONFIG_PPCDBG is not set > # CONFIG_DUMP is not set Building the kernel: $ make clean dep $ make zImage.initrd 4. Preparations for Net-Booting the LPAR ---------------------------------------- The LPAR did not have a CD-ROM drive. We booted and installed over the net, using tftp and nfs. The iBook was used as a boot server. We installed tftp and dhcp software on the iBook for this purpose: $ sudo apt-get install tftpd dhcp The configuration of tftpd is entirely in the command line contained in /etc/inetd.conf: tftp dgram udp wait nobody /usr/sbin/tcpd /usr/sbin/in.tftpd /tftpboot Since the tcp-rwapper tcpd was used, we had to allow access from the client machine's ip address in /etc/hosts.allow The dhcp configuration file is /etc/dhcpd.conf; rudimentary stanzas for the subnet and client were added: subnet 10.0.26.0 netmask 255.255.255.0 { } host p630lpar3 { hardware ethernet 00:02:55:af:c0:63; filename "zImage.initrd"; fixed-address 10.0.26.75; } We patched the iBook into the subnet with the target machine and copied the combined kernel-and-ramdisk image zImage.initrd from arch/ppc64/boot into /tftpboot. 5. Booting the LPAR ------------------- In the HSM workstation's GUI, we activated the LPAR and opened a console window. By pressing "8", we interrupted the boot sequence and got the Open Firmware Prompt. (It would have been possible to boot the LPAR from the SMS menus, but we prefer the OF prompt). Due to the way the LPAR was configures, we had to boot from the second ethernet interface. We set up a devalias for this purpose: 0 > devalias ibm,sp /vdevice/IBM,sp@4000 disk /pci@400000000211/pci@2,6/scsi@1/sd@8,0 scsi /pci@400000000211/pci@2,6/scsi@1 screen /vdevice/vty@0 net /pci@400000000211/pci@2/ethernet@1 network /pci@400000000211/pci@2/ /nvram@3fdbd800000 ok 0 > devalias net2 /pci@400000000211/pci@2,2/ethernet@ 0 > boot net2:10.0.26.74,,10.0.26.75 The first address in the boot-command is the iBook, the second one the target machine. So the iBook started sercing the kernel image, and the Debian installation system was activated on the LPAR. During out first trials we ran into trouble because the kernel could not find a console: we had not compiled in the hvc_console driver due to its dependencies on the magic sysrq key option, which was in the same general configuration section as the kernel debuggers. A situation is described in [4], where the tftp transfer gets interrupted. The solution appears to lie in manually setting the arp table entry for the client on the tftp server: # arp -s 10.0.26.75 00:02:55:AF:C0:63 We did this proactively, and never experienced a tftp transfer interrupt. 6. Debian Installation ---------------------- At this point we were following the instructions in [4] fairly closely. First, we edited the disklabel ("Partition a Harddisk"): sda1 Type 41: PPC PReP Boot 8.39M sda2 Type 82: Linux swap 1024.46M sda3 Type 83: Linux ext2 35368.47M As detailed in [4], we set the "bootable" flag on sda1. The following steps were "Initialize and Activate a swap partition", and "initialize a Linux Partition". We chose ext2 as filesystem. Next, we configured hostname and network. The base system was installed via nfs: since the LPAR had no CD-ROM, we mounted the Debina CD on a remote machine and exported it via NFS. We did not experience the difficulties described in [4]. Then we had to leave the Debian installation menu, because we had to set up custom kernel, boot-loader and console devices in the freshly installed base system. "Execute a Shell" got us a shell prompt. The new filesystem was mounted on /target. # chroot /target # mount /proc First, we got the kernel image off the iBook and into the new /boot directory. We used ftp to get the vmlinux file from the kernel build directory. For some more packages, we used apt. First, we had to adjust /etc/apt/sources.list. Since the target system had no access to our internet proxy, we used apt-proxy on a workstation to create a local mirror. Here is out /etc/apt/sources.list: deb http://workstation:9999/main unstable main deb http://workstation:9999/non-US unstable/non-US main In /etc/apt/apt.conf, we set up a large timeout value to allow for proxy latency: Acquire::http::Timeout "600"; We installed yaboot and ssh. Calling yabootconf (without parameters) installed the boot loader into sda1 As we discovered, we had to create a device special node in /dev for the pSeries Hypervisor console. # cd /dev # mknod hvc0 c 229 0 # chown root:tty hvc0 # chmod 600 hvc0 We had to modify /etc/inittab so the post-install routine would find the console. 1:2345:respawn:/usr/sbin/termwrap /usr/sbin/base-config /dev/hvc0 2>&1 Now everything was ready for the final reboot. We exited the chroot environment and stopped the LPAR. # umount /dev # exit # umount /target # halt 7. Reboot --------- Again, we activated the LPAR from the HSM workstation, and pressed "8" to get the Open Firmware prompt. 0 > boot disk We saw the yaboot pause for a moment, and then the Debian system booted into the post-install routine. We lost the console on our first few tries, because base-config was started on some other tty device. Also, after base-config exits, the console is lost again. This is the reason we installed ssh, so we would have a backdoor into the system when we lost the console. Editing the inittab to put a getty on /dev/hvc0 solved the problem. A. Benchmarks ------------- Debian GNU/Linux contains several benchmark packages. Since Debian is essentially identical on all supported platforms, one nice thing to do is install it on reputedly equivalent systems from different vendors and run benchmarks and load tests. bonnie++ -- disk i/o contest -- system responsiveness lmbench -- general tiobench -- threaded i/o siege -- httpd B. References ------------- [1] http://www-1.ibm.com/servers/eserver/pseries/linux/ [2] http://www.debian.org/ [3] http://penguinppc64.org/ [4] http://lists.debian.org/debian-powerpc/2002/debian-powerpc-200207/msg00858.html [5] http://people.debian.org/~debacle/cross.html [6] http://www.debian.org/releases/woody/powerpc/release-notes/ch-upgrading.en.html#s-dselectupgrade