By Jeffrey Carl
Boardwatch Magazine was the place to go for Internet Service Provider industry news, opinions and gossip for much of the 1990s. It was founded by the iconoclastic and opinionated Jack Rickard in the commercial Internet’s early days, and by the time I joined it had a niche following but an influential among ISPs, particularly for its annual ranking of Tier 1 ISPs and through the ISPcon tradeshow. Writing and speaking for Boardwatch was one of my fondest memories of the first dot-com age.
Even though many server admins prefer BSD Unix, there’s no denying that Linux is “where it’s at” for third-party software development. So, what’s a BSD admin to do?
In the bygone misty past of Unix (in the era historians call “the early ‘90s”), acceptance and market share were hampered by the diverging standards that forced application developers to write for only one version of Unix. This (as the youngsters today say) “really sucked,” and resulted in the Unix wars that left many syadmins hospitalized with “flame” wounds from heated Usenet exchanges. However, the tide has fortunately turned in favor of compatibility among the many *nixes.
To commercial software vendors, market share is everything. As such, if *BSD forced Linux vendors to develop for their OSes, it would be like your average Unix sysadmin saying, “If Cindy Crawford wants to go out, she can call me.” Fortunately, Jordan Hubbard of FreeBSD has indicated a prudent recognition that Linux’s API is becoming the default for third party developers (Indpendent Software Vendors, or ISVs) to develop for free *nixes. Therefore, it’s better for BSD users to accept the Linux ABI (Application Binary Interface) rather than to force ISVs to choose between Linux and BSD. This, in my opinion, is a very pragmatic and wise attitude.
FreeBSD, NetBSD and OpenBSD all include options to allow Linux binaries to run on their systems (at least for the Intel x86 architecture; on other architectures, your mileage may vary). For OpenBSD and NetBSD, you can check your system documentation or see their respective websites (www.openbsd.org and www.netbsd.org); for the purpose of this article, we’ll look at Linux binary compatibility on FreeBSD for x86. NOTE: Much of the information on the inner workings of Linux compatibility came from information graciously provided by Terry Lambert ([email protected]).
Under FreeBSD, Linux binary compatibility is managed by creating a “shadow” Linux file system with the requisite programs and libraries, and re-routing the Linux program’s system calls into this file system. These libraries then deal with a Linux kernel embedded into the FreeBSD kernel.
Setting Up Linux Compatibility
The development of an “executable class” loader for FreeBSD has been under development since 1994 (although when the project was started, Linux wasn’t one of the primary target OSes). There are three essential items that enable this functionality.
The first is a KLD (“Kernel LoaDable”) object called linux.ko, which is essentially a Linux kernel that can be loaded dynamically into the FreeBSD kernel. As a KLD, it can be loaded or unloaded without rebooting, using the kldload and kldunload commands. To check and see whether linux.ko is loaded properly on your system, use the kldstatcommand:
schnell# kldstat
Id Refs Address Size Name
1 4 0xc0100000 1d9a60 kernel
2 1 0xc1038000 3000 daemon_saver.ko
3 1 0xc1822000 4d000 nfs.ko
4 1 0xc1020000 10000 linux.ko
The second necessary item is the set of basic Linux libraries (pre-existing code that developers can link their programs to, so they don’t have to write every function from scratch) and binaries (like bash, cp, rm, etc.). These can be installed via the ports collection from /usr/ports/emulators/linux_base, which is based on a minimal Red Hat installation. As of this writing in late September, for the i386 architecture, the installed libraries matched a Red Hat 6.1-1 release including glibc 2.1.2-11, libc 5.3.12-31, glib 1.2.5-1, ld.so 1.9.5-11, libstdc++-2.9.0-24, gdbm-1.8.0-2 and XFree86 libs 3.3.5-3.
To be able to use a wider range of Linux apps, you may also wish to install the Linux development tools, found in the ports collection at /usr/ports/devel/linux_devtools. Note that you’ll want to do this after installing linux_base, since linux_devtools requires rpm (the Red Hat Package Manager application, also available as a FreeBSD app through the ports collection) and /compat/linux/etc/redhat-release to be installed. This currently includes (also for i386) kernel-headers-2.2.12-20, glibc-devel-2.1.2-11, make-3.77-6, cpp-1.1.2-24, egcs (plus egcs-c++ and egcs-g77)-1.1.2-24, gdb-4.18-4, and XFree86-devel-3.3.5-3.
Also, any necessary binaries/libraries can be set up manually by installing them into their original paths under the FreeBSD directory /compat/linux (e.g., install the Linux /usr/X11R6/lib/libX11.so.6.1 library onto FreeBSD as /compat/linux/usr/X11R6/lib/ libX11.so.6.1). This process is also necessary for installing libraries that aren’t part of the default ports collection sets.
The third item is “branding” Linux ELF (“Executable and Linking Format”) binaries. Although the GNU toolchain now brands Linux ELF binaries with the OS name (so newer Linux binaries should work without branding), some older binaries may not include this information (although ELF has replaced a.out as the primary binary format since Linux kernel 2.x and FreeBSD 3.0). To do this, use the brandelf command (which sounds like a move from “Dungeons and Dragons” to poke elves with hot sticks) with the “-t” (type) flag like:
schnell# brandelf -t Linux /usr/local/games/doom
How Linux Compatibility Works
Originally, Unix supported only one “binary loader.” When you ran a program, it examined the first few bytes of the file to see if it was a recognized binary type. If the file type that was attempted to being loaded and run wasn’t the singular binary type that it recognized, it would pass the file in question to the basic shell interpreter (/bin/sh) and try to run the program with that. Failing that, it would spit a nasty message back to you about how you couldn’t run whatever it was that you were trying to run.
Nowadays, FreeBSD supports a number of loaders, which include an ELF loader (by the way, Linux binaries can be stored on either a FreeBSD UFS or Linux ext2 file system, assuming you have mounted the ext2 file system the app is located on using the mount -t ext2fs command). If the “brand” is seen in the ELF file as “Linux,” it replaces a pointer in the “proc” structure of the binary to point automatically to /compat/linux/, and the binary is executed using the Linux libraries and binaries, instead of the FreeBSD ones. If it isn’t a FreeBSD or Linux binary, and the list of loaders (including those examining the first line for “#!” lines like Perl, Python or shell scripts) is exhausted without recognizing a “type,” then it is attempted to be executed as a /bin/sh script. If this attempt to run is unsuccessful, it is spit back out as an “unrecognized” program.
Linux and BSD Binaries: Both “Native?”
If a binary or library is required by a Linux program that is not included under the Linux compatibility “area,” the FreeBSD tree is searched. Thus, a Linux application that is expected to run with the Linux version of /compat/linux/usr/lib/libmenu.so – if it doesn’t find it – is checked to see whether it can run under the FreeBSD /usr/lib/libmenu.so. Only if a compatible library is not found in either the Linux-compatible “re-rooted (/compat/linux/whatever)” filesystem or the regular FreeBSD libraries is an executable rejected.
Because of this construction, it isn’t really accurate to say that Linux is “emulated” under FreeBSD, especially in the sense that, say, platforms are emulated under the MAME (www.mame.net) emulator, since it doesn’t involve a program which impersonates another platform on a binary level and translates all of its native system calls into local system calls. Linux binaries are simply re-routed to a different set of libraries, which interface natively with a (Linux kernel) module plugged into the FreeBSD kernel. Rather than “emulation,” this is largely just the implementation of a new ABI into FreeBSD.
Under the current implementation of binary compatibility, programs calling FreeBSD’s glue() functions are statically linked to FreeBSD’s libraries, and programs calling the Linux glue() function calls are statically linked to its own libraries, so their executions are dependent on completely different code libraries; however, the importance of glue()is on the wane. In the future, this might change, which would seriously open the question as to whether Linux apps under FreeBSD are any more “native” than apps written specifically for FreeBSD are.
Running Linux Binaries on BSD
Currently, the major problem with running Linux binaries on FreeBSD is similar to the major problem with using new apps on Linux: discovering which libraries they require and downloading and installing them. Unfortunately for BSD users, the only certain way to do this right now is to have access to a Linux system which already has the app in question installed, and run the ldd command on the application to list the libraries needed. If you don’t have access to a Linux system that already has the program installed, your next best bet is to check info on the program’s website, or search for it at Freshmeat (www.freshmeat.net).
Under current FreeBSD Linux binary emulation, most Linux binaries should work without problem. However, due to operating system differences, it won’t support those that heavily use the Linux /proc filesystem (which operates differently from the FreeBSD /proc), ones that make i386-specific calls like enabling virtual 8086 mode, or ones that use a significant amount of Linux-specific assembly code. While some people have reported that some Linux binaries run faster in this BSD environment than they do natively in Linux, this is a highly subjective claim, and one that should be taken with a grain of salt (at least until some real standardized testing is done with it). While Linux binary emulation on FreeBSD shouldn’t be considered as a major security risk, it is worth noting that a buffer overflow bug was discovered in August 2000 (www.securityfocus.com/frames/?content=/vdb/bottom.html%3Fvid%3D1628).
Popular Uses for Linux Compatibility
The most popular binaries for using Linux compatibility are the ISV applications that have been developed for Linux from popular commercial/semi-commercial vendors. These include programs like Sun’s StarOffice, Wolfram’s Mathematica, Oracle 8, and games like Bungie’s Myth II and id Software’s Quake III (there are already tutorials on installing Oracle 8.0.5 and Mathematica 4.x included in the FreeBSD Handbook; see below for more).
Other popular installations include the Netscape Navigator 4.6-7.x browser. Particular advantages to installing the Linux version (even though there is a FreeBSD version already) are that the Linux version under FreeBSD compatibility is reportedly more stable than the “native” FreeBSD version. Also, binaries for popular plug-ins (Flash 4, Real Player 7, etc.) are available, since “native” FreeBSD versions aren’t. Info on installing VMware for Linux on FreeBSD can be found at www.mindspring.com/~vsilyaev/vmware/.
Getting Help with Linux Compatibility
The primary reference on Linux binary emulation on FreeBSD is the “handbook” page at www.freebsd.org/handbook/linuxemu.html, including specific pages on Oracle (www.freebsd.org/handbook/linuxemu-oracle.html) and Mathematica (www.freebsd.org/handbook/linuxemu-mathematica.html). To keep up-to-date on Linux emulation under FreeBSD, subscribe to the mailing list freebsd-emulation (send e-mail to [email protected] with the text [in the body] subscribe freebsd-emulation).
Additional information on Linux compatibility (including instructions for older versions of FreeBSD), see www.defcon1.org/html/Linux_mode/linux_mode.html. For a list of Linux applications that are known to run on FreeBSD under Linux binary compatibility, see www.freebsd.org/ports/linux.html. For an example of setting up Word Perfect 8 in Linux compatibility – as well as an account of running Windows 2000 on top of VMware on top of Linux on top of FreeBSD, read the excellent article at BSD Today (www.bsdtoday.com/2000/August/Features252.html). You can find some (mildly outdated but still useful) instructions on installing StarOffice on FreeBSD at www.stat.duke.edu/~sto/StarOffice51a/install.html. And, for fun, there’s info on setting up an Unreal Tournament server on FreeBSD via Linux compatibility at www.doctorschwa.com/ut/freebsd_server.html.