make-redir (1) - Linux Manuals
make-redir: run "make" but automatically redirect its results to DESTDIR
NAME
make-redir - run "make" but automatically redirect its results to DESTDIRSYNOPSIS
make-redir [MAKE_ARGUMENT*]...[install]
- * DESTDIR=directory and -k/--keep-going are especially common.
DESCRIPTION
The make-redir program runs "make" in a way that redirects file installations to descend from the $DESTDIR directory, even if the provided makefile doesn't natively support $DESTDIR. In most cases, simply replace "make install" with "make-redir DESTDIR=directory install" (adding options as necessary).
The make-redir program attempts to solve a common problem for packagers of software and for users who directly install source code: Inadequate DESTDIR support. Software is typically installed on Unix/Linux from a source code package by typing "make install". Historically, this script would write directly to privileged directories to perform the installation (e.g., /usr/bin), or at least to a directory that merged all installed files (e.g., $HOME/bin). However, packagers of software using most modern packaging systems (such as .rpm and .deb) instead require that files be written to some intermediate directory, even though they will be run from a different filesystem location. Similarly, users who directly install source code often use programs such as GNU stow, encap, or toast. These programs install each package in its own separate directory, but work best if the installed programs are invoked through a different directory (which merges the contents of multiple packages, simplifying the PATH and supporting plug-ins). Thus, for both many packagers of software and for many users who directly install source code, a program should be installed in one directory, yet actually be run from a different directory. This is easy to do if the installation script supports the "DESTDIR convention"; simply run "make DESTDIR=something install", which would install the files descending from DESTDIR, even though the files are configured to run without the DESTDIR prefix. Unfortunately, many source packages don't support the DESTDIR convention.
The auto-destdir tools "make-redir" and "run-redir" will automatically redirect file reads/writes to follow the DESTDIR convention, without requiring any changes to the build files (e.g., a makefile). Attempts to write to directories that are normally root-privileged (such as /usr/bin) will instead be redirected to descend from a designated destination directory. This destination directory is specified by the environment variable REDIR_DESTDIR or DESTDIR, with REDIR_DESTDIR taking precedence.
You should use make-redir (not run-redir) if you are redirecting "make", because make-redir performs additional helpful actions. If you pass DESTDIR=something as a parameter, it automatically sets the DESTDIR environment variable. The make-redir command also sets the overrides INSTALL="install -p" CP="cp -p" MKDIR="mkdir" MKDIR_P="mkdir -p" mkdir_p="mkdir -p" LN="ln", so that makefiles with absolute pathnames of common commands will work correctly and preserve timestamps. (Add overrides on the command line if these are not acceptable.) By default make-redir also sets special SHELL and MAKESHELL values, so that even invocations with a full path name (like /bin/cp) will be redirected if they begin a line in a makefile.
Internally, make-redir runs "make" using run-redir; run-redir does the actual file redirection. Use run-redir if you need to redirect file writes without first running "make".
The auto-destdir tools are designed to "just work" given the limitations described below (in particular, only "wrapped" commands should read/write files that must be redirected). Filenames will not be redirected twice, so it's safe to use run-redir and make-redir even with makefiles that do support DESTDIR. If a write is redirected, all parent directories are first automatically created if they exist in the unredirected filesystem. This means that an attempt to "cp" some file to /usr/bin/xyz will create the directory "$DESTDIR/usr/bin" if /usr/bin already exists, then copy the file to "$DESTDIR/usr/bin/xyz". If a file is being read, a file written inside $DESTDIR will have precedence over a file that isn't; this nicely simulates the overwriting of an old file with a new one. Since make-redir and run-redir set the PATH, they will work even if there are many levels of indirection, and even if they use tools other than make (as long as they end up eventually invoking the usual commands such as cp and install).
If the "make install" works except that it uses a few "unwrapped" commands, consider using the "-k" ("--keep-going") option, so that make-redir will keep going instead of failing on an error. Then, re-perform the actions that would have failed. Alternatively, you may be able to perform one or two actions "by hand" first, and then invoke make-redir to do the rest.
The programs make-redir and run-redir can be run as root, but in most cases, you should not run them as root. The installation program itself may require root to run (e.g., to set ownership), even if files are written to DESTDIR. If you just want to record permissions (e.g., chown, chmod), consider also using "fakeroot" to record the intended permissions instead of running as root. When creating an rpm package, the permission-setting is typically irrelevant, since in most cases it is the rpm spec file that will ultimately set the permissions. If the permission-setting is irrelevant, you may want to run make-redir as non-root and pass it the "-k" ("--keep-going") option so that make will keep going instead of failing on an error.
A file/directory is not redirected if its full pathname begins as a typical user directory (specifically /home/ or /Users/) or temporary directory (/tmp/, /usr/tmp/, /var/tmp/, or /opt/tmp/). A file/directory is redirected if its full pathname begins with a directory name that is normally privileged: /usr/, /etc/, /boot/, /bin*/, /dev/, /lib*/, /opt/, /sbin*/, /selinux/, /srv/, /sys/, and /var/. If neither case applies, it is not redirected.
You can override some values, though few will need to do so. By default, make-redir invokes "make"; this can be changed by setting the environment variable REALMAKE to whatever "make" program should be invoked instead. By default, make-redir sets a special SHELL value (so it can override commands that give full pathnames); this special value does some processing and then invokes /bin/bash. Set the environment variable REALMAKESHELL to the name of the shell to use after doing this filtering if you do not want it to use /bin/bash. You can disable this filtering entirely by setting the make variables SHELL and MAKESHELL on the command line to the name of the shell program to use (e.g., make-redir SHELL=/bin/sh MAKESHELL=/bin/sh); if make can be called recursively, also set the environment variables with the same name. If you do not want any makefile variable overrides, set the environment variable DESTDIR and use "run-redir make ..." instead.
EXAMPLES
The following configures, builds, and then "installs" a program. The "installed" files will descend from /tmp/stuff, e.g., directory /tmp/stuff/usr/bin will contain files that were intended to be installed in /usr/bin:
-
./configure --prefix=/usr make make-redir DESTDIR=/tmp/stuff install
For example, if "cp myprogram
You can use make-redir inside an RPM .spec file.
Be sure to "BuildRequires" auto-destdir, and set the DESTDIR to be the
RPM build root, as follows in the spec file:
Some RPM spec authors may prefer to use %{buildroot} instead of $RPM_BUILD_ROOT.
The auto-destdir tools (including run-redir and make-redir)
work by modifying the PATH environment variable
and providing a few "wrappers" that override
typical installation commands like "cp" and "install".
These commands with wrappers, called "wrapped commands",
use the values of $REDIR_DESTDIR or $DESTDIR
(with REDIR_DESTDIR taking precedence),
and will redirect reads and writes of files under certain conditions.
The "redirection" comes by simply changing the name of the file source or
destination, and then calling the "real" program being wrapped.
The internal function redir_name takes the original filename,
and how it will be used (e.g., for reading or writing),
and determines if it should be renamed (and if so, to what).
In addition, make-redir redirects all shell commands (using SHELL
and a special wrapper for shell commands);
any command that begins with the prefix /usr/ or /usr/bin/, followed by a
wrapped command's name, will have that prefix removed.
This may seem to be an odd implementation approach, but it has many advantages.
The "obvious" way to implement redirection is to manipulate LD_PRELOAD and
wrap the open() and creat() calls.
But as the fakeroot(1) documentation notes, this approach
creates "problems, as demonstrated by the libtricks package.
This package wrapped many more functions, and tried to do a
lot more than fakeroot.
It turned out that a minor upgrade of libc (from one
where the stat() function didn't use open() to one with a stat()
function that did (in some cases) use open()), would cause unexplainable
segfaults (that is, the libc6 stat() called the
wrapped open(), which would then call the libc6 stat(), etc).
Fixing them wasn't all that easy, but once fixed, it was just a
matter of time before another function started to use open()...".
LD_PRELOAD-based approaches fail on commands that are statically linked,
and the commands usually used for installation (e.g., cp) are often
statically linked.
Others tools (like fakeroot) also use LD_PRELOAD, so using LD_PRELOAD
means that these other tools cannot be used simultaneously.
Other "obvious" approaches are to use ptrace(), or redirect file writes
using the kernel (e.g. with chroot),
but these turn out to have many problems too, as discussed in
<http://www.dwheeler.com/essays/automating-destdir.html>.
In contrast, since run-redir and make-redir don't manipulate LD_PRELOAD
to manipulate open() or creat(),
they are immune to deadlocks caused by changes in the C library,
work with statically-linked commands, and work with tools like fakeroot.
No special privileges are needed to run these programs, nor do they
require setuid installation, eliminating many security
concerns (compared to a chroot, setuid, or kernel-based approach).
No kernel driver is used, nor is ptrace() used,
so these tools should be fairly portable to any POSIX-based system with bash.
All work happens very quickly, since nearly all commands simply run as
usual, and even the wrapped commands merely run a short shell script
before running as usual.
The implementation is entirely a set of small bash scripts
with trivial dependencies, so the auto-destdir tools
can be easily included in a package's build process without slowing the
build process down (even if the build process requires re-installing
dependencies on each run).
In practice, this approach is sufficient to automate DESTDIR
for typical installation scripts.
Typical "make install" programs, even if they are deeply recursive,
only use a very few commands to actually write files to their
final locations in trusted directories,
and usually invoke commands without using the full pathnames of the commands
(e.g., "install -p myprog /usr/bin").
As a result, typical installation scripts are
easily redirected by this program.
More information about various implementation alternatives,
their pros, and their cons, is here:
http://www.dwheeler.com/essays/automating-destdir.html
Only wrapped programs will be redirected to the new destination directory;
any other program will not be redirected.
Currently, the "wrapped" commands are (alphabetically):
If the makefile invokes wrapped programs using absolute pathnames
(e.g., calling "/bin/cp" instead of "cp"), then the command will be
redirected anyway if it is the absolutely first command
at the beginning of a line in a makefile and is invoked directly
or via makefile variable expansions like $(CP).
Absolute pathnames at the beginning of a line are handled because
make-redir sets SHELL to a special shell wrapper that handles
this particular case.
However, absolute pathnames for wrapped programs are not handled in any
other place (doing so would require the shell wrapper to
re-implement shell parsing); e.g., program names stored in shell variable
expansions will not be intercepted.
Thus, makefile lines like
"cd X; /usr/bin/install myprogram /usr/bin" will not be redirected.
In these cases, you should change absolute pathname invocations like
/usr/bin/install
to unnamed-path invocations like install.
Capabilities built into the shell are not redirected, including
I/O redirection such as "<", ">", and ">>".
This means that "cat x > /usr/bin/program" will not be redirected;
use "cp x /usr/bin/program" or similar instead.
Similarly, "test" and "[" (which are normally shell built-ins)
aren't redirected.
Changing this would require auto-destdir
to re-implement a full shell.
The wrappers presume GNU command line options, and will not
understand other options even if the underlying tool would.
In practice, this is almost always irrelevant; "make install" typically
only uses highly-portable commands to copy files and make directories,
and does not use unusual options.
The "rm" and "rmdir" commands are not redirected simply because these
are unlikely commands during a "make install".
Similarly, there are no "whiteout" files to record when a
file is removed from $DESTDIR, because this does not seem useful for
"make install".
In cases where this wrapper is not sufficient, you may need to
modify the installation script.
If significant modifications to the installation script are required,
you may as well add DESTDIR support directly to the installation script.
The auto-destdir tools are designed to handle common cases, so that
people can focus their time on the cases that need special handling.
When writing the commands for "make install":
Supporting DESTDIR is still a good idea.
As noted above, only "wrapped" commands are redirected.
Some commands that aren't wrapped, and perhaps should be, include:
install-info, ldconfig, rm, rmdir, chown, chgrp, and libtool.
For permissions-related commands (like chmod, chown, and chgrp), consider
also using fakeroot or make's "-k" option.
Options with optional arguments cannot accept arguments that begin with "-",
due to limitations of getopt(1).
cp's --parents argument is supported, but it won't work if the
original files are being copied from a redirected location.
When running make-redir, the makefile's SHELL value is ignored.
If you need to use the SHELL value provided by the makefile,
run make-redir and provide "SHELL=..." with the correct shell value.
If you do this, wrapped commands must never be invoked using their
full pathname.
/bin/mkdir -p /tmp/stuff/usr/bin
/bin/cp myprogram /tmp/stuff/usr/bin
...
BuildRequires: auto-destdir
...
%install
rm -rf "$RPM_BUILD_ROOT"
make-redir DESTDIR="$RPM_BUILD_ROOT" install
IMPLEMENTATION APPROACH
LIMITATIONS
GUIDANCE FOR SOFTWARE CREATORS
BUGS
AUTHOR
David A. Wheeler
REPORTING BUGS
Report bugs to <dwheeler, at, dwheeler dot com>
COPYRIGHT
Copyright © 2009 Institute for Defense Analyses.
Released using the MIT license.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.