Distribution Constructor Usage TOI

Last Modified: November 13, 2008

Overview

Setting up to use Distribution Constructor

File Locations

The Simplest Usage Case

The Build Area

Common Usage Examples

Common Customization Tasks

Finalizer Script Examples

Example 1: Adding an IPS package from an alternate repository in a finalizer script
Example 2: Replace an IPS package with your own test version
Example 3: Adding a SVR4 package to the image for testing

Trouble Shooting/FAQ

Existing Bugs

References

Overview

The Distribution Constructor (DC) is a command line tool for building custom Open Solaris images. At this time, the tool takes a manifest file as input, and outputs an ISO image based on parameters specified in the manifest. Optionally, an USB image can be created based on the generated ISO image. Other output image formats are possible, but it is not produced by the DC team at this time. Thus, this document covers steps on building ISO and USB images. The DC provides the ability to checkpoint and resume from certain parts of the image generation process. This is designed to help speed up the process of developing and debugging images. We have verified that the DC is able to successfully build the Live CD image distributed as Open Solaris.

Setting up to use the Distribution Constructor

The DC can run on both Open Solaris or Solaris Nevada.

Set Up Needed for Open Solaris

Open Solaris 2008.11
The SUNWdistro-const package, installed from an IPS repo. For example: http://pkg.opensolaris.org

Set Up Needed for Solaris Nevada

Solaris Nevada build 99 or newer
The SUNWipkg, SUNWpython-cherrypy, and SUNWpython-pyopenssl package from the IPS gate
The SUNWinstall-libs, SUNWinstall, and SUNWdistro-const package, built from the slim_source gate.

Disk space requirements

At least 8 gigabyte should be allocated for the “build area” for each image to be generated.

File Locations

The Distribution Constructor package (SUNWdistro-const) installs its files in the following directories.

The Distribution Constructor application	/usr/bin/distro-const
For generating and copying USB images	/usr/bin/usbcopy /usr/bin/usbgen
For generating the ISO sort file	/usr/bin/proc_tracedata /usr/bin/proc_slist
Python library modules used by the DC	/usr/lib/python2.4/vendor-packages/osol_install/distro_const
Configuration, data, and utilities used by the DC application	/usr/share/distro_const
Configuration, data, and utilities used by the Slim CD only	/usr/share/distro_const/slim_cd
Configuration, data, and utilities used by the Auto Installer project	/usr/share/distro_const/auto_install
Documentations for the Distribution Constructor	/usr/share/doc/distro_const
Man page for the Distribution Constructor	/usr/share/man/

The Simplest Usage Case

Make a copy of the default manifest file (/usr/share/distro_const/slim_cd/slim_cd.xml) to somewhere on your system. In this example, I just put it in /
```
cp /usr/share/distro_const/slim_cd/slim_cd.xml /slim_cd.xml
```

Modify /slim_cd.xml to specify values that are specific to your environment. The values you need to modify/verify are the following.

XML tag name	Explanation
main url	It defaults to http://pkg.opensolaris.org. This will generally work, but it is very slow since a lot of people uses this repo. If you have your own repo, or you have access to another repo(ie: a Sun internal repo, if you work at Sun),You should use that instead.
build_area	This is the working area for the build. You have to make sure this is valid for your system. If you want to use checkpointing, this should either point to a ZFS dataset or a mount point that correlates to a ZFS data set. The ZFS dataset doesn't have to exist, it will be created it if it doesn't exist. We do require that the zpool you specified in this exists. If you don't want to use checkpointing, you can provide a regular mountpoint on a ZFS or UFS file system. At least 8G of free space should be available in the zpool for each image.

Everything else in the file can be left alone for the simple case.

When you are done with modifying the manifest file, you are ready to run the distro constructor. You have to run it as root.
```
/usr/bin/distro_const build /slim_cd.xml
```

The Build Area

The build area is where all the work is done. The Distribution Constructor creates the following structures in the build area.

Directory	Explanation
<build_area>/build_data/pkg_image	This is the package image area.
<build_data>/build_data/bootroot	The bootroot build area. This is where the bootroot is constructed, the content of this directory will be archived into the bootroot that gets put into the image.
<build_data>/build_data/tmp	The temp dir that is used by the the DC app as well as finalizer scripts.
<build_data>/media	The output images.
<build_data>/logs	The log files for the build. There are 2 log files for each run of the Distro constructor. The simple log file shows which steps are executed, and whether the build is successful. The detailed log file contains output from running all commands.

Common Usage Examples

The Distro Constructor has a resume and pause feature that will help users debug their selection of files, packages and scripts for creating images. To use the checkpointing feature, you must specify a ZFS dataset as your build area. Each checkpoint is referenced by a name.

To build an image, from start to finish:
```
/usr/bin/distro_const build <manifest>
```

To see which checkpoints are available:

/usr/bin/distro_const build -l <manifest>

Based on the checkpoints available, re-start from the bootroot initialization step (br-init), and run to completion:
```
/usr/bin/distro_const build -r br-init <manifest>
```
Based on the checkpoints available, re-start from bootroot initialization step(br-init), and stop at bootroot archiving(br-arch):
```
/usr/bin/distro_const build -r br-init -p br-arch <manifest>
```
Based on the checkpoints available, start from beginning, and stop at bootroot archive(br-arch):
```
/usr/bin/distro_const build -p br-arch <manifest>
```

Common Customization Tasks

The DC sets up the package image area, populates it with packages specified in the manifest, and customizes the image by executing finalizer scripts specified in the manifest. The manifest is the blueprint from which the image is generated, all your customizations should be specified in the manifest. See the Distribution Constructor design specification for details on each item in the manifest. Most of your customizations are done in the following sections.

<packages> section

                <packages>
                        <pkg name="SUNWcsd"/>
                        <pkg name="slim_cd"/>

                        <pkg name="SUNWcs"/>
                        <pkg name="slim_install"/>
                        <pkg name="SUNWslim-utils"/>
                        <pkg name="entire"/>

                </packages>

This is the list of packages or cluster of packages to be included in the image. Add or remove packages from this section to add/subtract content.

<bootroot_contents> section

               <bootroot_contents>
                        <base_include type="file">usr/sbin/pmadm</base_include>

                        <base_include type="file">usr/sbin/lofiadm</base_include>
                        <base_include type="file">usr/sbin/devfsadm</base_include>
                        <base_include type="file">usr/sbin/modload</base_include>

                        <base_include type="file">usr/sbin/i86/modload</base_include>
                        <base_include type="file">usr/sbin/mount</base_include>
                        <base_include type="file">usr/sbin/hostconfig</base_include>

                         .........
                        <base_include type="dir">var/svc/manifest</base_include>
                        <base_include type="dir">var/svc/profile</base_include>
                        <base_include type="dir">var/pkg/catalog</base_include>

                        <base_include type="dir">etc</base_include>
                        <base_exclude type="dir">etc/gconf</base_exclude>
                     <base_exclude type="dir">etc/brltty</base_exclude>

                     <base_exclude type="dir">etc/gtk-2.0</base_exclude>
               </bootroot_contents>

List of files and directories to be included in the bootroot of the image. All files and directories specified here are copied from the package image area. If a file or directory needs to be excluded, specify it with the "base_exclude" tag.

<finalizer> section

                        <finalizer>
                                <script name="/usr/share/distro_const/slim_cd/pre_bootroot_pkg_image_mod">
                                        <checkpoint name="im-mod" message="Image area Modifications"/>
                               </script>

                                <script name="/usr/share/distro_const/bootroot_configuration.py">
                                        <checkpoint name="br-config" message="Boot root configuration"/>
                                        <argslist>
                                               ?/usr/share/distro_const/slim_cd/slimcd_generic_live.xml?
                                               ?.livecd?
                                        </argslist>

                                </script>
                               <script name="/usr/share/distro_const/create_usb">
                                       <checkpoint name="usb" message="USB image creation"/>
                                </script>

                        </finalizer>

List of scripts for customizing the image. These scripts are provided by the user. A few standard scripts are also included as part of DC. These scripts can be found in /usr/share/distro_const. Scripts are executed in the order they are listed in the <finalizer> section. These can be Python programs, shell scripts or binaries. By default, the DC passes 5 arguments to all programs/scripts executed. The 5 arguments are:

server socket file name: server socket used for accessing manifest data
package image area path: path to the package image area
tmp dir: path to the /tmp directory for temp files.
Bootroot build area: this is where the bootroot gets put together.
Media area: this is where the output images will go

When specifying a finalizer script, you have to specify a name used for referencing this script during checkpointing. The checkpoint name can be any string as long as it is unique compared to other existing checkpoing names. The checkpoint message is optional. If it is not specified, the path of the script will be used as the checkpoint message.

Arguments to the finalizer scripts/programs are passed in by specifying them with the <argslist> tag. Spaces used for formatting the XML are also passed in, make sure your program/script strip them. Other information can also be passed to scripts by specifying them as key-value pairs. Stdout and stderr for the scripts are captured in the log files.

Finalizer Script Examples

When putting together an image, we find ourselves wanting to experiment with how an image will work when we add a few additional packages or remove a few packages from a known to be working set. This is fully supported by the Distribution Constructor. Additional packages can be added to the existing list in <package> section, and packages that you want to remove can be added to the <post_install_remove_package> section. Unfortunately, after you made your modification, you need to restart the build process from the very beginning, and re-download all the packages. That will take quiet a bit of time.

In the future, we will add support for downloading a major set of packages, then, allow people to add or subtract to this list, and have it be done in a separate checkpoint, so, people can easily experiment. At the mean time, you can do this yourself quiet easily with finalizer scripts.

These finalizer scripts will add or remove packages from the package image area. So, they should be the very first finalizer scripts in the <finalizer> section in the manifest. For example, if you have created a script “/export/home/user1/test_my_pkg” to test your own package, you would add it as follows in the finalizer section.

<finalizer>
      <script name=”/export/home/user1/test_my_pkg”>
           <checkpoint name=”my_test” message=”Running my test package”
      </script>
      <script name="/usr/share/distro_const/pre_bootroot_pkg_image_mod">
           <checkpoint name="im-mod" message="Image area modifications"/>
      </script>
      …........
</finalizer>

After you made the changes to the manifest, you should start your build from the beginning once. After that, if you make any changes to your package, you don't need to re-start from the beginning . You can generate an image with your modified package by starting at the checkpoint that runs your script. The following are the sequence of commands a user would execute to repeatedly test their modifications to packages.

Run the build from the very beginning after modifying the manifest. This just need to be done once.
```
distro_const build /my_updated_manifest.xml
```

Check steps that are resumable

distro_const build -l /my_updated_manifest.xml

Re-start from the step of your package modification. You can restart from this step as many times as you need while you debugging your modified package/contents.
```
distro_const build -r my_test /my_updated_manifest.xml
```

Example 1

A finalizer script to add an IPS package from an alternate repo specified in the manifest. The package will be added to the package image area. The name of the package to add is just hard coded in the script. This example also demonstrates how to use the ManifestRead program to get values from the manifest.

#!/bin/ksh
#
#
# Args:
#
#   5 arguments are passed in by default from the DC.
#
#   MFEST_SOCKET: Socket needed to get manifest data via ManifestRead object
#   PKG_IMG_PATH: Package image area
#   TMP_DIR: Temporary directory 
#   BR_BUILD: Area where bootroot is put together (not used in this example)
#   MEDIA_DIR: Area where the media is put (not used)
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

if [ "$#" != "5" ] ; then
        print -u2 "Usage: $0: Requires 5 args:"
        print -u2 "    Reader socket, pkg_image area, tmp dir,"

        print -u2 "    bootroot build area, media area"
        exit 1
fi

MFEST_SOCKET=$1

PKG_IMG_PATH=$2
if [ ! -d $PKG_IMG_PATH ] ; then
        print -u2 "$0: Image package area $PKG_IMG_PATH is not valid"
        exit 1
fi

PKGCMD="/bin/pkg"

#Hard code package to install
TEST_PKGS="SUNWcdrw"

#
# You would have specified the additional repository like this in the manifest
#
#                 
#   <pkg_repo_addl_authority>
#        <main url="http://localhost:10000" authname="localtest"/>
#   </pkg_repo_addl_authority>

# Get the alternate repository URL from the manifest
add_url=/usr/bin/ManifestRead ${MFEST_SOCKET} "distro_constr_params/pkg_repo_addl_authority/main/url"

# Get the alternate repository authority from the manifest
add_auth=/usr/bin/ManifestRead ${MFEST_SOCKET} "distro_constr_params/pkg_repo_addl_authority/main/authname"

added_authority=0

#
# Check to see if authority is already set in the package image area
# if not, add it in
#
${PKGCMD} -R $PKG_IMG_PATH authority $add_auth > /dev/null 2>& 1
if [ $? != 0 ] ; then
        ${PKGCMD} -R $PKG_IMG_PATH  set-authority -O ${add_url} ${add_auth}
        added_authority=1
fi

if [$? != "0" ] ; then
        print -u2 "$0: Unable to set additional authority"
        exit 1
fi

for t in ${TEST_PKGS} ; do
        pkg_name="pkg://${add_auth}/${t}"
        ${PKGCMD} -R $PKG_IMG_PATH install ${pkg_name}
        if [$? != "0" ] ; then
                print -u2 "$0: Unable to install ${pkg_name}"

                exit 1
        fi
done

# if we have added the additional authority, unset it so it doesn't pollute what's
# originally there
if [ $added_authority == 1 ] ; then
         ${PKGCMD} -R $PKG_IMG_PATH  unset-authority ${add_auth}
fi

return 0

Example 2

A package is already installed in the package image area because it is specified in the <package> section of the manifest. I have a test version of this package, I want to test out my version of the package. So, we need a finalizer script to replace the previously installed version with a test version from an alternate repo. This will be helpful for testing your own private copy of a package. This example assumes that you have a IPS repository server running on http://localhost:10000. The name of the package to replace is passed as an argument. This example also demonstrates how to get arguments passed in as finalizer script arguments. The script first uninstalls the existing version, then, install the test version from the alternate repo.

#!/bin/ksh
#
#
# Args:
#
#   5 arguments are passed in by default from the DC.
#
#   MFEST_SOCKET: Socket needed to get manifest data via ManifestRead object (not used in this example)
#   PKG_IMG_PATH: Package image area
#   TMP_DIR: Temporary directory
#   BR_BUILD: Area where bootroot is put together (not used in this example)
#   MEDIA_DIR: Area where the media is put (not used)
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

if [ "$#" != "6" ] ; then
        print -u2 "Usage: $0: Requires 6 args:"

        print -u2 "    Reader socket, pkg_image area, tmp dir,"
        print -u2 "    bootroot build area, media area, pkg_name"
        exit 1
fi


PKG_IMG_PATH=$2
if [ ! -d $PKG_IMG_PATH ] ; then
        print -u2 "$0: Image package area $PKG_IMG_PATH is not valid"
        exit 1
fi

PKGCMD="/bin/pkg"

#The test packages are passed in as arguments to this finalizer script
#You would have specified the argument like this in the finalizer section
#to pass in the argument
#
#
#  <finalizer>
#       <script name="/my/update_my_pkg_test">
#               <checkpoint name="update-pkg" message="Replaces an existing package with my own"/>
#                    <argslist>

#                       "SUNWcdrw"
#                    </argslist>
#        </script>
#  </finalizer>
#
TEST_PKG=$6

#hard code alternate repository and authority.  Assume that my test package resides in
#a repository running on port 10000 of the localhost.

# Get the alternate repository URL from the manifest
add_url="http://localhost:10000"

# Get the alternate repository authority from the manifest
add_auth="MY_TEST"

added_authority=0

# Check to see if authority is already set in the package image area, if not,
# add it in
${PKGCMD} -R $PKG_IMG_PATH authority $add_auth > /dev/null 2>& 1
if [ $? != 0 ] ; then
        ${PKGCMD} -R $PKG_IMG_PATH  set-authority -O ${add_url} ${add_auth}
        added_authority=1
fi


if [$? != "0" ] ; then
        print -u2 "$0: Unable to set additional authority"
        exit 1
fi

# Remove the package that's currently in the package image area.
${PKGCMD} -R $PKG_IMG_PATH uninstall ${TEST_PKG}
if [$? != "0" ] ; then
        print -u2 "$0: Unable to uninstall ${TEST_PKG}"
        exit 1
fi

# Install the package from test repo
pkg_name="pkg://${add_auth}/${TEST_PKG}"

${PKGCMD} -R $PKG_IMG_PATH install ${pkg_name}
if [$? != "0" ] ; then
        print -u2 "$0: Unable to install ${pkg_name}"
        exit 1
fi

# if we have added the additional authority, unset it so it doesn't pollute what's
# originally there
if [ $added_authority == 1 ] ; then
         ${PKGCMD} -R $PKG_IMG_PATH  unset-authority ${add_auth}
fi

return 0

Example 3

You have a package that's still in the SVR4 format. You want to include the content of this package into the image to see how it works before you convert it into an IPS package. This example shows you how to add a SVR4 package to the image using a finalizer script.

#!/bin/ksh
#
#
# Args:
#
#   5 arguments are passed in by default from the DC.
#
#   MFEST_SOCKET: Socket needed to get manifest data via ManifestRead object (not used)
#   PKG_IMG_PATH: Package image area
#   TMP_DIR: Temporary directory
#   BR_BUILD: Area where bootroot is put together (not used in this example)
#   MEDIA_DIR: Area where the media is put (not used)
#
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

if [ "$#" != "5" ] ; then
        print -u2 "Usage: $0: Requires 5 args:"

        print -u2 "    Reader socket, pkg_image area, tmp dir,"
        print -u2 "    bootroot build area, media area, pkg_name"
        exit 1
fi

PKG_IMG_PATH=$2
if [ ! -d $PKG_IMG_PATH ] ; then
        print -u2 "$0: Image package area $PKG_IMG_PATH is not valid"
        exit 1
fi

TMP_DIR=$3

#
# Install a SVR4 packages into the package image area
#

#create an admin file for non-interactive pkgadd's

ADMIN_FILE=${TMP_DIR}/admin.$$
cat << \ADMIN_EOF > $ADMIN_FILE
mail=
instance=unique
partial=nocheck
runlevel=nocheck
idepend=nocheck
rdepend=nocheck
space=nocheck
setuid=nocheck
conflict=nocheck
action=nocheck
networktimeout=60
networkretries=3
authentication=quit
keystore=/var/sadm/security
proxy=
basedir=default
ADMIN_EOF

#
# Path to your new packages
#
PKG_PATH=/path/to/my/test/svr4_pkg

#
# Test package name 
#
SVR4_TEST_PKG=SUNWmy-test

/usr/sbin/pkgadd -n -a ${ADMIN_FILE} -d $PKG_PATH -R ${PKG_IMG_PATH} ${SVR4_TEST_PKG}

if [ $? != ?0? ] ; then
        echo "installing package failed"

        exit 1
fi

/bin/rm ${ADMIN_FILE}

return 0

Trouble Shooting/FAQ

Q: I see some errors about downloading a package.

A: Make sure the pkg(1) command on your system is working correctly, and your connect with the IPS server is stable. Sometimes, IPS time out when trying to download a big cluster of packages. To check outside of the DC env, try to mimic what the DC does in terms of installing packages. Try the following commands as root, and make sure it works correctly.

pkg image-create  -F -a opensolaris.org=http://ipkg.sfbay.sun.com /tmp/test_img
pkg -R /tmp/test_img install SUNWcsd
pkg -R /tmp/test_ima install SUNWcs
pkg -R /tmp/test_img install slim_install
.....

Q: There is error about importing SMF services.

A: This is a known problem, caused by bug 6721855. The path for the pkg image area specified is too long for SMF to handle, after other SMF directories are appended to it. The workaround is to specify a shorter path for the pkg image area.

Existing Bugs

Here are some bugs that might affect your experience of using DC.

References

slim_source gate:

ssh://anon@hg.opensolaris.org/hg/caiman/slim_source

IPS gate:

ssh://anon@hg.opensolaris.org/hg/pkg/gate

Distribution Constructor Open Solaris page:

http://opensolaris.org/os/project/caiman/Constructor/

Distribution Constructor Documents:

ssh://anon@hg.opensolaris.org/hg/caiman/caiman-docs

distro_constructor/DC_DESIGN_DOC.odt (Design Specification)
distro_constructor/DC_TOI.odp (Technical TOI)
distro_constructor/DC_Usage_TOI.html (This TOI)

Mailing list for questions or comments:

caiman-discuss@opensolaris.org

Bugs and enhancement requests:

http://defect.opensolaris.org

product: distro-constructor

The Distribution Constructor Team:

OpenSolaris

Distribution Constructor Usage TOI

Overview

Setting up to use the Distribution Constructor

Set Up Needed for Open Solaris

Set Up Needed for Solaris Nevada

Disk space requirements

File Locations

The Simplest Usage Case

The Build Area

Common Usage Examples

Common Customization Tasks

<packages> section

<bootroot_contents> section

<finalizer> section

Finalizer Script Examples

Example 1

Example 2

Example 3

Trouble Shooting/FAQ

Existing Bugs

References