2008 Debate of Build and Release

[Charles Merriam]

Here's my write-up from the April Fools memo
(http://wiki.laptop.org/go/April_Fool_2008_Build_Process) and
Mini-conference presentation.   I'll get something on the wiki
shortly, probably at
http://wiki.laptop.org/go/2008_Debate_of_Build_and_Release.  I'd
appreciate any comments.

Thanks,

Charles
====

About The Debate

The 2008 Debate on Build and Release started when Charles Merriam (l)
wrote an April Fool's document named "April Fool 2008 Build Process"
(l).  The document is short and should be read before reading this
page.  The ideas were then presented as a set of slides (l) at the
April 4 Spring Miniconferce (l)   Much debate ensued.

    Build and Release processes and tools change over time as software
matures.  You can find references to the build and release debates of
2007 (L), which more more concerned with getting any process to work.

Debating
    Debating occurs in several venues:  the talk page of this article
(l); calls and conferences to be determined; and the devel and testing
mailing lists (l).   Please start *all* debate posts with "Build
Debate:".  If your posts relate to a section, mention the section.
For example, a good subject line might be "Build Debate: Time Based
Release has another problem".  Relevant information, conclusions, and
unsolved issues will be eventually be added into this article.   If
you want to track the progress and conclusions, you may want to watch
this page (l) on your watchlist (l).

Social vs Technical Problems
    Thinking about build and release management causes arguments
everywhere because it dredges up long standing concerns about
corporate strategy and implementation.  Build and release management
is not a solution to social and business problems, no matter how much
you wish it were.   Nor will build and release management squeeze
seven months of engineering into a six month cycle.  Be cognizant of
discussions that catapult from release management to general
management.

    Also, recognize that the OLPC project is transitioning rapidly
from realizing an "impossible dream" to managing many thousands of
laptops in diverse deployments.  People need time to adjust
expectations for this change.


Social Problems To Address

Adopt a Time Based Release System

* See Ubuntu's Wiki for an explanation (l) and release schedule [l].

Time Based Release is a practice and procdure to ship based on the
calendar instead of based on features completed.  The release ships
on-time and features still stabilizing will slip forward to ship in
the following release.  This change in philosophy provides a regular
heartbeat of new versions, so that slipped features have a reliable
next ship date.

The ultimate aim is to always be able to build a nearly shipping
version,  This means an exceptional circumstance requiring a special
release would only need "Disaster Insurance" final quality testing.
New features still gaining quality are developed in separate branches
until they can be incorporated into a solid branch.

This is a contentious discussion and took up most of the
MiniConference discussion time:
* Every project starts with feature based releases until it hits a
minimum functionality.  Stability and predictability become issues
after there are users relying on the project's functionality.  OLPC
now has many schoolchildren, teachers, and developers relying on
releases.
* Delaying releases to serve one market breaks other markets:  many
bug fixes and improvements will have been tested and ready each
release date.  Some countries will only pick up one release per year
and may not be able to adopt any changes if release dates slip.
* Time-based releases also trigger contemplation of the changing
management directives for each release.  Having specific release dates
may make the discussion easier.  Specifically, an inflexible ship-date
grounds discussions in reality and helps to evaluate trade-offs of
important features.  The build manager creates a release on a date
from whatever engineering is completed.

Build Manager Not Necessarily On Site in Cambridge
    This idea brings more contentious debate and contemplation.   The
role of the build manager is to make sure releases happen on specific
dates and to balance adherence to a release process with the
flexibility of exceptions to that process.  He or she does not get
features by "going down the hall to beat up engineers for patches."
Also, a growing percentage of development happens in the open source
community, which is off-site.

Buy-in From OLPC Foundation Before Work

Buy-in facilitates work that is used.  Work proceeds with only partial
commitment:  full adoption will take a year of successes to cement.
Participants at the Mini Conference believed the minimum buy-in for
work to commence was doable.

OLPC Foundation must agree to rename Update-1 to something with a year
designation.  The exact name is flexible, e.g., "2008 Release 1" and
"XO-08, April Edition" are both fine.   This commits to releases
annually, closer to a timely basis.

OLPC Foundation must agree to progressively challenging exceptions to
changing code ("freezes") near the release date.  Changing code after
certain dates requires convincing build engineer to make an exception.
  Convincing arguments include:
            The change fixes problems in existing functionality.
            The change is well tested and understood
            Release not useful to customers without change
As the ship date gets closer, the arguments need to be progressively
more convincing.  Higher risk items, such as late changes to the
Fedora core or programmer API may require compelling arguments.  The
agreement builds understanding that late changes in focus are "for the
next release".


= Actual Work.  The Technical Aspects =

The technical aspects generated far less debate than the social
aspects.  Not everything from the April Fool's memo is discussed.  The
idea is to succeed at the higher value opportunities one by one.

= Multiple Targets/Timely Builds=

    Current State
        A basic build system resulting in numbered builds exists
            This is the Build Announcer emails to dev for "build" and
"faster build"
        Titus has a build bot recompiling every twenty minutes
        Sugar JH build, last seen, depends on dozens of non-OLPC sites
to grab packages
        Pre-compiled packages for different platforms are heavily
used, but not based on known builds
            E.g., Jani's Ubuntu packages, LiveCD, VMWare packages

    Ordered List Of First Improvements
        Get Versions Up
            Get standard "released", "stable" (interim), and "joyride"
branches or tags into all projects and firmware.
            There is always exactly one most recent of each of these
three versions.
        Get Bundles Up
            Simply packaging in easy to grab sets
            Packages are Sugar, Sugar w/source, Sugar w/solid
applications, Sugar w/solid applications and source, and the big
package of everything in the GIT since last build.
        Get Deployment Bundles Up
            Set up standard descriptors to track for specific
deployments.  For example, "2008-Release 1 Version, Peru, Source"
would be the exact configuration and application packages of the
2008-Release 1 as it was deployed in Thailand plus source code.
There's some debate on naming schemes and how each deployment should
be recorded in the wiki.
        Probably modify olpc-update, etc., to understand Version and Bundle tags
        Get one or more Emulator Platforms, starting with Ubuntu, into
the build cycle.  (Emulator Platforms were called "formats" in the
April 1 memo and Mini-conference slides).  Try to get good builds, at
least for the "stable" and "released" versions so that Activity
developers have a solid base of development.

        A failing emulator fail would not halt a build or a deployment
but may be released when a patch to make it functional is released.
.  It is important for emulator versions to be available for "stable"
and "released" versions, and for LiveCD versions to be available for
"released" versions.  A failing emulator on an otherwise good build
might be released a week or two later with just patches added to make
the emulator function.  Emulator problems would, naturally, receive
less support and mind-share from Cambridge than the core XO platform.

        Create screen casts detailing how to start developing and
executing first program.  We want to make it easy for experienced
Python programmers to experiment with the Sugar interface and
development.

= GitWeb or dev.laptop.org =
    Current State
        Lots of projects and activities accretion
            Many do not work
            Many are abandoned.
        Manual Application Process via email
            Projects encouraged to start on other source control, and
then switch to GIT.

    Ordered List of First Improvements:
        Classify projects by solidity:
            "Vapor" are projects that are discussed but not ready for
others to run yet.
            "Liquid" are projects that run, have some functionality,
but have major missing features.
            "Solid" are projects that are part of core distributions,
including the firmware and OS.
        Allow any project
            Either automate method, or just approve every one.
Inappropriate projects may never get out of "Vapor".
        Show some metrics by project
            Meaning of metric number will change over time
            Make any estimate on Content, Coverage, and Components
                For example, has translation files, has any tests, has
an icon might give high numbers for the first version.
            Metrics provide goals towards making solid projects

= Conclusions =
    It's just work.
        Low hanging fruit aren't that much work.
    Nothing new.  No research papers.