2008 Debate of Build and Release

[david at lang.hm]

one thing to be very careful of with a date-based schedule is that as you 
near the release data you need to be extremely intolorent of regressions. 
it's useually better to delay a feature becouse you have to revert a patch 
to fix a regression then it is to break someone's existing setup becouse 
someone else thought a different feature was more important.

as long as the steps are forward, people will accept small steps, and many 
steps, but if they feel that they are playing russian roulete with each 
upgrade there will be problems.

David Lang

On Sat, 5 Apr 2008, Charles Merriam wrote:

> Here's my write-up from the April Fools memo
> (http://wiki.laptop.org/go/April_Fool_2008_Build_Process) and
> Mini-conference presentation.   I'll get something on the wiki
> shortly, probably at
> http://wiki.laptop.org/go/2008_Debate_of_Build_and_Release.  I'd
> appreciate any comments.
>
> Thanks,
>
> Charles
> ====
>
> About The Debate
>
> The 2008 Debate on Build and Release started when Charles Merriam (l)
> wrote an April Fool's document named "April Fool 2008 Build Process"
> (l).  The document is short and should be read before reading this
> page.  The ideas were then presented as a set of slides (l) at the
> April 4 Spring Miniconferce (l)   Much debate ensued.
>
>    Build and Release processes and tools change over time as software
> matures.  You can find references to the build and release debates of
> 2007 (L), which more more concerned with getting any process to work.
>
> Debating
>    Debating occurs in several venues:  the talk page of this article
> (l); calls and conferences to be determined; and the devel and testing
> mailing lists (l).   Please start *all* debate posts with "Build
> Debate:".  If your posts relate to a section, mention the section.
> For example, a good subject line might be "Build Debate: Time Based
> Release has another problem".  Relevant information, conclusions, and
> unsolved issues will be eventually be added into this article.   If
> you want to track the progress and conclusions, you may want to watch
> this page (l) on your watchlist (l).
>
> Social vs Technical Problems
>    Thinking about build and release management causes arguments
> everywhere because it dredges up long standing concerns about
> corporate strategy and implementation.  Build and release management
> is not a solution to social and business problems, no matter how much
> you wish it were.   Nor will build and release management squeeze
> seven months of engineering into a six month cycle.  Be cognizant of
> discussions that catapult from release management to general
> management.
>
>    Also, recognize that the OLPC project is transitioning rapidly
> from realizing an "impossible dream" to managing many thousands of
> laptops in diverse deployments.  People need time to adjust
> expectations for this change.
>
>
> Social Problems To Address
>
> Adopt a Time Based Release System
>
> * See Ubuntu's Wiki for an explanation (l) and release schedule [l].
>
> Time Based Release is a practice and procdure to ship based on the
> calendar instead of based on features completed.  The release ships
> on-time and features still stabilizing will slip forward to ship in
> the following release.  This change in philosophy provides a regular
> heartbeat of new versions, so that slipped features have a reliable
> next ship date.
>
> The ultimate aim is to always be able to build a nearly shipping
> version,  This means an exceptional circumstance requiring a special
> release would only need "Disaster Insurance" final quality testing.
> New features still gaining quality are developed in separate branches
> until they can be incorporated into a solid branch.
>
> This is a contentious discussion and took up most of the
> MiniConference discussion time:
> * Every project starts with feature based releases until it hits a
> minimum functionality.  Stability and predictability become issues
> after there are users relying on the project's functionality.  OLPC
> now has many schoolchildren, teachers, and developers relying on
> releases.
> * Delaying releases to serve one market breaks other markets:  many
> bug fixes and improvements will have been tested and ready each
> release date.  Some countries will only pick up one release per year
> and may not be able to adopt any changes if release dates slip.
> * Time-based releases also trigger contemplation of the changing
> management directives for each release.  Having specific release dates
> may make the discussion easier.  Specifically, an inflexible ship-date
> grounds discussions in reality and helps to evaluate trade-offs of
> important features.  The build manager creates a release on a date
> from whatever engineering is completed.
>
> Build Manager Not Necessarily On Site in Cambridge
>    This idea brings more contentious debate and contemplation.   The
> role of the build manager is to make sure releases happen on specific
> dates and to balance adherence to a release process with the
> flexibility of exceptions to that process.  He or she does not get
> features by "going down the hall to beat up engineers for patches."
> Also, a growing percentage of development happens in the open source
> community, which is off-site.
>
> Buy-in From OLPC Foundation Before Work
>
> Buy-in facilitates work that is used.  Work proceeds with only partial
> commitment:  full adoption will take a year of successes to cement.
> Participants at the Mini Conference believed the minimum buy-in for
> work to commence was doable.
>
> OLPC Foundation must agree to rename Update-1 to something with a year
> designation.  The exact name is flexible, e.g., "2008 Release 1" and
> "XO-08, April Edition" are both fine.   This commits to releases
> annually, closer to a timely basis.
>
> OLPC Foundation must agree to progressively challenging exceptions to
> changing code ("freezes") near the release date.  Changing code after
> certain dates requires convincing build engineer to make an exception.
>  Convincing arguments include:
>            The change fixes problems in existing functionality.
>            The change is well tested and understood
>            Release not useful to customers without change
> As the ship date gets closer, the arguments need to be progressively
> more convincing.  Higher risk items, such as late changes to the
> Fedora core or programmer API may require compelling arguments.  The
> agreement builds understanding that late changes in focus are "for the
> next release".
>
>
> = Actual Work.  The Technical Aspects =
>
> The technical aspects generated far less debate than the social
> aspects.  Not everything from the April Fool's memo is discussed.  The
> idea is to succeed at the higher value opportunities one by one.
>
> = Multiple Targets/Timely Builds=
>
>    Current State
>        A basic build system resulting in numbered builds exists
>            This is the Build Announcer emails to dev for "build" and
> "faster build"
>        Titus has a build bot recompiling every twenty minutes
>        Sugar JH build, last seen, depends on dozens of non-OLPC sites
> to grab packages
>        Pre-compiled packages for different platforms are heavily
> used, but not based on known builds
>            E.g., Jani's Ubuntu packages, LiveCD, VMWare packages
>
>    Ordered List Of First Improvements
>        Get Versions Up
>            Get standard "released", "stable" (interim), and "joyride"
> branches or tags into all projects and firmware.
>            There is always exactly one most recent of each of these
> three versions.
>        Get Bundles Up
>            Simply packaging in easy to grab sets
>            Packages are Sugar, Sugar w/source, Sugar w/solid
> applications, Sugar w/solid applications and source, and the big
> package of everything in the GIT since last build.
>        Get Deployment Bundles Up
>            Set up standard descriptors to track for specific
> deployments.  For example, "2008-Release 1 Version, Peru, Source"
> would be the exact configuration and application packages of the
> 2008-Release 1 as it was deployed in Thailand plus source code.
> There's some debate on naming schemes and how each deployment should
> be recorded in the wiki.
>        Probably modify olpc-update, etc., to understand Version and Bundle tags
>        Get one or more Emulator Platforms, starting with Ubuntu, into
> the build cycle.  (Emulator Platforms were called "formats" in the
> April 1 memo and Mini-conference slides).  Try to get good builds, at
> least for the "stable" and "released" versions so that Activity
> developers have a solid base of development.
>
>        A failing emulator fail would not halt a build or a deployment
> but may be released when a patch to make it functional is released.
> .  It is important for emulator versions to be available for "stable"
> and "released" versions, and for LiveCD versions to be available for
> "released" versions.  A failing emulator on an otherwise good build
> might be released a week or two later with just patches added to make
> the emulator function.  Emulator problems would, naturally, receive
> less support and mind-share from Cambridge than the core XO platform.
>
>        Create screen casts detailing how to start developing and
> executing first program.  We want to make it easy for experienced
> Python programmers to experiment with the Sugar interface and
> development.
>
> = GitWeb or dev.laptop.org =
>    Current State
>        Lots of projects and activities accretion
>            Many do not work
>            Many are abandoned.
>        Manual Application Process via email
>            Projects encouraged to start on other source control, and
> then switch to GIT.
>
>    Ordered List of First Improvements:
>        Classify projects by solidity:
>            "Vapor" are projects that are discussed but not ready for
> others to run yet.
>            "Liquid" are projects that run, have some functionality,
> but have major missing features.
>            "Solid" are projects that are part of core distributions,
> including the firmware and OS.
>        Allow any project
>            Either automate method, or just approve every one.
> Inappropriate projects may never get out of "Vapor".
>        Show some metrics by project
>            Meaning of metric number will change over time
>            Make any estimate on Content, Coverage, and Components
>                For example, has translation files, has any tests, has
> an icon might give high numbers for the first version.
>            Metrics provide goals towards making solid projects
>
> = Conclusions =
>    It's just work.
>        Low hanging fruit aren't that much work.
>    Nothing new.  No research papers.
> _______________________________________________
> Devel mailing list
> Devel at lists.laptop.org
> http://lists.laptop.org/listinfo/devel
>