Trac: release management

[Garrett Goebel]

On Wed, Jun 4, 2008 at 3:55 PM, Michael Stone <michael at laptop.org> wrote:
>
> If it pleases you, count the number and severity of bugs that have been
> untouched for longer than 3 months. Then ask yourself where #6454
> compares fits in to the lists of 'can be addressed quickly' and 'needs
> to be addressed quickly'.

It depends on what your definition of "addressed" means. Every ticket
ought to be addressed as in "responded to" within a couple days or a
week at the latest. #6454 has gone 4 months without any response or
comment from the ticket's owner or anyone at OLPC/1cc. How many other
tickets are out there like it?

I'll take you up on your offer for a cloned copy of the Trac db. If
the data is in there, I'll write you a query which will give all the
non-closed tickets which have never been changed by the owner.


> On Wed, Jun 04, 2008 at 02:50:30PM -0400, Garrett Goebel wrote:
>> Where is your list of priorities?
>
> http://wiki.laptop.org/go/Priorities-2008
>
>> How does that map to the list of open Trac tickets?
>
> It doesn't. See
>
>  http://lists.laptop.org/pipermail/sugar/2008-May/006006.html
>  http://lists.laptop.org/pipermail/sugar/2008-May/006007.html

Nice general overview of priorities.

There ought to be a way to generate it or something a bit more
detailed from the Trac database. You'd probably need some way of
designating a ticket as a meta priority, and then have it block on all
the tickets required to meet that goal. Speaking of which stuffing
multiple values into the 'Blocked By' column has a certain odor.

There seems to be some overlap between the Trac fields for Milestone
and Version. They seem to be being used for some combination of
'branch' and 'build'. Whereas where I think you are trying to go,
would be to have a Release and Schedule. Where 'Release' would be one
of 08.Autumn, 09.Spring, etc. and would signify that work was to be
targeted as an update to the branch for that Release and all
subsequent releases. and 'Schedule' would be the YY-MM you want it to
land.


>> Where do you track the severity/impact of a ticket? I.e. scope of who
>> is effected
>
> Some people try to indicate this information with the 'priority' field
> on the ticket. In practice, I actually try to skim every change to Trac
> looking for important issues. Then we discuss them via status summary
> emails or meetings, like the one we're going to have in 10 minutes at
> 2:00 PM EST in #olpc-meeting on irc.freenode.org.
>
>> Where do you track the difficulty? I.e., general estimate of time
>> require to address a ticket
>
> We don't track it formally; only via discussion and status updates.

Whatever you want to call it, you might find it useful to track the
scope and complexity of the changes required to fix an issue. Priority
doesn't get at that. It would allow you to collect historic data which
could be used to project how much time tickets will take to be
implemented and how many bug hours you'll get per change.

I understand that the process needs to be as lightweight as possible
so as not to get in the way of developers actually implementing
things. But there's the balance to be had with actually being able to
have historic data available to make possible an efficient use of
those developers.


>> How to you track defects back to the changes (ticket) which introduced them?
>
> We don't do so systematically.

It is hard to do if you want to track back to a specific git
changeset... or Trac ticket.

But you could start to get a handle on it by adding 'Found In', 'Fixed
In', and 'Tested In' relationships for tickets. How do you currently
figure out when/where a defect was introduced and fixed?


>> How many Full Time Equivalent hours does a given developer represent?
>
> A guesstimate: about 25 hrs/wk of coding and 30 hrs/wk of talking for
> social folks, maybe 30 hrs/wk of coding and 10 hrs/wk of talking for
> contractors; and 5-8 full days off a month (including weekends).

Is there any list of developers and which slot each fit into?


>> What components are the given developers capable of working on?
>
> I don't understand this question.

You've got folks who have particular areas of expertise. Or to put it
the other way, developers who can work in certain areas but not
others. If your Trac ticket classifies a ticket as belonging to a
particular area, you can then project how many FTE's you've got on
hand to work in that area.

I realize that this being an open source project leaves a lot open
ended. But if you collect the data in a way that you can get at it
effectively, you can use historic data to verify your assumptions and
track and make projections against non-employee/non-contractor
developers as well.


>> How long does the assigned developer think the specific ticket will
>> take to complete? How long did it take?
>
> The limiting factors seem to me to be:
>
>  a) how long is the critical path of changes necessary to close the
>     ticket?
>  b) how overloaded is the required developers?
>  c) how frequently are the required developers task-switching?

I was ambiguous. What I meant was a).

It'd be nice if there were a field in the ticket for the developer to
note down how long they think they actually worked on a ticket.

If you combine this with the earlier mentioned field for
scope/complexity (difficulty) then you can make some projections on
how many FTE outstanding ticket hours you've got based on historic
data. And you can assign each 'difficulty' level an average time to
completion based on a running aggregate of the historic data.

You can also make some projections on the average number of defect
hours you can expect will eventually be entered based on how the
ticket is classified.

Put it together with allowing tickets to have a 'schedule' where
schedule is YY-MM (ex: 08-07) And you can hope to create a realistic
schedule of what you hope to accomplish month by month.


> Also, you've got to be careful here to specify a ticket workflow. I do
> release work at the moment, so I don't consider issues to be "fixed"
> until they're fixed _in a released stable build_.

true.

I'm assuming by fixed you mean implemented and tested.


>> How long must a ticket sit dormant before it gets bumped and someone
>> takes notice?
>
> It's not a matter of taking notice. It's a matter of being reminded at a
> time when there's nothing more pressing on the priority queue.

I'm going to assume that we both agree that some action should
initially take place. Perhaps we don't. I think the person who filed
it should be able to see within a few days whether or not the ticket
is accepted or rejected, who it has been assigned to and how it has
been prioritized.

Define:
o  who should be reminded
o  'nothing more pressing'
o  'priority queue'

And then we can see if it can be automated...

I'm probably not the only person who has ignored things in my queues
because I'd rather work on the interesting problems or the low hanging
fruit.


> (Most developers seem to have actual work queues which are about 5
> tickets long. In practice, their Trac ownership lists are often 30-100
> tickets long. Go figure.)

I don't understand the difference between an 'actual work queue' and
'ownership'. Can you explain?


>> What is your rate of defects per change? How does that break down by
>> severity and difficulty?
>
> Are you measuring by source commits, packages, test builds, candidate
> builds, or releases?

Trac tickets.  Source commits might be better. Some analysis of the
composition of git changesets associated with a Trac ticket would be
better. But in practice Trac tickets with fields for complexity and
length of work required are probably just as good and a whole lot less
complicated.


>> Are tickets tested? Tested by someone other than the implementer?
>
> Sometimes. Also sometimes.

Is that what 'verified' indicates? How do you know what branch/build's
it was tested in?


>> Are tickets reviewed before being closed?  By someone other than the
>> implementer. Who?
>
> See #7014 for an example of the problem.

Looks like the big problems are easier to solve if you identify them
as a bunch of little problems problem. The ticket should probably have
been broken down into lots of smaller tickets and then updated to list
them as its blockers. There's mention of other tickets all
throughout... but you're not using the 'Blocked By' or 'Blocking'
fields...


>> You _talk_ about priorities and wise time management but certainly
>> haven't done a good job communicating how you plan to go about doing
>> it.
>
> What should he do instead? (Keep in mind that Peru is receiving
> something like 40k additional laptops next week that need new software
> which Scott is the only person working on full-time. [He's both the
> release engineer and he's fixing and testing individual issues as
> documented by his recent changes to Trac.])

This comment was taken out of context. It was in response to a mild
rant from Scott in which in addition to stating that he was overworked
and needed to prioritize... mis-characterized my original email as a
loudly insisting that my person problems be worked on *right now*.

None of which answered the original question as to how the OLPC
process can be so broken as to allow a newly filed tickets to be
completely ignored forever.

I wouldn't pretend to know what Scott should be doing with his time. I
was only asking that the OLPC improve its process so it does not waste
mine.


>> Besides, how can you hope to prioritize if you don't enumerate your
>> resources,
>
>  http://wiki.laptop.org/go/Available_Labor-2008

Since (o) employee/contract is grouped... it might be useful to break
them out. If you and I are both interested in my working on your issue
tracking system to make it more useful for resource planning.


> Also notice how I'm splitting release prioritization from development
> prioritization into separate management problems.

No I didn't notice. I don't see the words 'release' or 'development'
mentioned on that URL.

It would be interesting if there were a mapping from how you've
categorized these people to how Trac tickets are categorized. I was
assuming the 'component' field could be used for that purposed.


>> constraints, and interdependencies? How can you balance
>> work queues if you can't quantify them?
>
>> How can we as outsiders expect our interactions with the OLPC to be
>> addressed in a timely manner?
>
> By bartering for the time of the people whose help you need.

I can't even contact the owner of ticket #6454. Apparently, I'm
supposed to go on IRC, hang out and hope to catch him.

Barter with what? My overly polite and friendly disposition :-) ?


>> Even if addressed only means being notified that the given issue won't
>> be addressed.
>
> Maybe we should put it in big bold letters on the front of Trac, but if
> your ticket hasn't been commented on within a week, that means that no
> one is currently watching.

That much is obvious. The question is how you get a hold of someone to
find out if they ever intend to do anything about it.


>> How do I create a report? http://dev.laptop.org/wiki/TracReports tells
>> you about reports, but not how to create one...
>
> Sadly, you need to be a Trac admin to create reports. What would work
> better would be to work with a local copy of the Trac DB. We could
> probably set up a cron job to make one if it would help.

That would be nice. Should I open a ticket?


>> You folks are always asking people to open tickets. But how many of
>> those tickets result in n-way communication? How many open tickets are
>> out there that go unanswered? 610 have "Never Assigned" for their
>> Milestone. Does the OLPC have anyone out there tasked to track down
>> abandoned tickets? Why should I invest my time?
>
> OLPC has no one tasked to track down abandoned tickets. Who should they
> assign?

Who ever assigns tickets when tickets need to be assigned. Would that be you?

Make a sql query that does the work for you. Give me that copy of the
Trac database and I'll take a swing at doing it for you.

cheers,

Garrett