Monday, May 9, 2011

How I estimate web work

At my first job out of college I was asked by my manager how long I thought something was going to take after discussing design and approach. My response at that time was laughter, literally, I had no idea. Nor did I accept this question as important in any way. Building takes time and it takes the amount of time it takes me to build it.

At my second job, I was initially victim to someone else's estimation of work. Usually senior developers on the team, or the project manager would estimate given there was nobody else to do it. If something went wrong, then I would just blame the estimate, or the fact that web work is sometimes unpredictable and crazy things happen to blow up the amount of time something works.

While still at my second job, after my first major site launch, I was enterend into a more structured estimation process. We were estimating the build in large chunks prior to even design. It was a high level estimate for the client pitch. We estimated in buckets of easy, medium or hard, which was usually 20, 40 and 60 hours respectively. The estimates were wildly broad and large, but it was a major extension effort of an existing site we just finished, so we had a good known rhythm of the development team and client. This effort ended up being very close to the base estimates, we were roughly a few weeks late which is nothing relatively speaking.

Still at my second job, I somewhat moved up into the position to be the primary technical resource to come up with estimates for an upcoming new project. I thought to myself, "oh shit". How do I estimate a whole project, start to finish? Where do I start? Luckily, someone else already went through this and tossed me over a spreadsheet they used. I took it and ran, changed it around and made it work for my project.

Now at my third job, per release, we are estimating work. This will be a common thread for all developers as you age and are taken more seriously. Managers and clients will always want some reasonable schedule they can plan on.

Break it down

Here is what I did and do. This might seem entirely obvious, but I've never seen it written down, shared, or taught for that matter. Maybe I haven't really looked either.
  • Break it down as far as you can
That's it... 

Break it down further

Well I guess that's a large task. So, let's break that down further. 
  • Break down components
Alright, that is manageable, what are the primary major site functions of your site? 
  • Search
  • Registered customer area
  • Public API
  • Caching
  • Sign up
  • Checkout
That's enough for now, I am sure each site has many or fewer areas. Let's use sign up as an example. 

Components

Now let's break that down. What makes up the component of "sign up" on a web site?

A lot of what comes from breaking down the component is starting with the UI. What will the customer work with and experience? Starting from the other end (data) is a lot harder and I don't recommend it for web estimation. Maybe for data warehousing where all work is internal this makes sense (starting with data). On the web,  the interaction will drive the data, or at least the data that mostly matters. 

What makes up a sign up? Hmm....
  • Site header link
  • Sign up primary page 
  • Sign up form 
So we have a link in the header to sign up for an account. That directs to the primary page containing the sign up flow. That page contains a form. 

We've got some work now, we need to build the link and integrate the link into the existing information architecture of the page. We need the primary page developed. It likely fits into a template clamp and then contains unique content. The form will have elements such as user name, email, password and a submit button. The page will have styling, the form will have validation. 

Breadcrumb Model

Alright, let's organize this information. It's good to use a naming structure, kind of like breadcrumbs to organize the list within itself. This naming his helpful during search and sorting elements in a list or in a ticketing system and is why I like it. Let's breadcrumb:
  • Sign up - Header Link - Integration of page with existing header links
  • Sign up - Primary Page - Apply page styling and page element layout
  • Sign up - Primary Page - Form - Add form validation pre-submit 
  • Sign up - Primary Page - Form - Add form action  
Whoa, wait, the form action? What the hell am I going to do with this data? Well how about I implement some sort of controller to handle the data, transform it and store it. 
  • Sign up - Controller - Add sever side data element validation
  • Sign up - Controller - Add data storage logic
Gah!! The data, where does it go? I think we need a model. 
  • Sign up - Data - Develop user sign up model entity relationships
I think we are looking good now.... but what if something goes wrong. We need some error handling items as well. 
  • Sign up - Primary Page - Add error messaging highlights or modals for user
  • Sign up - Controller - Add server side error handling 
  • Sign up - Controller - Add user de-duplication 
What about integration? Usually data doesn't just sit on the web, it goes somewhere usually to a master or central location? Let' add some break down for third party integration synchronization of user accounts. 
  • Sign up - Third Party - Add third party controller
  • Sign up - Third Party - Implement real time data mapping to third party
What if the third party was down, having an outage?
  • Sign up - Third Party - Implement third party retry logic
Let's stop there, we can go on and on depending on requirements and scope, but let's keep it simple. 

Hours

Now, how long will it take? Given the vage details, let's gut check it by conceptualizing into easy, medium and hard. Then we can apply hour range as such:
  • Easy - 2 to 6
  • Medium - 6 to 10
  • Hard - 10 +
Overlap, yeah, it doesn't matter. There is a easy six and a medium six, trust me. Notice these times are different fromt he 20, 40 and 60 above. The range can apply to whatever you want, but in general for actual work that will be tasked out, keep it smaller.

What's with hard? Hard isn't capped because its hard. Sort of like my first experience of estimation at my first job. You might not know what's happening until you get in there and really figure it out. Maybe 16 is the cap, but beyond that, think about breaking down that complex task into generic parts. 

To apply time, lets organize our list and mark times. Along with times, you'll want to add an assumptions as well. You won't remember why you estimated a certain way unless you mark it. Let's just mark time for now:
  • Sign up - Header Link - Integration of page with existing header links - 2
  • Sign up - Primary Page - Apply page styling and page element layout - 8
  • Sign up - Primary Page - Form - Add form validation pre-submit - 6
  • Sign up - Primary Page - Form - Add form action - (wash per item below)
  • Sign up - Primary Page - Add error messaging highlights or modals for user - 2
  • Sign up - Controller - Add sever side data element validation - 6
  • Sign up - Controller - Add data storage logic - 4
  • Sign up - Controller - Add server side error handling - 2 
  • Sign up - Controller - Add user de-duplication  - 3
  • Sign up - Data - Develop user sign up model entity relationships - 8
  • Sign up - Third Party - Add third party controller - 2
  • Sign up - Third Party - Implement real time data mapping to third party - 3
  • Sign up - Third Party - Implement third party retry logic - 12
Let's explain some
  • Sign up - Primary Page - Apply page styling and page element layout - 8
    • Some thing front end work is easy. It's not. Layout, interaction, browsers, etc. This takes time, 8 might not even be enough. 
  • Sign up - Controller - Add data storage logic - 4
    • Only 4?
    • Yeah, the model development isn't in this task, find the record, insert or update it, done.
    • Be sure to clearly separate work so it's understood when which part is done
  • Sign up - Third Party - Implement third party retry logic - 12
    • Why so much? Might not be enough. 
    • We need a retry model, a schedule, etc. 
Organization column model

We've now conceptually finished the sign up. Now repeat. Organize this all in a big fat spreadsheet. Column model in the sheet is:
  • Component
  • Component Breadcrumb
  • Hours
  • Assumptions, reasoning, justification, whatever
If you want you can also add the following:
  • Phase - allows for scheduling in releases
  • Change - allows you to know if the item is a change per the client given original data
Possible component breakdown areas 

Each component section can be though in these areas as possible breakdown areas:
  • Page design, UI flows and site templates
  • Page to data integration
  • Controlling logic, or server processing
  • Data modeling 
  • Errors, exceptions, monitoring and logging
  • Performance
  • Caching
  • Security
  • Scheduling
  • Pure integration (might be a duplicate of server processing)
  • Go live, or release preparation
What about testing? Tests are a true aspect of all work, vereses security or performance, sometimes they just dont matter, but testing always does. Be sure to pad the time the developers will need to regression test their work. 

What about code reviews? Same as testing, pad the time the developers will need to get their work reviewed and incorporate changes. 

Finally

Next, load into ticketing system in batches per timing or major release, assign to developers, tell them to record actuals, then learn from what you missed and apply next time. I do not recommend loading all breakdown work into your ticketing system prior to the schedule time it can be done, use a few weeks as a cycle. Likely the work will change, and you will have a terrible amount of clean up to do on the ticketing system. 

Appendix: The work changes?

Oh yeah, version your sheet per major adjustment. Either naming, or literal version, you are going to need to be able to see the evolution of your break down estimation as you go. Similar to the loading issue, you might want to stop breaking down work until you are nearer to actually doing it because there is a likelihood it will change. 

Appendix: Don't over distribute work

Task breakdown and estimation might lead to the thought of "many developers progress more quickly to the goal line". Big mistake. Understand order, grouping and dependencies between tasks. Keep a person within a component, or two, not 8 people. You dont need 4 guys writing the sign up form. 

Appendix: Credits

I learn a lot from the people around me. I was helped a lot to gain this knowledge over some time and terrible experiences. I want to thank those people, but I won't mention them here. I will stick with my typical nameless and vague posts. If they demand credit, they can:
  1. Read this blog
  2. Ask for credit
Then I will gladly update the post. 

No comments:

Share on Twitter