Short Cycle Development
In the previous 2 releases, 4.0 and 3.5, we had a very, very long release cycle. 4.0 work was started in January 2012 for some people and everyone in the department shifted to it in early March. We released it in November, giving us 11 months of coding and grinding without actually giving the world any value in that period.
The second part of the problem was the way we were dealing with making features and stabilizing them. Developers would frantically make features ready for “Feature Complete”, at which time QA would dive in and find a bazillion bugs. This then quickly diverged into a war of bugs, where developers would be trenched in bug fixing for months. Literally from July to November, bugs were the primary thing everyone looked at, which turns a release into a death march.
We wanted to get out of this vicious cycle and at the same time improve the quality of the product.
Enter Short Cycles
Our solution to the problem was to push our entire cycle down to 2 months. 2 months from the first alpha to the final release. In reality the cycle is longer, since the features are obviously being built before the first alpha, but those 2 months is the rough timeframe we have between releases. There’s a whole range of consequences of this approach:
- We don’t know what is in each release. If stuff is ready for an alpha, it comes in, otherwise it goes to next release.
- We have overlapping cycles. Development of 4.2 has been ongoing for a long time while we are stabilizing 4.1.
- We get features out to users much more frequently than before. This gives us better feedback on the individual feature, thus better test coverage.
- Our betas are 100% feature complete. It is absolutely out of the question to introduce feature once in beta. This was not the case before when a release could be 4 months away.
- Our beta testers experience a better stability, thus are more willing to convert, thus better coverage.
Another benefit we reaped from this is that QA are now having their hands on features ahead of the alpha release. Previously we simply weren’t staffed for it, but this has changed and we are now having time to weed out the worst bugs before the alpha crew gets them. Our internal demands for getting stuff into trunk is simply higher than previously.
First I’d like to show you a “burnup” chart of the effort on 4.1. As you see the work started a long time ago and the bug fixing naturally takes off for real when we start the alpha cycle and slows down when we reach release candidates and raise the bar massively for what can enter. About 330 bugs have been fixed during 4.1
A lot of our test coverage comes from our fabulous community and we take them very seriously. As you see in the below chart, we have handled 97% of all reported incidents, this number will be 100 very soon. We have recieved 231 incidents in total, converted 181 to be reproducible bugs and 69.4% of those are either still active or have been resolved as fixed. As a bonus info I can say that our communities account for 36% of all the active/fixed bugs found in 4.1, thank you very much for that!
Enter the burnup for 4.2. This cycle is more representative of how it will go in the future, since 4.1 was our very first and lots of changes had to happen.
It’s interesting to notice that we released 4.2 alpha 1 a few days ago and we already have 200(!) bugs fixed in this release. Before even getting the first user on it, we have shaved a massive pile of technical debt away. This is exactly what we wanted to achieve!
More to come…
As you can tell from the above, we’re full speed ahead on version 4.2, which will include more good features for everyone and the astute reader will probably figure out that we are also having lots going on already for 4.3 and 4.4. This is going to be an exciting year. My personal goal is that we deliver the highest quality game engine you will find on the market and the quest for that continues ever onwards.