Subscribe to DSC Newsletter

Agile Business Intelligence (BI):

by Kesavan Hariharasubramanian

The aim of any Business Intelligence (BI) initiative is to help an organization to review its business, its competition and its external environment from time to time and take more smarter and faster decisions not only for survival and sustenance but also to stay one step ahead of its competition. Or in other words, BI serves to keep an organization agile and vigilant at the same time.

There is a fundamental difference between a typical application development project that we routinely come across in a software industry and a BI project. The difference is that the requirements in an application development exercise are well-defined and static by nature; whereas the requirements in a BI project are more amorphous and dynamic since these BI projects are closely aligned with the business goals. As the goals and objectives change in response to the demand-supply economics at a given point in time, the expectations from BI are also bound to change. For instance business consolidation arising out of mergers and acquisitions could lead to a total revamp of an organization's goals and objectives.

Few days ago, I came across an article on Agile Software methodology and it was very interesting indeed to note how similar its principles are to that of BI project methodology - especially certain principles enshrined in Ralph Kimball methodology. Not only that, the Agile approach also throws ideas which could be deployed in a BI implementation. Before we list down these similarities and examine these ideas, let us examine in brief what this Agile software development is all about.

Agile Methodology:
Agile software development is a methodology that promotes:- a project management process that encourages frequent inspection and adaptation- a leadership philosophy that encourages team work, self-organization and accountability- a set of engineering best practices that allow for rapid delivery of high-quality software and- a business approach that aligns development with customer needs and company goals.

The Agile approach chooses to do things in small increments with minimal planning, rather than long-term planning. Iterations are short time frames (known as 'timeboxes') which typically last from one to four weeks. Each iteration is worked on by a team through a full software development cycle, including planning, requirements analysis, design, coding, unit testing, and acceptance testing when a working product is demonstrated to stakeholders. This helps to minimize the overall risk, and allows the project to adapt to changes more quickly. Documentation is produced as required by stakeholders. An iteration may not add enough functionality to warrant releasing the product to market, but the goal is to have an available release (with minimal bugs) at the end of each iteration. Multiple iterations may be required to release a product or new features.

Team composition in an agile project is usually cross-functional and self-organizing without consideration for any existing corporate hierarchy or the corporate roles of team members. Team members normally take responsibility for tasks that deliver the functionality of an iteration. They decide for themselves how they will execute during an iteration.

Agile methods emphasize face-to-face communication over written documents. Most agile teams are located in a single open office to facilitate such communication. Team size is typically small (5-9 people) to help make team communication and team collaboration easier. Larger development efforts may be delivered by multiple teams working toward a common goal or different parts of an effort. This may also require a coordination of priorities across teams.

No matter what development disciplines are required, each agile team will contain a customer representative. This person is appointed by stakeholders to act on their behalf and makes a personal commitment to being available for developers to answer mid-iteration problem-domain questions. At the end of each iteration, stakeholders and the customer representative review progress and re-evaluate priorities with a view to optimizing the return on investment and ensuring alignment with customer needs and company goals.

Agile methods emphasize working software as the primary measure of progress. Combined with the preference for face-to-face communication, agile methods usually produce less written documentation than other methods. In an agile project, documentation and other project artifacts all rank equally with working product. Stakeholders are encouraged to prioritize them with other iteration outcomes based exclusively on business value perceived at the beginning of the iteration.

Similarities between Agile and BI Methodology

Ralph Kimball's paradigm states that a Data warehouse is the conglomerate of all data marts within the enterprise. In reality also, most DW implementations in enterprises across the world align with Ralph Kimball's idea. This is because most data warehouses start out as a departmental effort, and hence they
originate as a data mart. As more and more data marts are built later, they evolve into a data warehouse (DW).

So, as we see here, this data mart approach is similar to the incremental appproach adopted in Agile methodology where one working prototype is demonstrated to the stakeholders first followed by several iterations to add various functionalities. This not only serves to demonstrate the benefits of a DW to the
stakeholders but also mitigates any risks of abandonment of initiative by securing their buy-in and commitment to the BI initiative.

Now let us take team composition for a BI project. The team typically comprises people representing all the stakeholder functions across the enterprise since a BI initiative cannot progress without the connivance and acceptance of all stakeholders. Thus, as observed in the agile approach, the team is necessarily cross-functional by nature.

Owing to the changing nature of the requirements in a BI project, it is imperative to ensure constant communication flow among the stakeholders of the initiative. Regular face-to-face communications are recommended in order to help make team collaboration easier and ensure continued support from the
stakeholders for the BI exercise. Also quarterly reviews are essential to take stock of changes in the business, re-evaluate priorities and accomodate them using well-defined change management procedures.

Ideas from Agile Approach for effective BI implementation

Let us now turn our attention to some ideas that we could borrow from Agile methodology to better manage BI implementations. Extreme programming is an agile methodology that is gaining currrency nowadays. Proponents of extreme programming regard on-going changes to requirements as a natural, inescapable and a desirable aspect of software development projects.

There are two practices in extreme programming that could be considered for use in BI projects to reduce the costs and increase efficiency of BI implementation. These are: 1. Pair Programming 2. Test-driven development

Pair Programming

Pair programming is a software development technique in which two programmers work together at one keyboard. One types in code while the other reviews each line of code as it's typed in. The person typing is called the driver. The person reviewing the code is called the observer or navigator. The two programmers switch roles frequently (possibly every 30 minutes).

While reviewing, the observer also considers the strategic direction of the work, coming up with ideas for improvements and likely future problems to address. This frees the driver to focus all of his or her attention on the "tactical" aspects of completing the current task, using the observer as a safety net and guide.

Some of the benefits that accrue from this technique are as follows:-

Design quality: Better designs, fewer bugs. Pairs typically consider more design alternatives than programmers working solo, and arrive at simpler, more-maintainable designs, as well as catch design defects very early.

Reduced cost of development: With bugs being a particularly expensive part of software development, especially if they're caught late in the development process, the large reduction in defect rate due to pair programming can significantly reduce software development costs.

Decreased management risk: Since knowledge of the system is shared among programmers, there is less risk to management if one programmer leaves the team.

Fewer workstations required: Since two people use one workstation, half as many workstations are required.

Test-driven development

Test-driven development requires developers to create automated unit tests that define code requirements before writing the code itself. The tests contain assertions that are either true or false.

Or in other words, in test-driven development, each new feature begins with writing a test. This test must inevitably fail because it is written before the feature has been implemented. If it does not fail, then the proposed “new” feature is obviated. To write a test, the developer must clearly understand the feature's specification and requirements. The developer can accomplish this through use cases and user stories that cover the requirements and exception conditions.

This could also imply a variant, or modification of an existing test. This is a differentiating feature of test-driven development versus writing unit tests after the code is written: it makes the developer focus on the requirements before writing the code, a subtle but important difference.

Given the frenzy with which changes are rippling across the business landscape today as a result of globalization and removal of trade barriers, it wouldn't be wrong to conclude that 'agile business intelligence' has truly come of age.


Views: 1102


You need to be a member of AnalyticBridge to add comments!

Join AnalyticBridge

Comment by Matt Warden on May 25, 2010 at 1:32pm
I don't disagree that pair programming and the use of automated tests could be useful to BI. However, most BI development environments can barely be called multi-user. And doing automated tests with BI probably means you're stuck with something expensive like QuickTest Pro at this point. And neither of these will fix what I see as the biggest problem with BI, which is the inability to deliver what the user actually wants. Kimball's books are gold, but they do not remove the need to drive BI solutions from business needs.

You rightly observe the agile methodology available to custom development and wonder why BI is behind in the times. But it's worse than that. Custom development is not nearly as business and time critical as BI development. Missing a requirement in a custom app usually means go-live delays and inconvenienced and/or less-productive end users. But a missed requirement in a BI app means flying blind with decisionmaking, missing new business/market opportunities, and guessing where to allocate funding and where to cut costs.

The market is only now realizing this need. There is a report just out from Forrester Research that talks about Agile BI development approaches, and Forrester underscores the difficulties with the traditional "bottom-up" (data to user) BI development approaches, saying that very few tools get it right and use a top-town (user requirements to data) approach. Balanced Insight has free reprints of this Forrester report:
Comment by John A Morrison on January 25, 2009 at 1:55am
Great piece of work, thoroughly thought through, have you considered the 4th generation BI in terms of Predictive Analytics and the place for the Open Source Community, I think you would find that whole approach most interesting.

Follow Us

On Data Science Central

On DataViz

On Hadoop

© 2017 is a subsidiary and dedicated channel of Data Science Central LLC   Powered by

Badges  |  Report an Issue  |  Terms of Service