With existing networks heading towards the limit of their
capacity, the need for a new Internet architecture – sometimes referred to as the
Future Internet – is
becoming critical. Future Internet is the motivation driving breakthrough infrastructures,
applications, and services that will make it possible to continue growing the Internet
and lay the groundwork for new models we need to help business move forward –
whether in the cloud, through more social applications, or via mobile.
But how
do we develop applications for the Future Internet, and more importantly, how
do we make sure these applications are reliable, secure, and work as intended?
FITTEST, a European
project (2010-2013) aimed
at developing an integrated environment for the automated and continuous
testing of Future Internet applications was created to help solve this
challenge. As partners in the project, IBM researchers in Haifa, Israel
are contributing unique testing tools based on combinatorial test design (CTD),
and using the project’s tools and techniques to test a simulation system for
cloud management.
What is CTD?
Testing applications or services is all about examining how
the system will react under different scenarios. The difficult part is making
sure you take into account all the different situations that could possibly
arise. If you do more intuitive test planning, there’s always a chance you’ll
forget a good number of scenarios due
to the massive number of combinations that need to be considered. CTD
helps testers systematically model the gamut of possible scenarios to be
tested, and then works to optimize the number of tests so they cover these
situations.
Say, for example, you want to test a banking application
that allows users to carry out financial transactions on the Internet. These
transactions need to be examined to make sure they transfer the right amount of
money from the correct account, at the right time, to the right place. Beyond
that, you would need to verify that the application works correctly even if the
web site falls under cyber-attack, in which massive amounts of requests are
sent at the same time and the system crashes because of the overload. This is a
case where CTD can help make sure all possible scenarios are covered.
Aside from validating that testing is effective and of the
highest quality, it’s essential that the effort involved does not become
impractical. Another great benefit of CTD is its ability to dramatically reduce
the amount of testing required. IBM researchers found that CTD has the
potential to cut testing from an order of 10,000 hours to 100! In our
work with a large European insurance company to test their Internet
applications, it would have taken months to do the thousands of tests defined
in their testing plan. Using our CTD methods and tool, we managed to reduce the
plan to 30 tests – which were carried out in a matter of hours.
How does it work?
A few years ago, our team decided it was time to tackle the
challenges that plague the testing process. There were too many combinations to
deal with and it was taking too long to test all of them. We wanted to control
the risks we took by leaving out certain tests, yet be sure we covered all the
important scenarios. The solution was to find a method that maximizes the value
of each tested scenario so we could reduce the number of tests, while
controlling the risk of leaving out important scenarios.
To balance the cost and risks, we first define a systematic
testing plan that outlines all possible tests and then choose a subset of tests
that cover all the interactions of variables at some level of interaction
(pairs, three-way, etc.). We developed a CTD algorithm that identifies a small
test plan that covers 100 percent of a given interaction level. The end result
is what we call
IBM FOCUS
CTD - the IBM Functional Coverage Unified Solution.
As part of the FITTEST initiative, we contributed the IBM FOCUS CTD tool to help generate high quality, optimized test plans. IBM FOCUS
is independent of an application's domain and can be applied at different
levels of testing. It can also read existing tests, analyze their functional
coverage, select a subset of the tests that maintains the same coverage, and
generate new tests to close the coverage gaps.
What’s it good for?
CTD can help businesses carry out testing in all kinds of
industries, whether banking, health insurance, retail — essentially, any
business that involves a web service. CTD is especially effective in domains
where the testing efforts require huge amounts of resources and are expected to
cover large areas of complex business processes.
“I
believe we’re seeing a new revolution in the area of testing. CTD bridges
between formal methods for advanced software specification ... and the current state-of-the-practice in testing.”
-- Eitan Farchi
|
CTD bridges between formal methods for advanced software specification—which
is difficult for most people to use—and the current state-of-the-practice in
testing. Our workshops and training sessions have shown that anyone can use
this method, even people who are not programmers. We can teach people to use our
tools within a few hours and start to model in just one business
day.
The Future Internet will be a complex interconnection of
services, applications, content, and media, offering a rich user experience
beyond current hyperlink-based navigation. With society becoming increasingly
dependent on these services for business, entertainment, government, and
education, it’s vital that the applications running on top of the Future
Internet meet strict standards for quality and reliability. IBM FOCUS CTD is
one tool that is helping make this a reality.
If you're interested in finding out more about IBM FOCUS CTD, contact the author at farchi@il.ibm.com.
