Software Design

Engineering vs. Modeling

We often speak of software design as "software engineering." This is unfortunate since software design is not precisely equivalent to engineering. As software developers, we do not work with physical things that can be measured to determine their structural capabilities. Because it is not physical, software is perceived as infinitely malleable. Coding is perceived as just "typing at the computer" and we are expected to be able to alter the system significantly at any time. We must construct a system that is sufficiently rigid to be reliable, but flexible enough to be adaptable. Almost no other discipline has this problem.

It is more accurate to describe software design as a process of modeling. We describe the user's data and the processing to be performed on the data by creating a model of that data and processing. We then attempt to reduce that model to its most concise form. This is, in essence, a form of applied mathematics and should be treated as such.

Software design is thus the process of describing the model of the software system in such terms that the system can be constructed and verified. Design involves answering three questions:

• Why?
• What?
• How?

Why?

You have to know why you are building the software. The answer to why you are building the software is best expressed as a problem that needs to be solved stated from the perspective of a potential user of the software. Market analysis, customer interviews and research into new and existing technology are the activities that help you understand and describe why the software should be created or modified.

Typically, the developer produces a concept document and/or a requirements document that describe why the software is being created. The amount of detail in each document is determined by the size of the problem. For very large systems, there may be multiple levels of requirements specification. These documents may reference technology or software systems, but in general should only speak in terms of the user's problem. This is a subtle distinction that most software developers get wrong. Instead of limiting the description to the problem from the user's perspective, they fill the requirements document with software functions that are not part of the user's problem. For example, a user may have a problem tracking his customers. An incorrect requirements document might state that the user requires a database. In fact, a database is not a requirement; it is a solution. If the user only has ten customers, the best solution is a filing cabinet with a good filing system. A computerized database is a solution to the problem of having too many customers to keep track of using paper records.

The creation of a good requirements document is essential to successful software development. Not only does the document provide the starting point for the whole process, it also serves as a means of checking the validity of the completed system. At the end of development, you should be able to identify some place in the software where each requirement has been met.

What?

Once you have determined why the software should be created, you then can ask, "What functions does the software need in order to solve the user's problem?" This is the point when you must map the user's problem into a set of software modules and functions. This translation is the most difficult step in software design. In order to describe the software system, we have to translate a "real-world" activity into a "virtual-world" function that the computer can perform.

The simplest way to begin the design is with a one-to-one conversion of each requirement into a software function or data structure. This will not produce the most efficient design, but it is a good way to start. Once the list of functions is determined, they can be analyzed for coupling and cohesion and then rearranged into a more effective structure. This process is largely intuitive and requires a trial and error procedure. You try some combination, look for potential problems, and then restructure the design as needed. Good designers are usually developers that have been involved in several similar projects before. They can draw on past experience to see how the current system is similar or different and then apply their knowledge of past systems to the current system.

The software functionality is described in a functional specification and one or more design documents. As with requirements specifications, the size and number of documents is dependent on the size of the software system. Small systems, or maintenance projects, may be able to combine everything into one short document. Large systems will require several levels of design documents, including a architectural specification and a module design for each major component. It is important to keep these documents at the proper level of abstraction. In general, specific algorithms and code are inappropriate at this point. These documents should concentrate on major data structures and interfaces and not try to describe the internal operation of each function or class.

At this point in the design process, it may be possible to locate existing software that can be reused to provide much of the functionality of the system. If the development team does use existing software, the interface to that software will control much of the design process. Often it is possible, or necessary, to build adapters between the existing software and the new system. The developer must be on guard against force-fitting existing software into the system. If the match is not close, the adapters and other interface matching problems can create an inefficient, fragile, or unworkable system. The developer must also consider the long-term cost of using existing software systems. Often the developer will have no control over upgrades to the existing software and my find himself limited in choices when future changes to the system must be made.

How?

With a description of the functionality of the software system, you must then determine how the software is to be constructed. This is the point where the designer must develop the algorithms necessary to implement the system.

Traditionally, a developer would take the module design specification and produce a detailed design document. However, modern high-level languages are sufficiently abstract that you can often develop the code directly from the module design documents. In some cases, where the code is unusually complex, it may be better to create a detailed design document. In this document, you describe the algorithms, classes, etc., that will be implemented. Typical design techniques such as flow charts, class diagrams, sequence diagrams, and state machine diagrams are used to express the detailed design.

The major advantage of having the detailed design documented is that it allows other developers to readily understand and take over the code. Without a detailed design document, subsequent developers must study the code, usually poorly documented, and reverse engineer the design. For simple code, this is not a problem. For complex code, the detailed design document can save considerable effort. All too often, the only person in the world who understands the code is the person who wrote it originally. The result is that the knowledge in the head of the original developer must be continuously "dumped" into the head of the next programmer. Knowledge of the code becomes an oral tradition that is easily lost.

Levels of Abstraction

Our human minds have the ability to hold a vast amount of information and to recall that information when needed. Our ability to add to our store of information also seems to be unlimited. However, there is a limit to how much we can hold in our conscious mind at a particular point in time. It is practically impossible to have all the details of a complex software system "in the head" all at once. We overcome this limitation by using abstraction. Abstraction is simply the process of ignoring details so that a very large structure can be contemplated all at once. The three questions of why, what and how can be mapped to levels of abstraction as shown in the following diagram.

As we move from one question to another, we increase or decrease the level of abstraction and the corresponding amount of detail. This is the most fundamental concept in design and it is essential to understand its importance. You can never effectively manage a large design without the ability to separate out the levels of abstraction. The biggest mistake software designers make is to confuse the levels of abstraction. Inserting low level details into higher abstractions limits options in how the design may proceed. Likewise, failing to resolve the appropriate detail at a level of abstraction leaves part of the system without a design and may hide problems that will only be discovered later. The correct balance between abstraction and detail is one of the things that a software designer must learn through experience.

The Waterfall Model and Its Derivatives

As described above, during the development process we work through multiple levels of abstraction. This process of moving from high abstraction to detail is the basis of the standard model of software development, generally called the waterfall model. In this model, you start with an abstract view of the system expressed as the user's requirements. Each step in the process refines the abstractions to ultimately reach the level of code. The following diagram shows this model.

Although the number and names of each level may be different, the basic model is always the same. Each level of abstraction is scheduled and performed in-order. As a general model of software design, the waterfall model is generally correct. However, there are several problems with this model that every software designer eventually encounters.

The biggest problem with this model is that it assumes each level is visited once. In reality, the longer we work on a problem the more complete our understanding becomes. Requirements are specified based on the understanding near the beginning of the design process. As the system is developed, it is inevitable that additional requirements will be discovered. Sometimes this is due to insufficient analysis, but usually it is simply that we don't fully understand what we need until there is some basic system in place. Once the user can work with the system they have a broader understanding of what is possible and can realize additional functionality is required. The same learning process happens between design and coding. We start with what we believe to be a complete design, only to discover that we can't get the implementation to work. In essence, abstraction allows us to grasp large structures by hiding details, but the hidden details also hide problems with our high-level design. Thus, software designers know that they will have to iterate over various levels of abstraction during the design process.

Recognizing that we need to iterate, developers have added cycles, loops, whirlpools, rapid prototyping, and other such refinements to this model. Although the waterfall model and its derivatives express the concept of levels of abstraction correctly, even with iteration these models are misleading in that each implies each level is visited in a linear time-order. Realistically, the process of design looks more like a "random walk" than a stroll down a well-known path. We move up and down as needed, when needed until the system reaches an appropriate level of refinement.

The standard models also imply that design is always top-down, or mostly so. However, we can also work "inside-out" or even "bottom-up" depending on the situation. For example, a developer may start with a proposed module, examine the possible functions it can provide, then search out market requirements that the functions can satisfy. Leading-edge technology improvements often work this way. Likewise, we can start at the bottom with a useful piece of existing programming (e.g. chart drawing) and then develop upward into a commercial product. The bottom-up process is often required when attempting to create a software implementation of a formula or process from some other discipline. The details of the calculations are known in advance, and the programmer must build the system around that specific calculation. The important concept to grasp is that we work at the appropriate level of abstraction, and not that we must design in a straight, top-down manner.

Once you understand that levels of abstraction are a design concept, you can see one of the biggest mistakes in software design. The waterfall model and its derivatives mix design concepts with scheduled activities. The standard software development models all attempt to map abstractions into time and then schedule accordingly. Levels of abstraction are a design concept and are not sufficient to describe the activities of developers for scheduling purposes. In another chapter, I describe a list of actual activities.

An Alternate Model - Multiple Abstraction Hierarchies

Beyond the linear-time ordering and top-to-bottom assumptions, the standard software design models mix more than one type of abstraction. Note how the top levels of abstraction concentrate on user abstractions, while the lower levels deal with software abstractions. The model implies that software design is an increase in level of detail from user's requirements to code. Because of this, there is a very difficult transition from requirements to functional specification. At that point, the designer must reinterpret all the specification into software technology and the mapping between the two may not match. However, once we separate the concept of levels of abstraction from scheduling, we can create better and more useful abstraction hierarchies. The following diagram is one example.

This model shows that the software system can be viewed from the perspective of the user or from the perspective of the developer. The description of the software is not a refinement in level of detail of the user's requirements. Furthermore, the element of time is removed, since when each level is examined is determined by the nature of the software project and also because there will be a natural iteration over the levels of abstraction as the system is developed and extended over time.

Note also that the arrangement of this diagram does not mean that there is a one-to-one horizontal mapping of user abstractions to software abstractions, although some analogies do exist. It simply shows that two hierarchies exist in parallel and both must be dealt with equally.

During design of the system, the user's requirements and functional view of the system have to be mapped onto a software model. The two views of the system are orthogonal. One user function (F1) may use several software modules (A, B). A second function (F2) may share some software modules (A, B) and include additional modules (C). A third function (F3) shares some software functions with user functions F1 and F2, but requires the use of another module (D).

When developing the software, we can take a user function and see if existing software modules can be used to implement the function. If not, we modify the module functionality or add modules to the design as appropriate. Adding a module may involve changes to the architecture as well. Thus, we work up and down the levels of abstraction based upon the demands of the software functions.

Another approach is to improve the software functionality with new technology and then revisit user functions to see if they can be improved with the new functions. The technology additions may create the possibility of entirely new user functions.

This approach leads naturally to a component software strategy. Software design becomes largely a matter of locating the appropriate component to support the user functionality. Much of the design concentrates on interface specification, adapters, and integration rather than attempting to construct the entire system from scratch each and every time. Using this design approach, a single set of components can be recombined in order to produce multiple software systems. Each system, or product, is then a map of the product against the set of software components. Essentially, a third dimension to the diagram shown above.

Software modules can be mapped against other aspects of the system as well. For example, a set of modules may have build dependencies. With a list of modules in the system, the system builder can design a process to build the software by mapping build steps against the set of software components. Likewise, Q/A can build a map of tests against the software modules. Each mapping of this type adds yet another dimension to the model. Thus, software design will usually result in a N-dimensional orthogonal mapping of software components. The more of these mappings the software development team can discover and document during the design process, the better chance the team will have of successful software development.