Best Practices – What does it really mean?

It is so interesting to me that everyone talks about wanting training in “best practices”. This term is often used to describe the kind of consulting we might provide as well. What does that really mean? I believe all of us have fed into the myth that best practices can be the basis for training individuals. Best practices are not trained, they are experienced.

If we compare the best practice approach to nature, we know that it not only the strongest but the most prolific species that survive. That’s one of the reasons it is so difficult to change processes in an organization. Think about that for a moment. The strongest and most prolific probably describe the majority of individuals you have in just about any group in your organization. I have seen this over and over in the System Engineering field. The strongest and most prolific engineers have often been doing their job for many years and have a specific way they do this job. Asking them to try new techniques and tools is often futile, as they don’t know how this is going to improve the already wonderful job that they are doing. Their practice is going to survive if we continue to shove a best practice approach at them.

So what do we do? Best practices start with our own personal experience. All of the books you pick up and read around best practices are about the author’s personal experience in some area, whether it is use cases, requirements elicitation, system engineering, testing, etc. At some point an organization decides that it needs a consistent process across the organization. There are lots of good reasons for doing this. So a group is formed to come up with a process that reflects these best practices. Since there are many best practices in the group, guess who usually wins? The strongest and most prolific. And the resulting best practice usually results in a compromise between the best practice group and those in the organization who perform this role. Studies indicate it takes about five years to actually role out this process across the organization. Then along comes the consultants who notice these practices. They may see an opportunity for business and so they get behind the practices that will lead to more business for them. They begin to show up in conferences and trade shows touting these practices and writing books. (Think about the Agile approach.) Because they are looking for business they are focused on the practices that will provide the most business for them. We are always in search of best practices so we don’t need to know how often it works, just that it CAN work. And so we continue to push that practice throughout the organization, hoping to realize all of the benefits that have been talked about over the years.

We must remember that best practices are based on hindsight. The best practices for your job are developed as you learned about that job. We can certainly learn from each other. That is not the point of this discussion. The point is that we need to base our best practices primarily on our own practical experience. We must be discerning in how we apply best practices to our jobs and our organizations. Just don’t accept best practices because someone wrote a book about it or spoke about it a conference. Research the practice and determine the environment in which they were fostered. Is this environment comparable to the one you are in today? Often there is little commonality. Choose the tidbits from the best practices that apply to you and your job and will provide real benefit to you. Start with slight changes and build on the practice as it is applied in your organization. Know where it has succeeded and where it has failed and learn from this. Remember that if you have strong and prolific objectors to the practice, you will need someone with a practical understanding of the practice to help the objectors see how this practice can help them. Listen to them and address their concerns. Don’t just ignore them and hope they will go away. They will not.

In other words, just saying the words “best practice” does not mean it will be a best practice for you.

By: Marcia Stinson

Non-Functional Requirements – they can bring a project to its knees

Often during system development we become very focused on making sure we meet all of the functional requirements for a system. Although functional requirements are indeed very important, they are not the only way a system will be evaluated by end users and business owners. If a user can indeed retrieve a credit card statement, but it takes five minutes to do so, the user will likely not be happy with the product.

Consider this example. In a book called “Angle of Attack” by Mike Gray he discusses the impact of requirements on the development effort to send the first man to the moon. He mentions two non-functional requirements that came from President John F. Kennedy. First, was that the astronauts were returned safely. (Seems reasonable to me. Who would volunteer to go if that weren’t a requirement?) Second, that the project would be completed within the current decade. Reading this book made me realize how a simple requirement like “during this decade” can change the entire scope and focus of the development effort. In this case, instead of going through normal system engineering practices and performing risk analysis, all of the development efforts were focused on finding a solution quickly through rapid trial and error. The entire process changed based on that single requirement. The requirement to bring the astronauts back safely resulted in a whole new project which involved getting the astronauts off the moon and back into the spaceship to return home.


I have my own example of the impact of non-functional requirements. When working on a weapon control system we had a reliability requirement. When we began looking at the reliability we needed to achieve we found the only way to achieve the required reliability was to add a second processor as a backup. It seemed like a reasonable thing to do. In the end it became a nightmare. The effort involved in backing up data and processes, and keeping track of the exact state at the time the processes and data were offloaded was very tricky. Testing this was a whole different problem. There were so many possible permutations of the situation that it was impossible to test them all. The effort required to support this single requirement was much more than any of us anticipated.

The more non-functional requirements there are in a system, the more expensive to both develop and test and the higher the risk. In particular the requirements that affect the entire system have the biggest impact on schedule and cost.

So what do we do with all of those non-functional requirements? It just isn’t feasible to just throw them out, of course. The best we can do is look at each non-functional requirement carefully. When users specify non-functional requirements ask them why this is important. Verify that the requirement is really necessary by understanding its importance to the end users. Verify that the pass/fail criteria are valid. In some cases users will say something like the system must be available 100% of the time without thinking about what they are saying so ask if this is a hard number or what the user would like to see.

When users state non-functional requirements like “user friendly” don’t just let that little requirement slide into your document without understanding what this means to the user. Ask how the user will test the system to determine it is user friendly. With the user, create criteria that can be used to determine if the requirement is met. Documenting these criteria up front with users will make sure everyone understands the intent of the requirement.

In short, make sure you know why the non-functional requirement is important to the user and how success will be measured. Getting this information up front will help you create a better solution.

By: Marcia Stinson

Implementing a Requirements Management Tool on a Complex System

Many years ago I spent several years working on a very complex weapon control system. As you can imagine the requirements were large, complex, and changed often. We spent a lot of time just trying to manage those pesky changes that continued to be submitted, both from customers and from the developers. In those early days, we did not have any requirements management tools to help use assess these changes. We were using Interleaf and Excel (I can hear groans of pain now). Everything was manual, including our complex traceability. We had a couple of folks who did nothing but maintain the traceability matrices and assess the impact of changes. At this time we only had traceability from the Concept of Operations to System Requirements to Subsystem Requirements. I say “only” but at that time just having this level of traceability was a big accomplishment.

When we had enough changes we issued a new system requirements document and new subsystem requirements document. Those poor contractors had to go through the massive subsystem requirements and manually determine what had changes. I can’t imagine the time the contractors spent just trying to figure out what changes they needed to be concerned about.

It was in the middle of this upgrade project that the customer said enough and tasked my team with evaluating and selecting a requirements management tool. The tool we selected is not important to this particular discussion, but what we learned from this tool selection and implementation is important. Here are some lessons learned.

(1) - There is not a single tool which is going to please everyone. We had users who loved our selection and those who fought us every step of the way. Without a customer supporting and enforcing the change it would not be possible on a large program like this one. One user complained about the column size of the tool generated traceability matrix, totally ignoring the fact that it saved him days of manual effort.

(2) - Our manual traceability was not very clean. Once we imported all of our information into the tool and linked it up we found many gaps in the traceability. What was more disturbing was that we had links that really didn’t make any sense. We had to do a lot of work to clean up our traceability matrices.

(3) - Just tracing requirements was great, but now we could use the same effort to link requirements to test plans, and went so far as to link subsystem requirements to design documents that we could review. This didn’t happen overnight, but it did happen. Eventually we could trace system requirements to a subsystem requirement to a design document to a code module. We even used a tool to determine the complexity of code modules and used this to help determine how difficult a change would be to implement and test.

(4) - Metrics from a requirements tool are key to understanding completeness of testing activities. We often thought we were 50% complete with testing. After all, 50% of the tests were completed. However, what we found was that we were prone to testing the simplest and most understood parts of the system first. So even thought we were 50% complete, everything left was very high risk. We learned to prioritize our testing by looking at requirements priorities and software complexity, information we could not determine through manual traceability.

(5) - It was very easy to get overwhelmed. Start simple. We had to back off our ambitious ideas and begin with a simple traceability model. As we learned and gained more experience with the tool, we added more information to our model. We were constantly assessing our process to figure out what else we could do to make it better.

- Don’t skimp on training and mentoring. We trained everyone on the project and created experts who helped users get over initial hurdles. We sent our experts to our contractors for weeks at a time to help them get up to speed in using the tool. We even had our own internal users group. Be prepared for this kind of effort.

What a great leargning experience this was for me. If you´re interested in embarking on a change like this to improve your requirements process, contact Visure Solutions. We will be happy to discuss your process with you.

By: Marcia Stinson