Understanding test executives

Contents
• Introduction
• Control test operations
• Instrument control has many levels
• Flexibility can be confusing
• Advantages and disadvantages
• Selecting a test executive
• Remember the little things

Introduction
When you have to develop a test system, much of your effort will involve creating, documenting, and supporting the test software. Software functions such as test sequencing, limit checking, and data logging occur in all test systems. By standardizing these tasks in programs, test engineers can significantly reduce test development and maintenance time. A test executive (TE) will help in this effort.

A test executive provides a core set of common software functions that you can use throughout the test development process. It contains the software that manages the test environment and provides a structure for all your test plans. A TE is like an operating system for your test station, and it includes the test station's interfaces to the outside world.

Test executives can be used to automate test at many stages of a product's lifecycle, from design verification testing in the R&D lab, through pilot production, to board-level functional and final test on the factory floor. Common applications for test executives include functional test in high-volume manufacturing (such as cell phone test), where performance and reproducibility of test are critical; in high-mix manufacturing (such as contract manufacturing), where production lines need to be quickly changed over to produce new products; and in applications where quality and quality tracking are critical (such as aerospace and medical electronics manufacturing).

Figure 1 shows some of the users of a test executive, and the work items they produce using the TE. The people involved with the test process require different types of test data and generate different types of outputs. Some will determine test strategies, choose hardware and software components, write software to integrate the components and link them to the enterprise, and link the test system to enterprise systems such as databases. Others implement specific test plans for specific units under test (UUTs), which involves interpreting the product specifications, implementing fixturing and wiring, writing drivers and test code, creating the sequence of tests or test plan, and debugging the system.

You and your colleagues may perform several merged functions and work well together as a team, but without a well-defined process for controlling the flow of information and data, you will have inefficient operation, poor documentation, and untraceable results. A TE can provide tools to help you define and enforce a smooth, well-documented flow of procedures and data.

Developing a test plan is one of the primary uses for a TE. Usually you base your test plan on test specifications that define measurement parameters and pass/fail limits. The test plan may be developed iteratively, as the specifications of the test and of the product to be tested (UUT) fluctuate during its design cycles. To produce a test plan using a TE, you select routines from a library and use them to perform test setups, measurements, and other required actions. You also specify the pass/fail criteria for the test, including specific limits. The test executive stores the results (measurement values and pass or fail) for further processing.

The routines will be stored in either the TE's libraries or in libraries that you and your colleagues create. If you can reuse routines from libraries supplied with the TE (or available separately as options), or if you can easily store your own test routines for later reuse, you will find that developing each new test plan goes faster because you are developing less new code each time. You will also find that your test systems are more maintainable, because the test code is consistent and time-tested.

Some TE's have search capabilities that make it easy to find and reuse low-level routines (such as instrument setup and measurement routines). Some test executives let you reuse entire test plans with different sets of limits or parameters - for example, one set of limits for environmental tests and another set for quality-assurance tests. All these features make it simpler for you to modify your existing test plans for a new UUT, and make your test systems more reliable and maintainable over time. Code reuse is particularly critical to produce supportable test applications for worldwide deployment in a global manufacturing environment.

When you can't find a needed test routine in the libraries, you'll have to develop a routine and add it. The TE may restrict the programming languages you can use. It may support only one or a short list of languages, or it may require you to implement a defined interface – such as a specific set of parameters to a DLL or COM interface. If the TE has its own test-routine-generation language, then it probably has a built-in development environment. When a TE supports a variety of languages, those languages usually provide their own development and debug environments.

It is important for you to decide how much variety of language and style you want to support in your test environment. Your TE may need to support many languages to allow the use of code that you inherit from older test systems, code that is passed to you from the R&D lab, or code developed by previous manufacturers of the product (if an acquisition or a contract manufacturer is involved). However, it is useful to impose a degree of consistency wherever possible, to facilitate maintenance and to allow one test engineer to reuse another's work. Even when two engineers use the same programming language, their style can vary greatly. Some TEs include optional tools, such as code generators, that help impose a consistent style without precluding the use of legacy code. Of course, you may choose to adopt a couple of languages because of their suitability to different purposes – for example, Visual Basic for user interfaces and C for performance-critical measurement code.
Back to top

Control test operations
Test executives can be used to control test operations such as sequencing. Test sequencing involves more than simply stepping from one test to another. For example, the results of one test (pass or fail) may determine which test is done next. More elaborate procedures may require the program to branch to different locations in the test procedure, or even to change the test setup, depending upon the value of the result. Looping - another technique - lets a test fail multiple times before the TE determines that your UUT is bad.

Almost all test systems have switching elements to connect the ports on the test fixture to the measuring instruments. Some test executives have features that make switching easier to implement, such as tools to define switch paths that have meaningful names (Figure 2). These switch paths can then be used by the setup and measurement routines without having to deal with specific switch closure commands.

A TE can reduce test plan debugging time by letting you step through the test plan and set breakpoints within the setup and measurement routines. If you stop the program at a breakpoint, you can control instruments manually or loop on a measurement to see immediately how your modifications affect test results and how repeatable those results are. You can also trace variables to find software problems such as incorrect calculations, or trace I/O activity to find logic errors in the commands you are sending to an instrument or to the UUT. Some TEs also provide tools to help you measure and optimize the performance of your testplan. So, TEs can make it easy to fine-tune test and measurement procedures. Figure 3 shows a TE screen used for debugging.

External software applications such as databases, spreadsheets, statistical quality control programs, diagnostics, and word processors are often used to process test results. The TE provides an interface to these applications, often using industry standards for either programmatic (e.g., COM) or data (e.g., XML) interfaces.
Back to top

Instrument control has many levels
Your test executive must handle a hierarchy of common instrument control software. At the highest level, the test program for a specific UUT may say "Measure parameter X." That command goes to the measurement software, which in turn instructs the appropriate source, measurement, and switch drivers to have their devices perform the required functions.

At the lowest level, each instrument's software interface sends its commands in its own data format – such as GPIB or SCPI (Standard Commands for Programmable Instruments). The TE will often have a menu of measurement modules or some other simple tool to help you select the instrument and the measurement routine. Some TEs will also let you send commands directly to instruments without writing code.
Back to top

Flexibility can be confusing
A test executive can save you a lot of time and trouble by helping you develop and manage test plans and tests, communicate with instruments, and integrate your test with external applications. It's important that the TE be flexible and powerful enough to do all these things well. However, all this functionality can lead to a very complex and confusing software package! So it is also important that the functionality be well organized, and that the TE provide tools and shortcuts to make your most common tasks easy.

You should be able to see the information you need without spending a lot of time rearranging windows. The normal flow of control of a test sequence should be obvious from reading the sequence. Errors and problems should be readily apparent. Figure 3 shows a screen with a large, but typical, amount of information displayed in several windows of a TE.

The TE should make many tasks possible, but should make common tasks simplest. Since creating, modifying, and debugging test plans is extremely common, the TE should make those tasks simple and provide tools to assist with them. Since it is usual to test several UUTs in parallel, the TE should provide you with tools to easily configure the number of UUTs being tested by a test plan. Since you often need to add or reconfigure an instrument in a test system, the TE should help you do that, and should not require you to modify and recompile code to accommodate configuration changes.
Back to top

Advantages and disadvantages
As with any tool, using a TE offers both advantages and disadvantages. The advantages include:

• fast test system development because much of the material you need resides in the test routine libraries and in tools such as switch handlers and multiple-up interfaces;
  
• a common look and feel for all users since they all use the same resources and can reuse interfaces, testplans and measurement code;
  
• easier maintenance due to common structure and shared code;
  
• performance optimization due to performance-tuned execution engines and profiling tools

Among the disadvantages are:

• inferior performance in a few cases, where the imposed structure of the TE is not as fast as hand-crafted code for a specific application;
  
• programming language or operating system limitations: you may have difficulty finding a TE that is compatible with your existing software;
  
• difficulty in customizing, though not necessarily for technical reasons: often all the users want their own screen layout, icons, report contents, and specific formats. Reconciling these conflicting demands may present problems.

Selecting a test executive
If you decide that the advantages of using a TE outweigh the disadvantages, your next step is to find a TE that will work with your application. The key to selecting a TE is to specify the operating environment properly.

The environment includes the operating system and the interfaces to other applications - such as manufacturing databases for error tracking or for factory automation. Often, you won't have a choice since your corporate test strategy or IT strategy may define the environment. The environment also includes the users of the test software – test engineers, technicians and operators – and tools that they use, such as compilers and bar-code readers. You should thoroughly understand which elements of the environment are fixed and which are negotiable.

Next, determine if the candidate TEs support standard, familiar programming languages such as Visual Basic and C/C++. If a prospective TE has its own proprietary language or requires the use of a language unfamiliar to your test engineers and programmers, you may want to eliminate it from consideration. Then, does the TE support the language(s) in which you have already developed test code? Will you have to change much of your code to integrate with the TE's calling conventions, error handling, and other services?

Does the TE require programming every time you want to change a test plan? Programming can be kept to a minimum with reusable libraries. You can also save a lot of programming time by taking advantage of menu-driven access to common functions such as switch handling, multiple-UUT testing, and control of message-based instruments.

The TE you choose should merge into your testing operation without disrupting it. To ensure a smooth transition into the test environment, you may want to retain some applications and functions that were effective in your previous environment. Perhaps you developed some of the applications for special requirements, and modifying TE software for those needs isn't economical. Or perhaps many users find an existing application "comfortable", and there is no need to switch.

If you want to reuse some applications, look for a TE that will interface directly with them, or that will provide standard data interchange formats such as XML. You may also want a TE that can work with your existing database applications so you can maintain a test history database. The test-history database will prove useful for statistical analysis as well as when a product arrives for warranty repair.

Pay attention to the instrument I/O support provided by the TE. Make sure it supports the standard instrument I/O libraries and drivers - such as VISA, SCPI, and VXIplug&play – that you want to use. If not, you may need to develop instrument drivers yourself.

Determine the flexibility of the user interface: Does it meet the demands of all potential users? The engineers who integrate test systems and create test plans need a powerful user interface that nonetheless makes it simple for them to perform their most common tasks. The operators on the factory floor need a very different user interface, usually a simple screen that allows input from barcode readers and button-presses and that is not cluttered with unneeded data (Figure 4). You should be able to customize the user interface of your test executive to meet the needs of your engineers and operators. If you can customize or write your own user interface using a standard, easy programming language such as Visual Basic, you will be able to meet changing demands of multiple users.

Remember the little things
Considerations that may seem secondary, such as support for reusability, can be significant. If the test executive doesn't let you easily share common test functions among different test plans, you'll have to do extra work each time you produce a new plan. You should be able to easily search libraries of test routines and retrieve even the simplest segments of code for reuse. This is a feature you may not notice much at first, but by its nature, it saves you more and more effort as time goes on. Code reuse is not a little thing.

Make sure you can easily upgrade and expand your TE. Does the vendor have a track record of improving the TE and its libraries? Is the vendor conscious of your needs, and do they make improvements to the TE that really improve your job as a test engineer?

Is the TE's documentation usable, searchable, and helpful? Are there tutorials, code examples, and training classes to help you get started quickly? Does the TE help you document your measurements and test plans so that others can reuse them without calling you for help?

Does the vendor provide support, not only in your geographic location but everywhere you may want to deploy your test plans?

In summary, analyzing the test executive requirements up front allows you to make the right choices - resulting in faster test system development and lower costs. Test engineers can concentrate on writing the test routines specific to the device under test and not waste time developing the same software over and over again.
Back to top

Author biography
Laura Johnson holds a B.S.E. in computer engineering from the University of Michigan and is an R&D project manager at Agilent Technologies in Loveland, CO. She has worked for Hewlett-Packard and Agilent Technologies since 1986 on projects related to communications test, network management software, and test system software. In her spare time, Laura enjoys hiking in the Colorado mountains and volunteering in raptor rehabilitation programs.