Smooth test bed integration

DLR

What to look for in cost-effective and safe test bed integration

As you know, you can not only connect systems like ERP and PLM to your test process management system to receive or deliver information about requirements and tests as well as results. You can also connect test beds and other systems to a higher-level system. But since there is no "one" test bed automation system with standardized interfaces, your test bed integration may prove difficult.

Here you will learn how to master typical challenges when integrating your test bed, what solutions are available to you and how to implement them.

Different test bed automation systems pose different challenges

Test bed automation systems differ in several aspects, complicating your test bed integration:

They differ

  • in the execution of your tests and the format of the sequences used for them
  • in the way your test orders are received and processed
  • in the formats in which your data is recorded
  • in the available metadata that is required for a test and is to be edited or supplemented
  • in the type of access to the data obtained on the test bed

For your enterprise-wide test management, there may be additional aspects of diversity that arise from the individual implementations of the processes in your individual departments. This can lead to the fact that even for nominally identically structured systems, you still have to provide for different workflows and data flows in order to meet the requirements of the respective processes - if standardization of the processes is not possible.

Requirements that your interfaces must meet for smooth data exchange

Due to the existing differences between the systems involved in your test process management, a number of requirements arise for your interfaces for data exchange, depending on the type and intended use of the data. These must be mastered if you want to integrate your test beds efficiently and securely and thus optimize your workflows:

Test order delivery

The test execution information collected in the test planning phase must be transmitted to the test bed so that the test bed operator has all the information needed to execute and track the tests. An automated interface that allows transmission without media breaks avoids error-prone manual steps and reduces the time required for test preparation work.

It is important here that the test order must be interpreted by the test bed automation system in the same way as in the higher-level test process management system. Formats and contents must therefore be coordinated with each other.

Data exchange – via file transfer and via REST interfaces

A common way for data exchange is to provide the data in files. The test bed and test planning must use the same transfer directory and a common file format for this. In addition, it must be ensured that no "half-finished" files are picked up during the writing process.

Another, often chosen way is also the data retrieval via the programming interfaces of the test planning systems. These are often equipped with a REST interface or allow direct database access to their dataset.

Consistent data and secure data transfer

In order to be able to compare test data more easily later and to simplify processing in analyses, data harmonization is often carried out in the course of the data transition from the test bed to the test management system. This involves standardizing different file formats, but also making adjustments with regard to naming conventions, units, etc.

Since large amounts of data usually come from the test bed, the data is usually transferred as a file transfer. During data transfer from the test bed, no data may be irretrievably lost due to system failures. It is therefore important to select a secure transfer mode with confirmation of complete and correct receipt and to temporarily store the data until successful transfer is completed.

Synchronizing master data is also essential for data consistency: the test bed and test management system must always use the same master data. These requirements also apply to other interfaces such as data transfer between the test bed and other connected systems, e.g. for the development of test sequences and methods, but also for setups and settings, operating and setup personnel for documentation, e.g. according to ISO 17025, and for resources.

It is extremely important that consistent data is available to the test bed and test management. Only if both use the same data with the same meaning it is ensured that orders can be defined and executed correctly. For this purpose, lists and enumerations can ideally be better used in both systems instead of free text fields, which can then be synchronized between the systems.

Sensible distributed data storage

Where the master data is managed also depends on its type: resources, personnel and equipment can be managed very well in the test management system, where they can be reconciled with other higher-level systems such as ERP if required. Master data can also be transferred to the test bed as part of the test order. Test sequences, test methods and setup information are often held as proprietary formats in the test bed automation.

Therefore, this information is better kept there. The test bed automation system tells the test management system which elements are available and which "capabilities" are available. The main element is always the test order. Which sequences and resources were used to process a test order should therefore always be stored and passed on considering the context of the test order.

Transmission of status information

Especially when many test beds or long-running tests have a significant share, the interface for the transmission of status information from the test beds becomes very important. This includes information such as error messages, but especially the operating status of the test bed and the execution status of the tests. In addition, parameter values can also be useful for status monitoring. And in the case of endurance tests, regular data records are added as intermediate statuses that can be used for trend evaluation.

A good knowledge of the current state of the test bed and the test sequence is of great importance: On the one hand, central monitoring allows less staff presence to be required at the test bed during long-running tests. This means that less staff time is required and employees can concentrate on important other tasks.

On the other hand, the early detection of faults and abnormalities on the test bed allows test beds and devices under test to be protected from destruction in the event of slowly developing material failure. Failed tests can then be detected at an early stage, aborted and restarted. The test bed is thus utilized much more efficiently. The automation of monitoring by means of rules (or even AI methods) also enables predictive maintenance of the test beds.

The interface between the test beds and the test management system can be implemented in different ways. For example, the test beds can use the web service API of the test management system to make status updates there. However, this requires that all test beds implement this. Therefore, an industrial standard interface such as OPC-UA, which is widely used, is more suitable. OPC-UA can be used primarily for a process data server, which can run either at each test bed or also centralized as middleware.

In any case, certain risks must be taken into account when directly accessing the test bed during test execution: On the one hand, test execution can potentially be disrupted if, for example, write access is also permitted instead of exclusively read access. On the other hand, an inadequately secured process server carries the risk of data being viewed by unauthorized persons. Concepts and architectures must therefore be selected for the interface for transmitting status information that take both efficient operation and security into account as important aspects.

Test sequence transfer

In addition to interfaces resulting directly from the main process of testing, there are other interfaces resulting from the accompanying processes. One of these interfaces is used to provide test procedures - commonly referred to as test sequences.

Test sequences are often developed and released centrally. They consist of the test steps that are to be executed in automatic sequence. Before execution, they are often parameterized with variables. Initial basic settings and parallel background monitoring can also be part of the sequences. Sometimes the test sequences are also provided with pre- and post-test steps, e.g. for calibrations or checklists with user instructions for set-up and dismantling.

For this purpose, a central development and release process makes sense from both an efficiency and a quality point of view. This is the only way to establish a good quality assurance process and achieve standardization, which ultimately leads to the desired gain in efficiency.

When centrally developing and managing the test procedures in the test management system, existing process and data management mechanisms can often already be used. Likewise, the sequences and configuration can already be selected during test planning in the test management system, which leads to considerable time savings on the test bed.

For the interface to the test bed, care must be taken to ensure that no conversion of sequences and parameterizations has to take place. Instead, the sequences should be available in the native format of the test bed automation. Alternatively, at least standardized formats should be used which are converted into the necessary formats.

The transfer of the released sequences can be done via network drives. However, since accompanying information is often also required, such as the type of test, other ways are usually more suitable. Either the sequence can be attached to the test order as a file, or - if sequences are to be managed independently of the orders for organizational reasons - concepts of version management systems from software development are used. This allows versions of test sequences to be managed centrally and retrieved by the test bed for local storage.

Transmission of analysis scripts

Analysis procedures can be handled in a similar way to test sequences: On the one hand, derived variables are often calculated online from the measured values at the test beds in order to be able to display and record the relevant parameters. On the other hand, initial evaluations of the data are already carried out at the test beds, e.g. to validate that the test was carried out correctly before the device under test is removed from the test bed.

It is important here that the information from the test definition can be integrated into the evaluation as parameterization. The corresponding scripts for performing the analyses are very similar to the test sequences: they are files that should undergo a release process for quality assurance as part of the development. This means that the same procedures for management and development come into play as for test sequences: the test management can support a release-based development cycle and provides the configuration and parameterization of the analyses for the individual tests.

It makes perfect sense to perform analyses decentrally, because this reduces the load on the test bed and frees it up for further tests at an early stage. Especially when it comes to long-running and computationally intensive evaluations, a decentralized analysis makes sense. However, there are also reasons for carrying out the analysis directly at the test bed, because then it is possible to analyze the data promptly and, if necessary, to repeat a test immediately, as long as the test bed and the device under test are still available. Also, the test bed operator can already evaluate and comment on the information of the analyses before all data are transmitted collectively to the test data management.

Since there is a certain similarity between the processing of test sequences and analyses, the distribution of sequences and analyses can be done via similar procedures. For the execution of the analyses, it is necessary to include appropriate analysis tools, which requires internal interfaces within the automation that are adapted to the tools used.

Updating the test bed software

This interface is similar to the interface to test sequences and analysis scripts. The software on the test beds consists of several components: Operating system, runtime engines, utilities such as firewalls, antivirus software, PDF viewers, etc., as well as the actual test bed automation software and analysis tools, if any. These must be transferred to the test beds and installed there. They must then be maintained and kept up to date.

With a large number of test beds, this can be a major challenge:

  • Different versions need to be managed.
  • Efficient planning of updates in the context of test bed utilization and availability is an important aspect.
  • For quality assurance purposes, it must be documented which software versions were installed on a test bed at which point in time.
  • Comparability of results must be ensured, and errors can creep into manual update processes.

As with test sequences and analysis scripts, a software version management tool is a good choice.

Software update scheduling can be integrated with the test management system to align scheduling with test bed utilization. Full integration of test management and version management also makes it possible to detect when tests are being run on test beds with outdated versions.

The ideal solution is to use update services on the test beds that search for new versions, download them and prepare them for installation. Then the installation can be triggered by IT during idle phases through remote access or performed directly by the operator at the test bed. But at the very least, centralized management and distribution should be in place.

System failure

For integrated systems, system failures are a particular challenge. Especially for the test bed, for example, the network connection between the test bed and the higher-level test process management system is of elementary importance. If the network connection to the higher-level system is interrupted, the test bed automation system must draw attention to this at an early stage via an alarm so that the IT and system administrators can fix the existing problems as quickly as possible. For this case, the option of local temporary storage of all already available test orders directly on the test bed and the use of a simple local test management is recommended. From this, tests can be temporarily selected for execution.

In order to be able to continue working smoothly even if connection problems persist for a longer period of time, the local test management should also provide the option of being able to create and manage test orders locally. This means that a test definition can also be created locally on the test bed as a complete data set if necessary.

The local test management system is then well equipped for the subsequent transfer of test order data and test result data to the higher-level system. "Test result data" here refers to the entirety of all information occurring at the test bed. In addition to the measurement results of all connected systems, this also includes the photos, comments and reports stored for a test order, which may already have been generated by initial analyses at the test bed.

However, the higher-level system must also offer the necessary interface functions for the transfer. A flexible system strategy that takes open interfaces into account is the best choice for these cases.

For serious failures of the test bed automation system, such as a failure of a server computer, a redundant, identically designed system can also provide protection.

Concrete solutions for your smooth test bed integration

Simple and error-free test bed integration through a smart test bed

Ideally, your test bed itself is equipped with the necessary intelligence to automatically integrate itself into the process. It has interfaces to your central test management, through which it automatically receives new test orders, provides result data and allows continuous condition monitoring.

In addition, with a smart test bed you can exchange and synchronize master data, test and analysis procedures, and even the operating software. Ideally, your smart test bed has a local test management to be able to edit the received test orders locally and to document their execution. It collects all data from the test bed and sends it bundled to the central test management. In addition, the smart test bed supports the reconciliation of master data as well as procedures and software statuses.

Your advantages:

  • More efficiency through seamless integration into the perfect test sequence.
  • Fewer errors and higher user acceptance due to convenient operation on the test bed.
  • You meet the highest quality standards by automating work steps and checking data for validity.
  • Likewise, it offers you the possibility to work self-sufficiently in case of system or network failures.

For this solution, the development costs for a single test bed are somewhat higher. However, they can be compensated with a consistent platform concept that is rolled out to several or all test beds of the test field, because this distributes the costs accordingly to several test beds.

More efficiency through smart test bed integration

If your test beds do not offer their own interface to the test management, an automatic adapter process between test management and test bed makes sense. However, your test bed must provide a certain basic functionality of a local test management so that it can be integrated into the higher-level test data management.

The automatic adapter process receives the test orders from the test management and converts them into a format that your test bed understands. Likewise, the adapter receives the results from your test bed and transmits them to the test data management in a format they can understand. The adapter can also manage the operational status of the test bed and share it with the test management. The adapter is also able to share all other information (such as sequences and software states).

However, your test bed should still have a local test management. It can display and process test orders and document their execution. In addition, it can supply all data recorded for a test order to the adapter process working in the background. This enables seamless integration into your test process.

If your test bed has appropriate interfaces, convenient operation is also possible. Since the adapter process acts as an interface between test management and test bed automation, the original test bed software does not need to be adapted, which offers you considerable cost savings.

The adapter should be based on a platform approach that provides core functionality and brings configurable plug-ins for customization to the individual interfaces. This means that you only have to implement the interfaces once in the direction of the test management for orders and results. You should adapt the interfaces in the direction of the test beds according to the automation systems, but here you can almost always find certain commonalities that you can bundle in reusable modules.

Easy test bed integration through interactive adapters

If you have simple test beds, they probably don't have their own test management to help you select the right test and run it correctly using instructions. You can only execute the test and record the data in the process. To integrate these test beds into a test management system, you need an interactive adapter. In addition to the interface functions, this adapter also handles the local test management and provides you with a user interface. You can use it to call up the tests pending for the test bed from the test management and display their details. The tests can be "started" and "finished" from the adapter, whereby this essentially means that the data recorded from now on is immediately assigned to the selected test.

With this concept, you can integrate even simple test beds that do not have the necessary capabilities into your test process. The costs here are relatively low if the same adapter implementation can be used for several test beds. Only the interfaces to the test bed itself have to be implemented specifically. However, the implementation work is usually limited to monitoring data directories or starting automation software. You only have to enter data from the test order partly manually into the automation software, because the test bed automation of simple test beds may not be able to take over the data from the test management, or not completely.

Also for the adapter process, it is only possible to transfer the result data automatically if it is clear which data belongs to the test. If this is not possible due to the nature of the automation and its data storage, then you must compile files manually. The interactive adapter can nevertheless support you in this work if it allows you to assign files, e.g. via drag-and-drop, to the individual tests managed by the adapter. However, as always with manual steps, there is a risk of manual assignment errors and data may be lost halfway through the process if it cannot be transferred consistently from the test management via the adapter to the test bed or back again.

Therefore, you should always consider a higher level of integration by means of improved automation systems. For simple test beds in existing test fields, however, where improved automation systems are out of proportion to the costs, the use of such an interactive adapter is a very good way to significantly improve the efficiency and quality of your processes.

Here's what you should definitely pay attention to when integrating your test bed

The data exchange between test bed and higher-level test process management is very diverse: test orders, result data, condition information, master data, test methods, analysis methods and test bed software are all components of this highly heterogeneous world.

The different types of data and uses require communication processes tailored to them. A cleanly defined and implemented process ensures maximum efficiency in your test field and, through appropriate automation, you also increase the quality of your process.

The integration of test beds into your test process benefits from good IT architectures and powerful IT infrastructure tailored to the task. Smart test beds are certainly the ideal solution for integration into your test process. However, you can also integrate simple test beds with suitable background processes or even interactive adapters into your process as cost-effective platform solutions.

In all cases, a well thought-out structuring of the processes and a clever choice of architecture appropriate to the task are important.

Integrate test beds cost-effectively into your test process - with Werum's HyperTest Test Bed Adapter

If you are interested in smart adapters that allow you to integrate your test beds into your test process and increase its efficiency and quality, feel free to take a closer look at the HyperTest Test Bed Adapter:

Continue to HyperTest TBA