Advanced Programmable System Hardware Design

This Technical Brief describes how to utilize the combination of the MLE Linux environment and a powerful multi-boot configuration management for efficient FPGA design, test and development. After describing the boot flow of the MLE “Soft” Hardware Platform we will introduce an interface to the configuration management. Afterwards we will present options for failsafe in-field upgrades of FPGA designs without the need of direct physical access to the system. We also show, how such a system benefits from a watchdog mechanism that prevents the system from becoming unresponsive.

Copyright © 2010 Missing Link Electronics, Inc. All rights reserved. Missing Link Electronics, the stylized Missing Link Electronics MLE logo are the service mark and/or trademark of Missing Link Electronics, Inc. All other product or service names and trademarks are the property of their respective owners.

 

Programmable Systems as used in Next-Generation Embedded Systems and networked compute platforms for Industrial, Scientific and Medical (ISM) systems require high degrees of optimization of the hardware and the software components. For designing such systems Missing Link Electronics provides platforms where a configurable microcontroller can be optimized towards the Open Source Software that runs on it and the particular application that the system must serve. Most functionality is implemented as an FPGA-based System-on-Chip which runs a full software stack comprising a boot-loader, a Linux operating system kernel and userspace application software. Because of the tight integration of hardware and software the CPU instruction sets, the kernel device trees, the interrupt hierarchy, etc may be different for each implementation and as a result these components may not be interchangeable.

Therefore, MLE has dedicated special attention to ensure that components depending on each other can be associated and loaded / booted together. The concept is to use a so-called Unified Design Image (UDI) which combines an FPGA configuration bitfile, a boot-loader executable, a Linux kernel image, and possibly even userspace application software (see Figure 1). Now, during the system design phase it is important to try out many different system configurations - for functionality, performance and for robustness testing. To facilitate this, MLE has implemented a powerful and flexible multi-boot concept for Programmable System hardware and software which can hold multiple UDIs and offers failsafe system update functionality even for remote access [1].


PIC

Figure 1: Unified Design Images for MLE

The configuration management functionality is described in a triplet of MLE Technical Briefs: [2] goes into details of the implementation using CPLD devices and how to operate this functionality in general. [3] (this paper) describes how to use this functionality from a hardware designers point of view. [4] explains how to take advantage of this functionality for efficient firmware and software design.

Most Field Programmable Gate Arrays (FPGAs) incorporate volatile memories for the implementation of logic functions. This so-called configuration has to be loaded after every power cycle. There are various ways for configuring FPGAs. In the following we will show how a Complex Programmable Logic Device (CPLD) can be used for configuring an FPGA and controlling and monitoring the whole system. As CPLDs incorporate non-volatile configuration memory technology, they are fully functional immediately after applying power to them and thus can operate without first being configured. The MLE 1000 Series Rapid Prototyping System as one of the many instantiations of the MLE “Soft” Hardware Platform uses such CPLD devices to implement a very powerful configuration management.

The boot process of the MLE 1000 Series Rapid Prototyping System as seen in Figure 2 consists of several stages. Initially, after the power comes up, control over the system is transferred to the CPLD. The CPLD selects which out of eight system configuration UDIs shall be used according to user configurable DIP-Switches. These system configuration UDIs are stored in eight partitions in the on-board Flash memory, one partition for each UDI.


PIC

Figure 2: Boot process of MLE 1000 RPS

Each partition holds one UDI’s components: An FPGA design, a bootloader executable and an operating system kernel. The CPLD configures the FPGA with the configuration data found in the selected partition. Once the FPGA is configured the embedded processor within the FPGA design executes the associated initial bootloader executable, which in turn loads the corresponding operating system kernel. After initialization of the MLE Linux kernel the system boots, just like any other workstation, into a userspace environment. From there one can set the next configuration to boot from and then issue a reboot, an aspect which we will describe in this paper.

In the following we will show, how FPGA developers can easily test new FPGA designs with existing software, even if the MLE 1000 Series Rapid Prototyping System is deployed in field. Using the powerful configuration management there is no need for direct physical access to the MLE 1000 Series Rapid Prototyping System to try out System-on-Chip designs. As we will show, it is sufficient to have remote network access, an aspect which enable resource sharing and collaboration among an engineering team, possibly operating from multiple locations.

As described above the MLE 1000 Series Rapid Prototyping System supports eight system configuration UDIs stored in on-board Flash memory. The last UDI at number 7 is reserved for a small standalone MLE Linux Rescue System. The rescue system provides the functionality to update any of the other system configurations and allows to reboot the system using a specified system configuration. Using the default implementation both operations, configuration management and system reboot, are possible via an secure remote network connection. For example, update functionality may be implemented by a TFTP server and reboot commands can be executed after logging in via telnet. Of course, when deployed in field, an additional layer of security, for example through encrypted services like secure shell access (SSH) and secure copy (SCP) are highly recommend and fully supported by the MLE “Soft” Hardware Platform.

As mentioned above, one complete system configuration UDI comprises three components: FPGA bitfile, bootloader executable, and MLE Linux kernel image. These three components can be stored in partitions in the on-board Flash memory. Table 1 shows the arrangement, sizes and content of these partitions. The last column shows the partition’s content for an exemplary MLE 1000 Series Rapid Prototyping System delivered by MLE.


PIC


Table 1: Physical flash memory partition layout

From within the MLE Rescue System as well as from many other MLE Linux systems, the partitions in the on-board Flash memory and therefore the stored configuration UDIs can be fully accessed at runtime. Because MLE Linux uses a Memory Technology Device (MTD) to provide access to the configuration memory, configuration UDIs can be read and written just like any other memory device.

Obviously, this low-level access is potentially dangerous as writing incorrect or inconsistent data may render an MLE system non-operable. To recover from such a situation the MLE 1000 Series Rapid Prototyping System comes with certain safeguards (see [2] for details) ensuring you are always able to get back to a working configuration.

To update a configuration component with new data the following command can be used to write the contents of filename into the mtd partition X:

        root@mle1000:~# dd if=<filename> of=/dev/mtdblockX

Any data written can be read back and compared to the data intended to be written. The MLE Linux ships with various GNU/Linux tools for comparing the data – for example hashing functions like md5sum, sha1sum or byte by byte comparisons via binary diff.

Once a new configuration data has been stored in the appropriate UDI partition, the new configuration can be tested. Normally one would flip the DIP switches to choose the appropriate configuration UDI and manually press the reset button to restart the system using the system configuration just written – functionality which is still available in the MLE 1000 Series Rapid Prototyping System. However a much more convenient way is to access the CPLD from within a running MLE Linux and instruct it to reboot the system to the new configuration. Control over the CPLD is provided by an access file within the MLE Linux virtual /proc filesystem. When reading the file /proc/driver/cpld_partition the user gets a list of values separated with the tab-character (\t):

        root@mle1000:~# cat /proc/driver/cpld_partition  
        3       3       f

The first value in the output is the configuration to boot on next boot. The second value indicates the configuration used for the system currently running. The third value is a control and status flag. This flag identifies whether the “configuration to boot from next” was set by software or by DIP switches.

In the above example the output indicates that the configuration to be booted from next will be number 3 (i.e. the fourth one). The system currently running has been booting from configuration number 3. The decision to boot this configuration again was made by software (f, forced), overriding the position of the DIP switches.

The following example shows the effect of writing the desired configuration number to
/proc/driver/cpld_partition:

        root@mle1000:~# echo 4 > /proc/driver/cpld_partition  
        root@mle1000:~# cat /proc/driver/cpld_partition  
        4       3       f

As the output shows, the configuration to boot from next can easily be changed, in this case to configuration number 4.
Sending a value ’c’ to the CPLD will activate the use of the DIP switches to select the configuration UDI for the next reboot:

        root@mle1000:~# echo c > /proc/driver/cpld_partition  
        root@mle1000:~# cat /proc/driver/cpld_partition  
        2       3       c

The c flag stands for clear and means, that there is no force bit: it has been cleared. This results in the configuration to boot from next is determined by the DIP switches, which will be number 2 in this example (yes, one can also read the DIP switches with MLE Linux!).

To summarize: when writing a number in the range of 0-7 to the CPLD access file, the next configuration to boot from is set; via character c the setting is cleared. Then issuing a simple reboot will boot the partition set in /proc/driver/cpld_partition. So, from within a running MLE Linux, a different system configuration UDI, including an FPGA bitfile, can be written to the on-board Flash memory and be booted.

Of course, this can be a possibly dangerous feature. Corrupt, incomplete or otherwise wrong configuration data can find its way into Flash memory. That is why the MLE 1000 Series Rapid Prototyping System features a failsafe watchdog which will automatically boot the rescue system in case it detects repeated failure. Details on this watchdog are described in [2].

Obviously, the presented techniques do not require any direct physical access to the MLE 1000 Series Rapid Prototyping System. With various network interfaces like Ethernet, Bluetooth, WLAN and 3G mobile networks the MLE 1000 Series Rapid Prototyping System can be accessed remotely. This allows the MLE 1000 RPS to be placed in almost any location where multiple developers from around the world can access and share one single MLE 1000 RPS. Of course to share means to agree on a schedule of usage, as it is well understood from other shared multi-user, multi-tasking Linux systems. Each developer can be using his dedicated set of configuration UDI partitions. Following this rule, a developer B shall not overwrite and therefore interfere with developer A’s work, because he has his own system configuration UDI.

The configuration management features described in the previous paragraphs not only facilitate development, but they also offer the important capability of remote system updates. In the following paragraphs we will have a short glance at other possible usage scenarios.

All examples make use of the manifold networking capabilities of the MLE 1000 Series Rapid Prototyping System. For detailed information on these capabilities we recommend reading the Technical Brief [1]. Figure 3 visualizes the connectivity used for the following scenarios.


PIC

Figure 3: Networkedness of MLE 1000 Series Rapid Prototyping System

The first example is taken from our own in-house development process. Missing Link Electronics engineering team utilizes its own rescue system during production for initial programming of the MLE 1000 Series Rapid Prototyping System. For initializing the MLE 1000 Series Rapid Prototyping System we use our JTAG over USB solution and Ethernet. Via JTAG we download an image of the rescue system (partition 7) into RAM and boot it. Afterwards the rescue system receives the contents for each partition via Ethernet and stores this data into the on-board Flash memory. This is a reliable approach to almost automatically initialize systems for a group of users. Obviously, images can be put under version or access control to ease engineering management.

The second example is more advanced. In this scenario the MLE 1000 Series Rapid Prototyping System autonomously updates itself. This can be implemented via a small script that periodically checks a website on the Internet for update information and compares the currently installed versions of FPGA design and operating system to the available versions. Whenever a new version is detected the script can download the new version, install it and schedule a reboot. Of course extra security concerns have to be taken into account in such a case. For example the communication between the MLE 1000 Series Rapid Prototyping System and the webservice can be encrypted via SSL and/or the data received can be digitally signed and/or encrypted via gnupg. This way it can even be guaranteed that only data intended for one certain single MLE 1000 Series Rapid Prototyping System reaches its destination.

In the third scenario updates are not automatically deployed to our MLE 1000 Series Rapid Prototyping System. Instead the support engineer logs in to the MLE 1000 Series Rapid Prototyping System, pushes updates onto the Flash memory and reboots the system. This scenario can easily be implemented using SSH and SCP. Other possibilities are via a webserver running on the MLE 1000 Series Rapid Prototyping System and a custom CGI script or via a custom TCP/IP based service.

Combinations of the scenarios are possible. For example, the MLE 1000 Series Rapid Prototyping System can search for new updates and send an SMS or email to the support engineer, who then triggers an update.

In this technical brief we have shown how the MLE “Soft” Hardware Platform featuring a Linux environment enables advanced hardware development in terms of FPGA design update and selection from different designs. Even without physical access to the real hardware different designs can be tested and replaced. We have provided a set of ideas on how to implement typical update procedures for in field deployed MLE hardware and shown that those updates can be done fault-tolerant.

 

References

[1] MLE TECHNICAL BRIEF 20100218. Network Connectivity of MLE 1000 Series, 2010.
      http://www.missinglinkelectronics.com/MLE-TB20100218.
[2] MLE TECHNICAL BRIEF 20100817. Advanced Programmable System Configuration
      Management, 2010.
      http://www.missinglinkelectronics.com/MLE-TB20100817.
[3] MLE TECHNICAL BRIEF 20100818. Advanced Programmable System Hardware De-
      sign, 2010.
      http://www.missinglinkelectronics.com/MLE-TB20100818.
[4] MLE TECHNICAL BRIEF 20100819. Advanced Programmable System Software De-
      velopment, 2010.
      http://www.missinglinkelectronics.com/MLE-TB20100819.