How do you upgrade firmware on a USP (TR-369) device?

If you plan CPE firmware upgrades, sending the file is the easy part. The hard questions come after it: did the image validate, did the device boot it, and what happens to the subscriber if it did not. TR-369 answers them in its theory of operation for firmware, Appendix II, built around one data model table and two commands, Download() and Activate(). This guide walks through it using only what the specification says.

Where does USP keep a device’s firmware?

Firmware lifecycle management is one of the extensions TR-369 lists in its scope (§1.2.2), and it is kept apart from the rest of the software on the device. USP software module management covers every piece of software “with the exception of the primary firmware” (Appendix I.2). The primary firmware is managed through the Firmware Image Objects of the data model instead.

Those objects form the table Device.DeviceInfo.FirmwareImage.{i}., where each instance holds one image, and an instance is active if it holds the firmware that is currently running (Appendix II.1). The appendix works with four elements:

How does a new image get onto the device?

The Controller invokes Download(), through an Operate message, on an instance of the firmware image table. The command downloads the referenced file into that instance and makes the Agent validate it: “any normal system validate of a firmware image”, plus the checksum the Controller supplied in the Download() command (Appendix II.1).

Which instance the Controller targets depends on the device (Appendix II.1):

• One image. The Controller invokes Download() on the active instance with the AutoActivate argument, so the new firmware is activated as soon as it has been downloaded. Such a device typically implements neither BootFirmwareImage nor Activate().
• Two or more images. The Controller typically invokes Download() on a non-active instance, to preserve the current image “in case of an error while upgrading the firmware”.

1. OperateDownload()Invoked on a non-active FirmwareImage instance, with the image checksum.
2. AgentValidateThe Agent checks the image and its checksum. The result arrives as an OperationComplete notification.
3. OperateSwitch images
   • Activate() on the instance
   • or BootFirmwareImage plus Reboot()
4. NotifyConfirm the boot
   • Boot! with FirmwareUpdated
   • ValueChange on SoftwareVersion

Once a device holds several images that are downloaded, validated and available, a Controller can query which images are on it, which one is active, and configure which one to activate. It then has two ways to switch (Appendix II.2.1): set Device.DeviceInfo.BootFirmwareImage to the instance with the desired image and reboot the device with a Reboot() command, or invoke Activate() against that instance.

How does the Controller know the download finished?

Download() reports its result through the OperationComplete notification. The specification uses it as the example of a command behind that notification (§7.6.4.2.5), and OperationComplete exists to signal that an asynchronous operation finished, “either successfully or unsuccessfully” (§7.6.3.3).

Asynchronous operations follow a set pattern (§7.7.2):

1. The Agent creates an instance in its Request table and returns a reference to it in the Operate response. It removes that instance when the operation completes, whether it succeeded or failed.
2. A Controller that wants the result creates a Subscription with NotificationType set to OperationComplete and a ReferenceList that includes the path of the command.
3. When the operation completes, the Agent sends a Notify that includes the command_key the Controller chose in its Operate request. The notification carries either the output arguments or a CommandFailure with an error code and a message (§7.6.4.2.5).

A Controller can also cancel a request that is still in the table with Device.LocalAgent.Request.{i}.Cancel() (§7.7.2). One detail matters for firmware: asynchronous operations are not expected to persist across a reboot of the Agent, and a command still in process when the Agent reboots “is considered to have failed”, unless the command description says it is retained (§7.7.2.1). Subscriptions and Notify are covered in USP notifications and subscriptions.

How do you confirm the device booted the new image?

A successful download does not prove the device runs the new firmware. When switching images, the specification recommends that the Controller subscribe to a ValueChange notification on DeviceInfo.SoftwareVersion or to the Boot! event. If the software version has not changed, or the FirmwareUpdated argument of Boot! is false, the device may have had problems booting the target image (Appendix II.2.1). In the specification’s own example, a Boot! notification carries Cause, CommandKey, ParameterMap and FirmwareUpdated (§7.6.4.1).

The same signals work the other way round. If the version changes unexpectedly, or FirmwareUpdated is true when no switch was planned, the device may have had problems booting a particular image (Appendix II.2.3).

Once the device is back, check the service as well as the version. On a fiber ONT, the first number to read is the optical power: see what RX power is healthy on a GPON ONT.

What happens when an upgrade fails?

This is where a second image earns its place. If a device cannot boot a target image, it can try another one (Appendix II.2.3):

• With two images, it simply tries the alternate image, which ideally holds the previous version of the firmware.
• With more than two, it can try any of the other available images, ideally the previously known working firmware.

The device also has to leave a trail. If it boots an image other than the one BootFirmwareImage points to, it must not change that parameter to point to the running image, because doing so “could make troubleshooting problems” with the switch more difficult (Appendix II.2.3). The Controller can still see which image it asked for.

How do you upgrade a whole fleet in a short window?

Normally, the download and the switch both have to happen inside the maintenance window. Bandwidth limits may cap how many devices can download at the same time, and the upgrade may then need several maintenance windows (Appendix II.2.2).

Devices with more than one image change that. The provider can push the image to a set of devices at any point during the day, then use a smaller maintenance window to switch them all, which the specification says can result in shorter system-wide upgrades (Appendix II.2.2). Switching later also reduces the impact on the subscriber, and spreading the download over the entire day, or over several days, can help minimize the impact on the provider’s network (Appendix II).

Does firmware still matter as device software goes modular?

Yes, although the specification expects it to change less often. Operators and manufacturers are “moving away from monolithic firmware images” toward modular software, which lets operators shorten device firmware versioning lead times, “typically 12 to 18 months”, and update individual services independently of the overall firmware, including quick patches for security issues (Appendix VI.1).

The primary firmware stays outside that model, managed through its own Firmware Image Objects (Appendix I.2). The base image of the device still goes through Download() and Activate(). For the bigger picture of what TR-369 covers, read what is TR-369.