April 25, 2006 - Enterprise Networks vs. Cisco Vulnerabilities

by Paul Proctor

First, some context. I've been in security for 20 years and started my career as a kernel programmer. However, at Gartner, my job is to serve large enterprise clients (revenue $1B and up). It‚s fun to play both sides with technical knowledge and the big, strategic business context but let me be up front about one thing; I gave up my hands on technical skills long ago and now I talk for a living.

When the Cisco vulnerability story first broke I was immediately drawn to the impact on enterprises. The fear and loathing that comes with someone owning your routing infrastructure was the perfect juxtaposition with the fact that few of our clients update their networking devices—ever.

Many of our clients didn't understand the 2-stage nature of the attack and thought IPV6 was the problem. Cisco is responding but much of their response affects internal devices and not the precious edge devices. The best news of all is that the fear that major Internet backbone systems would be taken over by these types of threats is not as bad as I originally thought. The multi-million dollar devices that run the backbone are already protected. The bad news is that the biggest problem is going to be getting large enterprises to act differently and treat networking devices like vulnerable, patchable platforms.

The following are some excerpts from an upcoming Gartner research note advising our enterprise clients to get their act together because this time it‚s for real.

It has been proven that Cisco IOS can be vulnerable to remote shell exploitation. While Cisco has been responsive to initial vulnerabilities organizations must develop and adhere to Cisco IOS security maintenance practices

Organizations have traditionally had great confidence that IOS was not vulnerable to remote shell exploitation but one demonstrated case showed that it was possible. This was a two-stage vulnerability for which Cisco offered a corrected versions on all affected versions of IOS for the first known stage on August 11, 2005 and for the second stage on November 2, 2005. Another vulnerability that could lead to a remotely accessible shell was reported in November 2005.

"When the Cisco vulnerability story first broke I was immediately drawn to the impact on enterprises."

Cisco's modular operating system ION (Q106 planned) makes it easier to patch certain subsystems without a reload but ION will only run on the 6500 series through 2006. ION is an internal name. The external name will be ModularIOS.

Cisco is working on full IOS In-Service-Software-Upgrade (ISSU) functionality to support reloading IOS without service interruption. This is particularly important for upgrading edge routers that point at or within the service provider cloud. Initial deliveries of ISSU, beginning on the Catalyst 12000, are projected by Cisco to be Q206.

IOS xR is patchable to an even finer granularity than ION but will only run on Cisco's Carrier Routing System 1 routers (CRS-1 routers), as well as the Catalyst 12000, which are largely not applicable to enterprise networks

Predictions

Given the increased focus on Cisco IOS in the underground community Gartner predicts that more vulnerabilities that may lead to remotely accessible shells will be found in IOS by YE06. The second vulnerability was found within 6 months of disclosure of the first.

Recommendations

Organizations should upgrade IOS as quickly as practicable with all fixes for identified vulnerabilities that may lead to remotely accessible shells. They should consider „staging‰ the upgrades for emergency deployment in case an exploit is detected "in the wild."

Cisco should continue to make all versions of IOS more manageable to patch/upgrade by porting modular IOS to all supported hardware architectures and continuing to make improvements in the in service upgrade capabilities of IOS.

Cisco should enhance their existing patch/upgrade program for IOS to match other software vendors including the separation of security and functional enhancements. Cisco should create a special designation for vulnerabilities that may lead to remotely accessible shells.

Analysis

Organizations need to understand the severe consequences of kernel vulnerabilities that may lead to a remotely accessible privileged shell. It means that the attacker can completely control routing in their organization and this would be very difficult to detect. Historically, there have been over 100 vulnerabilities identified in various versions of IOS since its inception but only two that lead to remotely accessible shells. Organizations need to know the difference and address these severe vulnerabilities.

Understanding the Vulnerability

The first known vulnerability to permit a remotely accessible shell, demonstrated at Blackhat 2005, was a two stage exploit. The first stage is the execution of a known buffer overflow in [the user level of IOS] that makes it possible to execute the second stage in the IOS kernel. Not all buffer overflows make the second stage possible. Only two were publicly known by November 2005, one of which was the IPv6 vulnerability that was demonstrated at Blackhat.

The second stage is the execution of the underlying vulnerability in the timer nodes of a linked list used for memory management in the IOS kernel. It is through this second stage where it becomes possible to execute a remotely accessible privileged shell in IOS. The attacker must know specific code offsets which are unique to each build of monolithic IOS and dependent on the architecture of the device on which it is running making for hundreds of possible code offsets.

Cisco has closed this known vulnerability in the IOS kernel and expresses confidence that the vault is now secure. Gartner believes that the IOS kernel is considerably more difficult to exploit than [the user level] code in IOS but that other kernel vulnerabilities will be found. Given the increased focus on Cisco IOS in the underground community Gartner predicts that more vulnerabilities that may lead to remotely accessible shells will be found in IOS by YE06. The second vulnerability was found within 6 months of disclosure of the first.

Organizations need to put these threats into perspective without underestimating the ramifications. This is very hard to execute for a variety of reasons but the severity of a successful attack and that fact that other exploits are likely to be found makes it something organizations should address in a timely manner.

Addressing the Challenge of Cisco Network Device Patching

Traditional, monolithic IOS is a proprietary operating system that runs Cisco routing and most switching devices. Traditionally it has required an image replacement and reboot to upgrade. IOS is not able be patched while running. The criticality of routers to the operation of a network means that rebooting a router typically takes down the network and severs all connections which can crash applications and have serious consequences for businesses that rely on uninterrupted network connectivity.

The monolithic nature of traditional IOS where a patch required a new binary image resulted in over 850 discrete production builds of IOS over 20-30 product families. This situation raises the complexity of IOS management in most large enterprises. Most enterprises are not able to keep track of which build is running on each of their hundreds or thousands of networking devices. Cisco has gone through a streamlining process to reduce the number of IOS versions about 75% from 850 discrete builds to about 150 discrete builds by YE05. Despite the reduction in IOS builds, most large organizations find themselves running dozens of different IOS versions in their network.

To address the full image replacement and reboot issue Cisco has created two new modular versions of IOS based on the Neutrino micro-kernel called ION and IOS-xR. The Neutrino kernel supports modularity by decoupling router operation from the control plane to allow patching a running IOS without full image replacement and reboot. ION is intended for enterprise use and IOS xR is for high-end carrier-class devices

ION will be supported on 6500 series devices in Q106 (planned). According to Cisco, the 6500 series represents about 50% of enterprise switching devices overall and the vast majority of enterprise switches in the Fortune 500 that Cisco sells. Cisco is also delivering IONized MPLS and IPv6 which will be leveraged by 6500 series deployments in the Service Provider segment. There are no plans to support older devices through 2006.

Another new feature in ION is an interpreted scripting language to identify simple situations and to allow the router itself to respond. It uses measures such as buffers, CPU utilization, and access control list information to provide heuristics that an enterprise can use to dynamically adjust the behavior of the device.

While ION will help the upgrade problem it's not a fix for the routers that are most prone to attack from the outside. The 6500 is a core switching platform and will be inside the firewall. Outside attacks will be against the 17xx, 18xx, 26xx, 28xx and 38xx series of branch office routing platforms that are often connected to the Internet. Cisco is working on full IOS In-Service-Software-Upgrade (ISSU) functionality to support reloading IOS without service interruption. Initial deliveries of ISSU are projected by Cisco to be Q206 but it is unclear which platforms will be supported.

Gartner recommends that organizations upgrade to modular IOS over the next year and develop more mature processes to address network device patching. To begin with, enterprises should familiarize themselves with Cisco‚s disclosure process and subscribe to and review Cisco vulnerability disclosures. This is a shift in perspective for most organizations. Network device patching will likely remain more challenging than server operating system patching but organizations need to move towards improved vulnerability management processes. Some Gartner clients have not upgraded IOS in over 5 years on certain devices. This is unacceptable given the changing threats against IOS, especially on edge routers.

A New Risk with Modularity

Modularity provides for easier live patching but this comes with additional risk. The Neutrino micro-kernel uses fixed code offsets potentially providing consistent entry points for malware. This reduces one of the inherent barriers (hundreds of code offset possibilities) in monolithic IOS that reduces the threat of malware spreading. Basically, a single piece of exploit code will only work on one build and this will reduce the ability of the exploit to spread because of the number of different builds most enterprises run.

Neutrino is not inherently less secure than monolithic IOS, but the ramifications of a successful exploit are more significant. With fixed code offsets, all builds of IOS look the same to the exploit code. In a worst case, a successful exploit may be able to spread like a worm through many devices which is highly unlikely with monolithic IOS. Gartner believes that modularity to support a mature patching process is appropriate and worth this risk but clients need to understand it.

According to Cisco, ION (run-time ModularIOS) supports position-independent code where memory offsets can still vary depending upon the actual build or release of code still making it difficult to derive any consistent entry points.

IOS xR however is more vulnerable to this threat but xR only runs in very high end devices (CSR-1, 12000) in carrier environments. Relatively few of these have shipped, they exist in critical high availability environments and they are patched very aggressively. Cisco does not plan to port IOS xR into enterprise class devices.

Cisco should remain aware of this code-offset issue and continue to minimize the threat of rapidly spreading malware. However, enterprises need to be aware of this trend and use this as further motivation to more aggressively patch IOS in the future.