Taking Video Services to the Next Level of Network Reliability and Revenue Assurance

05.13.03

SCTE Cable Tec-Expo 2003
Mark Davis (BigBand Networks) and Anthony Bowling (Comcast Cable)

Introduction

The video service provider landscape has changed dramatically over the last three years. With the introduction of unregulated local programming, satellite providers have launched a new competitive weapon into the MSO's market place. This has leveled the playing field in the quest for video subscribers. MSO's can only win the market share game by going beyond plain old video services and on to advanced products such as Video-On-Demand (VOD), High-Definition-TV (HDTV), High-Speed-Data (HSD) and Telephony. However, the secret to successfully holding or even gaining market share will be the MSO's who are offering the highest quality of service at the most reasonable costs.

As a part of this quest, several related technical and business trends within the cable industry are driving MSOs to redesign and build a more cost effective, reliable and fault tolerant video backbone network.

• Larger local markets have transitioned from multiple independent operators to a single dominant MSO driving the need to consolidate headends and lower operating costs. In addition, frequent channel line-up changes and programming additions in a 300 channel universe are much more complex when dealing with a multitude of disparate headends. Once a simple procedure, this process is now risky and costly to orchestrate without painful customer impacts.
• In growing large metropolitan systems, headend consolidation must take place without sacrificing high-margin ad insertion revenues and the ability to offer multiple targeted geographic ad zones.
• The transition from a few monthly analog Pay-Per-View (PPV) and premium subscription services to heavier PPV or on-demand services have placed a new penalty on MSO revenues when network availability is not at 100%. In short, no access to on-demand or more transaction-based channels equates to no incremental revenues.
• The video network has evolved to a complex analog and digital network of over 300 video and music channels. Customers who now have a choice in video service providers increasingly demand specific programming options consistent with their personal interest and no longer tolerate individual channel outages.
  The good news is that operators with large metropolitan networks can now take advantage of new intelligent digital video switching technologies designed to optimize digital video transport over low-cost Gigabit Ethernet networks. When coupled with advanced media processing, the same technology platform is used to provide a new, fully integrated, and low cost approach to headend consolidation, service level redundancy, grooming, rate shaping and zoned digital ad insertion. This paper will review the design considerations and technologies that are now being deployed in Comcast Cable's metro Atlanta region to support a fully integrated, fault tolerant digital video backbone system that generates incremental revenue and offers a new level of revenue assurance for minimal investment.

Why Invest in Video Network Reliability Improvements Now

As in any revenue generating service, operators must determine the level of revenue risk they are willing to take and then buy the appropriate level of reliability insurance that would keep network related outages within those limits. An appropriate balance of network capital investment and network availability will ensure that the MSO is optimizing its return on investment, protecting its customer base, mitigating customer churn, and protecting transactional and subscription based revenues. In short, competition has forced a renewed focus on customer service and revenue assurance. Master headends are the heart of a video network and can impact the largest number of customers should a serious or catastrophic outage occur. Many MSOs, especially those launching "lifeline" telephony services, have recognized this exposure and have taken steps to build new weather and fire-resistant headend buildings with back-up power systems. Over the last three years, digital penetration and digital programming have steadily increased to a point that now requires MSOs to seriously consider hardening the video headend and key distribution systems. Many Network Operations managers will attest that video service is now considered a "life line" service. Just ask anyone who has lived through a wide-spread video outage during a major Pay-Per-View event, Final Four tournament, or major football game. A major fight event for a medium size MSO region can generate over a million dollars in revenue in a single night. Having a major outage during this event will also guarantee a lot of very unhappy customers. In addition, cable television is quite literally a true life-line service during local or national emergencies. Urgent news and critical public information is quickly disseminated via the cable network, warning the public of severe weather or other impending dangers. Cable Television today is far from the old saying "it's just video" when it comes to outages and revenue assurance.

Where to Invest in Video Network Reliability Improvements

The MSO can invest in a variety of locations throughout the Hybrid-Fiber-Coax (HFC) network. Many factors are considered when looking for the right place to invest in improving network reliability including Mean Time Between Failure (MTBF) and Mean Time to Repair (MTTR) histories of a given network element. However, the most important place to start investing in reliability insurance is where the network is most exposed to wide-spread outages or a single point of failure that could affect the largest amount of customers. This is commonly referred to as a customer or network failure group. Normally, the smaller the failure group size, the smaller return on the network reliability investment. A special focus should be placed on all critical headend and backbone components that can impact the largest number of video programs or customers.

Architecture and Network Element Design Considerations

The recommended architectural and network element improvements required to design and build the ideal high-availability digital video backbone are discussed below:

• Optimize the headend and hub architecture for metro areas that have consolidated under a single dominant MSO. In larger systems, establish at least two regional Master headends with full signal acquisition of all video programming sources. Connect these two locations with a route-diverse fiber ring that can minimize any fiber cut exposure. The video distribution network is soon headed to an all digital platform so it only makes sense to convert the core analog video backbone to a high-quality digital signal today. This digital backbone can be used to push secondary digital hubs (equivalent to full master headends without signal acquisition) closer to analog hub serving areas that have distance limitations from a Master headend using traditional centralized, amplitude modulated (AM) 1550nm backbone optics. Re-homing the existing analog hubs in a diverse ring architecture to a nearby master headend or new digital hub can also improve reliability and picture quality by eliminating the need for dual 1550nm mid-split and/or EDFA-pumped long distance analog transport systems. Again, the fewer network elements to maintain and operate, the better the performance and higher the reliability. Fewer elements to fail also translate into fewer technical resources and replacement equipment investments required to maintain these devises. Eventually migrating 100% of all programs to digital formats will facilitate moving this same all-digital backbone architecture down to the current analog hub level. These improvements, including eliminating long distance dual 1550nm analog optical links (some routes were over 90 miles) and re-homing all analog fed hubs to near by digital headends or digital hubs have recently been completed in Comcast Cable's metro Atlanta region.
• Leverage new low-cost optical transport platforms based on Gigabit Ethernet. Convert all digital programming from native satellite receiver source feeds such as Digital Video Broadcast - Asynchronous Interfaces (DVB-ASI) or Digital Headend Expansion Interfaces (DHEI) into a GigE protocol that can easily and inexpensively be transported to secondary digital hubs and redundant master headends. Intelligence on the video switching and processing platform can recognize preset quality or failure conditions - from a single low quality stream to an entire transponder outage, and switch to an alternative source via the GigE backbone. In most cases, fiber routes feeding both master headends and digital hubs must be fully diverse and have no single point of failure if a fiber cut were inflected. In addition to a ring layout, a simple linear transport architecture between two master headends can also be used for automatic backup feeds in locations where a fiber ring is not feasible. If the two master headends are spaced beyond 40 (east to west) miles apart, annual solar outage impacts to video satellite feeds could also be completely avoided by switching to the alternate site during the solar interference. Although Ethernet technologies bring a very low cost point to MPEG transport, they do have some limitations that must be overcome. MPEG transport over Ethernet tends to introduce jitter which needs to be removed at some point. Since media processing is required to convert content from GigE back to the required native media formats such as DVB-ASI or DHEI, it's economically prudent to perform the de-jitter process on the same platform used to perform grooming, statistical multiplexing and protocol conversion. The two Master headends and two secondary digital hubs in Atlanta are connected with a fully redundant, carrier class OC192 SONET transport system that has GigE video, data and TDM voice traffic interfaces. Converging all traffic types, including High-speed Data and voice services onto a single transport platform simplifies backbone provisioning, network management and minimizes training and spare requirements.
• Minimize video channel interruptions by reducing the number of separate network pizza boxes or video processing appliances that can introduce more failure points, cabling problems, and have hardware or software compatibility issues with various discrete boxes and suppliers. In addition, these same pizza box appliances are normally optimized for a small and narrowly focused application and cannot be upgraded when technical and operating requirements change. This limits the return on investment and drives the need to change out or "forklift upgrade" these boxes on a more frequent basis. A better approach would be to maintain maximum hardware availability by installing a modular multi-function platform with fault-tolerant redundant processors, power supplies, cooling, and control systems on all mission critical common equipment. Equipment redundancy can be in the form of 1 to 1 or N + 1. 1 to 1 redundant systems provide the highest level of fault tolerance by providing a corresponding backup card or system for every major component. However, on a robust platform, N+1 systems (where a single card or component provide a backup source for N or many cards or systems) still provides a very high MTBF with minimal investment. In addition to a good redundancy design, all equipment must have a hardened, carrier-class design to resist heat and vibration damage, and support the use of software upgradeable and hot-swappable redundant components. The MPEG switching and protocol conversion platform must be, able to sense a program outage and automatically switch over to a redundant Headend feed within milliseconds. The Atlanta backbone is fully redundant with carrier-class transport and MPEG switching and cross-connect platforms deployed.
• All headend equipment deployed must have the capacity and flexibility to process all digital programs in order to facilitate and simplify channel line-up changes and additions without introducing customer impacting outages and excessive headcount requirements. Wire all program feeds to a central MPEG digital video switch or cross-connect point once and rely on the switch fabric to facilitate and simplify future channel changes. In other words, be able to cross-connect any digital program input to any output. This avoids costly and labor intensive headend rewire projects each time a channel moves from analog to digital tiers or each time a digital multiplex changes content or frequency assignment. In addition, the switch and headend layout must easily accommodate the introduction of new programming services such as additional Standard Definition (SD), Video On Demand (VOD) or High Definition (HD) programming. It is a known fact since the inception of cable television that new revenue sources and programming will constantly change and expand over time. Designing the flexibility to accommodate these changes in every Master headend will further ensure reduced operations costs and improve network availability. A simple, easy to use, drag-and-drop graphical user interface (GUI) on the MPEG switch should be deployed for all digital video grooming and provisioning. All digital channels in each Atlanta Master Headend and secondary digital hub are routed through the MPEG switching and cross-connect platform and have facilitated two flawless channel line-up changes in the last 12 months with no customer impacting service interruptions.
• While headend consolidation reduces operating costs, the network design should preserve existing ad insertion zones in order to maximize localized ad revenues. If the smaller zones are not maintained, the headend serving area would be so large that local ad time would be too costly for many smaller businesses and many of the homes covered would be too far away from their establishment. Consequently, the same smaller businesses are normally willing to pay a higher premium for very geographically targeted advertising spots. A good network design should not only maintain existing zones, but should also include an expansion plan for adding additional zones. In addition, the MPEG cross-connect or video switching platform should be able to easily accommodate splitting and adding additional zones and integrating digital program insertion through an internal splicing engine. The Atlanta design deployed today, with two Master headends and two secondary digital hubs, inserts ads on 24 digital programs across 13 different ad zones. Each digital hub or master headend is designed to expand into a separate zone per analog hub for 50 total zones.

Customer and MSO Impacts

Implementing this new redundant video backbone and intelligent MPEG switching platform has a number of bottom line cash flow benefits by meeting today's video network challenges.

Market consolidation has driven the need to unify disparate networks and consolidate headends.The simple effect of consolidating these headends saves on facilities costs, simplifies channel line-up changes, lowers costs of adding additional services, and minimizes manpower requirements to maintain complex equipment. Redundant headends, possibly serving secondary digital hubs, will ensure higher reliability, better service and improved picture quality.

Minimize network and field operations expenses by automating headend related video channel outage recovery. Improved headend technician efficiency is facilitated by the core MPEG video switch automatically transferring to the redundant equipment or diverse fiber route and resolving equipment failures during normal working hours. New MPEG-based switching systems can detect a loss of signal at the program level, automatically switch to an alternate source and send an alarm message of the incident to a national or local network operations and monitoring center. This automation drastically reduces headend related channel outages.

Call center resources are better managed because headcount capacity can be flattened out when they are not as exposed to large spikes in call volumes. Outages and call center call volumes are too often linked to customers demanding service credits for subscription services. By having automatic recovery of headend related video outages, the large majority of "no trouble found" trouble tickets can be eliminated. Most call center Customer Service Representatives (CSRs) attempt to accommodate a customer complaint quickly by scheduling a video-channel-out truck roll the following day. While steps are taken to avoid taking trouble tickets on network outages by quickly identifying large video outages to CSRs, inevitably many tickets make it into the dispatch pools the following morning. Useless $30 to $60 truck rolls translate into inflated service technician headcount requirements and higher operations costs. The same resources could have been used to speed up rather than delay the clearing of legitimate CPE and distribution network trouble calls.

Minimizing video outages of key programming also reduces churn by keeping customers who may be on the fence and ready to pull the disconnect trigger when irritated by channel outages. Unlike the old days without video competition, a small, single outage that is very important to an individual customer could be the "last straw" that pushes them "over the edge" to give satellite a try and eliminate at least 12 months of recurring MSO revenue.

Non-subscription video services like Pay-Per-View (PPV) events and VOD revenues are not collected when the service is not available for customers to buy due to a headend related outage. If a video outage in a single 30,000 home passed hub occurs during a rainy Friday night when standard PPV and VOD buys are known to peak, 900 buys or $45,000 of revenue could be lost.

Increasing ad insertion revenue is a great way to off-set programming cost increases, minimize customer rate increases, and improve the customers' value perception. The localized ad insertion business is one of the most profitable products a cable operator can offer. More zoning of the ad insertion network helps to drive higher margins and a larger total available market. Expanding digital ad-insertion into more digital channels is necessary to preserve and grow ad-sales revenue as more customers are spending more time viewing digital programming.

Summary

We have reviewed the key drivers that are forcing MSOs to re-evaluate, redesign and build a more flexible, cost-effective, reliable and fault tolerant video backbone network. Increasing competition for video subscribers, pressure for higher returns on investment, minimizing operating and capital expenses, improving margins, and generating incremental revenues are just a few examples. Technology improvements such as automated redundant switching and digital video processing platforms and significant cost reductions in backbone transport equipment have now enabled engineers to harden the core digital video network for a minimal investment. A reduction of video related outages is already taking place in Atlanta and we are confident that the improved redundant digital video backbone network will continue to validate these assumptions in the coming months. In the end, it is easy to see that the secret to successfully minimizing churn and holding or even gaining market share will be the MSOs who take advantage of this opportunity now and make the smart network investments that ensures customers are offered the highest quality of service at the most reasonable costs.