PROBLEM:
Starting late Thursday, March 26, errors began to occur on DACT systems, not only on calls primarily originating from Comcast network customers but also other originator’s networks. These errors started to increase exponentially. The calls were coming from multiple regions across North America. All DACT formats were impacted during this episode. When actually measuring the timing of Contact ID DTMF signals, it appeared the inter-digit spacing was increased to as much as 70 milliseconds. With other formats there appeared to be echo and waveform distortion. Typically in VoIP systems, this is usually indicative of either poor network performance or the use of other than G711 CODEC.
This problem occurred before signals were handed off to affected central stations, so there was little any central station could have done to mitigate this problem, even if it is served by multiple carriers.
Many central stations were overwhelmed with the numbers of calls as panels tried to communicate, then redialing up to 16 times. This has the additional effect of reducing available capacity on incoming lines for other than Comcast customer panels.
THE WAY PSTN NETWORK WORKS TODAY:
Virtually all calls these days are transported by VoIP, somewhere along the way, even if the customer’s panel is connected to a legacy PSTN.
Regarding the issue at hand, this diagram shows how these signals are transported from a customer’s panel beginning with the Comcast network, then ultimately to the central station.
In this instance Comcast calls that were originated from Comcast customers traveled though the Comcast network but are then handed off to an interexchange carrier. In this case the interexchange carrier was Peerless Networks. Since Comcast uses multiple vendors for this traffic, some Comcast calls would be effected, others would not. A very high percentage of the calls routed through Peerless seemed to fail.
WHAT WAS DONE:
Several of the national monitoring/alarm companies contacted Comcast at a high level on March 30, and by the morning of March 31 Comcast was able to move the traffic away from Peerless. This, for the most part, slowed down the error rate to manageable levels. Central stations may still be receiving communication troubles messages and other signals as communications start to restore.
NOW WHAT:
We are continuing to follow this issue. AICC has started a project to work with the carriers. But what we really need is evidence. So that we effectively move forward, please complete the AICC DACT Dropped Call Survey.
