Scrum Ways that Led to 3G iPhones

One spring day in the year 2002, Chicago’s AWS (AT&T Wireless Services) FSA One’s OMC (Operations Management Center) led in the data migration project from the outdated reporting software to new.  Failure of the project would result in regional disruption of service for three states. This data dump to its upgrade followed the implementation of the 3G rollout in the prior year

The OMC quant data reports detail the activities of field operations. These reports include personnel data, fleet information, switch and cell site locations, and tower and cell site leasing agreements, RF Engineering data, and their relationship for three states. And most important, the system transmits alerts to field operatives and potential administrative challenges that carry response windows. Rehousing data meant regional updating would also impact the NOC (National Operations Center) in Washington State.

Not only were we tasked with being the lead in this rollout, but we also had two weeks to formalize and test our work within the new, untested software environments. With the help of NOC personnel and project lead, my focus and project description was to gather and migrate as much operational data as possible into the new reporting system. I decided on a dry run deliverable within five days, understanding that predictable good outcomes were problematic.

3G footprint implementation came after 9/11. Anticipating intended and unintended disruptions to the area cell sites and switches made tension high in field operations and RF engineering.

After four long days of data migration, we checked and rechecked reports. Fully prepared for an unanticipated fallout, we’d only six days remaining before the expected project completion.

The NOC (National Operations Center) techs would be the first to receive notifications of any out-of-the-ordinary alarms from cell sites or switches and the first line before escalation.

The wait was not long.

Two hours later, while shopping for a new personal phone, my AWS iPhone began to beep, that familiar tone that says a switch is having a hard time reading incoming data. The issue would escalate to field management in a matter of minutes, just before customer impact.

Speeding down a busy Kennedy going north, nighttime highway lights flashing by, and I’m taking frantic call after frantic call. The fixes resulted in late nights for all concerned. The morning reveal was this.

The solutions turned out to be minor and fixable, simple manual data entry of omissions of contact data for a few field operatives, contractors for cell site maintenance.

The early software runs had little or no impact on consumer service. Cell and switch alarm protocols set in place worked to suppress exposure to failure. The new reporting software captured the outer tendrils of the spider web of data.

By the morning of the grand reveal, we were eager to lean forward during the conference calls.  We participated in lively and productive coast-to-coast conversations centered around how the system worked, its flaws, and what to look forward to in the series of national rollouts to come.

Shorten timelines on deliverables helped to benchmark potential failures that included reports detailing the problems and subsequent fixes. The final rollout ran as smooth as glass.

Although the process was stressful, especially in moments of alarms and the confusion of gathering missing data, protocols mimicking Scrum helped in ferreting out hidden worries and risks while offering ample opportunity to apply solutions.

And the NOC had full disclosure for the production of procedural manuals in how and what to anticipate in future rollouts of reporting systems.

I look back on the days of pre-Scrum and now appreciate its principles. I wonder why it took so long to formalize these concepts into something coherent and usable across all industries. The Scrum process is vital to building trust within lean project teams while developing confidence in reporting systems. Both are crucial tools of Project Management.