One spring day in the year 2002 Chicago’s AWS (AT&T Wireless Services) FSA One’s, OMC (Operations Management Center) was tasked to lead in the data migration project from the outdated reporting software to new. Failure of the project would result in regional disruption of service for three states. This data dump to its upgrade followed the implementation of the 3G roll out in the prior year
The OMC quant data reports detail the activities of field operations. This includes its personnel, fleet information, switch and cell site locations and tower and cell site leasing agreements, RF Engineering data, and their relationship for three states. And most important the system transmits alerts to field operatives and management of potential issues. There’s a prescribed time limit to respond. It was understood this regional updating would also be impactful to the NOC (National Operations Center) in Washington State.
Not only were we tasked with being the lead in this rollout, we had two weeks to formalize and test our work within the new untested software environments. With the help of NOC personnel and as project lead my focus and project description was to gather and migrate as much operational data as possible, into the new reporting system. I decided on a dry run deliverable within five days, understanding that predictable good outcomes were problematic.
This was post 9/11 and the new 3G footprint implementation environment. All these intended and unintended upheavals and company wide tensions high.
After four days of data migration, checking rechecking reports I left the Chicago OMC offices after initiating the new monitoring system. We had six days before expected completion and I was fully prepared for fall out.
I was project lead… and had confidence in the hands of kind strangers…. the NOC techs. They were the first to monitor for notification of any out of the ordinary alarms from cell sites or switches and the first line before escalation.
The wait was not long.
Two hours later found me shopping for a new personal phone when my AWS iPhone began to beep, that familiar tone that says a switch is having a hard time reading incoming data and we’re escalating in a few minutes.
Speeding down a busy Kennedy going north, night time highway lights flashing by and I’m taking agitated call after agitated call. The fixes resulted in late nights for all concerned. The morning reveal was this.
The solutions turned out to be small and fixable, simple manual data entry of omissions of contact data for a few field operatives…. contractors for cell site maintenance.
The early software runs had little or no impact on consumer service. Cell and switch alarm protocols set in place worked to suppress exposure to failure. The new reporting software captured the outer tendrils of the spider web of data.
By the morning of the great reveal we were eager to lean forward during the conference calls. We participated in lively and productive coast to coast conversations centered around how the system worked, its flaws, and what to look forward to in the series of national rollouts to come.
Shorten timelines on deliverables helped to benchmark potential failures that included reports detailing the problems and subsequent fixes. The final rollout ran as smooth as glass.
Although the process was at times stressful, especially in moments of alarms and the confusion of gathering missing data. Protocols mimicking Scrum helped in ferreting out hidden worries and risks while offering ample opportunity to apply solutions.
And the NOC had full disclosure for production of procedural manuals in how and what to anticipate in future rollouts of reporting systems.
I look back on the days of pre-Scrum and now appreciate its principles. I wonder why it took so long to formalize these concepts into something coherent and usable across all industries. The Scrum process is important to building trust within lean project teams while developing confidence in reporting systems. Both are vital tools of Project Management.
My experiences during this project made living with 3G much more comfortable.