arrow Back to Alumni Connection Articles

Infrastructure Solutions’ Automated Change Management Solution Helps Clients Improve Productivity and Lower Costs

Infrastructure Solutions’ Automated Change Management (ACM) helps clients improve productivity and lower costs by automating the change processing required by many clients who are running Microsoft Windows operating systems and using Citrix-based access.

Team Members (l to r): Garry Scheer, Laura Sellers, Alex Deininger, Joe Anaya, Steve Bartolini, Renee Wayland, Florence Campbell and Larry Ray.
Not pictured: Joe Bengfort, Mark Eimer, Dak Frisbie, Alex Gonzales, Josh Petrone, and Brian Queary.

Infrastructure Solutions’ service delivery teams use Perot Systems’ Operational Process Application Suite (OPAS) to manage changes to our clients’ IT environments. Clients with 100+ servers in a large, “server farm,” especially those running Microsoft Windows operating systems and using Citrix-based access, follow vendor- recommended ”best practices,” which include weekly server reboots. These weekly reboots require multiple, recurring changes and the closing of the change records is manually intensive and time consuming. The OPAS application team is often asked to allow users to programmatically close change records to reduce the labor needed and to ensure adherence to change windows.

Team Members that participated in creating ACM:

Joe Bengfort
Josh Petrone
Laura Sellers
Renee Wayland
Brian Queary
Dak Frisbie
Steve Bartolini
Joe Anaya
Larry Ray
Alex Deininger
Michael Steptoe
Florence Campbell
Angela Harris
Alex Gonzalez
Stuart McKay

Missing a change window, i.e. an unsuccessful change, can mean a missed service level for a client or a missed key performance indicator for a service delivery team. Change window adherence is particularly rigorous for the Stanford account, and an automated change management process needs methods that are recoverable and auditable. To help meet Stanford’s requirements, the OPAS, Automation and Monitoring, and Batch Enablement teams collaborated with the Stanford account team to create an effective solution to automate change processing.

As a result of the collaboration, ACM was designed as a solution that will put a change record in a WIP (work-in-progress) status, update the installation notes when necessary, and close the change record using a disposition code that is related to the completion status of the change. Data is passed between AutoSys (the tool for the Batch Enablement service), the Event Consolidator and OPAS.

Mark Eimer, director, Technology and Innovation, and ACM project sponsor said, “Today, we have 150 servers in our Citrix farm and plan to add 200 more in a very short period of time. We are already experiencing the benefits from implementing the Automated Change Management solution, our support costs are down and FTE costs associated with the manual effort are now being redirected to focus on other automation solutions using the Automated Change Management solution as a foundation that has been built to further reduce costs and increase productivity.”

“Automated Change Management provides our clients with capabilities to automate recurring change processes and there are many accounts that have expressed the need,” says Garry Scheer, service delivery manager for Business Systems. “The use of Automated Change Management can significantly improve accuracy of change management records which can significantly improve Service Level compliance.”

The creative solution designed to address the needs of the Stanford account will now be seen across the company, as many clients prepare to implement ACM in order to reap the benefits of higher productivity and lower costs.

How Does Automated Change Management Work?

The Automated Change Management (ACM) process depends on a recurring change for the Configured Items (the servers to be rebooted or patched) in OPAS and a set of related AutoSys jobs. The AutoSys jobs are scheduled to kick off a few minutes after the recurring change is scheduled to begin. The AutoSys jobs will send events to OPAS via the Event Consolidator. The Event Consolidator processes the events received from AutoSys and passes the necessary data to OPAS. OPAS workflow detects the type of event and takes the necessary action against the recurring change. OPAS passes the recurring change record ID back to the Event Consolidator, which then passes the recurring change information to the AutoSys jobs. If OPAS cannot locate a recurring change then OPAS will generate an incident and the AutoSys job will fail resulting in the need to complete the recurring change manually.

Advanced Monitoring is deployed on the servers targeted in the recurring change records and will be configured to execute a recovery action after a reboot. The recovery action will execute a health check script which will validate the system health, and depending on the state of the system, determine a disposition code. The disposition code is used to close the change record; valid codes are: 0=Successful, 1=Backed-Out, 2=Incomplete or 3=Completed with issues.