Published by Toggle magazine
(https://www.togglemag.com/case-studies/lynn-haueter-family-childrens-services/)
The last thing you want to happen to technology for the largest behavioral health organization in Oklahoma is to have the system crash — there’s too much at stake for a vulnerable and under-resourced population seeking mental health assistance, family services and suicide prevention.
The gravity of a potential failure drove Lynn Haueter of Family & Children’s Services to act when he arrived in 2012 as the director of IT, infrastructure and operations. Coming from a Fortune 100 company and a 25-year career, he inherited a struggling IT department experiencing up to three catastrophic server crashes a week, leaving 450 employees to resort to manual processes to get the job done.
Gaining support from F&CS CEO Gail Lapidus, Haueter developed a plan to turn operations around. With two major phases to the project now completed, there hasn’t been a system failure in over three years.
“Here’s a group of people dedicated to those they serve,” Haueter says. “Watching my coworkers care for people in such dire situations inspired me to be a better person, motivating me to help them do what they do.”
Scope of the situation
Incorporated in 1921 as the Family Welfare Society of Tulsa, the nonprofit has diversified into one of the largest service organizations in the state with $73 million in revenue. It offers more than 60 programs to Oklahoma’s underprivileged population in seven centers across metro Tulsa.
Currently, 750 employees offer child, adult and family services addressing severe psychiatric, abuse, and trauma needs, along with child and adult suicide prevention.
Those services were hard to deliver, however, when the system’s infrastructure was failing. Haueter says he inherited a system that lacked standardization, proper alignment with business units and had a revolving door of technical staff. The worst part, however, was multiple server crashes weekly.
“We were bleeding,” Haueter says. “There was a lot of pain and no trust.”
With the support of Lapidus, who invested 46 years — 34 years as CEO — in the system herself, he gained the funding and the freedom to design a new system.
“None of it would be possible without the trust and backing of Gail and our CFO Jan Dietrich,” Haueter says. “They valued my ability to treat people well and support my vision of where we needed to go.”
Getting to work
The two-phased project entailed a monumental rebuilding of the F&CS infrastructure. The first phase, in 2013, took two years to complete with staff working more than 90 hours per week.
Promoted to CIO in 2013 to take over operations, Haueter first sought to establish trust with the nine members of his team. Together, they untangled a web of infrastructure problems, poorly designed applications and integrations.
Basing the design on ITIL standards — the Information Technology Infrastructure Library — he devised best practices from the servers to application development. He also stuck to Tier 1 manufacturers, including Microsoft, HP Enterprise and Rubrik, as automated backup solutions for F&CS’ disaster recovery strategies.
“We got away from personal flavor of application development style and focused on something that’s standardized,” Haueter says.
The initial project included upgrades to servers, storage and the network to support the company’s electronic health record system and develop the data center.
“The complexity of the project was intense,” Haueter recalls. “HP Enterprise was a tremendous strategic partner based on the resiliency of products. When we upgraded the system the second time it was a hundred times easier,” he says, noting that all the original engineers are still on his team today.
Second time around
By 2019, the infrastructure required Phase 2 of a system refresh. Haueter says he wouldn’t have gotten the job done without the extraordinary support and performance of account executive AJ Matheson of PCM who coordinated efforts and kept the PCM team on point.
The challenge for Haueter was time. As a nonprofit, the fiscal year changes on June 30, so when the CFO approached him eight weeks before the deadline, the system upgrades needed to be designed, ordered, invoiced and delivered by that date.
With four weeks to go before the June 30 deadline, he was not close to completing the upgrades that needed to be made for Phase 2. Hardware and software still needed to be ordered and the invoices for all aspects of the project had to be dated before the end of the month to comply with accounting.
“It was amazing how Matheson cracked the whip and orchestrated presales engineering, manufacturing and got the costs down,” Haueter says of the new partnership. “It was crunch time.”
When all the equipment arrived, it was a matter of replacing items that fatigue over time — enterprise servers using HPE Synergy, new SAN using HPE fiber SAN switches, HPE Nimble for storage, and Cisco for all new NexGen firewalls, core switches and edge switches.
“Running the business 24/7, we’re talking about a lifesaving measure; you can’t afford an outage — one misstep can cost a life,” Haueter says.
In addition to the upgrades in 2019, the organization activated the new data center in July 2020, using its former site for disaster recovery — a process that once took days and now takes minutes.
Fits like a glove
Tackling these types of problems is second nature to Haueter, who’s been in IT since graduating from Northeastern State University with a degree in electronics technology in 1987.
From field engineer at MicroAge in 1990 to technology consultant at MAPCO in 1994, he’s “done it all.” However, his longest commitment was 16 years at Alliance Coal — first as a senior systems engineer beginning in 1996, and then as a planning and compliance officer starting in 2010.
Searching for his next set of challenges brought him to F&CS.
“Challenges motivate me. I always want to be in the middle of solving problems, but I didn’t realize the level of issues here,” he jokes.
Being responsible for every technology in the system, what he didn’t expect to find was the human element and connection to the mission. It’s different from working at a straight-up energy company, he notes. Here, human lives hang in the balance.
“I have the greatest job in the world. Working with these people has been an unparalleled experience,” Haueter says.
