Founded in 1736, Manhattan's Bellevue Hospital Center provides comprehensive medical, psychiatric and social services, including inpatient, outpatient and emergency care, annually treating 26,000 inpatients and handling nearly 400,000 outpatient clinic visits, while the hospital's world famous Emergency Service provides help for another 100,000 people each year. The hospital has an attending physician staff of 1,200 and a house staff of more than 500 residents and interns.
Bellevue is a member hospital in the South Manhattan Healthcare Network (SMHN), a division of the New York City Health and Hospitals Corporation. The IT staff at Bellevue is therefore also responsible for the other facilities in the network: Gouverneur Healthcare Services, which maintains the largest outpatient diagnostic and treatment center in New York State; and Coler-Goldwater Specialty Hospital and Nursing Facility, a 2000-bed health center that provides medical, subacute, rehabilitative, and long-term specialty services. In addition, the New York University Medical School is responsible for clinical services at Bellevue, and the IT staff also oversees the in-hospital portion of the school's network.
Looking for More InsightThe IT department at Bellevue is thus responsible for a network with more than 5000 users and a mix of LAN and inter-hospital WAN links, as well as hundreds of different applications ranging from patient admission and other database-oriented systems to a PACS (Picture Archiving and Communication System) that can deliver multi-hundredmegabyte radiological images to workstations throughout the network. IT depended on device-oriented network management systems such as HP OpenView for fault management, and CiscoWorks for utilization information, but lacked a management solution that would give them an end-to-end view of network and application activity.
"After an experience with the PACS system, where it took three different teams of outside consultants to pinpoint a server problem that had been blamed on the network, we realized we needed something that would give us better insight into the conversations taking place across the network, and what the various computers were doing," says Ben Aheto, network manager for Bellevue. "Without that information, we were spending far too much time defending the network from the typical "the network is slow" complaints."
Faster Troubleshooting, No GuessworkThat changed when Bellevue installed the ACE Live appliances-one at each hospital: Bellevue, Gouverneur, Coler, and Goldwater. "The [ACE Live] appliance captures metrics that no other tool we looked at sees, and the way it breaks down application response is very helpful to us," notes Mr. Aheto. "In fact, I find the Response Time Composition Chart that displays that breakdown especially useful. I keep it open on my desktop all the time, so when I get a complaint, I can drag the IP addresses involved into the chart and see immediately whether I should be looking at the network or bounce the problem back to the server team."
The ACE Live Response Time Composition Chart breaks down end-to-end application response into its component parts: TCP setup time, server response time, network time, time lost to packet loss, and latency, enabling network managers to very quickly distinguish between server, network, and application problems.
Overall, the ACE Live appliance has delivered faster, more efficient network troubleshooting. "The Problem Management Dashboard illuminates potential problems before they fully manifest themselves. And, the breadth of statistics recorded lets us automatically rule out different scenarios and focus in on the most likely cause of a problem," he says. "Before, we had to speculate about what the cause might be and then test our hypothesis using other tools, like Sniffer, that didn't give us a direct view of end-to-end performance. Now we can use the same tool for several aspects of the troubleshooting process."
The Problem Management Dashboard applies a number of advanced statistical and correlative techniques to automatically flag problems, identify their likely origin, and allow further, focused drill-down analysis and resolution.
Keeping up with Change
A healthcare network is an incredibly dynamic environment. Doctors in particular view the network as a utility and expect it to be available anywhere, like electricity. "If a doctor goes from his office at Bellevue to his office at Gouverneur, or relocates to a new office," notes Aheto, "he expects all the same information to be just as available, including huge PACS x-ray files. Before we installed the [ACE Live] appliance, we erred on the side of caution, allocating more bandwidth than might be needed. The unnecessary bandwidth came at the expense of both resources and money."
Change management was also eased by the flow-based technology of the ACE Live appliance, which makes the information it delivers largely independent of changes in the underlying physical infrastructure. "We used to have to upgrade our device-oriented systems frequently to keep up with changes, and sometimes had to do so on an emergency basis before we could troubleshoot problems on a segment," says Aheto. "A solution that would fit into our environment had to be independent of both the installed network infrastructure and the type of data that travels through it."
More Cooperation, Better Use of Existing Resources
The improved visibility furnished by the ACE Live appliances has also improved collaboration with the various server teams in the SMHN and improved the utilization of existing systems. For instance, the Bellevue IT department relies on Microsoft SMS servers to maintain and upgrade desktop applications, but, as Mr. Aheto observes, "In SMS, distribution servers are particularly important for scalability, but before we had the [ACE Live] system, we couldn't tell the other server teams just how much traffic they'd have to handle if they shared one of their machines for SMS. But now we can forecast the load very accurately, which makes their job a lot easier."As well, he notes, "The server people are finding the tool invaluable, too, because now they can see who is using a server and how much load they're imposing on it, which makes their planning much more efficient."
A Versatile, Budget-Friendly Investment
"The ability of the [ACE Live] appliance to quickly distinguish server, network, and application problems was justification enough for the purchase," says Mr. Aheto. "And its flexibility enables us to continually find new ways to use it, allowing us to apply it to vastly different situations." And, he notes, it's helped make IT more nimble in meeting and even anticipating hospital needs. "The versatility of the [ACE Live] box, the wide range of metrics it lets us see, and its relative independence from changes in the physical infrastructure, has greatly increased IT staff productivity. And the less time we spend troubleshooting and upgrading, the more time we have for new projects to make the network even more useful to the hospital system."
The ability of the [ACE Live] appliance to quickly distinguish server, network, and application problems was justification enough for the purchase. And its flexibility enables us to continually find new ways to use it, allowing us to apply it to vastly different situations.
Ben Aheto, Network Manager, Bellevue Hospital Center