If yours is like many IT organizations, the adoption of cloud may have gotten ahead of your ability to manage it. Nine in ten enterprises currently use multiple cloud vendors, and eight in ten share data between public-cloud and on-premises applications, according to a survey of 502 cloud professionals by Dimensional Research. But when things go wrong, more than half of those IT professionals said they struggle to resolve those issues.
Why is that? Many organizations have older, outdated monitoring tools that don't span on-premises and public-cloud infrastructures, and the metrics available for public-cloud monitoring are more limited. As a result, 64% of respondents said it's more difficult to remedy issues with applications located in the public cloud than it is for on-premises applications, and a whopping 96% said they're resolving public-cloud application issues too slowly. That, they say, is having a negative impact on their operations—and their businesses.
Here's why cloud apps are harder for IT Ops teams to monitor and manage, and what your team can do about it.
Hybrid cloud has made debugging difficult
According to the study, “Hybrid Cloud Usage Is Growing Rapidly Creating New Problems and Challenges for Monitoring Solutions,” commissioned by Micro Focus, resolving public cloud-based application issues involves extra work, such as additional time to schedule and coordinate remediation with employees from the cloud vendor. Resolving cloud-based application issues also requires organizations to invest more resources to mitigate business loss.
Source: Dimensional Research
This is likely to become a bigger problem in 2018, as the use of the public cloud continues to expand.
IT struggles to debug apps in the public cloud because it's difficult to get a comprehensive view, or topology, of the entire system that supports those applications. Forty-two percent of respondents said this was their top challenge. The problem has been made worse by a lack of access to the cloud infrastructure, as well as a failure of organizations' infrastructure monitoring tools to supply the needed information in a hybrid IT environment, the report states.
Forrester Research analyst Lauren Nelson agreed that this is part of the problem. “When using cloud apps and cloud infrastructure, the responsibilities of management, governance, and security vary. Some layers are managed by the vendor; others are managed by the customer,” she said. “Sometimes the monitoring insights available are not something the enterprise is able to select or control.”
This often makes it difficult for companies to get the monitoring they want. Some organizations even question the accuracy of the information they're getting. And they complain about needing new cloud-monitoring and management tools for the cloud versus what they use for on-premises workloads. In this way, it is more difficult, Nelson said.
Source: Dimensional Research
“IT isn't responsible for all of the daily tasks of maintaining the cloud infrastructure, provisioning it, and making sure it’s got proper heating and cooling—that’s the good side,” said David Gehringer, principal at Dimensional Research. “The bad side is, it’s not yours, so you don’t get to connect to it and get the detailed information you need."
"Sometimes public cloud providers will give you metrics—but only what they want to provide.”
—David Gehringer
For some companies and some types of metrics, that’s a complete black hole, he said. For other people, it’s just a gray hole—they see some of the metrics, but not all. “So that’s one of the other issues: When you’re trying to triage a problem, you don’t have visibility into that side of it to know if there’s a hardware problem or an application problem or maybe a software problem,” Gehringer said.
More metrics needed
While companies have control of the applications in their on-premises environments, they have almost no control of their cloud-based apps, and that makes it difficult to get the metrics they need to resolve issues, said Michael Procopio, product marketing manager at Micro Focus.
“For example, for Amazon's compute capability, they only give you seven metrics at five-minute intervals,” he said. “If you want to pay them extra, you can get them at one-minute intervals. But you only get seven metrics, and if you want more, you have to pay for them." That's an additional cost to monitoring your cloud. "And the question is, Are those the metrics you need to solve a problem when it happens?”
David Linthicum, chief cloud strategy officer at Deloitte Consulting, said cloud vendors do provide the necessary metrics, but companies need to turn on logging and other data retention services. The problem is, most IT operations management teams don’t know how to turn them on, or gather only the basic data points.
“The trade-off is cost. You have to pay for the processing and storage to gather the data, not the metrics and data ongoing,” he said. “If you turn on everything, it can be costly, as with on-premises systems.”
Forrester's Nelson said that public-cloud users rely on their providers, third parties, and APIs to gather the metrics they require.
“Hybrid cloud managers also provide dashboards that show all cloud workloads in the same dashboard with the same metrics,” she said. “This consolidation and view is often determined by the hybrid cloud management [HCM] vendor. These HCM tools use APIs and partnerships to serve this up.”
This issue has always existed, but it’s more pronounced now because more organizations are moving to hybrid cloud environments.
“Your cloud application is working in conjunction with on-premises systems,” Procopio said. “You can go to your cloud-monitoring console, which the cloud provider gives you, to get part of the picture. Then you go to your on-premises monitoring to get the rest of the picture. The question is, How do you tie those two together?”
Nelson said she hasn’t seen a strategy dictated by lack of monitoring insights or ability to correlate insights for a single workload. This is usually determined by balancing cost, speed, and performance for a single app, an ecosystem of apps, and across an entire environment.
“There is also not a notable difference in monitoring available between public and private clouds,” she said.
But given the rapid progression to hybrid IT and the additional obstacles in resolving cloud-based application issues, IT professionals are further challenged with inadequate tools, the survey found.
“Hybrid IT is requiring functionality that previous/legacy tools never needed and thus don’t have,” Gehringer said. He noted that the top four challenges cited by survey respondents "are nearly tied, at 42% and 41%, respectively—things like topology across private and public clouds, and limited access to public cloud infrastructures.”
Some respondents noted that cloud-based access needed for monitoring can in fact create new security risks. The problem is that for these new hybrid environments, the old tools just don’t work.
Better tools needed for hybrid cloud environments
Are comprehensive monitoring systems required to properly operate a hybrid IT environment? Nearly all survey participants (98%) said they wanted a sophisticated monitoring system, enabled with machine learning, that automates responses and repairs to complex hybrid cloud environments.
“I was . . . surprised by the automatic actions from machine learning,” Gehringer said. “Many IT professionals tend to be a little afraid of giving too much automated action to anything." Because automated actions are like any other piece of code, they're subject to errors. And if you get into a sequence or a cascade of actions and one of them is wrong, you can create a bigger problem than you’re solving, he added. "So I was surprised at how willing they were to look at that. That’s something I wasn’t expecting.”
Source: Dimensional Research
Companies need enterprise monitoring tools that include both on-premises and cloud-based infrastructure. The problem, Linthicum said, is that many organizations haven't matched their investment in public cloud with modern management tools designed to manage across hybrid cloud environments.
“Most of the time people are debugging applications with tools that are three to five years old.”
“You have the monitoring and resource management and resource governance tools that you traditionally use on premises, but you have to upgrade to add the cloud capabilities,” Linthicum said. “You can still look at it under a single pane of glass. You can see your on-premises and public-cloud assets under one management platform. And you're able to automate learning, maintenance, and everything you have to do within the data centers.”
IT teams are now considering moving to tools that allow them to instrument many aspects of the applications as they’re running, whether on premises or in the cloud, Gehringer added.
Expectations are high: 62% of respondents expect tools to monitor hybrid cloud to improve availability and performance, 59% want tools that improve efficiencies, and 53% expect those tools to reduce expenses. In a nutshell, they want a dedicated hybrid IT monitoring system that delivers better performance at less cost.
Is automation the answer?
Automation can help here, Gehringer said. “IT needs monitoring that looks across everything. It needs tools that can flag when there's a problem and provide automated remediation. It needs a system that’s intelligent enough to learn to take basic actions that operators currently do manually."
Nearly every cloud professional surveyed (98%) saw value in incorporating machine-learning automation, including for dynamic load balancing; adding compute capability to overloaded systems; detection and monitoring of new systems; and restarting stopped application processes.
What you can do about it
To ensure that they’re better able to monitor and manage cloud applications, organizations need to learn more about their cloud service offerings and the tools available for gathering and analyzing the data, Linthicum said.
“Most in IT don’t spend a lot of time understanding public clouds, and it shows in their monitoring capabilities,” he said.
Keep learning
Choose the right ESM tool for your needs. Get up to speed with the our Buyer's Guide to Enterprise Service Management Tools
What will the next generation of enterprise service management tools look like? TechBeacon's Guide to Optimizing Enterprise Service Management offers the insights.
Discover more about IT Operations Monitoring with TechBeacon's Guide.
What's the best way to get your robotic process automation project off the ground? Find out how to choose the right tools—and the right project.
Ready to advance up the IT career ladder? TechBeacon's Careers Topic Center provides expert advice you need to prepare for your next move.
