Effective GCP monitoring for performance and costs optimization

By monitoring your Google Cloud, you ensure optimal performance, security of your data and applications, and pinpoint areas for improvement.

Performance is a key parameter in any infrastructure, and one of the main reasons for using a cloud provider as GCP. With its distributed networks and computing powers, GCP offers, in addition to all other cloud providers, important advantages in terms of sheer performance. Nevertheless, only by monitoring the performance can you ensure that there are no losses at any point.

Google's default software, Cloud Monitoring, aggregates metrics, logs and trace data, displays them in various dashboards, and thus monitors GCP resources and services. GCP performance monitoring focuses on basic metrics such as CPU usage of VM, network usage and bandwidth, ratio of the number of active connections to managed databases, error rates in a virtual network, and storage usage of the various cloud applications. Depending on how you use your Google Cloud, your monitoring may include many more metrics.

To help with GCP performance monitoring, Google provides the Cloud Profiler. This focuses on collecting CPU and memory usage from your cloud applications, generating a statistical profile of each of them. Such a profile is helpful to make changes that improve the applications’ performance. Cloud Profiler attributes metrics to the specific parts of the application’s source code, so you can identify what part of the application is consuming the most resources, enabling costs to be limited, and optimizing performance.

GCP performance monitoring also includes a load balancer. As the virtual semaphore for virtual networks and applications, this plays a key role in distributing loads in the most efficient way, avoiding overloading a single resource or service. Within GCP monitoring, the Google Cloud Monitoring tool collects and analyzes metrics coming from the GCP load balancers, including latency, requests count, byte count for each request, and packet count. Keeping an eye on these metrics will help you realize how efficiently the load balancer is working and if fine-tuning is useful.

The area of GCP network monitoring focuses on virtual networks. Whether you have a cloud-only or a hybrid infrastructure, it is important to ensure that the pieces interconnecting it all are working, optimally configured, and efficient. This is possible by using GCP monitoring tools.

GCP network monitoring is primarily used to guarantee that the basic network connectivity is not disrupted or interrupted, and to monitor how the actual networks are performing. Making sure that the paths between all the components forming a cloud infrastructure can communicate among each other is the first step with GCP network monitoring. Identifying the traffic status, trends, and possible bottlenecks is the next natural step. Other vital parts of network monitoring are firewalls, DNS, and VPNs.

GCP comes with the Network Intelligence Center, a console for Google clouds and its observability, monitoring, and troubleshooting. It has five modules to visualize network topology, packet loss and latency, to view the usage of firewall rules, to analyze network issues, and to test network connectivity. These modules integrate metrics that overlap with GCP performance and GCP security monitoring. The Network Intelligence Center is a simple product that is more than sufficient to get you started or for small networks, but shows its limits when the infrastructure grows into thousands of hosts and services. Third-party monitoring tools like Checkmk can provide more efficient and scalable monitoring insights, and is not limited to only monitoring GCP clouds.

GCP security monitoring involves actively checking your GCP resources and services to detect intrusions, security threats, and potential vulnerabilities. It is the area of cloud monitoring that is charged with protecting your GCP resources by detecting possible security incidents, and inform you about them.

Securing clouds involves many efforts and checks. It is on the one hand checking the safety of data and communications and on the other hand preventing and, hopefully, never needing to resolve security incidents. GCP logs are often monitored to identify suspicious behaviors and possible unsafe events in GCP applications and resources. GCP includes Google Logging, a fully managed log management system, which includes storage, search, analysis, and alerting of logs. Google Logs Explorer is integrated in the Google Cloud Console and is the main instrument for consulting the logs on the GCP platform.

However, not only logs are important in GCP security monitoring. Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) actively monitor networks to identify unauthorized accesses and intrusions, and hopefully prevent them. Google includes the former in GCP with Cloud IDS. It detects network threats such as malware, spyware, and monitors intra- and inter-VPC (Virtual Private Cloud) communication. It is not an IPS system, but it helps to detect security-related issues in their GCP infrastructure.

For monitoring GCP applications and containers’ security, a useful tool is the Google Web Security Scanner. This identifies vulnerabilities in the App Engine, Google Kubernetes Engine (GKE), and Compute Engine applications. The Security Scanner acts as a crawler of public web URLs, spotting vulnerabilities by repeatedly submitting as many inputs and event handlers as possible. Not a complete GCP security monitoring for cloud applications by any stretch, but nonetheless a useful tool to keep in mind.

Whether your infrastructure makes great use of the power of Compute Engine’s VMs or leans more towards leveraging the App Engine for running applications on GCP: Both services drive a business – making the monitoring of these two services a must.

Google Cloud Monitoring and Google Logging collect metrics from cloud applications and virtual machines. These make part of the Google Cloud operation suite, formerly known as Stackdriver. It not only integrates GCP logging and monitoring, but provides you with a few more features. It can alert following pre-established policies, offload the management and scaling of Prometheus infrastructure, set up custom dashboards, perform uptime checks, and conduct security audits. Even though it is a fairly large monitoring solution, it is limited to GCP clouds.

GCP also offers an error reporting solution, named Google Error Reporting. This can identify and help you to understand errors in applications, thus providing a basic form of GCP applications monitoring. It supports many popular languages such as Go, Java, .NET, PHP, Node.js, Python, and Ruby. Through its integration with Google Logging, it can send errors as logs for further analysis.

GCP offers Cloud Storage as a service for hosting data. With different tiers for different desired levels of availability, Cloud Storage is a flexible service for storing unstructured data. Files are a part of web applications and as such are of vital importance for the health of your infrastructure. Monitoring GCP Cloud Storage is thus a logical addition to any monitoring efforts.

Google Cloud Monitoring and Google Logging come to provide help with that. Key metrics such as storage usage, data transfer rates, error rates, and security-related events are collected to give insight on how the storage service is used, by whom, and if scaling is necessary.

While Google Cloud Monitoring is an integrated and easy to use GCP monitoring tool, it may be insufficiently powerful in larger infrastructures and in more advanced use cases. Third-party Google monitoring tools may be better suited to your requirements. Checkmk and similar tools allow more flexible customization and efficiency, without making the tools integrated by Google itself redundant.

Like any other cloud provider, GCP offers databases. Cloud SQL, Cloud Firestore and Cloud Bigtable are only a few of the choices you have when setting up a database on the Google cloud. Needless to say, all of these must be part of your GCP monitoring strategy.

GCP database services expose a series of metrics that include CPU utilization, memory usage, disk I/O, network traffic, query latency, connection pool usage, and more. Along with the support for collecting database logs in Google Cloud Monitoring, there is a fairly complete range of data for assessing databases’ health.

In addition to health, database performance is also very important. GCP monitoring includes a specific tool available for this task: Query Insights. It is an analyzer of queries for Cloud SQL databases, helping you in detecting, diagnosing, and preventing query performance problems. It is a helpful troubleshooter for SQL queries, and useful to pair with general GCP logging and monitoring.