What is GCP database monitoring?
GCP (Google Cloud Platform) database monitoring is a collection of the processes and software that deal with the monitoring of any of the GCP database options. It is the equivalent of the general cloud database monitoring for the GCP cloud database services. A solution for GCP database monitoring should collect the various metrics related to the many types of databases offered by GCP, present these metrics through one or more dashboards, and has a plethora of features for alerting, notifying, and automating your monitoring efforts.
GCP database monitoring covers both completely cloud-based environments, and hybrid ones in which only a part of your infrastructure is hosted on Google's servers. In this article we will focus only on Google database services monitoring, leaving on-premises databases to the more generic area of database monitoring and covered in a separate guide. Here we will instead discuss the why, the how, and the best practices in the monitoring of any of the available GCP database options.
Why is GCP database monitoring important?
A fundamental piece of many applications is a database, and that includes cloud-based solutions. If your cloud databases are hosted on Google Cloud Platform, GCP database monitoring is a necessary component of your IT team's efforts to maintain an efficient infrastructure. As with any type of database, no matter where it may be run from, keeping track of how it is working and if there are potential problems is of primary importance to enable your organization's services and operations to run efficiently and reliably without hiccups.
Cloud databases are no different in scope from on-premises ones. Thus, what is important for database monitoring is also valid when monitoring any of the Google database services. It is critical to ensure that valuable data kept in databases is in good hands and cannot be compromised or accessed by unauthorized eyes. Cloud databases delegate part of the database security to cloud vendors, but that does not completely exclude risks and misconfigurations. Monitoring GCP databases is therefore important to know that your data is safe.
GCP database monitoring can also inform you about your cloud costs, which can often skyrocket. With good monitoring you can be aware of the workloads on your cloud databases, be informed when you are reaching pre-configured usage thresholds, and scale up or down your infrastructure according to your needs. This makes for efficient spending, not wasting resources on workloads that do not need them.
Plus, as is natural in every aspect of IT, automating time-consuming tasks saves human time, allowing your IT experts to concentrate on more fruitful tasks. At the very least, monitoring any of the GCP database options will let your cloud administrators be worry-free knowing that the databases are being tracked for disruptions, potential threats, performance bottlenecks, and any possible issue.
As with any type of monitoring, the more you know, the better you can optimize and maintain your operations.
Monitoring GCP database services
Google has set up a large range of databases on their cloud. Not only in terms of the classic GCP SQL and NoSQL databases, but also various subtypes for more specific applications. Let's discover what these databases are, and how they can be monitored.
GCP SQL databases
Speaking of GCP relational databases, Google primarily provides a choice between three cloud databases: Cloud SQL, AlloyDB, and Cloud Spanner. The most popular of these is Cloud SQL. This offers managed MySQL, PostgreSQL, and SQL Server databases on the GCP cloud, and its monitoring is not too dissimilar from the various SQL databases in other cloud services.
The main metrics, available through the Google Cloud Monitoring tool, are CPU and memory utilization, storage usage, read/write operations, input and output network traffic, and replication lag (in the case of read replication from the database). These are visible in the default Cloud SQL monitoring dashboard and are available for external use by third-party GCP database monitoring tools, such as Checkmk.
With the AlloyDB alternative there is a further option: the integrated AlloyDB Monitoring dashboard. In a way not very different to the Google Cloud Monitoring dashboards, the same metrics seen above with Cloud SQL are with a couple of clicks visible on the GCP web interface. As a GCP SQL database AlloyDB offers compatibility only with PostgreSQL, but it is optimized for higher workloads than with Cloud SQL.
The last of these GCP relational databases that can be monitored is Cloud Spanner. It is another PostgreSQL-compatible cloud database, like AlloyDB, but even better optimized for extreme scalability and availability. Cloud Spanner can be monitored through the default Google Cloud Monitoring tool using a dedicated dashboard.
All these GCP relational databases also have their own client libraries that give third-party GCP database monitoring tools access to a range of metrics for checking the health of your GCP databases. In addition to pure metrics-based monitoring, all Google database services also provide access to their logs, which can be analyzed via Cloud Logging or any other GCP database monitoring solution.
It is worth mentioning that as well as these cloud-native databases it is perfectly possible to lift and shift your on-premises databases and host them on the GCP cloud. Monitoring-wise this will mean either using a custom dashboard in Google Cloud Monitoring or an external monitoring software that supports the monitoring of GCP sites. Checkmk is an example of the latter case.
GCP NoSQL databases
Moving on from Google relational databases to GCP NoSQL databases, the choice is far from unimportant. GCP has its main NoSQL database in Google Firestore. It is a GCP document database, fully-managed and easily scalable. Google Cloud Monitoring has its own dedicated dashboard which collects metrics such as read, write and delete docum̄ent count, active connections, total requests, and evaluation count.
It is not a GCP document database, however Google BigTable is an important member of the GCP NoSQL database family. As an Apache HBase compatible database, it offers higher scalability, availability, and lower latency than Google Firestore. It has its own set of monitoring metrics, such as attempt and operation latency, retry count, server latency, connectivity error count and more. These are in addition to, and this is valid for Google Firestore as well, the more generic metrics relating to CPU, memory, disk, and network usage.
As far as GCP graph databases go, Google does not have a direct tool that supports this type of database. Thus, no official GCP graph database exists, however JanusGraph is an open source project that involves the likes of Amazon, Google, IBM and others. This can integrate with BigTable as a backend, it has an official guide, and monitoring-wise it operates as if it is monitoring a BigTable database alone.
Similarly, GCP time series databases do not exist, but can be emulated by using BigTable and an appropriate schema.
MemoryStore for Redis is instead offered by GCP, and it has its own set of metrics to pair up with the classical usage ones. Check values for client connections, blocked clients, number of calls per minute, number of stored keys, cache hit ratio and much more to get an idea of how healthily this type of database is running.
GCP database monitoring best practices
When starting to implement a GCP database monitoring strategy it is important to follow best practices to start with the right foot. Firstly, define the relevant metrics for your database type. Checking the basics like CPU usage, memory utilization, free and occupied storage space, network traffic, query latency, and error rates is the minimum. Then, depending on what metrics are exported by GCP through their API or you can collect with monitoring agents, you should define a few more that are database-specific, like we just saw in the previous sections.
Once metrics have been established, setting up alerts and relative notification is the next logical step. If your GCP database monitoring system allows it, and it most probably does, set up custom dashboards so that you can see at a glance what matters to you instead of being inundated by a mass of data. Google Cloud Monitoring allows for the creation of custom dashboards, as does our Checkmk.
Regardless of the need to monitor GCP SQL databases or GCP NoSQL databases, keeping an eye on query performance is a must. Identifying slow-running or resource-intensive queries and optimizing them is key to improving the overall system efficiency, not just of the databases themselves but of all of the applications that depend on them. Look for query-specific metrics such as execution time, query latency or throughput to identify bottlenecks and misconfigurations.
If you are using database replication for high availability and disaster recovery, monitoring the replication lag of your Google database services is helpful to ensure data consistency across replicas. Also do not forget to generally check for database availability metrics to detect disruptions or downtimes of any sort.
Any GCP database option comes with its own logs, and these should never be ignored when doing GCP database monitoring. Google has its own service for reading logs, Cloud Logging, and most other GCP monitoring softwares have their own solutions.
Conclusion
Whichever solution you choose for your GCP database services, an effective GCP database monitoring strategy and system needs to be set up. Not only are databases too critical in any infrastructure to be ignored, but no monitoring solution is complete without including every component of a hybrid, cloud, or on-premise infrastructure.
Checkmk comes with extensive support for the monitoring of GCP database options. It offers all the features that make following the GCP database monitoring best practices a breeze, and is flexible enough to be used locally or from the cloud itself. Whatever type of GCP database you are using there is a high chance that a monitoring plug-in already exists in Checkmk or can be easily written.
FAQ
A GCP database migration service for Cloud SQL and AlloyDB for PostgreSQL databases is available. A migration can work from on-premises, other clouds or other Google database services. This GCP database migration service has a relatively easy to use guide for migration, with built-in scripts and data validation before the new databases are added to production.