Access to a wide range of system metrics is essential for operating reliable and scalable applications in the cloud. For Amazon Relational Database Service, users need system-level visibility to monitor performance and diagnose potential issues.
Amazon CloudWatch provides detailed visibility into multiple metrics related to resources launched in AWS, including database clusters and instances managed in RDS. Even though CloudWatch metrics are a critical area to monitor, they don’t offer much visibility into OS-level activity. However, Enhanced Monitoring in Amazon RDS could provide these deeper insights.
RDS Enhanced Monitoring publishes metrics gathered by an agent running on the OS on which a particular database instance is launched. This additional set of data provides detailed metrics for database deployments managed by RDS that can be useful for the following:
- Early issue detection.
- Enhanced troubleshooting.
- Performance improvements.
They can also be used to find optimal cloud resource allocations, which can reduce cost. The feature is available for all RDS database engines: MariaDB, MySQL, PostgreSQL, Db2, Oracle and Microsoft SQL Server.
How Enhanced Monitoring gathers metrics
One important consideration regarding Enhanced Monitoring is that the OS data is not published as standard CloudWatch metrics. Instead, OS metrics are published into CloudWatch Logs — by default, into the RDSOSMetrics log group. Users must analyze OS log data and act upon it using a different approach compared to other RDS CloudWatch metrics.
CloudWatch Logs Insights is a useful tool to query data stored in CloudWatch Logs. It uses a query syntax that supports specific data filtering and aggregation based on specific fields. CloudWatch metric filters can detect patterns in CloudWatch Logs and convert them into CloudWatch metrics. This can trigger anomaly detection, alarm notifications and actionable events based on OS log data. For simplified analysis and troubleshooting, admins can visualize these custom metrics in CloudWatch dashboards. For more detailed analysis and actions, use CloudWatch Logs subscription filters to automatically export incoming log data to other AWS services, such as Kinesis, Data Firehose and Lambda.
With Enhanced Monitoring, a unique log stream in the RDSOSMetrics log group ingests each database instance or cluster. This enables the separation of log data based on the source database, which is useful for data filtering when using the CloudWatch Logs console. The RDS console also supports a view of Enhanced Monitoring data using graph widgets, which simplifies the analysis of OS data within the context of specific RDS instances.
Overview of metrics
Metrics are published to CloudWatch Logs in JSON format, and each record contains source data identifiers. These include the associated RDS instance, database engine, record timestamp, number of virtual CPUs and amount of time the database has been active. Then, there are over 80 detailed metrics related to the following:
- CPU utilization.
- Disk utilization.
- File system.
- Process load.
- Memory utilization.
- Network utilization.
- Resource consumption.
The same set of metrics is available for all database engines, except for Microsoft SQL Server. This engine supports a slightly narrower range of metrics.
Enabling RDS Enhanced Monitoring
Admins can turn on Enhanced Monitoring with the AWS SDK, CLI, CloudFormation or console. This is done at the cluster or instance level, depending on the RDS deployment configuration. If users enable a cluster, they must configure it at the cluster or instance level. Configure it by setting a value for the MonitoringInterval parameter — e.g., 0, 1, 5, 10, 15, 30 or 60 seconds. When this parameter is set to zero, Enhanced Monitoring is disabled.
Specify MonitoringRoleArn, which is an Identity and access Management role that grants log ingestion permissions to RDS. These parameters can be configured during cluster/instance launch or as an update operation to an existing RDS resource.
RDS Enhanced Monitoring pricing
The Enhanced Monitoring feature itself does not incur any RDS cost. However, given that data is published to CloudWatch Logs, there is data ingestion and storage usage incurred in CloudWatch.
Data ingestion costs $0.50 per gigabyte in the N. Virginia region, and 1 TB of data storage costs $4.50 per month. There is an OS metrics granularity configuration in RDS that supports log ingestion every 1, 5, 10, 15, 30 and 60 seconds. The lower the granularity, the higher the data ingestion and storage cost is in CloudWatch Logs.
According to RDS, log data published every second — the highest available frequency — incurs approximately 16 GB per month, which costs about $8 in data ingestion fees. Users can also configure CloudWatch Logs retention to expire records after a period of time or to keep them indefinitely. Depending on application needs, the right balance between ingestion granularity and log retention can optimize cost.
Ernesto Marquez is owner and project director at Concurrency Labs, where he helps startups launch and grow their applications on AWS. He enjoys building serverless architectures, building data analytics solutions, implementing automation and helping customers cut their AWS costs.
