📊 GCP Monitoring

Introduction

Google Cloud provides comprehensive observability tools: Cloud Monitoring for metrics and alerts, Cloud Logging for centralized logs, and Cloud Trace for distributed tracing. Together, they enable complete visibility into your applications and infrastructure.

Key Learning Outcomes

By the end of this lesson, you'll understand:

• Cloud Monitoring dashboards and metrics

• Setting up alerts and notifications

• Cloud Logging for centralized log aggregation

• Structured logging best practices

• Cloud Trace for performance analysis

• Log-based metrics

• Integration with third-party tools

Cloud Monitoring

Create a Monitoring Dashboard

# Create dashboard via CLI
gcloud monitoring dashboards create --config-from-file=dashboard.json

dashboard.json:

{
  "displayName": "My Application Dashboard",
  "mosaicLayout": {
    "columns": 12,
    "tiles": [
      {
        "width": 6,
        "height": 4,
        "widget": {
          "title": "Request Rate",
          "xyChart": {
            "dataSets": [{
              "timeSeriesQuery": {
                "timeSeriesFilter": {
                  "filter": "metric.type=\"cloudfunctions.googleapis.com/execution_count\""
                }
              }
            }]
          }
        }
      },
      {
        "width": 6,
        "height": 4,
        "widget": {
          "title": "Error Rate",
          "xyChart": {
            "dataSets": [{
              "timeSeriesQuery": {
                "timeSeriesFilter": {
                  "filter": "metric.type=\"cloudfunctions.googleapis.com/errors\""
                }
              }
            }]
          }
        }
      }
    ]
  }
}

Set Up Alerts

# Create alert policy
gcloud alpha monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="High Error Rate" \
  --condition-display-name="Errors > 10/min"

Cloud Logging

View Logs

# View logs for a service
gcloud logging read "resource.type=cloud_run_revision" --limit 50

# Filter by severity
gcloud logging read "severity=ERROR" --limit 20

# Stream logs in real-time
gcloud logging read --follow

Structured Logging

Node.js:

function logStructured(severity, message, data = {}) {
  const logEntry = {
    severity,
    message,
    timestamp: new Date().toISOString(),
    ...data
  };
  console.log(JSON.stringify(logEntry));
}

// Usage
logStructured('INFO', 'User created', { userId: 123, email: 'user@example.com' });
logStructured('ERROR', 'Database connection failed', { error: 'timeout' });

Python:

import json
import logging
from datetime import datetime

def log_structured(severity, message, **kwargs):
    log_entry = {
        "severity": severity,
        "message": message,
        "timestamp": datetime.utcnow().isoformat(),
        **kwargs
    }
    print(json.dumps(log_entry))

# Usage
log_structured("INFO", "User created", userId=123, email="user@example.com")
log_structured("ERROR", "Database connection failed", error="timeout")

Create Log-Based Metrics

# Create metric from logs
gcloud logging metrics create error_count \
  --description="Count of ERROR severity logs" \
  --log-filter='severity=ERROR'

Cloud Trace

View Traces

# List recent traces
gcloud trace list

# View specific trace
gcloud trace describe TRACE_ID

Distributed Tracing in Application

Node.js with Express:

const express = require('express');
const cloudTrace = require('@google-cloud/trace-agent');

// Initialize Trace Agent
cloudTrace.start();

const app = express();

app.get('/api/users/:id', async (req, res) => {
  const tracer = cloudTrace.get();
  const span = tracer.createChildSpan({ name: 'fetchUser' });
  
  try {
    // Your code here
    const user = await fetchUserFromDB(req.params.id);
    res.json(user);
  } finally {
    span.endSpan();
  }
});

Best Practices for Observability

1. Structured Logging Format

// Good: Structured logs
console.log(JSON.stringify({
  severity: 'INFO',
  message: 'Request processed',
  method: req.method,
  path: req.path,
  status: res.statusCode,
  duration_ms: duration,
  user_id: userId
}));

// Avoid: Unstructured logs
console.log('Request from user ' + userId + ' took ' + duration + 'ms');

2. Log Levels

// ERROR: Errors that may require immediate attention
logStructured('ERROR', 'Failed to process payment', { orderId: 123 });

// WARNING: Issues that should be investigated
logStructured('WARNING', 'Slow database query', { duration_ms: 5000 });

// INFO: Important business events
logStructured('INFO', 'User signed up', { userId: 456 });

// DEBUG: Detailed debugging information
logStructured('DEBUG', 'Cache hit for key', { key: 'user:123' });

3. Request Tracing

const { v4: uuidv4 } = require('uuid');

app.use((req, res, next) => {
  const traceId = req.headers['x-trace-id'] || uuidv4();
  req.traceId = traceId;
  res.setHeader('x-trace-id', traceId);
  
  logStructured('INFO', 'Request started', {
    traceId,
    method: req.method,
    path: req.path
  });
  
  next();
});

4. Error Tracking

app.use((err, req, res, next) => {
  logStructured('ERROR', 'Unhandled error', {
    error: err.message,
    stack: err.stack,
    url: req.url,
    method: req.method,
    traceId: req.traceId
  });
  
  res.status(500).json({ error: 'Internal server error' });
});

Key Takeaways

• **Cloud Monitoring** provides metrics, dashboards, and alerting

• **Cloud Logging** centralizes logs from all applications

• **Structured logging** enables better querying and analysis

• **Cloud Trace** shows request flows across services

• **Log-based metrics** convert logs into metrics for alerting

• **Alerts** notify you of issues in real-time

Next Steps

Learn about error reporting for automatic error tracking, or explore Profiler for identifying performance bottlenecks.