Lab 4.6: X-Ray + ServiceLens + Synthetics (Deep Dive)

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  Lab 4.6 - X-Ray + ServiceLens + Synthetics Workshop
  Deploys: DynamoDB, SNS, Lambda x2 (with X-Ray), API Gateway (with X-Ray),
  S3 bucket for canary artifacts, CloudWatch Alarm  

Parameters:
  NotificationEmail:
    Type: String
    Description: Email address for SNS notifications (you must confirm the subscription)
    AllowedPattern: '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'

Resources:

  # ============================================================
  # DynamoDB Table
  # ============================================================
  OrdersTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: Orders
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: orderId
          AttributeType: S
      KeySchema:
        - AttributeName: orderId
          KeyType: HASH

  # ============================================================
  # SNS Topic + Email Subscription
  # ============================================================
  OrderNotificationsTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: OrderNotifications

  EmailSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrderNotificationsTopic
      Protocol: email
      Endpoint: !Ref NotificationEmail

  # ============================================================
  # S3 Bucket for Synthetics Canary Artifacts
  # ============================================================
  CanaryArtifactsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'lab46-canary-artifacts-${AWS::AccountId}'
      LifecycleConfiguration:
        Rules:
          - Id: ExpireOldArtifacts
            Status: Enabled
            ExpirationInDays: 31

  # ============================================================
  # IAM Role for Lambda Functions
  # ============================================================
  LambdaXRayRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: Lab46-LambdaXRayRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
        - arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
        - arn:aws:iam::aws:policy/AmazonSNSFullAccess
        - arn:aws:iam::aws:policy/service-role/AWSLambdaRole

  # ============================================================
  # Lambda: NotifyService
  # ============================================================
  NotifyServiceFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: NotifyService
      Runtime: python3.12
      Handler: index.handler
      Role: !GetAtt LambdaXRayRole.Arn
      Timeout: 30
      TracingConfig:
        Mode: Active
      Environment:
        Variables:
          TOPIC_ARN: !Ref OrderNotificationsTopic
      Code:
        ZipFile: |
          import json
          import boto3
          import os
          from aws_xray_sdk.core import xray_recorder, patch_all

          patch_all()

          sns = boto3.client('sns')
          TOPIC_ARN = os.environ['TOPIC_ARN']

          def handler(event, context):
              order_id = event.get('orderId', 'unknown')

              subsegment = xray_recorder.current_subsegment()
              if subsegment:
                  subsegment.put_annotation('orderId', order_id)
                  subsegment.put_annotation('service', 'NotifyService')

              with xray_recorder.in_subsegment('PrepareNotification') as seg:
                  message = {
                      'orderId': order_id,
                      'status': 'CREATED',
                      'message': f'Order {order_id} has been created successfully'
                  }
                  seg.put_metadata('notification', message)

              with xray_recorder.in_subsegment('PublishSNS'):
                  sns.publish(
                      TopicArn=TOPIC_ARN,
                      Subject=f'New Order: {order_id}',
                      Message=json.dumps(message)
                  )

              print(json.dumps({
                  'level': 'INFO',
                  'message': 'Notification sent',
                  'orderId': order_id
              }))

              return {'statusCode': 200, 'body': 'Notification sent'}          

  # ============================================================
  # Lambda: OrderAPI
  # ============================================================
  OrderAPIFunction:
    Type: AWS::Lambda::Function
    DependsOn: NotifyServiceFunction
    Properties:
      FunctionName: OrderAPI
      Runtime: python3.12
      Handler: index.handler
      Role: !GetAtt LambdaXRayRole.Arn
      Timeout: 30
      TracingConfig:
        Mode: Active
      Code:
        ZipFile: |
          import json
          import boto3
          import uuid
          import time
          from aws_xray_sdk.core import xray_recorder, patch_all

          patch_all()

          dynamodb = boto3.resource('dynamodb')
          table = dynamodb.Table('Orders')
          lambda_client = boto3.client('lambda')

          def handler(event, context):
              http_method = event.get('httpMethod', 'GET')

              subsegment = xray_recorder.current_subsegment()
              if subsegment:
                  subsegment.put_annotation('httpMethod', http_method)
                  subsegment.put_annotation('service', 'OrderAPI')

              if http_method == 'POST':
                  return create_order(event)
              elif http_method == 'GET':
                  return get_order(event)
              else:
                  return list_orders()

          def create_order(event):
              body = json.loads(event.get('body', '{}'))
              order_id = str(uuid.uuid4())[:8]

              with xray_recorder.in_subsegment('ValidateOrder') as seg:
                  product = body.get('product', 'Unknown')
                  quantity = body.get('quantity', 1)
                  seg.put_annotation('orderId', order_id)
                  seg.put_metadata('orderDetails', body)
                  if quantity <= 0:
                      return response(400, {'error': 'Invalid quantity'})

              with xray_recorder.in_subsegment('ProcessPayment'):
                  time.sleep(0.1)

              with xray_recorder.in_subsegment('SaveOrder'):
                  item = {
                      'orderId': order_id,
                      'product': product,
                      'quantity': quantity,
                      'status': 'CREATED',
                      'timestamp': int(time.time())
                  }
                  table.put_item(Item=item)

              with xray_recorder.in_subsegment('InvokeNotify'):
                  lambda_client.invoke(
                      FunctionName='NotifyService',
                      InvocationType='Event',
                      Payload=json.dumps({'orderId': order_id})
                  )

              print(json.dumps({
                  'level': 'INFO', 'message': 'Order created',
                  'orderId': order_id, 'product': product
              }))

              return response(201, {'orderId': order_id, 'status': 'CREATED'})

          def get_order(event):
              params = event.get('queryStringParameters') or {}
              order_id = params.get('orderId')
              if not order_id:
                  return response(400, {'error': 'orderId required'})

              subsegment = xray_recorder.current_subsegment()
              if subsegment:
                  subsegment.put_annotation('orderId', order_id)

              result = table.get_item(Key={'orderId': order_id})
              item = result.get('Item')
              if not item:
                  print(json.dumps({
                      'level': 'ERROR', 'message': 'Order not found',
                      'orderId': order_id
                  }))
                  return response(404, {'error': 'Order not found'})
              return response(200, item)

          def list_orders():
              with xray_recorder.in_subsegment('ScanOrders'):
                  result = table.scan(Limit=20)
              return response(200, {'orders': result.get('Items', [])})

          def response(status_code, body):
              return {
                  'statusCode': status_code,
                  'headers': {
                      'Content-Type': 'application/json',
                      'Access-Control-Allow-Origin': '*'
                  },
                  'body': json.dumps(body, default=str)
              }          

  # ============================================================
  # API Gateway: REST API
  # ============================================================
  OrdersRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: OrdersAPI
      Description: Lab 4.6 - Orders API with X-Ray tracing

  OrdersResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId: !Ref OrdersRestApi
      ParentId: !GetAtt OrdersRestApi.RootResourceId
      PathPart: orders

  # --- GET /orders ---
  OrdersGetMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      RestApiId: !Ref OrdersRestApi
      ResourceId: !Ref OrdersResource
      HttpMethod: GET
      AuthorizationType: NONE
      Integration:
        Type: AWS_PROXY
        IntegrationHttpMethod: POST
        Uri: !Sub >-
          arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${OrderAPIFunction.Arn}/invocations

  # --- POST /orders ---
  OrdersPostMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      RestApiId: !Ref OrdersRestApi
      ResourceId: !Ref OrdersResource
      HttpMethod: POST
      AuthorizationType: NONE
      Integration:
        Type: AWS_PROXY
        IntegrationHttpMethod: POST
        Uri: !Sub >-
          arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${OrderAPIFunction.Arn}/invocations

  # --- Lambda Permission for API Gateway ---
  ApiGatewayLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref OrderAPIFunction
      Action: lambda:InvokeFunction
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub 'arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${OrdersRestApi}/*'

  # --- Deploy API to 'dev' stage with X-Ray ---
  ApiDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn:
      - OrdersGetMethod
      - OrdersPostMethod
    Properties:
      RestApiId: !Ref OrdersRestApi

  ApiStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      RestApiId: !Ref OrdersRestApi
      DeploymentId: !Ref ApiDeployment
      StageName: dev
      TracingEnabled: true
      MethodSettings:
        - HttpMethod: '*'
          ResourcePath: '/*'
          LoggingLevel: INFO

  # ============================================================
  # CloudWatch Alarm: Lambda Errors
  # ============================================================
  OrderAPIErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: OrdersAPI-Lambda-Errors
      AlarmDescription: Alert when OrderAPI Lambda has errors
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
        - Name: FunctionName
          Value: !Ref OrderAPIFunction
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 2
      Threshold: 5
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref OrderNotificationsTopic

# ============================================================
# Outputs
# ============================================================
Outputs:
  ApiUrl:
    Description: API Gateway Invoke URL (use this for testing and canary)
    Value: !Sub 'https://${OrdersRestApi}.execute-api.${AWS::Region}.amazonaws.com/dev'

  ApiUrlOrders:
    Description: Full URL to /orders endpoint
    Value: !Sub 'https://${OrdersRestApi}.execute-api.${AWS::Region}.amazonaws.com/dev/orders'

  SNSTopicArn:
    Description: SNS Topic ARN
    Value: !Ref OrderNotificationsTopic

  CanaryBucketName:
    Description: S3 bucket for Synthetics canary artifacts
    Value: !Ref CanaryArtifactsBucket

  DynamoDBTableName:
    Description: DynamoDB table name
    Value: !Ref OrdersTable

  OrderAPIFunctionArn:
    Description: OrderAPI Lambda ARN
    Value: !GetAtt OrderAPIFunction.Arn

  NotifyServiceFunctionArn:
    Description: NotifyService Lambda ARN
    Value: !GetAtt NotifyServiceFunction.Arn

import json
import boto3
import os
from aws_xray_sdk.core import xray_recorder, patch_all

patch_all()

sns = boto3.client('sns')
TOPIC_ARN = os.environ['TOPIC_ARN']

def handler(event, context):
    order_id = event.get('orderId', 'unknown')
    
    # Add X-Ray annotation (searchable)
    subsegment = xray_recorder.current_subsegment()
    if subsegment:
        subsegment.put_annotation('orderId', order_id)
        subsegment.put_annotation('service', 'NotifyService')
    
    # Custom subsegment for business logic
    with xray_recorder.in_subsegment('PrepareNotification') as seg:
        message = {
            'orderId': order_id,
            'status': 'CREATED',
            'message': f'Order {order_id} has been created successfully'
        }
        seg.put_metadata('notification', message)
    
    # Publish to SNS (auto-traced by patch_all)
    with xray_recorder.in_subsegment('PublishSNS'):
        sns.publish(
            TopicArn=TOPIC_ARN,
            Subject=f'New Order: {order_id}',
            Message=json.dumps(message)
        )
    
    print(json.dumps({
        'level': 'INFO',
        'message': 'Notification sent',
        'orderId': order_id
    }))
    
    return {'statusCode': 200, 'body': 'Notification sent'}

import json
import boto3
import uuid
import time
import os
from aws_xray_sdk.core import xray_recorder, patch_all

patch_all()

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Orders')
lambda_client = boto3.client('lambda')

def handler(event, context):
    http_method = event.get('httpMethod', 'GET')
    
    # X-Ray annotations
    subsegment = xray_recorder.current_subsegment()
    if subsegment:
        subsegment.put_annotation('httpMethod', http_method)
        subsegment.put_annotation('service', 'OrderAPI')
    
    if http_method == 'POST':
        return create_order(event)
    elif http_method == 'GET':
        return get_order(event)
    else:
        return list_orders()

def create_order(event):
    body = json.loads(event.get('body', '{}'))
    order_id = str(uuid.uuid4())[:8]
    
    with xray_recorder.in_subsegment('ValidateOrder') as seg:
        product = body.get('product', 'Unknown')
        quantity = body.get('quantity', 1)
        seg.put_annotation('orderId', order_id)
        seg.put_metadata('orderDetails', body)
        
        if quantity <= 0:
            return response(400, {'error': 'Invalid quantity'})
    
    # Simulate processing time (visible in X-Ray)
    with xray_recorder.in_subsegment('ProcessPayment'):
        time.sleep(0.1)  # Simulate payment processing
    
    # Write to DynamoDB (auto-traced)
    with xray_recorder.in_subsegment('SaveOrder'):
        item = {
            'orderId': order_id,
            'product': product,
            'quantity': quantity,
            'status': 'CREATED',
            'timestamp': int(time.time())
        }
        table.put_item(Item=item)
    
    # Invoke NotifyService async (auto-traced)
    with xray_recorder.in_subsegment('InvokeNotify'):
        lambda_client.invoke(
            FunctionName='NotifyService',
            InvocationType='Event',  # Async
            Payload=json.dumps({'orderId': order_id})
        )
    
    print(json.dumps({
        'level': 'INFO', 'message': 'Order created',
        'orderId': order_id, 'product': product
    }))
    
    return response(201, {'orderId': order_id, 'status': 'CREATED'})

def get_order(event):
    params = event.get('queryStringParameters') or {}
    order_id = params.get('orderId')
    
    if not order_id:
        return response(400, {'error': 'orderId required'})
    
    xray_recorder.current_subsegment().put_annotation('orderId', order_id)
    
    result = table.get_item(Key={'orderId': order_id})
    item = result.get('Item')
    
    if not item:
        print(json.dumps({
            'level': 'ERROR', 'message': 'Order not found',
            'orderId': order_id
        }))
        return response(404, {'error': 'Order not found'})
    
    return response(200, item)

def list_orders():
    with xray_recorder.in_subsegment('ScanOrders'):
        result = table.scan(Limit=20)
    return response(200, {'orders': result.get('Items', [])})

def response(status_code, body):
    return {
        'statusCode': status_code,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body': json.dumps(body, default=str)
    }

# Set API URL
API_URL="https://xxxxxx.execute-api.region.amazonaws.com/dev"

# Tạo order
curl -X POST "$API_URL/orders" \
  -H "Content-Type: application/json" \
  -d '{"product": "Laptop", "quantity": 2}'

# Response: {"orderId": "abc12345", "status": "CREATED"}

# Get order
curl "$API_URL/orders?orderId=abc12345"

# List orders
curl "$API_URL/orders"

# Tạo thêm vài orders để có data
for i in $(seq 1 10); do
  curl -s -X POST "$API_URL/orders" \
    -H "Content-Type: application/json" \
    -d "{\"product\": \"Product-$i\", \"quantity\": $i}"
  sleep 1
done

Client → API Gateway → OrderAPI (Lambda)
                            ├── DynamoDB (Orders)
                            └── NotifyService (Lambda)
                                    └── SNS (OrderNotifications)

Trace Timeline (ví dụ POST /orders):
├── API Gateway (5ms)
│   └── OrderAPI Lambda (350ms)
│       ├── Initialization (150ms) ← Cold start!
│       ├── Invocation (200ms)
│       │   ├── ValidateOrder (1ms)
│       │   ├── ProcessPayment (100ms) ← Simulated delay
│       │   ├── SaveOrder - DynamoDB PutItem (15ms)
│       │   └── InvokeNotify - Lambda Invoke (5ms)
│       └── Overhead (2ms)
└── Total: ~355ms

# Tìm tất cả POST requests
http.method = "POST"

# Tìm orders cụ thể (dùng annotation)
annotation.orderId = "abc12345"

# Tìm requests chậm (> 500ms)
responsetime > 0.5

# Tìm errors
error = true

# Tìm requests tới OrderAPI service
service("OrderAPI")

# Combine filters
annotation.service = "OrderAPI" AND http.method = "POST" AND responsetime > 0.3

# Gọi GET không có orderId → 400 error
curl "$API_URL/orders?orderId=nonexistent"

# Gọi POST với invalid data
curl -X POST "$API_URL/orders" \
  -H "Content-Type: application/json" \
  -d '{"product": "Test", "quantity": -1}'

import random

# Thêm vào đầu function create_order():
def create_order(event):
    # Simulate random failures (30% chance)
    if random.random() < 0.3:
        print(json.dumps({
            'level': 'ERROR',
            'message': 'Database connection timeout',
            'errorType': 'TimeoutError'
        }))
        raise Exception('Simulated database timeout')
    
    # ... rest of code

# Gọi 30 requests để tạo mix success/failure
for i in $(seq 1 30); do
  curl -s -X POST "$API_URL/orders" \
    -H "Content-Type: application/json" \
    -d "{\"product\": \"StressTest-$i\", \"quantity\": 1}" &
done
wait
echo "Done sending requests"

ServiceLens Service Map → Thấy OrderAPI node đỏ
  → Click node → Overview tab → Fault rate 30%
  → Traces tab → Filter: error = true
  → Click trace → Thấy Exception: "Simulated database timeout"
  → Related logs → Thấy ERROR log với details
  → Kết luận: Database connection issue

const { URL } = require('url');
const synthetics = require('Synthetics');
const log = require('SyntheticsLogger');

const apiCanaryBlueprint = async function () {
    const API_URL = 'https://xxxxxx.execute-api.region.amazonaws.com/dev';
    
    // Step 1: Create Order (POST)
    log.info('Step 1: Creating order...');
    let createResponse = await synthetics.executeHttpStep(
        'Create Order',
        new URL(`${API_URL}/orders`),
        {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({
                product: 'CanaryTestProduct',
                quantity: 1
            })
        }
    );

    // Parse response
    let responseBody = JSON.parse(createResponse.body || '{}');
    let orderId = responseBody.orderId;
    log.info(`Order created: ${orderId}`);
    
    if (!orderId) {
        throw new Error('Failed to create order - no orderId returned');
    }
    
    // Step 2: Get Order (GET)
    log.info(`Step 2: Getting order ${orderId}...`);
    let getResponse = await synthetics.executeHttpStep(
        'Get Order',
        new URL(`${API_URL}/orders?orderId=${orderId}`)
    );
    
    let orderData = JSON.parse(getResponse.body || '{}');
    log.info(`Order data: ${JSON.stringify(orderData)}`);
    
    // Verify order data
    if (orderData.product !== 'CanaryTestProduct') {
        throw new Error(`Expected product 'CanaryTestProduct', got '${orderData.product}'`);
    }
    
    // Step 3: List Orders (GET)
    log.info('Step 3: Listing orders...');
    let listResponse = await synthetics.executeHttpStep(
        'List Orders',
        new URL(`${API_URL}/orders`)
    );
    
    let listData = JSON.parse(listResponse.body || '{}');
    log.info(`Total orders: ${listData.orders ? listData.orders.length : 0}`);
    
    log.info('All API tests passed!');
};

exports.handler = async () => {
    return await apiCanaryBlueprint();
};

# Thêm vào đầu handler:
def handler(event, context):
    # Force error for canary testing
    raise Exception("Intentional outage for testing")

Service Map với Synthetics:

[Canary: orders-api-health] → API Gateway → OrderAPI → DynamoDB
                                                      → NotifyService → SNS

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 10

┌─────────────────────────────────────────────────────┐
│ OrdersApp-Observability Dashboard                    │
├──────────────┬──────────────┬───────────────────────┤
│ API Avail:   │ API Latency  │ API 5XX Errors        │
│   99.5%      │ [line graph] │ [line graph]          │
├──────────────┴──────────────┴───────────────────────┤
│ Lambda Duration (p50/p99)  │ Lambda Errors/Throttles│
│ [line graph]               │ [line graph]           │
├────────────────────────────┴────────────────────────┤
│ DynamoDB RCU/WCU                                     │
│ [line graph]                                         │
├──────────────────────────────────────────────────────┤
│ Recent Errors (Logs Insights)                        │
│ 2025-02-14 10:23 ERROR Order not found orderId=xyz  │
├──────────────────────────────────────────────────────┤
│ Alarms: ✅ All OK                                    │
└──────────────────────────────────────────────────────┘