4.1.3 Query Logs to Find Relevant Data
Query Logs to Find Relevant Data
CloudWatch Logs Insights
-- Find errors
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20
-- P99 latency
stats pct(@duration, 99) as p99 by bin(1h)
-- Cold starts count
filter @message like /Init Duration/
| stats count(*) as coldStarts by bin(5m)
-- Top 10 slowest invocations
fields @duration, @requestId
| sort @duration desc
| limit 10
-- Error rate
stats count(*) as total,
sum(strcontains(@message, "ERROR")) as errors
| display errors/total * 100 as error_rate
-- Find specific request
fields @timestamp, @message
| filter @requestId = "abc-123-def"
Log Structure
Log Group: /aws/lambda/my-function
└── Log Stream: 2025/01/15/[$LATEST]abc123
└── Log Events
Log Retention
| Setting | Mô tả |
|---|
| 1 day → 10 years | Configurable |
| Never expire | Default (costs accumulate!) |
| Export to S3 | Long-term archival |
| Subscription filter | Stream to Lambda/Kinesis/OpenSearch |
CloudTrail Logs (API Audit)
fields @timestamp, eventName, errorCode, errorMessage, userIdentity.arn
| filter errorCode = "AccessDenied"
| sort @timestamp desc
| limit 20
Amazon Athena (Query Logs trong S3)
Athena là serverless interactive query service, dùng SQL để query data trực tiếp trong S3.
| Feature | Mô tả |
|---|
| Engine | Presto-based SQL |
| Pricing | Pay per query ($5/TB scanned) |
| Data formats | CSV, JSON, Parquet, ORC, Avro |
| Schema | Glue Data Catalog |
| Use case | Ad-hoc queries trên S3 data, log analysis |
Athena vs CloudWatch Logs Insights
| Athena | CloudWatch Logs Insights |
|---|
| Data source | S3 | CloudWatch Logs |
| Query language | Standard SQL | Custom query syntax |
| Best for | Large-scale log analysis, archived logs | Real-time log troubleshooting |
| Cost | Per TB scanned | Per GB scanned |
| Performance | Tốt hơn cho large datasets | Tốt cho recent logs |
Common Use Cases cho Developer
-- Query CloudTrail logs exported to S3
SELECT eventname, sourceipaddress, errorcode
FROM cloudtrail_logs
WHERE errorcode = 'AccessDenied'
AND eventtime > '2025-01-01'
ORDER BY eventtime DESC;
-- Query ALB access logs trong S3
SELECT target_status_code, count(*) as cnt
FROM alb_logs
WHERE target_status_code >= 500
GROUP BY target_status_code;
Optimize Athena Queries
| Technique | Benefit |
|---|
| Parquet/ORC format | Columnar → scan ít data hơn |
| Partitioning | Chỉ scan relevant partitions |
| Compression | Giảm data scanned |
| LIMIT clause | Giảm output size |
Exam Tip: Logs Insights = ad-hoc queries on CloudWatch Logs. Athena = SQL queries trên S3 data (archived logs, large datasets). CloudTrail for API call auditing. Parquet format giảm Athena cost. Set log retention — default never expires.