aws-connect-new

Source: ZoosGlobal/aws-connect-new Visibility: Private This page is automatically synchronized from the repository README. Do not edit this generated file directly.

---

S3 → Datadog Log Forwarder

---

---

S3 Event → Lambda → Parse File → Batch → Datadog Logs API → Searchable Logs

Serverless pipeline that watches one or more S3 buckets, parses uploaded files (TXT, LOG, JSON, CSV), and forwards them to Datadog as structured logs — with automatic batching, retry, and failure persistence built in.

---

Configuration

Before deploying, update the following files with your environment-specific values.

Copy terraform.tfvars.example to terraform.tfvars

Provide the following values:

• AWS Access Key ID
• AWS Secret Access Key
• AWS region
• Datadog API Key

Example:

aws_access_key_id     = "YOUR_AWS_ACCESS_KEY"
aws_secret_access_key = "YOUR_AWS_SECRET_ACCESS_KEY"
aws_region            = "Your S3 Region"

dd_api_key = "YOUR_DATADOG_API_KEY"

Update variables.tf

Modify the bucket_prefixes variable to match your environment by specifying the S3 bucket names and folder prefixes to monitor.

Example:

bucket_prefixes = {
  "your-s3-bucket-name" = "zoos/"
}

Deploy

Run the following commands:

terraform init
terraform apply

❓ The Problem — Why This Exists

Teams routinely need to get ad-hoc files — exports, batch job outputs, application logs, CSV reports — into a centralized observability platform, but doing this reliably at scale has real friction:

• No agent-based path for ad-hoc files — Files dropped into S3 by upstream jobs, partners, or scheduled exports have no Datadog Agent sitting on them; there’s nothing watching the bucket and shipping content out by default.
• One-off scripts don’t survive failure — A simple “read file, POST to Datadog” script works until Datadog has a blip, a network hiccup occurs, or a payload is malformed — at which point logs are silently lost with no record they ever existed.
• Different file types need different parsing logic — TXT, JSON (single object or array), and CSV all carry data differently; a single naive parser either drops structure or breaks entirely on a CSV that wasn’t expected.
• Per-log API calls don’t scale — Sending one HTTP request per log line is slow and burns through API rate limits fast once file volume grows beyond a handful of rows.
• Multi-bucket / multi-team setups need isolation — Multiple buckets, each with their own folder structure and ownership, need to be watched independently without one team’s files leaking into another’s blast radius — or requiring a new Lambda per bucket.
• Standing up infrastructure for this is overkill — A dedicated log shipper, EC2 instance, or container running 24/7 just to occasionally forward files is wasted spend for a job that’s fundamentally event-driven.

---

✅ The Solution — What We Built

We built a lightweight, serverless AWS Lambda function that triggers on S3 upload events, parses the file based on extension, batches the resulting logs, and ships them to Datadog — with failed deliveries automatically persisted to S3 and retried on the next invocation, so no log is ever silently dropped.

Architecture

S3 Bucket (file uploaded to configured prefix)
        ↓
  S3 ObjectCreated:* Event
        ↓
  Lambda: s3-datadog-parser
        ↓
  ┌──────────────────────────────────────────┐
  │ 1. Retry any queued failed/ payloads       │
  │ 2. Read & parse new file (txt/json/csv)    │
  │ 3. Build structured log objects            │
  │ 4. Batch logs (default 100/request)        │
  └──────────────────────────────────────────┘
        ↓
  Datadog Logs Intake API
        ↓
   ✅ Success → done        ❌ Failure → s3://bucket/failed/*.json (retried next run)

Key Design Decisions

Decision	Rationale
Single Lambda, multi-bucket via config map	BUCKET_PREFIXES maps bucket → prefix; adding a new bucket is a one-line config change, not a new deployment
Batched Datadog delivery, not per-log POSTs	Logs are grouped into batches (default 100) before sending, cutting API calls dramatically for large CSV/JSON files
Exponential backoff on send failures	Transient Datadog issues (rate limits, brief outages) are retried with increasing delay instead of failing immediately
Failed payloads persisted to S3, not dropped	Every undelivered batch is written to a failed/ prefix as JSON — nothing is lost even on sustained Datadog downtime
Failed-file retry runs before new file processing	Every new trigger first clears the backlog for that bucket, so failures self-heal as soon as traffic resumes
Parallel retry via thread pool	Multiple queued failed files are retried concurrently (bounded by MAX_WORKERS), not serially
Extension-based parser dispatch	TXT/LOG, JSON (object or array), and CSV each get dedicated handling so structure is preserved, not flattened into raw text
IAM scoped to configured buckets only	The Lambda’s S3 policy is built from data.aws_s3_bucket lookups — no wildcard Resource: "*"
Self-trigger loop prevention	Events from the failed/ prefix are explicitly ignored so the Lambda never re-triggers itself on its own retry writes

Outcome

• Files dropped into any configured S3 bucket/prefix appear as structured, searchable logs in Datadog within seconds — no manual export or shipping step.
• Datadog outages or rate-limit blips no longer mean lost logs — failed payloads queue in S3 and are automatically recovered on the next trigger.
• Adding a new bucket to monitor is a one-line Terraform variable change, not new infrastructure.
• The entire pipeline runs serverless with no idle compute cost — Lambda only runs when a file actually lands.

---

📁 Directory Structure

.
├── lambda/
│   └── lambda_function.py     # Main handler: parse, batch, send, retry
├── main.tf                    # Terraform: IAM, Lambda, S3 notifications
├── variables.tf                # Input variable declarations
├── terraform.tfvars            # Your values — gitignored
└── README.md                   # This file

⚠️ lambda_function.py must live inside a lambda/ directory — data.archive_file zips the contents of ./lambda directly, and the handler is referenced as lambda_function.lambda_handler.

---

📊 Supported File Types Reference

.txt / .log — Raw Content

One log per file

Field	Description
content	Full file content as-is

---

.json — Structured Objects

One log per array item, or one log for a single object

Input Shape	Behavior
{ ... } (object)	Fields merged directly into the base log
[ {...}, {...} ] (array)	One log generated per array item
[ "a", "b" ] (non-dict items)	Each item wrapped in a content field
Invalid JSON	Caught and persisted to failed/ with status: invalid_json 🔴

---

.csv — Tabular Rows

One log per row

Field	Description
row_data	Parsed row as a dict via csv.DictReader

---

Unsupported Extensions

Field	Description
status	unsupported_file_type — still logged for visibility, not silently dropped

Every log, regardless of type, carries this base schema:

Field	Description
service	aws-connect
source	aws
bucket	Source bucket name
file_name	File name only (no path)
file_path	Full S3 key
file_type	File extension
message	File processed from S3: <key>
ddtags	env:dev,team:devops

---

⚙️ System Requirements

Requirement	Version
Terraform	≥ 1.x
AWS Provider	Configured with IAM/Lambda/S3 permissions
Python Runtime	3.12 (Lambda-managed)
Existing S3 Bucket(s)	One per entry in bucket_prefixes
Datadog	API key with Logs ingestion permission

---

1️⃣ Configure Terraform Variables

variables.tf

variable "dd_api_key" {
  type      = string
  sensitive = true
}

variable "bucket_prefixes" {
  type = map(string)
}

terraform.tfvars (never commit — add to .gitignore)

dd_api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

bucket_prefixes = {
  "datadog-file-processing-divyanshu"  = "zoos/"
  "datadog-file-processing-divyanshu2" = "input/"
  "datadog-file-processing-divyanshu3" = "logs/"
}

---

2️⃣ Deploy

terraform init
terraform plan
terraform apply

main.tf provisions:

[1/6]  IAM role for Lambda execution
[2/6]  IAM policy — S3 access scoped to configured buckets only
[3/6]  AWSLambdaBasicExecutionRole attachment — CloudWatch Logs
[4/6]  Lambda function (Python 3.12, 1024 MB, 300s timeout)
[5/6]  S3 bucket notifications — ObjectCreated:* per bucket
[6/6]  Lambda resource permissions — one per bucket

---

3️⃣ Manual Validation

Always test with a sample file before relying on production traffic.

# Upload a test file to a configured prefix
aws s3 cp sample.csv s3://datadog-file-processing-divyanshu2/input/sample.csv

Expected CloudWatch log output:

Processing s3://datadog-file-processing-divyanshu2/input/sample.csv (type=csv)
Datadog attempt 1: HTTP 202 (12 logs)

Verify in Datadog: Logs → Search → service:aws-connect file_name:sample.csv

---

4️⃣ Destroy

terraform destroy

📌 CloudWatch log groups auto-created by Lambda on first run are not managed by this Terraform config and will persist after destroy. Clean up manually via Console or CLI if needed.

---

5️⃣ Execution Timeline

File uploaded → Lambda triggered
  ├── Retry pass
  │     └── Replay any queued failed/ payloads for this bucket  → parallel, ThreadPoolExecutor
  │
  ├── Parse pass
  │     ├── .txt / .log   → single log, full content
  │     ├── .json         → 1 log (object) or N logs (array)
  │     └── .csv          → 1 log per row
  │
  └── Delivery pass
        ├── Batch logs (size: DD_BATCH_SIZE)
        ├── POST to Datadog Logs API
        │     ├── Success → done
        │     └── Failure → retry × DD_MAX_ATTEMPTS (exponential backoff)
        │           └── Still failing → persist to s3://bucket/failed/*.json

---

6️⃣ Recommended Datadog Monitors

🔴 Forwarder Errors — Immediate Alert

Query   : sum(last_15m):sum:aws.lambda.errors{functionname:s3-datadog-parser}.as_count() > 0
Alert   : > 0
Message : 🔴 s3-datadog-parser Lambda is erroring — check CloudWatch Logs.

⚠️ Failed Payload Backlog Growing

Query   : avg(last_1h):avg:aws.s3.bucket_size_bytes{bucketname:*, filtername:failed/} > 0
Alert   : > 0
Message : ⚠️ Failed payloads are accumulating in S3 — Datadog delivery may be degraded.

⚠️ Lambda Duration Approaching Timeout

Query   : max(last_15m):max:aws.lambda.duration{functionname:s3-datadog-parser} > 250000
Warning : > 250s (timeout is 300s)
Message : ⚠️ Lambda execution time is approaching the configured timeout — consider larger files or splitting input.

---

7️⃣ Datadog Dashboard Queries

Widget	Query
Logs ingested per source bucket	count:aws-connect.logs{*} by {bucket}
Logs by file type	count:aws-connect.logs{*} by {file_type}
Failed / invalid JSON events	count:aws-connect.logs{status:invalid_json}
Processing failures	count:aws-connect.logs{status:processing_failed}
Unsupported file type hits	count:aws-connect.logs{status:unsupported_file_type}
Lambda invocation count	sum:aws.lambda.invocations{functionname:s3-datadog-parser}
Lambda error rate	sum:aws.lambda.errors{functionname:s3-datadog-parser}

---

🔧 Environment Variables (Lambda Runtime)

Variable	Default	Description
DD_API_KEY	— (required)	Datadog API key for the DD-API-KEY header
FAILED_PREFIX	failed/	S3 prefix where undelivered payloads are stored
BUCKET_PREFIXES	{}	JSON map of bucket → prefix, mirrors the Terraform variable
DD_BATCH_SIZE	100	Logs bundled per Datadog API request
DD_MAX_ATTEMPTS	3	Retry attempts before a batch is marked failed
MAX_WORKERS	5	Thread pool size for parallel record/retry processing

---

🛡️ Production Features

Feature	Status
Multi-bucket support via config map	✅
TXT / LOG raw content forwarding	✅
JSON object & array parsing	✅
CSV row-by-row parsing	✅
Batched Datadog delivery	✅
Exponential backoff on send failure	✅
Failed payload persistence to S3	✅
Automatic retry of queued failures	✅
Parallel retry via thread pool	✅
Self-trigger loop prevention	✅
IAM scoped to configured buckets only	✅
Unconfigured bucket/prefix skip (no false processing)	✅
Invalid JSON captured, not dropped	✅

---

✅ Production Checklist

• terraform.tfvars populated with real bucket names and Datadog API key
• terraform apply completed successfully
• Test file uploaded and confirmed in CloudWatch Logs
• Logs visible in Datadog Logs Explorer (service:aws-connect)
• failed/ prefix checked — empty after successful test run
• IAM policy reviewed — no wildcard Resource: "*"
• Monitor created for Lambda errors
• Monitor created for failed payload backlog
• Default ddtags (env:dev,team:devops) updated for production environment

---

🚨 Troubleshooting

Issue	Fix
Logs not appearing in Datadog	Check CloudWatch Logs for Datadog attempt lines and HTTP status codes
ResourceAlreadyExistsException on terraform apply	A CloudWatch log group from a prior deploy already exists — import it: terraform import aws_cloudwatch_log_group.lambda_logs /aws/lambda/<function-name>
File uploaded but nothing happens	Confirm the key starts with the exact prefix configured in bucket_prefixes for that bucket
invalid_json status in logs	Source JSON file is malformed — check the original file, not the Lambda
Files queuing in failed/ and not clearing	No new files have triggered the bucket since the failure — retry only runs on new uploads
terraform apply shows Resource = "*" still in plan	You’re running an older version of main.tf — pull the latest scoped IAM policy
Lambda timing out on large files	Increase timeout in aws_lambda_function.s3_parser, or reduce file size per upload

---

👤 Author

Name	Divyanshu Kumar
Title	DevOps Engineer
Organisation	Zoos Global
Email	[email protected]

---

Version 1.0.0

Serverless, batched, and built to never silently lose a log.