A Guide to Kubernetes Topology Aware Routing

Contents

A practical overview of Topology Aware Routing, a feature for optimizing network traffic in Kubernetes

Overview

Topology Aware Routing (formerly Topology Aware Hints) was introduced as a beta feature in Kubernetes 1.23 and renamed to its current name in 1.27. Amazon EKS supports it from 1.24. It aims to reduce network latency and costs by keeping traffic within the same Availability Zone.

However, it does not work in every environment and is effective only when certain conditions are met. Alongside the theory, this guide covers various practical situations and their solutions.

Naming History

Core Concepts

Basic Operation

Topology Aware Routing makes kube-proxy use topology information when choosing Service endpoints.

yaml
# Service 예시
apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.kubernetes.io/topology-mode: "auto"
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080

Key Benefits

  1. Reduced network latency: Minimize latency through communication within the same AZ
  2. Data transfer cost savings: Eliminate cross-AZ traffic costs
  3. Automatic optimization: Automatically optimize routing without manual configuration

How It Works

The EndpointSlice Controller's Role

The EndpointSlice Controller assigns hints through the following process.

  1. Check Pod distribution: Count the Pods in each zone
  2. Proportional allocation: Allocate endpoints in proportion to the Pod count in each zone
  3. Generate hints: Add a hints.forZones field to each endpoint
yaml
# EndpointSlice 예시
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: my-service-abc123
endpoints:
- addresses:
  - "10.1.2.3"
  hints:
    forZones:
    - name: "zone-a"

kube-proxy Behavior

kube-proxy prioritizes endpoints with hints.

bash
# iptables 규칙 예시 (실제로는 더 복잡)
-A KUBE-SVC-XYZ -m comment --comment "zone-a endpoints only" \
  -j KUBE-SEP-LOCAL-ZONE

Requirements and Limitations

Conditions for Use

  1. Even distribution: Workloads must be evenly distributed across all zones
  2. Sufficient endpoints: At least 1 endpoint must exist in each zone
  3. Service types: ClusterIP, NodePort, and LoadBalancer are supported

Conditions That Prevent Use

yaml
# 불균등 분산 예시 (동작하지 않음)
# Zone A: 10 pods
# Zone B: 2 pods  
# Zone C: 1 pod

Common Practical Mistakes

1. A Single Pod in Development (Does Not Work)

yaml
# 개발환경에서 흔한 실수
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 1  # ← 파드 1개만 있으면 TAR 비활성화!
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.kubernetes.io/topology-mode: "auto"  # ← 설정해도 무시됨

2. A Redundant Setup with 2 Instances (Usually Does Not Work)

yaml
# 비용 절약을 위한 2대 이중화 - TAR 제한적
spec:
  replicas: 2  # 3개 zone에서는 불균등 분산으로 TAR 비활성화
  
# 실제 배치 예시:
# Zone A: 1개 파드
# Zone B: 1개 파드  
# Zone C: 0개 파드 ← 빈 zone으로 인해 TAR 비활성화

Limitations

Configuration

1. Set the Service Annotation

yaml
apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    # 자동 모드 (권장)
    service.kubernetes.io/topology-mode: "auto"
spec:
  selector:
    app: my-app
  ports:
  - port: 80

2. Deployment Strategy

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 6  # zone당 2개씩 균등 배포
  template:
    spec:
      # Zone 간 균등 분산을 위한 설정
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: my-app

Performance Impact Analysis

Latency Improvements

Scenario Default Configuration Topology Aware Hints Improvement
Same-AZ communication ~1ms ~0.5ms 50%
Cross-AZ communication ~3-5ms ~0.5ms 80-90%

Cost Savings

bash
# AWS 데이터 전송 비용 예시
# 크로스-AZ: $0.01/GB
# 동일-AZ: $0.00/GB

# 월 1TB 데이터 전송 시
# 기존: $10.24 (크로스-AZ)
# 적용 후: $0 (동일-AZ)
# 절약: 100%

Monitoring and Troubleshooting

1. Check the Cluster's Zone Configuration

bash
# Zone 수 확인 (가장 중요!)
kubectl get nodes -o jsonpath='{.items[*].metadata.labels.topology\.kubernetes\.io/zone}' | tr ' ' '\n' | sort -u | wc -l

# Zone 목록 확인
kubectl get nodes -o custom-columns="NAME:.metadata.name,ZONE:.metadata.labels['topology\.kubernetes\.io/zone']"

# Zone별 노드 분포 확인
kubectl get nodes -o json | jq -r '.items[] | .metadata.labels["topology.kubernetes.io/zone"]' | sort | uniq -c

Example output:

bash
   2 ap-northeast-2a
   2 ap-northeast-2b  
   2 ap-northeast-2c
# → 3개 zone, TAR 적용 가능 ✅

2. Comprehensive Zone Information Check Script

bash
#!/bin/bash
# zone-check.sh - TAR 적용 가능성 자동 판단

ZONE_COUNT=$(kubectl get nodes -o jsonpath='{.items[*].metadata.labels.topology\.kubernetes\.io/zone}' | tr ' ' '\n' | sort -u | wc -l)

echo "=== Topology Aware Routing 적용 가능성 분석 ==="
echo "📊 총 Zone 수: $ZONE_COUNT"

case $ZONE_COUNT in
    1)
        echo "❌ TAR 적용 불가 (Single AZ)"
        echo "💡 권장: 기본 로드밸런싱 사용"
        ;;
    2) 
        echo "⚠️ TAR 제한적 적용 가능"
        echo "💡 권장: 파드 2개 이상 + 균등 분산 필수"
        ;;
    3)
        echo "✅ TAR 최적화 가능"
        echo "💡 권장: 파드 최소 3개 (zone당 1개씩)"
        echo "🚀 이상적: 파드 9개 이상 (zone당 3개씩)"
        ;;
    *)
        echo "✅ TAR 최적화 가능"
        echo "💡 권장: 파드 최소 ${ZONE_COUNT}개 (zone당 1개씩)"
        ;;
esac

3. Check EndpointSlice Hints

bash
# EndpointSlice 힌트 확인
kubectl get endpointslices -o yaml | grep -A5 hints

# 특정 서비스의 EndpointSlice 상세 확인
kubectl describe endpointslices -l kubernetes.io/service-name=my-service

2. Check Service Status

bash
# Service 이벤트 확인
kubectl describe service my-service

# 힌트 관련 로그 확인
kubectl logs -n kube-system -l component=kube-proxy | grep topology

3. Troubleshoot Common Problems

Hints Are Not Generated

bash
# Pod 분산 상태 확인
kubectl get pods -o wide --show-labels | grep my-app

# Zone별 Pod 수 확인
kubectl get nodes --show-labels | grep topology.kubernetes.io/zone

Resolve Uneven Distribution

yaml
# PodDisruptionBudget을 통한 안정적 배포
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app

Precautions and Practical Recommendations

1. TAR Adoption Guide by Environment

Development/Test Environments

yaml
# 비용 우선 → TAR 비활성화 권장
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 1-2  # 최소 리소스
---
apiVersion: v1
kind: Service  
metadata:
  annotations:
    # TAR 명시적 비활성화 (불필요한 오버헤드 방지)
    service.kubernetes.io/topology-mode: "disabled"

Production Environments

yaml
# 성능 + 비용 절약 → TAR 적극 활용
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 6  # zone 수(3) × 2 = 균등 분산
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.kubernetes.io/topology-mode: "auto"

2. Optimal Configuration by Zone Count

Zone Count Minimum Pod Count Recommended Pod Count TAR Effect
1 N/A N/A ❌ Not possible
2 2 4 or more ⚠️ Limited
3 3 9 or more ✅ Optimal
4+ Zone count zones × 3 ✅ Optimal

3. Cost vs Performance Analysis

Cost Perspective

bash
# 시나리오 1: TAR 미적용 (2대 이중화)
월 인스턴스 비용: $200 × 2 = $400
크로스-AZ 데이터 전송: $10-50/월 (트래픽에 따라)
총비용: $410-450/월

# 시나리오 2: TAR 적용 (3대 분산)  
월 인스턴스 비용: $200 × 3 = $600
크로스-AZ 데이터 전송: $0-5/월 (거의 제거)
총비용: $600-605/월

# ROI 계산: 높은 트래픽일수록 TAR이 유리!

4. Availability vs Performance Trade-offs

yaml
# 높은 가용성이 필요한 경우 비활성화 고려
metadata:
  annotations:
    service.kubernetes.io/topology-mode: "disabled"

5. Dynamic Scaling Considerations

yaml
# HPA 사용 시 zone 균형 유지 필수!
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3   # zone 수와 동일하게 설정
  maxReplicas: 12  # zone 수의 배수로 설정

Hands-on Example

Complete Configuration Example

yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
      # 균등 분산 보장
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web-app

---
apiVersion: v1
kind: Service
metadata:
  name: web-service
  annotations:
    service.kubernetes.io/topology-mode: "auto"
spec:
  selector:
    app: web-app
  ports:
  - name: http
    port: 80
    targetPort: 80
  type: ClusterIP

Validation Script

bash
#!/bin/bash
# topology-check.sh

echo "=== Topology Aware Hints 검증 ==="

# 1. Service 힌트 모드 확인
echo "1. Service 어노테이션 확인:"
kubectl get service web-service -o jsonpath='{.metadata.annotations}' | jq

# 2. EndpointSlice 힌트 확인
echo "2. EndpointSlice 힌트 확인:"
kubectl get endpointslices -l kubernetes.io/service-name=web-service -o yaml | grep -A10 hints

# 3. Pod 분산 상태 확인
echo "3. Pod zone별 분산 상태:"
kubectl get pods -l app=web-app -o wide | awk '{print $7}' | sort | uniq -c

# 4. kube-proxy 로그 확인 (선택사항)
echo "4. kube-proxy 토폴로지 관련 로그:"
kubectl logs -n kube-system -l component=kube-proxy --tail=50 | grep -i topology || echo "로그 없음"

Performance Measurement

Benchmark Test

bash
# 성능 측정용 Pod 배포
kubectl run perf-test --image=busybox --rm -it --restart=Never -- sh

# 서비스 응답 시간 측정
for i in {1..100}; do
  time wget -qO- http://web-service/health
done | grep real | awk '{sum+=$2} END {print "평균:", sum/NR "초"}'

Conclusion and Practical Decision Guide

Topology Aware Routing can significantly improve network performance and cost efficiency in Kubernetes clusters. However, it is not suitable for every environment, so careful evaluation is needed.

TAR Adoption Decision Flowchart

bash
1. 클러스터 Zone 수는?
   └─ 1개 Zone → ❌ TAR 불가, 끝
   └─ 2개 Zone → 다음 단계
   └─ 3개+ Zone → 다음 단계

2. 환경 성격은?
   └─ 개발/테스트 → ❌ TAR 불필요 (비용 우선)
   └─ 프로덕션 → 다음 단계

3. 트래픽 특성은?
   └─ 저트래픽 → △ TAR 효과 미미
   └─ 고트래픽 → 다음 단계

4. 파드 확장 가능성?
   └─ 2대 고정 → ❌ TAR 제한적
   └─ 3대+ 가능 → ✅ TAR 도입!

Final Recommendations by Environment

Development/Test Environments

yaml
# 결론: TAR 적용 불필요
replicas: 1-2
service.kubernetes.io/topology-mode: "disabled"
# 이유: 비용 절약 > 성능 최적화

Staging Environments

yaml
# 결론: TAR 테스트 목적으로만 적용
replicas: 3-6
service.kubernetes.io/topology-mode: "auto"
# 이유: 프로덕션 환경 사전 검증

Production Environments

yaml
# 결론: 적극적 TAR 도입
replicas: zone_count × 3  # 최적화
service.kubernetes.io/topology-mode: "auto"
# 이유: 성능 + 비용 절약 효과 극대화

Key Points

Good candidates for adoption

Cases where it should not be adopted

Cases requiring careful consideration

Practical Checklist

Before adoption:

After adoption:

Recommendations

  1. Phased adoption: Start with services that have low business impact
  2. Sufficient testing: Validate in staging first
  3. Continuous monitoring: Measure hint generation status and effects
  4. Team training: Share TAR principles and troubleshooting methods with the operations team
  5. Documentation: Document the decision process and configuration criteria

Topology Aware Routing is not a universal solution for every environment; it is a tool that delivers substantial benefits under the right conditions.

References

  1. Kubernetes Official Documentation - Topology Aware Hints
  2. AWS Blog - Exploring the effect of Topology Aware Hints on network traffic in Amazon EKS
  3. Kubernetes Enhancement Proposal - Topology Aware Routing
  4. Amazon EKS User Guide - Topology Aware Hints

Diagram

100%

Drag to pan; pinch or use the wheel to zoom. Keyboard: arrow keys pan, + − zoom, 0 fits, Esc closes. The same content is in “Read the diagram as text” below the figure.