My Kubernetes Networking Journey

Model

• All Pods should get their own IP Address, and the container should be able to see itself as the same IP that others see it
• All containers can communicate with each other without NAT
• All nodes can communicate with all containers (and vice-versa) without NAT

Network communication

• Patterns
  • Pod to Pod
  • Pod to ClusterIP
  • Node to Pod
  • Node to ClusterIP
• Key points
  • When requests to ClusterIP service’s IP:port pair are made, they are intercepted and NAT’ed (by Netfilter) to a Pod that serves as that ClusterIP’s service endpoint.
  • Pod network to Service will use DNAT, external network (not in Pod IP CIDR) to Service will use SNAT and DNAT, and Pod connecting to a Service IP it serves will use hairpin NAT.
  • All NodePort traffic is forwarded to the ClusterIP Service, which then gets distributed to Pods which back that service.
  • Pod to Pod, and Pod to ClusterIP traffic appear as Pod to Pod traffic.
  • Node to Pod, and Node to ClusterIP traffic will both appear as Node to Pod traffic.
  • Requests to a Pod made from the same Node the Pod lives on are handled by the node’s virtual Ethernet bridge. Requests handled by this bridge never reach the wire.

Pod Networking (CNI)

Amazon EKS supports native VPC networking with the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes. This plugin assigns an IP address from the VPC to each Pod.

• Plugin components
  • L-IPAM daemon – Responsible for creating network interfaces and attaching the network interfaces to Amazon EC2 instances, assigning secondary IP addresses to network interfaces, and maintaining a warm pool of IP addresses on each Node for assignment to Kubernetes Pods when they are scheduled.
  • CNI plugin – Responsible for wiring the host network (for example, configuring the network interfaces and virtual Ethernet pairs) and adding the correct network interface to the Pod namespace.
• Each Amazon EC2 Node is deployed to one subnet. Each Node is assigned a private IP address from a CIDR block assigned to the subnet. Each Node is deployed with the Pod networking (CNI) which, by default, assigns each Pod a private IP address from the CIDR block assigned to the subnet that the Node is in and adds the IP address as a secondary IP address to one of the network interfaces attached to the instance.
• Communication within a VPC (such as Pod to Pod) is direct between private IP addresses and requires no source network address translation (SNAT). When traffic is destined for an address outside of the VPC, the Amazon VPC CNI plugin for Kubernetes translates the private IP address of each pod to the primary private IP address assigned to the primary network interface (network interface) of the Amazon EC2 node that the pod is running on, by default. i.e. SNAT
• External SNAT in the CNI plugin: The CNI plugin doesn’t translate a Pod’s private IP address to the primary private IP address assigned to the primary network interface of the Amazon EC2 instance node that the pod is running on, when traffic is destined for an address outside of the VPC.
  
  References:

https://aws.amazon.com/blogs/networking-and-content-delivery/using-vpc-flow-logs-to-capture-and-query-eks-network-communications

https://docs.aws.amazon.com/eks/latest/userguide/eks-networking.html

https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html

https://docs.aws.amazon.com/eks/latest/userguide/external-snat.html

https://www.cncf.io/blog/2020/01/30/kubernetes-networking-demystified-a-brief-guide/