JerryTech
All posts

AWSEKSENIAL2023

Upgrading AWS EKS to AL2023: Troubleshooting Custom Networking and nodeadm

A node startup failure, a config-source mismatch, and the checks that helped us find it.

JerryPublished Updated

Current guidance, checked on September 21, 2026: This post describes a past troubleshooting case. AWS advises against running nodeadm init again in EC2 user data or while building a custom AMI with the official EKS AL2023 Packer scripts. For dynamic settings, the current documentation describes writing a YAML or JSON file under /etc/eks/nodeadm.d. The boot process then applies it. See the AWS AL2023 guide.

This English version keeps the original case and examples. It adds scope notes around the old workaround and networking settings. No new cluster tests were run for this translation.

AL2023 uses a node setup tool called nodeadm. Its YAML configuration changes how we write user data compared with AL2 and bootstrap.sh.

During our EKS upgrade, we ran into a problem with VPC CNI custom networking. We needed a node label that depended on the node's Availability Zone. The script wrote the right configuration file, but the node still could not join the cluster.

This post follows the error, the cause, and the workaround we used at the time.

Background: why our setup needed extra work

Our EKS environment used custom networking so that Pods could use a different subnet from their nodes.

An AZ-based setup can select an ENIConfig whose name matches the Availability Zone, such as ap-northeast-1a. Our setup used a prefix to separate workloads, such as a billing system:

  • AZ: ap-northeast-1a
  • ENIConfig: billing-ap-northeast-1a

We needed a label on each node to select the right configuration:

eks.amazonaws.com/custom-eni-config=billing-ap-northeast-1a

In AL2, we could get the AZ in the bootstrap script and pass the label through --kubelet-extra-args. Moving to AL2023 changed that flow.

The label key also needs to match the VPC CNI configuration. In this setup, the intended key was eks.amazonaws.com/custom-eni-config; setting a node label alone is not enough if the CNI reads a different key.

The problem: the node could not join the cluster

AL2023 supports passing NodeConfig YAML in MIME multipart user data. But our node group could launch nodes in different AZs. Our Terraform configuration did not know the AZ of each future instance at plan or apply time.

We needed a shell script to read the AZ from the Instance Metadata Service (IMDS) during startup, then build the label.

We changed user data to generate a local configuration file. After Terraform apply, the node group stayed in Creating for about 20 minutes and then timed out.

Read the logs

We connected to a node that appeared in the EC2 console but had not joined Kubernetes. We checked our custom /var/log/user-data.log and the system journal with journalctl.

The original log showed:

[ec2-user@ip-10-xxx-xxx-xxx ~]$ sudo cat /var/log/user-data.log
...
+ echo 'Configuration file written to /etc/eks/nodeadm-config.yaml'
Configuration file written to /etc/eks/nodeadm-config.yaml
+ echo 'Executing nodeadm init...'
Executing nodeadm init...
+ /usr/bin/nodeadm init
info init/init.go:55 Checking user is root..
info cli/options.go:45 Using default config sources...
info init/init.go:65 Loading configuration.. {"configSource": ["imds://user-data"], "configCache": ""}
warn configprovider/chain.go:30 Encountered error in config provider {"error": "could not find NodeConfig within UserData"}
fatal cli/main.go:35 Command failed {"error": "no config in chain"}

The useful errors were could not find NodeConfig within UserData and no config in chain.

Find the cause

The log showed the configuration source used by that invocation: imds://user-data. We had run /usr/bin/nodeadm init without a source argument, so it read user data through IMDS.

  • With a suitable NodeConfig in user data, it could load the configuration.
  • Our user data held a shell script. The script created a YAML file, but that file was not the source nodeadm was reading.

The file existed at /etc/eks/nodeadm-config.yaml. Creating the file did not tell nodeadm to use it.

The workaround at the time: set --config-source

We checked the nodeadm help and found a configuration-source option:

$ nodeadm init --help
Usage:
  nodeadm init [flags]

Flags:
  -c, --config-source string   Source of the configuration.
                               Allowed schemes: "file:", "imds:"
                               (default "imds://user-data")

The file:// scheme let us point to the local file explicitly.

The original Terraform user data

The following is the simplified script from the original case. It records the old workaround, not the current recommended boot setup for an EKS-optimized AL2023 AMI. It contains placeholders, including ... in the label list. Cluster values and the service CIDR must match the real cluster.

user_data = base64encode(<<-EOF
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
set -ex
# Debug
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# 1. Get IMDSv2 Token & AZ
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
AZ=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/placement/availability-zone)

# 2. Build the label, including the dynamic custom-eni-config
# This line uses both Terraform interpolation and Bash variables.
FINAL_LABELS="...,eks.amazonaws.com/custom-eni-config=${var.node_group_name}-$AZ"

# 3. Write the NodeConfig YAML
cat > /etc/eks/nodeadm-config.yaml <<CONFIG
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: ${var.cluster.cluster_id}
    apiServerEndpoint: ${var.cluster.cluster_endpoint}
    certificateAuthority: ${var.cluster.cluster_certificate_authority_data}
    cidr: 192.168.0.0/16
  kubelet:
    flags:
      - --node-labels=$FINAL_LABELS
CONFIG

# 4. Historical workaround: run nodeadm init with a file source
# Use file:/// with three slashes for an absolute path.
/usr/bin/nodeadm init --config-source file:///etc/eks/nodeadm-config.yaml

--==MYBOUNDARY==--
EOF
)

For an absolute path, the URI is file:///absolute/path. The example keeps the MIME multipart wrapper from the original setup.

For a new setup, follow the current AWS guidance linked at the top. The EKS AMI already runs nodeadm through its boot services. Running it again can break the expected order. Dynamic configuration can go in /etc/eks/nodeadm.d without adding another manual nodeadm init call.

Also check the VPC CNI settings

After fixing startup, the node became Ready. That did not prove Pod networking was working. The aws-node DaemonSet also needed the correct settings.

  1. AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true enables custom networking. Without it, the CNI does not use the custom-networking setup described here.
  2. ENI_CONFIG_LABEL_DEF=eks.amazonaws.com/custom-eni-config tells the CNI which node label key to read in this setup. Check the selected ENIConfig, including its subnet and security groups. An ENIConfig annotation can take priority over a label. See the VPC CNI configuration reference.
  3. ENABLE_PREFIX_DELEGATION=true was another setting we checked for our high Pod-density setup. Prefix mode can provide more Pod IP addresses. A value such as maxPods: 110 alone does not prove that the instance and subnet have enough IP capacity. Check the instance type, CNI settings, and available subnet prefixes. See AWS prefix mode guidance.

These settings solve different parts of the problem. A Ready node can still have Pods waiting for network setup if IP allocation fails. See the AWS custom networking guide for the feature's scope.

Check the result

After we changed user data and the CNI settings, the node joined the cluster. We then checked whether custom networking was actually in use.

1. Check the node label

The label needed to point to the prefixed ENIConfig:

kubectl get node <node-name> --show-labels | grep custom-eni-config
# Output: eks.amazonaws.com/custom-eni-config=billing-ap-northeast-1d

2. Compare the Pod and node IP addresses

In our setup, the node used the primary subnet, while the Pod was expected to use the custom subnet. The original example showed:

$ kubectl get pod -o wide
NAME                              IP              NODE
billing-service-xxxx             10.23.139.130   ip-10-23-233-92...
  • Node IP: 10.23.233.92
  • Pod IP: 10.23.139.130

We used aws ec2 describe-subnets to check that the Pod IP belonged to the chosen custom subnet CIDR, 10.23.128.0/20. This confirmed the subnet selection in that case.

What this case taught us

A configuration file can be correct and still have no effect if a program reads a different source. The log's configSource field helped us find that mismatch.

The explicit file source explained our historical fix. For a new AL2023 setup, use the current AWS boot flow and dynamic configuration guidance. Then check the CNI settings, node labels, and Pod IPs separately. Successful node startup is only one part of working Pod networking.

Read more posts