AI Portfolio Advisor — Self-Hosted Inference on EKS (Phase 2)¶

Phase 2 design. Phase 1 ships first against AWS Bedrock (Claude 3.5 Sonnet) so the product can be validated end-to-end before any infrastructure investment. This page documents the deliberate jump from a managed inference API to a self-hosted model on Amazon EKS, including why that jump is worth making, not just how.

The full implementation backlog is tracked in the internal plan at app_portfolio_tracker_aurora/docs/ADVISOR_PLAN.md.

Operational status — 2026-06-14: the Phase 2 implementation is now live in-repo. The project added cloudformation/eks-genai-stack.yaml, cloudformation/eks-bootstrap.ps1, and k8s/vllm-stack.yaml. In the real AWS account, the original core stack VPC was too small and had no NAT, so the EKS stack now extends that same VPC with a secondary CIDR, dedicated EKS subnets, and a NAT gateway.

Operational status — 2026-06-14: two real-world corrections were required during deployment: (1) the original proposed secondary CIDR 10.0.1.0/24 was rejected by AWS as restricted, so the live deployment moved to a different extension CIDR; and (2) the first GPU node-group attempt on spot stalled with no g5.xlarge worker ever launching, so the replacement live deployment path uses on-demand GPU capacity to guarantee convergence.

1. Business context & product requirements¶

The Portfolio Tracker is a personal-finance application that ingests broker CSVs (Moomoo, Tiger, WeBull), normalises them into a Postgres schema (TRANSACTION, FUND_TRANSACTION, EXTERNAL_CASH_FLOW, BALANCE_SNAPSHOT, DAILY_PRICE, FX_RATE), and renders dashboards, holdings, IRR, and currency-normalised NAV history.

The AI Advisor extends this with four capabilities:

Capability	What the user gets	Why it requires an LLM
Performance review	A written assessment of returns, drawdowns, concentration, and attribution by ticker/sector.	Requires synthesising 9 quantitative tools (NAV, IRR, allocation, monthly returns…) into one narrative.
Qualitative rebalancing	Plain-English suggestions of which positions look overweight/underweight relative to the user's stated risk profile.	The LLM has to weigh trade-offs across asset class, currency, and broker constraints — no deterministic rule covers all cases.
Quantitative trade tickets	A structured JSON list `[{ticker, side, qty, rationale, confidence}]` that the UI renders as a table.	The model uses tool-calling to read live holdings, then emits a schema-validated proposal that the disclaimer banner labels as "educational only".
Free-form Q&A	A chat-style box: "How did my SG-listed REITs perform vs my US tech holdings this year?"	Open-ended; requires planning which tools to call and in what order.

Functional requirements¶

Streaming responses. First token must arrive in <2 s; full answer (~1 K tokens) in <15 s. The UI uses Server-Sent Events so the user sees text appear progressively, matching the ChatGPT/Claude UX users now expect from any AI feature.
Tool-calling / function-calling. The model must be able to call typed Python tools (get_holdings, get_irr, get_nav_history, …) and incorporate the JSON results into the next reasoning step. This rules out completion-only models.
Structured output. The "trade tickets" preset must return JSON that conforms to a fixed schema, validated server-side before persisting.
Auditability. Every advisor session is persisted to ADVISOR_REPORT with the user prompt, model ID, tool-call trace, token counts, latency, and final markdown — both for debugging and for showing the user their own history.
Disclaimer enforcement. Any output that resembles a trade recommendation must be visibly labelled as educational. This is enforced both in the system prompt and in the UI rendering layer.

Non-functional requirements¶

Requirement	Target	Driving constraint
Latency (p50 first token)	< 2 s	UX research: users perceive >2 s as "broken".
Throughput	≥ 30 concurrent users	Sized for "hundreds-to-thousands of users with peak concurrency around 30" — see §3.2.
Availability	99.5 % monthly	Single-AZ acceptable for a personal-finance side project; multi-AZ would double cost without changing user-facing SLO meaningfully at this scale.
Data residency	All portfolio data stays inside the existing VPC.	RDS Postgres holds real broker statements; we minimise the blast radius by never sending raw rows over the public internet.
Cost ceiling	< \$700/month incremental, predictable.	This is a portfolio/learning project, not a funded product.
Backend-swap time	< 5 minutes, zero code change.	The Flask abstraction must let us flip between Bedrock and self-hosted inference behind one environment variable.

2. Why two phases, and why this order¶

A naïve plan would be "build it on EKS from day one". That is the wrong sequencing for three independent reasons:

Product risk dwarfs infrastructure risk at this stage. Until we have shipped a working advisor and watched a real user interact with it, we do not know whether the prompts, the tool surface, or the structured-ticket format are useful. Sinking ~\$500/month into GPU infrastructure before that validation is the most expensive way to discover that a prompt needs rewording.
The Bedrock path validates the full product loop in <2 days. No quota requests, no CloudFormation for EKS, no Helm. Just boto3.client("bedrock-runtime").converse_stream(...) from the existing Flask EC2. Tokens cost roughly \$0.024 per advisor query at Claude 3.5 Sonnet pricing — affordable for hundreds of test runs.
The Flask code does not change between phases. By introducing a ModelClient ABC — short for Abstract Base Class, a Python pattern (from abc import ABC, abstractmethod) where you define a class that declares methods but does not implement them, and any concrete subclass must implement those methods or Python refuses to instantiate it — with BedrockClient and OpenAICompatClient as concrete implementations (vLLM exposes an OpenAI-compatible API), the migration to self-hosted is a config flip. The two phases are not competing architectures; Phase 2 is an alternative backend behind the same abstraction.

This is the standard "managed first, self-hosted later" pattern: pay a premium per request while you discover the product, then amortise the savings of self-hosting only once load is predictable and the prompts are stable.

3. What are pods? (Kubernetes / EKS primer)¶

Phase 2 introduces Kubernetes — the largest piece of new conceptual surface area in this project. Before reading the architecture diagram, a quick mental model:

3.1 The layered model¶

Container (Docker image — e.g. vllm/vllm-openai:latest)
    ↓ wrapped in
Pod (1+ containers sharing a network namespace, an IP, and storage volumes)
    ↓ scheduled onto
Node (a worker — for us, an EC2 instance like g5.xlarge)
    ↓ part of
Cluster (a control plane + a fleet of nodes — for us, one EKS cluster)

3.2 A pod, precisely¶

A pod is the smallest deployable unit in Kubernetes. It is:

One or more containers that always run together on the same node and share a Linux network namespace (so the containers inside a pod talk to each other on localhost).
One IP address. The pod gets a routable IP; every container inside it shares that IP. When the pod restarts, it gets a new IP — which is why we don't talk to pods directly, we talk to them via a Service (a stable virtual IP and DNS name that load-balances to whichever pods are currently healthy).
A shared storage namespace. Volumes mounted to the pod are mountable into any of its containers, which is how sidecar patterns (logging agents, proxies) work.
Ephemeral. Pods are not pets. If a node dies, the pod dies, and Kubernetes creates a new pod (possibly on a different node) to replace it. State must live outside the pod (e.g. in RDS or an EBS-backed PersistentVolume).
Declared, not commanded. You don't tell Kubernetes "run a pod". You tell it "I want a Deployment of 1 replica of this pod spec" and the Kubernetes controller manager continuously reconciles reality toward that declaration.

For us, the vLLM pod is one container (the vLLM server image) requesting 1 NVIDIA GPU, mounted with one EBS volume for the HuggingFace model cache. Kubernetes places it on whichever node satisfies the GPU request — in our cluster, the single g5.xlarge Karpenter spins up under the gpu NodePool.

3.3 Why pods exist at all (instead of "just run a container")¶

The pod abstraction lets the model server (vllm) and any future sidecars (e.g. a Prometheus exporter scraping /metrics, an Envoy sidecar enforcing mTLS) be co-scheduled and co-lifecycled automatically. They start together, die together, are rescheduled together, and reach each other on localhost. This is much cleaner than running Docker containers manually on EC2 and trying to coordinate their lifecycles with systemd.

3.4 Pod vs. node — are they the same thing?¶

No. This is the single most common point of confusion when starting with Kubernetes.

Concept	What it is	In our cluster
Node	A physical (or virtual) machine. In EKS, almost always an EC2 instance.	`t3.medium` (system) and `g5.xlarge` (gpu).
Pod	A logical group of 1+ containers, scheduled onto exactly one node.	The vLLM pod, the CoreDNS pod, etc.

Key relationship rules:

One pod is scheduled to exactly one node. A pod never spans two nodes.
One node can host many pods simultaneously, limited by (a) CPU/memory/GPU resource requests of all the pods on it, and (b) the per-instance ENI limit of the AWS VPC CNI plugin (e.g. a t3.medium supports ~17 IPs total ≈ ~15 pods after subtracting host-reserved IPs).
You can also force "1 pod per node" using taints and tolerations or node selectors. We do exactly this on the GPU node: only pods that request nvidia.com/gpu: 1 are allowed there, which in practice means only vLLM lands on g5.xlarge.

Layout of pods on nodes in our cluster:

flowchart LR
    subgraph SysNode["system node — t3.medium (2 vCPU, 4 GiB RAM)"]
        direction TB
        cdns["CoreDNS pod ×2<br/>(resolves *.svc.cluster.local)"]
        lbc["aws-lb-controller pod ×1<br/>(reconciles ALBs)"]
        karp["karpenter pod ×1<br/>(provisions new nodes)"]
        kp["kube-proxy<br/>(DaemonSet — 1 per node)"]
        an["aws-node CNI<br/>(DaemonSet — 1 per node)"]
    end
    subgraph GpuNode["gpu node — g5.xlarge (4 vCPU, 16 GiB, 1× A10G GPU)"]
        direction TB
        vllm["vLLM pod ×1<br/>requests: 1 GPU,<br/>~12 GiB RAM, 3 vCPU"]
        nvd["nvidia-device-plugin<br/>(DaemonSet — exposes the GPU<br/>as a schedulable resource)"]
        kp2["kube-proxy<br/>(DaemonSet)"]
        an2["aws-node CNI<br/>(DaemonSet)"]
    end

    classDef workload fill:#e1f5ff,stroke:#0277bd
    classDef ds fill:#f5f5dc,stroke:#8b7500
    class vllm,cdns,lbc,karp workload
    class kp,an,nvd,kp2,an2 ds

So the system node runs roughly 7 pods (counting both CoreDNS replicas and the per-node DaemonSets); the GPU node runs the vLLM pod plus 3 DaemonSets. It is not "1 EC2 instance = 1 pod".

A DaemonSet is a special workload type that automatically runs exactly one pod per node — that's how kube-proxy, aws-node, and nvidia-device-plugin are guaranteed to be on every node without anyone manually scheduling them.

3.5 Sidecars — what they are and when we'd use them¶

A sidecar is a secondary container inside the same pod as your main application container. Same pod = same network namespace = same lifecycle = shared volumes. The classic uses are:

Sidecar use case	Example
Observability	A `prometheus-exporter` container that reads vLLM's `/metrics` endpoint on `localhost:8000/metrics` and exposes them on `localhost:9100` in the Prometheus scrape format.
Service mesh / mTLS	An Envoy / Istio sidecar that intercepts all network traffic in/out of the pod and adds TLS, retries, circuit breaking.
Log shipping	A `fluent-bit` container that tails a shared `/var/log` volume the main container writes to, and ships to CloudWatch / Loki.
Config reload	A `consul-template` container that watches Consul and rewrites the main container's config on change.
Auth proxy	An `oauth2-proxy` sidecar that terminates auth and forwards already-authenticated requests to the main container on `localhost`.

Anatomy of a hypothetical sidecar-enabled vLLM pod:

flowchart LR
    subgraph Pod["vLLM pod (one IP, one lifecycle)"]
        direction TB
        Main["vllm container<br/>listens on :8000<br/>(main process)"]
        SideMetrics["prometheus-exporter sidecar<br/>scrapes localhost:8000/metrics<br/>exposes localhost:9100"]
        SideEnvoy["envoy sidecar<br/>terminates mTLS on :8443<br/>forwards plaintext to localhost:8000"]
        Vol[("shared emptyDir volume<br/>/var/log")]
        Main -.writes logs.-> Vol
        SideMetrics -.reads logs.-> Vol
    end
    Client["External traffic"] --> SideEnvoy
    SideEnvoy --> Main
    Scraper["Prometheus server"] --> SideMetrics

    classDef main fill:#e1f5ff,stroke:#0277bd
    classDef side fill:#fff3e0,stroke:#ef6c00
    class Main main
    class SideMetrics,SideEnvoy side

We don't have any sidecars in vLLM today. The pod is just the single vllm/vllm-openai container. If we later add Prometheus scraping or mTLS, those would be added as sidecars to the same pod rather than as separate pods, precisely so they share the network namespace and lifecycle.

3.6 Karpenter — what it actually is, what it isn't¶

Your understanding is largely correct, with one important refinement.

What Karpenter is: a pod that runs on the system node. Its job is node provisioning — when Karpenter sees pods that can't be scheduled because no existing node has room, it calls the EC2 API to launch a new EC2 instance, registers it with the cluster, and lets the pending pods land on it. When nodes go underutilized, Karpenter consolidates and terminates them.

What Karpenter is NOT: the thing that decides which pod runs on which node. That's the kube-scheduler, a built-in Kubernetes component that runs inside the EKS control plane (managed by AWS, not on your nodes). The two work in sequence:

kube-scheduler tries to place a pending pod on an existing node. If it can, done.
If no node fits, the pod stays Pending.
Karpenter watches for Pending pods, picks a NodePool that matches the pod's constraints, and launches an EC2 instance.
The new node joins the cluster; kube-scheduler now places the pod on it.

So Karpenter is the horizontal scaler for nodes; kube-scheduler is the bin-packer onto existing nodes. Both of them speak to the Kubernetes API server in the control plane through a WATCH connection that is HTTP and persistent. They do not communicate with each other directly. All communication goes through the EKS control plane. The EKS control plane is the source of truth.

After Karpenter has called AWS EC2 API to launch the node and assign it to the cluster, Karpenter will then send a new HTTP request to update the API server of the new node being available. kube-scheduler now sees that the new node is now available and binds the previously previously unschedulable and pending pods to that node. Afterwards, kube-scheduler updates the API server of its decision.

kubectl is just a bash client to send the Pod manifest to the Kubernetes API server.

See §7.1 below for the full sequence diagram of this dance.

3.7 Why system pods can't tolerate interruption but the GPU pod can¶

This is a deliberate placement decision, not a property of the pods themselves. Three reasons system pods stay on on-demand instances:

They are control-plane-adjacent. If CoreDNS dies, every pod's getaddrinfo() calls start failing, including vLLM trying to resolve huggingface.co or rds-endpoint.amazonaws.com. If AWS LB Controller dies, the ALB's target group stops being reconciled — when the vLLM pod gets a new IP, the ALB keeps sending traffic to the old (dead) IP. If Karpenter dies and a GPU node is simultaneously reclaimed, there is nothing alive to launch the replacement node. These are multiplicative failures, not local ones.
They are small and cheap. A t3.medium is ~\$30/mo on-demand. Saving 60 % by going spot saves ~\$18/mo — not worth the operational risk above.
They tend toward singletons. Karpenter typically runs as 1 replica; AWS LB Controller as 1–2; CoreDNS as 2. Coincident spot interruptions could wipe out the only running copy.

The GPU pod is the opposite on every axis: - It's the only workload on the GPU node — its failure is local, not multiplicative. - It dominates the cost line (~\$320/mo even on spot), so the 60 % spot discount is meaningful. - The application layer has a Bedrock fallback — if vLLM is down, flipping ADVISOR_BACKEND=bedrock restores service in under a minute. - vLLM restarts cleanly in ~90 s (model weights are already on the EBS cache from the previous boot).

Hence: system pods → on-demand; GPU pod → spot.

3.8 EBS gp3 and why it's cost-per-IOPS efficient¶

Your intuition is exactly right: our workload is modest in raw size but bursty in IOPS (pod cold-start reads 16 GB of model weights as fast as possible, then almost no I/O during steady-state serving). IOPS matters more than total throughput, and throughput matters more than capacity.

gp3's defining feature: IOPS, throughput, and capacity are independently provisioned.

Knob	gp2 (the previous default)	gp3
IOPS	3 IOPS per GB (so 100 GB = 300 IOPS — you had to buy capacity to get IOPS)	3000 IOPS baseline included on every volume, regardless of size. Provision up to 16,000 IOPS independently.
Throughput	Scaled with IOPS (linked)	125 MB/s baseline included. Provision up to 1000 MB/s independently.
Capacity	What you wanted, but it was the only knob	Independent; you can have a 20 GB volume with 16,000 IOPS.
Price	~\$0.10 / GB-month	~\$0.08 / GB-month + \$0.005 per provisioned IOPS above 3000 + \$0.04 per MB/s above 125

For our 60 GB GPU-node volume:

We get the 3000 IOPS baseline free — more than enough for vLLM's cold-start read pattern (loading 16 GB sequentially is throughput-bound, not IOPS-bound, and the 125 MB/s baseline handles that comfortably too).
We pay ~\$4.80/month for the 60 GB capacity. No extra IOPS or throughput charges.
On gp2, getting equivalent baseline IOPS would have required buying a 1000 GB volume (~\$100/mo) — a 20× cost penalty driven by the IOPS-to-size coupling.

The cost-per-IOPS efficiency comes from decoupling. You only pay for the resource axis you actually need (in our case: a little capacity + the included baseline IOPS), instead of being forced to over-buy along a coupled axis. For high-IOPS-small-volume workloads like ours, gp3 is dramatically cheaper. For high-throughput streaming workloads, the same decoupling lets you buy throughput without buying IOPS.

3.9 Operator tooling — Helm, kubectl, and kubeconfig¶

The EKS cluster is operated through a small set of tools and configuration files. The important distinction is this: CloudFormation creates AWS infrastructure, kubeconfig tells local tools how to reach the Kubernetes API server, kubectl sends direct Kubernetes API requests, and Helm packages many Kubernetes resources into one installable release.

Thing	What it is	Why you need it
EKS cluster	Managed Kubernetes control plane	Runs Kubernetes without you managing the API server, scheduler, controller manager, or etcd
Kubernetes API server	Front door for Kubernetes instructions	`kubectl`, Helm, controllers, and kubelets all read/write desired state through this API
Node group	EC2 worker machines registered to the cluster	Runs ordinary system and application Pods
GPU node group	EC2 worker machines with NVIDIA GPUs	Runs GPU-bound workloads such as the vLLM inference Pod
VPC CNI	AWS pod networking plugin	Gives Pods VPC IP addresses so they can participate directly in VPC networking
kube-proxy	Service networking helper on each node	Routes Kubernetes Service traffic to the right Pod endpoints
Controller	Reconciliation loop	Watches desired state in the API server and keeps real infrastructure matching that state
Deployment controller	Built-in Kubernetes controller	Keeps the desired number of matching Pods running for a Deployment
AWS Load Balancer Controller	AWS-specific Kubernetes controller	Watches Ingress/Service resources and creates or updates AWS ALBs/NLBs
kubectl	Kubernetes command-line API client	Applies YAML manifests, checks Pods/Services/Ingresses, reads logs, and troubleshoots cluster state
kubeconfig	Local Kubernetes client configuration file	Stores the cluster endpoint, certificate authority data, user/auth method, and contexts used by `kubectl` and Helm
kubeconfig context	Named combination of cluster, user, and default namespace	Lets you switch between clusters/namespaces safely instead of passing every option manually
`aws eks update-kubeconfig`	AWS CLI helper that creates or updates kubeconfig entries for EKS	Teaches local tools where the EKS API server is and how to authenticate through AWS IAM
Helm	Kubernetes package manager	Installs and upgrades complex Kubernetes applications without manually applying every individual manifest
Helm chart	Versioned package of Kubernetes templates plus default values	Describes an installable app such as the AWS Load Balancer Controller, Karpenter, or vLLM stack
Helm values	Configuration overrides passed into a chart	Customises chart output for this cluster, such as cluster name, service account, region, image, resources, and annotations
Helm release	A specific installed instance of a chart in a namespace	Lets you upgrade, roll back, inspect, or uninstall the app as one unit
Helm repository	Index of available charts	Lets Helm find and download charts, for example from the AWS EKS charts repository
EKS access entry	IAM-to-Kubernetes access mapping	Lets a specific IAM principal authenticate to the cluster
EKS access policy	Kubernetes permission policy associated with an access entry	Grants admin/read/write-style permissions inside the cluster

The most common confusion is that kubeconfig is connection configuration, not permission by itself. It can point kubectl or Helm at the right API server, but the IAM principal still needs an EKS access entry and access policy before the API server authorises cluster actions.

3.10 Bootstrap and deployment flow — from AWS infrastructure to vLLM traffic¶

This is the practical sequence used to bring the Phase 2 EKS backend online. Each step hands off to the next layer: AWS infrastructure first, then cluster access, then Kubernetes add-ons, then the vLLM workload.

Create AWS infrastructure with CloudFormation. CloudFormation creates or updates the infrastructure layer:
EKS cluster
managed node groups
GPU node group
IAM roles
security groups
VPC/subnet/NAT extensions where required
optional EKS add-ons
Grant cluster access. The deployment identity is mapped into Kubernetes through EKS access management:

aws eks create-access-entry ...
aws eks associate-access-policy ...

This gives the selected IAM principal Kubernetes permissions inside the EKS cluster. Without this, a valid kubeconfig can still fail because the API server does not authorise the caller.

Create or update kubeconfig. The local machine is taught how to reach and authenticate to the EKS API server:

aws eks update-kubeconfig ...

After this, both kubectl and Helm know the target cluster through the current kubeconfig context. A good sanity check is:

kubectl config current-context
kubectl get nodes

Install the AWS Load Balancer Controller with Helm. Helm pulls the chart, renders Kubernetes manifests with the supplied values, and creates a Helm release in the cluster:

helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller ...

The installed controller can then watch Kubernetes Ingress/Service resources and create AWS ALBs or NLBs on behalf of the cluster.

Install GPU support with kubectl. The NVIDIA device plugin is applied as Kubernetes manifests:

kubectl apply -f <nvidia-device-plugin-manifest>.yaml

This lets Kubernetes see GPU resources such as nvidia.com/gpu, so the scheduler can place GPU-requesting Pods onto GPU-capable nodes.

Deploy the vLLM stack with kubectl. The application manifests are applied:

kubectl apply -f k8s/vllm-stack.yaml

This creates the Namespace, Deployment, Service, and Ingress for the vLLM backend.

Let controllers reconcile the desired state. Kubernetes and AWS controllers now react to the submitted resources:
The Deployment controller creates the vLLM Pod.
The scheduler places the Pod on a GPU-capable node.
The AWS Load Balancer Controller sees the Ingress.
The controller creates an internal ALB and registers the vLLM Pod IP as a target.
Route Flask traffic to vLLM. Flask sends OpenAI-compatible HTTP requests to the internal ALB:

POST /v1/chat/completions

The request path is:

Flask → internal ALB → Ingress rule → Service → vLLM Pod → GPU model

4. ECS vs. EKS — a feature-by-feature comparison¶

Both services run containers on AWS. Choosing between them is one of the more consequential decisions in this project, and the answer is not "EKS is more modern, use EKS". Here is the actual trade-off.

4.1 Side-by-side feature comparison¶

Dimension	Amazon ECS	Amazon EKS
API surface	AWS-proprietary (Tasks, Services, Task Definitions). Learn-once, AWS-only.	Upstream Kubernetes API (Pods, Deployments, Services, Ingress, CRDs). Same API as GKE, AKS, k3s, self-hosted clusters.
Portability	Lock-in to AWS.	Manifests run on any conformant k8s cluster. Multi-cloud and on-prem feasible.
Control plane cost	Free. AWS runs it at no charge.	~\$73/month per cluster (\$0.10/hour) on top of nodes.
Data plane options	EC2 (you manage), Fargate (serverless containers).	EC2 (managed or self-managed node groups), Fargate, Karpenter-managed EC2, hybrid (EKS Anywhere).
GPU workloads	Supported on EC2 launch type only; Fargate has no GPUs. Manual AMI / driver setup.	First-class via the NVIDIA device plugin DaemonSet. Karpenter natively understands GPU capacity types. Massive ecosystem of GPU-aware operators (Kueue, KubeRay, Run.ai).
Autoscaling — pods	Service Auto Scaling (target tracking on CPU/memory/ALB request count).	Horizontal Pod Autoscaler (CPU/memory/custom metrics via Prometheus Adapter), Vertical Pod Autoscaler, KEDA (event-driven, e.g. SQS depth).
Autoscaling — nodes	Capacity Providers + EC2 Auto Scaling Groups. Reactive, ASG-based.	Cluster Autoscaler (ASG-based) or Karpenter (groupless, sub-minute bin-packing, spot-aware). Karpenter is significantly more efficient.
Networking model	`awsvpc` mode gives each task an ENI and a VPC IP. Simple. ENI density limits how many tasks fit on a node.	Each pod gets a VPC IP via the AWS VPC CNI plugin. Same ENI-density consideration but easier to mitigate (prefix delegation, custom CNI like Cilium).
Service discovery	AWS Cloud Map (DNS) integration.	Built-in Kubernetes DNS (CoreDNS). Services are reachable at `service.namespace.svc.cluster.local`.
Load balancing	ALB / NLB / Service Connect, integrated declaratively in the Task Definition.	AWS Load Balancer Controller watches `Ingress` / `Service type=LoadBalancer` and provisions ALB/NLB. Slightly more moving parts but more flexible.
Secrets management	Inject Secrets Manager / SSM Parameter Store values directly into env vars via Task Definition.	External Secrets Operator or AWS Secrets and Configuration Provider for the Secrets Store CSI Driver. Two extra components to install.
IAM integration	Task Role + Execution Role. Maps 1:1 to a Task Definition.	IRSA (IAM Roles for Service Accounts) — pod-level IAM via OIDC federation. More granular than ECS but more setup. EKS Pod Identity (newer, simpler) is now also available.
Observability	CloudWatch Container Insights out of the box.	Same Container Insights option, plus the entire CNCF observability ecosystem (Prometheus, Grafana, Loki, Tempo, OpenTelemetry Operator).
GitOps / declarative deploys	Possible via CDK / CloudFormation; not idiomatic.	Native fit: ArgoCD, Flux. Manifests in Git, controllers reconcile to desired state.
Extension model	Limited; you wait for AWS to add features.	Custom Resource Definitions (CRDs) let you teach the cluster about new object types (e.g. `vLLMService`, `Notebook`, `Spark Application`). The entire AI/ML ecosystem (KServe, KubeRay, Kueue, Karpenter itself) is delivered as CRDs + operators.
Multi-tenancy	Coarse — separate clusters or capacity providers per team.	Fine-grained via Namespaces, NetworkPolicies, ResourceQuotas, RBAC, OPA/Gatekeeper, Kyverno.
Learning curve	Hours to days. AWS console covers most workflows.	Weeks. You need `kubectl`, YAML manifests, Helm, RBAC, and at least three operators (LB controller, Karpenter, NVIDIA device plugin) before you have a usable GPU cluster.
Operational burden	Low. AWS handles the most.	Moderate to high. You own cluster upgrades, CNI versions, node AMI cadence, add-on compatibility matrices. EKS Auto Mode (2024+) reduces this materially.
Idiomatic GenAI ecosystem fit	Sparse. Few AI/ML projects ship ECS Task Definitions.	Dense. vLLM, Ray Serve, KServe, NVIDIA Triton, JupyterHub, KubeFlow, Argo Workflows all ship Helm charts and assume Kubernetes. The reference repo we're working from (`aws-samples/sample-genai-on-eks-starter-kit`) literally has "EKS" in the name.

4.2 When ECS is the right answer¶

ECS is the better choice when:

The workload is stateless HTTP services that scale linearly (a typical Flask / Express / Spring Boot API).
The team has no Kubernetes expertise and the cost of acquiring it is not justified by the workload.
You want Fargate (serverless containers) and don't need GPU.
You will never leave AWS and value lower operational burden over portability.
You want to minimise the per-cluster control-plane cost (especially relevant when you'd otherwise run many small clusters for isolation).

Most "we just need to run a container behind an ALB" workloads belong on ECS Fargate.

4.3 When EKS is the right answer¶

EKS is the better choice when:

The workload depends on CNCF ecosystem projects (vLLM, Ray, KServe, Argo, Kueue, KubeFlow) that assume Kubernetes APIs.
You need GPU scheduling, especially with mixed instance types, spot pools, and bin-packing — Karpenter is materially better here than ECS Capacity Providers.
You want declarative, GitOps-driven infrastructure that survives a multi-cloud or hybrid future.
You want fine-grained multi-tenancy in one cluster (Namespaces + RBAC + NetworkPolicy + ResourceQuota).
The team is investing in Kubernetes as a transferable skill — this is the dominant container orchestration API in the industry, and competence is portable across clouds and employers.

4.4 Why EKS for the Advisor (and not ECS)¶

The deciding factor is not GPU — ECS technically supports GPU. The deciding factor is the GenAI ecosystem assumption. Every reference implementation we want to consult — aws-samples/sample-genai-on-eks-starter-kit, the official vLLM Helm chart, KServe, Karpenter NodePools tuned for inference, KubeRay for distributed serving when we eventually outgrow one node — speaks Kubernetes. Porting any of them to ECS Task Definitions would be a constant uphill battle against the grain of the ecosystem.

Justification under the "real-world simulation" lens. Given the expected usage for our portfolio tracker and advisor application — a single technical user today, peaking at perhaps a few requests per day — it would make more sense to use Bedrock indefinitely and never stand up a cluster at all. However, the purpose of this project is to demonstrate the design and operational thinking required to run AI inference at a scale of hundreds-to-thousands of users, where pay-per-token economics flip in favour of self-hosting, where p99 latency must be controlled with predictable hardware, and where the system pods (DNS, ingress, autoscaler, observability) become load-bearing. Phase 2 simulates that production reality on the smallest viable footprint (1 system node, 1 GPU node) so that the design decisions are honest — they would survive a 100×-traffic scale-up by adjusting NodePool limits and HPA targets, not by re-architecting.

This is also why the Phase 2 cluster intentionally includes Karpenter, the AWS Load Balancer Controller, and a separate system NodePool even though a single node could technically host everything. Those components are the realistic shape of a production EKS cluster, and removing them would make this a toy.

5. The architecture, in detail¶

5.1 Layout principles¶

Reuse the existing VPC. No VPC peering, no Transit Gateway. EKS worker nodes go into the same private subnets that RDS uses; the internal ALB goes into the same public subnets as the existing public ALB but without a public IP. Network traffic between Flask EC2 and vLLM never leaves the VPC, satisfying the data-residency requirement in §1.
One AZ for GPU, two AZs for system. GPU spot capacity is tightest in any single AZ; we let Karpenter pick whichever AZ has stock. The system NodePool spans both AZs so that CoreDNS, Karpenter itself, and the AWS LB Controller never have a single-AZ outage take down the cluster's control loops.
Internal ALB, never public. The vLLM endpoint must not be reachable from the internet — both for security (we have no auth gateway in front of it yet) and to keep the data path inside the VPC.
Single GPU pod, replicas=1, always-on. A 90-second cold-start (model weight download + GPU memory load) is unacceptable for an interactive feature. We pin one replica and pay the steady-state cost rather than scaling to zero.
Spot for GPU, on-demand for system. Spot interruption on g5.xlarge in ap-southeast-1 is historically ~2–5 %/day, and vLLM restarts cleanly in <90 s; the cost saving (~60 %) is worth the occasional brownout. System pods cannot tolerate interruption, so they stay on-demand.

5.2 Why ALB, NAT, and EBS are each load-bearing¶

Internal Application Load Balancer. Pods have ephemeral IPs that change on every restart, so something has to give us a stable network endpoint. The AWS Load Balancer Controller watches Ingress objects and provisions an ALB whose target group is kept in sync with the live pod IPs. Without this, the Flask EC2 would have no reliable way to address vLLM.
NAT Gateway. Worker nodes live in private subnets — they cannot be reached from the internet, which is the secure default. But they still need outbound internet for one-time bootstrap operations: pulling the vLLM container image from a public registry, downloading ~16 GB of model weights from HuggingFace on first pod start, and reaching the EKS control plane endpoint. The existing VPC already has a NAT Gateway in each public subnet, so this is a zero-cost reuse.
EBS (gp3). The GPU node needs persistent local disk for the OS root volume, the container image layer cache, and (most importantly) the HuggingFace model cache, so that pod restarts don't re-download 16 GB every time. We use gp3 for cost-per-IOPS efficiency.

5.3 What stays in the AWS managed plane¶

EKS control plane itself — API server, scheduler, etcd, controller manager — runs on AWS-owned infrastructure across 3 AZs. We never SSH in.
Bedrock (Phase 1 backend) remains available throughout Phase 2 as a fallback. If the cluster is down for maintenance, flipping ADVISOR_BACKEND=bedrock restores service immediately.

6. Architecture diagram¶

flowchart TB
    User([Browser])

    subgraph VPC["VPC (ap-southeast-1) — shared with existing app"]
        direction TB

        subgraph PublicSubnets["Public Subnets (AZ-a, AZ-b)"]
            NAT["NAT Gateway<br/>(existing)"]
            ALB_public["Public ALB<br/>(existing — Flask app)"]
            ALB_internal["Internal ALB<br/>(NEW — fronts vLLM)<br/>scheme: internal"]
        end

        subgraph PrivateSubnets["Private Subnets (AZ-a, AZ-b)"]
            Flask["Flask EC2<br/>(existing)<br/>portfolio-tracker.service"]
            RDS[("RDS Postgres<br/>(existing)<br/>+ NEW table:<br/>ADVISOR_REPORT")]

            subgraph EKS["EKS Cluster (NEW)"]
                direction TB
                SysNode["system node<br/>t3.medium × 1<br/>(on-demand)"]
                GpuNode["gpu node<br/>g5.xlarge × 1<br/>(spot, A10G 24GB)"]

                subgraph SysPods["pods on system node"]
                    CoreDNS["CoreDNS"]
                    LBC["AWS LB Controller"]
                    Karp["Karpenter"]
                end

                subgraph GpuPods["pods on gpu node"]
                    vLLM["vLLM<br/>Mistral-7B-Instruct<br/>OpenAI-compatible API"]
                end

                SysNode -.runs.-> SysPods
                GpuNode -.runs.-> GpuPods
            end
        end
    end

    subgraph AWS_Managed["AWS-Managed (outside VPC)"]
        EKS_CP["EKS Control Plane<br/>API server + etcd<br/>~$73/mo"]
        Bedrock["Bedrock<br/>Claude 3.5 Sonnet<br/>(Phase 1 backend)"]
        HF[("HuggingFace<br/>(model weights,<br/>pulled once)")]
    end

    User -- HTTPS --> ALB_public
    ALB_public -- HTTP --> Flask

    Flask -- "psycopg2" --> RDS

    %% Phase 1 path (active today)
    Flask -. "Phase 1:<br/>boto3 converse_stream" .-> Bedrock

    %% Phase 2 path
    Flask == "Phase 2:<br/>POST /v1/chat/completions<br/>(OpenAI-compatible)" ==> ALB_internal
    ALB_internal == "routed by<br/>AWS LB Controller" ==> vLLM

    %% Control + bootstrap traffic
    EKS -. "kubectl, control" .-> EKS_CP
    GpuNode -- "model download<br/>(one-time)" --> NAT
    NAT --> HF

    classDef new fill:#e1f5ff,stroke:#0277bd,stroke-width:2px
    classDef existing fill:#f5f5f5,stroke:#9e9e9e,stroke-width:1px,stroke-dasharray:3 3
    classDef managed fill:#fff3e0,stroke:#ef6c00,stroke-width:1px

    class ALB_internal,EKS,SysNode,GpuNode,vLLM,CoreDNS,LBC,Karp new
    class Flask,RDS,ALB_public,NAT existing
    class EKS_CP,Bedrock,HF managed

Legend — Blue = new in Phase 2; grey dashed = pre-existing and reused; orange = AWS-/third-party-managed.

7. Deep dive — how the cluster components interact¶

The architecture diagram above shows the static layout. This section shows the dynamic behaviour: what API calls each component makes, in what order, when something happens. Read this if you want to know exactly what is going on between the moment you helm install vllm and the moment a user sees streamed tokens in the browser.

7.1 Karpenter provisioning a GPU node from scratch¶

Triggered the first time the vLLM Deployment is applied, when no GPU node yet exists.

sequenceDiagram
    autonumber
    actor Operator
    participant kubectl
    participant API as K8s API Server<br/>(EKS control plane)
    participant Sched as kube-scheduler<br/>(EKS control plane)
    participant Karp as Karpenter pod<br/>(on system node)
    participant EC2 as EC2 API
    participant Kubelet as kubelet<br/>(on new GPU node)
    participant vLLM as vLLM container

    Operator->>kubectl: helm install vllm ./chart
    kubectl->>API: POST /apis/apps/v1/.../deployments
    API-->>kubectl: 201 Created
    API->>Sched: WATCH event: new pod, requests nvidia.com/gpu:1
    Sched->>Sched: filter existing nodes — none have a free GPU
    Sched->>API: pod phase=Pending, reason=Unschedulable
    Karp->>API: WATCH /api/v1/pods?fieldSelector=spec.nodeName=
    API-->>Karp: pending pod (gpu:1, 12Gi RAM, 3 vCPU)
    Karp->>Karp: select NodePool "gpu" → instance type g5.xlarge, capacityType=spot
    Karp->>EC2: RunInstances(g5.xlarge, spot, AMI=amazon-eks-gpu-node-1.30)
    EC2-->>Karp: instance-id i-abc123
    Karp->>API: POST /api/v1/nodes (provisional Node object)
    Note over Kubelet: instance boots — bootstrap.sh joins cluster,<br/>NVIDIA driver loaded, nvidia-device-plugin DaemonSet starts
    Kubelet->>API: PATCH node status: Ready=true, capacity.nvidia.com/gpu=1
    API->>Sched: node now schedulable
    Sched->>API: bind pending pod → new node
    API->>Kubelet: WATCH says pod assigned to you
    Kubelet->>Kubelet: pull image vllm/vllm-openai (~6 GB)
    Kubelet->>vLLM: containerd run + GPU device passthrough
    vLLM->>vLLM: download Mistral-7B-Instruct weights (~16 GB) from HuggingFace
    vLLM-->>Kubelet: probe /health 200 OK
    Kubelet->>API: PATCH pod status: phase=Running, Ready=true

Key APIs used:

Karpenter → EC2: RunInstances — the actual EC2 launch.
Karpenter → K8s API: WATCH pods — long-lived HTTPS connection with chunked responses; how all controllers learn about state.
kubelet → K8s API: PATCH node/pod status — how nodes and pods report their own state back.

7.2 AWS Load Balancer Controller wiring the internal ALB to vLLM¶

Triggered when the Ingress manifest is applied. Runs in parallel with §7.1.

sequenceDiagram
    autonumber
    participant Operator
    participant API as K8s API Server
    participant LBC as aws-lb-controller<br/>(on system node)
    participant ELBv2 as Elastic Load Balancing v2 API
    participant EC2API as EC2 API
    participant ALB as Internal ALB<br/>(physical AWS resource)
    participant vLLM as vLLM pod

    Operator->>API: kubectl apply -f vllm-ingress.yaml<br/>(annotations: scheme=internal, target-type=ip)
    API-->>Operator: Ingress object created
    LBC->>API: WATCH /apis/networking.k8s.io/v1/ingresses
    API-->>LBC: new Ingress with class=alb
    LBC->>EC2API: DescribeSubnets (filter tag kubernetes.io/role/internal-elb=1)
    EC2API-->>LBC: private subnet IDs subnet-a, subnet-b
    LBC->>ELBv2: CreateLoadBalancer(scheme=internal, Subnets=[a,b], Type=application)
    ELBv2-->>LBC: LoadBalancerArn
    LBC->>ELBv2: CreateTargetGroup(targetType=ip, Port=8000, Protocol=HTTP, HealthCheckPath=/health)
    ELBv2-->>LBC: TargetGroupArn
    LBC->>ELBv2: CreateListener(Port=443, Protocol=HTTPS, DefaultActions=[forward→TG])
    LBC->>API: WATCH /apis/discovery.k8s.io/v1/endpointslices?labelSelector=service=vllm
    API-->>LBC: EndpointSlice — vllm pod IP = 10.0.42.17, port 8000
    LBC->>ELBv2: RegisterTargets(TG, [{Id: 10.0.42.17, Port: 8000}])
    ELBv2->>vLLM: GET /health (every 30 s)
    vLLM-->>ELBv2: 200 OK
    ELBv2->>ELBv2: target state = healthy
    Note over ALB: traffic now flows and pod IP rotation handled<br/>by LBC re-watching EndpointSlices

Key APIs used:

LBC → ELBv2: CreateLoadBalancer / CreateTargetGroup / RegisterTargets — standard AWS ELB v2 calls. The LBC essentially translates k8s Ingress objects into AWS ELB API calls.
LBC → K8s API: WATCH EndpointSlices — this is how the LBC learns when pods come and go and keeps the target group in sync without polling.

7.3 CoreDNS — what it actually serves in our cluster¶

Common misconception alert. CoreDNS is not on the Flask → vLLM hot path. The Flask EC2 lives outside the cluster; it resolves the internal ALB's DNS name using the VPC Route 53 Resolver and never sends a packet to CoreDNS. The hot path described in §7.4 does not touch CoreDNS at all.

CoreDNS only ever answers DNS queries that originate from a pod inside the cluster. In our cluster there are three concrete workflows where that happens — only two of them actually exist today.

Case A — vLLM resolving external names at pod startup (the workflow we actually run most)¶

When the vLLM pod cold-starts, it must fetch the Mistral-7B weights from HuggingFace. The container does a DNS lookup on huggingface.co. Every pod's /etc/resolv.conf is injected by the kubelet at startup and points nameserver at the CoreDNS Service ClusterIP — so even external-name lookups go via CoreDNS first.

sequenceDiagram
    autonumber
    participant vLLM as vLLM container<br/>(GPU node)
    participant Libc as glibc resolver<br/>(/etc/resolv.conf)
    participant CoreDNS as CoreDNS pod<br/>(system node)
    participant R53 as VPC Route 53 Resolver<br/>(upstream)
    participant NAT as NAT Gateway
    participant HF as huggingface.co

    Note over Libc: nameserver = 10.96.0.10 (CoreDNS ClusterIP)<br/>injected by kubelet at pod start
    vLLM->>Libc: getaddrinfo("huggingface.co")
    Libc->>CoreDNS: UDP :53 A-record query
    CoreDNS->>CoreDNS: not *.cluster.local — apply "forward" plugin
    CoreDNS->>R53: UDP :53 forward A "huggingface.co"
    R53-->>CoreDNS: A 18.x.x.x (public IP)
    CoreDNS-->>Libc: A 18.x.x.x
    Libc-->>vLLM: 18.x.x.x
    vLLM->>NAT: TCP 443 outbound
    NAT->>HF: source-NATed
    HF-->>vLLM: 16 GB of weights (streamed)

So in our cluster, CoreDNS's busiest role is as a forwarding proxy to the VPC resolver — not as an authoritative resolver for *.svc.cluster.local names. The *.svc.cluster.local case (shown in Case C below) is the less common one for us.

Case B — System pods talking to the Kubernetes API server¶

Karpenter, the AWS LB Controller, kube-proxy, and CoreDNS itself all reach the Kubernetes API server via the in-cluster Service hostname kubernetes.default.svc.cluster.local. CoreDNS resolves this internally (no upstream forward) using its in-memory cache of Services + EndpointSlices, returning the ClusterIP of the kubernetes Service in the default namespace. From there, kube-proxy's iptables/IPVS rules DNAT to the actual API server endpoint.

This means every long-lived WATCH connection in §7.1 and §7.2 was opened to a hostname that CoreDNS resolved — quietly, once, at controller startup.

sequenceDiagram
    autonumber
    participant Sys as System pod<br/>(e.g. Karpenter)
    participant CoreDNS
    participant Kproxy as kube-proxy<br/>iptables rules
    participant API as K8s API Server

    Sys->>CoreDNS: UDP :53 A "kubernetes.default.svc.cluster.local"
    CoreDNS->>CoreDNS: in-cluster lookup (kubernetes plugin)
    CoreDNS-->>Sys: A 10.96.0.1 (kubernetes Service ClusterIP)
    Sys->>Kproxy: TCP 443 to 10.96.0.1
    Note over Kproxy: DNAT rule: 10.96.0.1:443 → real API endpoint
    Kproxy->>API: TCP 443
    API-->>Sys: WATCH stream opens (long-lived)

Case C — In-cluster client of the vLLM Service (does not exist today)¶

If we later add a second pod inside the cluster that calls vLLM directly — for example a Jupyter notebook running evaluations, or a CronJob that nightly batch-scores prompts — that pod would resolve vllm.default.svc.cluster.local via CoreDNS, get back the vLLM Service's ClusterIP, and have kube-proxy DNAT it to the live vLLM pod IP. This is the case the previous version of this section depicted.

We do not run any such pod today. Our only in-cluster client of vLLM is the external Flask EC2 going via the internal ALB. Case C is documented here for completeness because it is the defining workflow of Kubernetes Services — but in our current architecture it is purely hypothetical.

Summary table — does CoreDNS touch this request?¶

Workflow	Where the request originates	Uses CoreDNS?
Browser → public ALB → Flask	Internet	No (public Route 53, not in VPC)
Flask EC2 → internal ALB → vLLM (the hot path, §7.4)	Outside the cluster (EC2 instance)	No — VPC Route 53 Resolver
Flask EC2 → RDS Postgres	Outside the cluster	No — VPC Route 53 Resolver
vLLM pod → HuggingFace (Case A)	Inside the cluster	Yes — CoreDNS forwards to Route 53
Karpenter / LBC / kubelet → K8s API server (Case B)	Inside the cluster	Yes — CoreDNS resolves the in-cluster Service
Hypothetical in-cluster pod → vLLM Service (Case C)	Inside the cluster	Yes — CoreDNS resolves `*.svc.cluster.local`

The takeaway: CoreDNS is a cluster-internal facility. It does not appear on the Flask EC2's request path. It is on the path for any DNS lookup performed by a pod, regardless of whether the name being resolved is internal (*.svc.cluster.local) or external (huggingface.co).

7.4 End-to-end inference request (the Flask → vLLM hot path)¶

This is the path traversed every time a user clicks Analyze in the UI. It combines components from all three diagrams above.

sequenceDiagram
    autonumber
    actor User
    participant Browser
    participant Flask as Flask EC2<br/>portfolio-tracker.service
    participant R53 as VPC Route 53 Resolver
    participant ALB as Internal ALB
    participant TG as ALB Target Group<br/>(kept current by LBC)
    participant vLLM as vLLM pod<br/>(GPU node)
    participant RDS as RDS Postgres

    User->>Browser: click "Analyze"
    Browser->>Flask: POST /api/advisor/analyze<br/>(opens SSE connection)
    Flask->>Flask: build prompt + load 9 tool schemas
    Flask->>R53: resolve internal-vllm-12345.ap-southeast-1.elb.amazonaws.com
    R53-->>Flask: A 10.0.10.42 (ALB ENI in private subnet)
    Flask->>ALB: POST /v1/chat/completions<br/>{stream:true, tools:[...], messages:[...]}
    ALB->>TG: choose healthy target (least-outstanding-requests)
    TG->>vLLM: forward HTTP to 10.0.42.17:8000
    vLLM-->>Flask: SSE chunk "Looking at your portfolio..."
    Flask-->>Browser: SSE data: "Looking at your portfolio..."

    vLLM-->>Flask: SSE chunk tool_use={name:get_holdings, args:{currency:SGD}}
    Flask->>RDS: SELECT * FROM "TRANSACTION" ... (calculations.py)
    RDS-->>Flask: rows
    Flask->>Flask: format JSON tool_result

    Flask->>ALB: POST /v1/chat/completions<br/>(history extended with tool_result)
    ALB->>TG: route
    TG->>vLLM: forward
    vLLM-->>Flask: SSE chunk "Your top holding is SGX:O39..."
    Flask-->>Browser: SSE data: "Your top holding is SGX:O39..."

    vLLM-->>Flask: SSE chunk [DONE]
    Flask->>RDS: INSERT INTO ADVISOR_REPORT (user_prompt, response_md, tool_calls, ...)
    RDS-->>Flask: id=42
    Flask-->>Browser: SSE event: done {report_id: 42}
    Browser->>User: render final markdown + permalink

Components and their jobs in this flow:

Component	Role in the hot path
Flask EC2	Orchestrates the tool-use loop. Talks SSE to the browser on one side and HTTP-with-tools to vLLM on the other. Persists final report.
Route 53 Resolver	Resolves the ALB's DNS name to its private IP. AWS provides this for free inside every VPC.
Internal ALB	L7 reverse proxy. Picks a healthy target, terminates the connection, opens a new one to the pod.
ALB Target Group	The membership list of "currently-healthy vLLM pod IPs", kept fresh by the AWS LB Controller watching EndpointSlices.
vLLM pod	Runs the actual inference. Implements the OpenAI-compatible `/v1/chat/completions` API including streaming and tool_use events.
RDS Postgres	Source of truth for portfolio data (read by tools) and audit store for advisor reports (written at end).

CoreDNS, Karpenter, and the AWS LB Controller do not participate in this hot path — they're cluster-internal control loops that ran earlier (§7.1, §7.2) to make this happen. They only re-engage when something changes: a node dies, a pod's IP changes, a new Deployment is applied.

7.5 What happens during a spot interruption (the failure path)¶

Useful to read in conjunction with §3.7 (why GPU pods can tolerate this but system pods can't).

sequenceDiagram
    autonumber
    participant EC2 as EC2 / Spot Service
    participant Kubelet as kubelet (dying GPU node)
    participant API as K8s API Server
    participant LBC as aws-lb-controller
    participant ELBv2 as ELB v2 API
    participant Karp as Karpenter
    participant Flask as Flask EC2

    EC2->>Kubelet: spot interruption notice (~2 min warning)
    Kubelet->>API: PATCH node taint=node.kubernetes.io/unschedulable
    LBC->>API: WATCH EndpointSlices — vllm pod going away
    LBC->>ELBv2: DeregisterTargets(TG, [10.0.42.17])
    Note over ELBv2: stops sending new requests to the doomed pod
    EC2->>Kubelet: SIGTERM (node terminates ~2 min later)
    Kubelet->>Kubelet: dies along with the node
    Karp->>API: WATCH — pod is now Pending again (no node)
    Karp->>EC2: RunInstances(g5.xlarge, spot, different AZ)
    Note over Karp,Kubelet: ~60-90 s: new node joins, image pulls,<br/>vLLM starts, model weights load from EBS cache
    Kubelet->>API: pod Ready=true on new node
    LBC->>ELBv2: RegisterTargets(TG, [new pod IP])
    ELBv2->>ELBv2: health check passes
    Note over Flask: meanwhile, any inflight requests<br/>get 502/503 — Flask falls back to Bedrock

This is why §11 calls out "Karpenter-mediated outage of up to ~2 minutes" as a known limitation, and why the Bedrock fallback in the Flask client is more than a Phase 1 artifact — it's the HA story for Phase 2 as well.

8. Request flow (Phase 2) — short version¶

The user clicks Analyze in the advisor UI. The browser opens a POST /api/advisor/analyze SSE connection to Flask.
Flask's OpenAICompatClient calls POST http://<internal-alb>/v1/chat/completions with stream: true, the JSON Schema tool definitions, and the user's prompt.
The AWS Load Balancer Controller has registered the vLLM pod IP as a target on the internal ALB's target group; the request reaches vLLM.
vLLM streams tokens back via SSE. When the model emits a tool_use event, Flask pauses the stream, executes the local Python tool (which calls calculations.py against RDS), wraps the result as a tool_result message, and re-invokes vLLM with the conversation history extended.
The loop terminates when the model emits a stop event or the iteration cap (ADVISOR_MAX_TOOL_ITERATIONS=8) is hit.
Flask persists the full transcript, tool-call audit trail, latency, and token counts to the ADVISOR_REPORT table in RDS, then closes the SSE connection.

The browser has been rendering tokens incrementally throughout steps 4–5 via marked.js Markdown streaming, so the user perceives a continuous Claude-style chat experience even though the backend is performing multi-turn tool execution.

9. Design decisions, made explicit¶

Decision	Alternatives considered	Why we chose this	What would change our mind
Bedrock first (Phase 1)	Skip straight to EKS; or use OpenAI's API.	Fastest validation of the product loop. No infra. No new IAM. Same Flask abstraction will accept the EKS backend later.	Bedrock pricing changes drastically, or a CCoE policy bans third-party hosted inference.
Self-host with vLLM (Phase 2)	Bedrock forever; SageMaker JumpStart endpoint; Ray Serve on EC2.	vLLM has the best throughput-per-GPU-dollar in 2026, an OpenAI-compatible API (zero client code change), and is the de facto standard in the open-weights serving ecosystem.	SageMaker introduces a managed vLLM equivalent with materially lower TCO for our scale.
EKS over ECS	ECS Fargate (no GPU), ECS on EC2 (GPU possible).	The entire GenAI ecosystem ships Helm charts; ECS would mean rewriting all of it. See §4.4.	The workload becomes a plain stateless HTTP service with no GPU and no Kubernetes-native add-ons.
Karpenter over Cluster Autoscaler	Cluster Autoscaler on EC2 Auto Scaling Groups.	Sub-minute scale-out, groupless bin-packing, native spot diversification, and first-class GPU support.	We standardise on a heavily-templated ASG-per-pool model for compliance reasons.
Spot for GPU, on-demand for system	All on-demand; all spot.	Spot saves ~60 % on the dominant cost line; vLLM tolerates restarts; system pods do not.	Spot interruption rate in `ap-southeast-1` g5 capacity rises above ~10 %/day sustained.
One GPU node, replicas=1, always-on	Scale-to-zero with cold start; multiple replicas for HA.	A 90 s cold start kills the interactive feel; the workload at our scale never saturates one GPU.	Concurrent QPS sustainably exceeds ~20, at which point we add a second replica and turn on HPA.
Reuse existing VPC	New VPC + VPC peering / Transit Gateway.	Zero network egress charges between Flask and vLLM; one SG model; data never leaves the existing perimeter.	EKS networking requirements outgrow what the existing CIDR can supply.
Internal ALB	Service-type `LoadBalancer` (NLB); direct pod-to-pod via headless Service.	ALB integrates cleanly with the AWS LB Controller, gives us L7 routing for future endpoints, and matches the team's familiarity.	We need raw TCP throughput for a non-HTTP protocol.
Helm for vLLM, CloudFormation for AWS infra	All-CloudFormation (no Helm); all-Helm (also for AWS infra).	CloudFormation is the team standard for AWS resources; Helm is unavoidable for Kubernetes workloads — there is no CFN-native equivalent for templated k8s manifests, and rewriting them by hand is brittle.	AWS introduces a first-class `AWS::EKS::HelmChart` resource that materially closes this gap.

10. Cost analysis¶

Phase 2 monthly, `ap-southeast-1`¶

Item	Cost	Notes
EKS control plane	~\$73	Flat per cluster. Doesn't scale with workload.
`t3.medium` system node (on-demand, 24/7)	~\$30	Hosts CoreDNS, Karpenter, AWS LB Controller.
`g5.xlarge` GPU node (spot, 24/7)	~\$320	On-demand would be ~\$800. Spot interruption tolerated.
Internal ALB	~\$22	Plus tiny per-LCU charge — negligible at our QPS.
EBS gp3 (60 GB on GPU node)	~\$5	Model weight cache + container layers.
Existing VPC / NAT / RDS / Flask EC2	\$0 incremental	Already paid for.
Total incremental	~\$450–\$600/month

Phase 1 monthly (Bedrock pay-per-token), same `ap-southeast-1`¶

Usage profile	Approx. cost
100 advisor queries/month (\~3 K in + 1 K out each)	~\$2.40
1,000 queries/month	~\$24
10,000 queries/month	~\$240
50,000 queries/month	~\$1,200 → break-even with Phase 2 around here

The break-even is a useful sanity check: self-hosting only becomes economically rational above roughly 30–50K advisor queries per month. Below that, Bedrock is cheaper. Phase 2 is therefore explicitly justified on learning + portfolio-demonstration grounds, not on cost — and that justification is honest because the architecture would scale through the break-even point and beyond without redesign.

10.1 What we actually deployed vs. the original cost-optimised plan¶

Original design target: spot g5.xlarge for the GPU plane.
Observed live behaviour on 2026-06-14: the first spot-based GPU node group remained in CREATING with no backing EC2 instance, so spot availability — not quota — became the blocker.
Current implementation choice: parameterise the GPU node-group capacity type and use on-demand for the fresh deployment stack so the cluster can finish provisioning reliably.
Implication: the real active-testing monthly cost is temporarily higher than the idealised spot estimate, which makes the shutdown runbook below even more important.

10.2 Immediate shutdown levers for cost control¶

If you are not actively testing the advisor backend, the biggest savings come from shutting down all four of these:

the Flask bastion EC2
the EKS cluster (control plane + node groups)
the NAT gateway used by the EKS extension subnets
the internal vLLM ALB created from the Kubernetes Ingress

Deleting the dedicated Phase 2 CloudFormation stack is the fastest way to remove items 2 and 3 together.

11. Trade-offs and known limitations¶

No high availability for the GPU plane. One spot GPU node means a Karpenter-mediated outage of up to ~2 minutes after a spot interruption. Acceptable for an interactive advisor with a Bedrock fallback; would not be acceptable for a billing-critical path.
No request-level auth on the internal ALB yet. Anything inside the VPC can call vLLM. This is mitigated by the SG-level restriction to the Flask EC2's instance SG, but a future iteration should add an API-key check via a Lambda authoriser or sidecar.
Single region. A regional outage takes the advisor down. Multi-region active-active is out of scope.
Operational complexity step-change. Phase 2 adds Kubernetes upgrade cadence, CNI compatibility, Karpenter version drift, and Helm release tracking to the operational surface area. This is an explicit cost paid in exchange for the ecosystem benefits described in §4.

12. Skills demonstrated by Phase 2¶

This phase exists in part to make tangible the following competencies that are otherwise difficult to evidence:

Sequencing of risk. Validating product fit on managed infra before paying for self-hosted infra.
Abstraction design. A ModelClient interface that genuinely lets us swap backends without touching application code.
AWS networking literacy. Reusing a VPC across compute paradigms (EC2, RDS, EKS) without resorting to peering.
Kubernetes operational depth. Karpenter NodePools, IRSA, the AWS Load Balancer Controller, GPU device plugins, and the Helm-on-top-of-CloudFormation hybrid pattern.
Cost engineering. Explicit break-even analysis, spot vs on-demand placement decisions, and acknowledging when a decision is for learning rather than economics.
Infrastructure as code discipline. CloudFormation for AWS resources, Helm for Kubernetes workloads, bootstrap script as the documented seam between them.