In my last employment, we faced an intriguing challenge: we needed to forward traffic from the same URL to both Orka One and Orka Two sites while keeping the old Orka One site accessible to external users. The journey toward achieving this goal involved overcoming obstacles in user request flow direction. This article will describe the challenge and the solution we implemented on AWS.

Challenge

The initial user request flow direction, depicted in the graph, outlined the path from hiorka.com through DNS, CDN, and ALB to ECS in the production environment. However, when CDN forwarded user requests to ALB, two critical problems emerged:

1. Source IP Transformation: The request source_ip changed from the user's IP to the CDN's IP, complicating the identification of Orka internal IP.

2. Cache Challenges: CDN might provide cache without forwarding the request to ALB, irrespective of the user's IP.

Architecture

The solution to this intricate challenge involved leveraging AWS CloudFront Function – a feature enabling lightweight functions to run on each CDN edge node, allowing modification of user requests and server responses.

Layer 1: Create a CloudFront Function (CDN Edge Node)

The first layer in the solution was to create a CloudFront Edge Function that would run on each CDN edge node. The function would modify the user request by adding a custom header cloudfront-custom-function-isinsiderip based on whether the user's IP is internal or external. In addition, we also modify the cache key to included this custom header, ensuring that the cache is separate for internal and external users.

Layer 2: Change the ALB Target Group

The second layer was to change the ALB Target Group to use the custom header cloudfront-custom-function-isinsiderip as the health check. This change ensured that the ALB would only forward requests to the ECS service if the custom header was present in the request.

A security concern arose from this change: the ALB would forward requests to the ECS service if the custom header was present in the request, even if the request was from an external user. To mitigate this concern, we added another custome header X-Cloud-Header to a secret value. The ALB service would only serve requests if the custom header X-Cloud-Header was present in the request and the value of the custom header X-Cloud-Header matched the secret value.

Layer 3: Duplicate ECS Service

The third layer was to duplicate the ECS service and create a new ECS service with the same task definition. The new ECS service would be used to serve internal users, while the old ECS service would continue to serve external users.