Shadowing

Documentation

Resiliency

Shadowing

Copy live production traffic to a shadow environment or service so that you can try out, analyze, and monitor new software changes before deploying them to production.

About traffic shadowing

When releasing changes to a service, you want to finely control how those changes get exposed to users. This progressive delivery approach to releasing software allows you to reduce the blast radius, especially when changes introduce unintended behaviors. Traffic shadowing, also referred to as traffic mirroring, is one way to observe the impact of new software releases and test out new changes before you roll them out to production. Other approaches to slowly introduce new software include canary releases, A/B testing, or blue-green deployments.

When you turn on traffic shadowing for an app, K8sGateway makes a copy of all incoming requests. K8sGateway still proxies the request to the backing destination along the request path. It also sends a copy of the request asynchronously to another shadow destination. When a response or failure happens, copies are not generated. This way, you can test how traffic is handled by a new release or version of your app with zero production impact. You can also compare the shadowed results against the expected results. You can use this information to decide how to proceed with a canary release.

When a copy of the request is sent to the shadow app, K8sGateway adds a -shadow postfix to the Host or Authority header. For example, if traffic is sent to foo.bar.com, the Host header value is set to foo.bar.com-shadow. This way, the app that receives the shadowed traffic can determine if the traffic is shadowed or not. This information might be valuable for stateful services, such as to roll back any stateful transactions that are associated with processing the request. To learn more about advanced traffic shadowing patterns, see this blog.

To observe and analyze shadowed traffic, you can use a tool like Open Diffy. This tool create diff-compares on the responses. You can use this data to verify that the response is correct and to detect API forward/backward compatibility problems.

ℹ️

To enable traffic shadowing, you must set up an Upstream resource for the app that you want to shadow traffic for and for the app that receives the shadowed traffic.

Before you begin

Follow the Get started guide to install K8sGateway, set up a gateway resource, and deploy the httpbin sample app.

Get the external address of the gateway and save it in an environment variable.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system gloo-proxy-http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

kubectl port-forward deployment/gloo-proxy-http -n gloo-system 8080:8080

Set up traffic shadowing

Create a namespace for a second httpbin app that you use to receive shadowed traffic.
```
kubectl create ns shadow
```

Deploy the httpbin shadow app.

kubectl -n shadow apply -f https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/policy-demo/httpbin.yaml

Verify that the httpbin shadow app is running.
```
kubectl -n shadow get pods
```

Create an Upstream resource for the httpbin shadow app.

kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: shadow
  namespace: gloo-system
spec:
  kube:
    serviceName: httpbin
    serviceNamespace: shadow
    servicePort: 8000
EOF

Create another Upstream resource for the httpbin app that you deployed as part of the Get started guide.

kubectl apply -f- <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: httpbin
  namespace: gloo-system
spec:
  kube:
    serviceName: httpbin
    serviceNamespace: httpbin
    servicePort: 8000
EOF

Create a RouteOption resource to define your shadowing rules. The following example shadows 100% of the traffic to the shadow Upstream resource that you just created.

kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: shadowing
  namespace: httpbin
spec:
  options:
    shadowing:
       upstream:
         name: shadow
         namespace: gloo-system
       percentage: 100
EOF

Create an HTTPRoute resource for the httpbin app that you want to shadow traffic for and reference the RouteOption resource that you created. Note that shadowing requires you to route traffic to the httpbin Upstream and not to the httpbin service directly.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin-shadow
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: gloo-system
  hostnames:
    - shadowing.example
  rules:
    - filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.solo.io
            kind: RouteOption
            name: shadowing
      backendRefs:
        - name: httpbin
          kind: Upstream
          group: gloo.solo.io
          namespace: gloo-system
EOF

Create a reference grant to allow the HTTPRoute resource to access Upstream resources in the gloo-system namespace.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: shadow-rg
  namespace: gloo-system   
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: httpbin
  to:
    - group: "gloo.solo.io"
      kind: Upstream
EOF

Send a request to the httpbin app on the shadowing.example domain. Verify that you get back a 200 HTTP response code.

curl -vik http://$INGRESS_GW_ADDRESS:8080/headers -H "host: shadowing.example:8080"

curl -vik localhost:8080/headers -H "host: shadowing.example"

Example output for a successful response:

...
{
 "headers": {
   "Accept": [
     "*/*"
   ],
   "Host": [
     "timeout.example:8080"
   ],
   "User-Agent": [
     "curl/7.77.0"
   ],
   "X-Envoy-Expected-Rq-Timeout-Ms": [
     "20000"
   ],
   "X-Forwarded-Proto": [
     "http"
   ],
   "X-Request-Id": [
     "0ae53bc3-2644-44f2-8603-158d2ccf9f78"
   ]
 }
}

Get the logs for the shadow httpbin app and verify that you see a copy of that request.

kubectl logs $(kubectl get pod -l app=httpbin -o jsonpath='{.items[0].metadata.name}' -n shadow) -n shadow -c httpbin

Example output:

go-httpbin listening on http://0.0.0.0:8080
time="2024-06-12T14:08:37.4174" status=200 method="GET" uri="/headers" size_bytes=442 duration_ms=0.17 user_agent="curl/7.77.0" client_ip=10.XX.X.XX

Get the logs for the httpbin app and verify that you see the same log entry for your requests.

kubectl logs $(kubectl get pod -l app=httpbin -o jsonpath='{.items[0].metadata.name}' -n httpbin) -n httpbin -c httpbin

Example output:

go-httpbin listening on http://0.0.0.0:8080
time="2024-06-12T14:10:23.4605" status=200 method="GET" uri="/headers" size_bytes=338 duration_ms=0.09 user_agent="curl/7.77.0" client_ip=10.XX.X.XX:38808

Cleanup

You can remove the resources that you created in this guide.

kubectl delete httproutes httpbin-shadow -n httpbin 
kubectl delete routeoption shadowing -n httpbin
kubectl delete upstream shadow -n gloo-system
kubectl delete upstream httpbin -n gloo-system 
kubectl delete referencegrant shadow-rg -n gloo-system
kubectl delete -f https://raw.githubusercontent.com/solo-io/gloo-mesh-use-cases/main/policy-demo/httpbin.yaml -n shadow
kubectl delete ns shadow

Retries Timeouts