Fault injection

Documentation

Resiliency

Fault injection

Test the resilience of your apps by injecting delays and connection failures into a percentage of your requests.

About fault injections

You can set two following fault injection types in K8sGateway.

Delays: Delays simulate timing failures, such as network latency or overloaded upstreams.
Aborts: Aborts simulate crash failures, such as HTTP error codes or TCP connection failures.

Delays and aborts are independent of one another. When both values are set, your requests are either delayed only, delayed and aborted, or aborted only.

💡

Faults can be configured for a route by using a RouteOption resource. Gateway-level faults are not supported.

For more information, see the Fault API.

Before you begin

Follow the Get started guide to install K8sGateway, set up a gateway resource, and deploy the httpbin sample app.

Get the external address of the gateway and save it in an environment variable.

export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system gloo-proxy-http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS

kubectl port-forward deployment/gloo-proxy-http -n gloo-system 8080:8080

Aborts

Use a RouteOption resource to abort all incoming requests to a specific route.

Create a RouteOption custom resource to specify your fault injection rules. In the following example, 50% of all requests are rejected with a 503 HTTP response code.

kubectl apply -n httpbin -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: faults
  namespace: httpbin
spec:
  options:
    faults:
      abort:
        percentage: 50
        httpStatus: 503
EOF

Create an HTTPRoute resource for the httpbin app that references the RouteOption resource that you created.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin-faults
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: gloo-system
  hostnames:
    - faults.example
  rules:
    - filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.solo.io
            kind: RouteOption
            name: faults
      backendRefs:
        - name: httpbin
          port: 8000
EOF

Send a few requests to the httpbin app on the faults.example domain. Verify that some requests succeed with a 200 HTTP response code and other requests are rejected with a 503 HTTP response code.

curl -i http://$INGRESS_GW_ADDRESS:8080/status/200 -H "host: faults.example:8080"

curl -i localhost:8080/status/200 -H "host: faults.example"

Example output for a successful response:

HTTP/1.1 200 OK
access-control-allow-credentials: true
access-control-allow-origin: *
date: Tue, 23 Apr 2024 17:12:13 GMT
x-envoy-upstream-service-time: 0
server: envoy
transfer-encoding: chunked

Example output for a denied request:

HTTP/1.1 503 Service Unavailable
content-length: 18
content-type: text/plain
date: Tue, 23 Apr 2024 17:12:08 GMT
server: envoy

Optional: Remove the resources that you created.

kubectl delete httproute httpbin-faults -n httpbin
kubectl delete routeoption faults -n httpbin

Delays

Use a RouteOption resource to deny incoming requests to a specific route.

Create a RouteOption custom resource to specify your fault injection rules. In the following example, 50% of all requests are delayed by 5 seconds.

kubectl apply -n httpbin -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: RouteOption
metadata:
  name: faults
  namespace: httpbin
spec:
  options:
    faults:
      delay:
        percentage: 50
        fixedDelay: '5s'
EOF

Create an HTTPRoute resource for the httpbin app that references the RouteOption resource that you created.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: httpbin-faults
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: gloo-system
  hostnames:
    - faults.example
  rules:
    - filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.solo.io
            kind: RouteOption
            name: faults
      backendRefs:
        - name: httpbin
          port: 8000
EOF

Send a few requests to the httpbin app on the faults.example domain. Verify that some requests succeed immediately and other requests are delayed by 5 seconds.

curl -i http://$INGRESS_GW_ADDRESS:8080/status/200 -H "host: faults.example:8080"

curl -i localhost:8080/status/200 -H "host: faults.example"

Example output:

HTTP/1.1 200 OK
access-control-allow-credentials: true
access-control-allow-origin: *
date: Tue, 23 Apr 2024 17:18:51 GMT
x-envoy-upstream-service-time: 0
server: envoy
transfer-encoding: chunked

Optional: Remove the resources that you created.

kubectl delete httproute httpbin-faults -n httpbin
kubectl delete routeoption faults -n httpbin

Retries