Fault injection

Fault injection

Test the resilience of your apps by injecting delays and connection failures into a percentage of your requests.

About fault injections

You can set two following fault injection types in K8sGateway.

  • Delays: Delays simulate timing failures, such as network latency or overloaded upstreams.
  • Aborts: Aborts simulate crash failures, such as HTTP error codes or TCP connection failures.

Delays and aborts are independent of one another. When both values are set, your requests are either delayed only, delayed and aborted, or aborted only.

💡
Faults can be configured for a route by using a RouteOption resource. Gateway-level faults are not supported.

For more information, see the Fault API.

Before you begin

  1. Follow the Get started guide to install K8sGateway, set up a gateway resource, and deploy the httpbin sample app.

  2. Get the external address of the gateway and save it in an environment variable.

    export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system gloo-proxy-http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
    echo $INGRESS_GW_ADDRESS  
    kubectl port-forward deployment/gloo-proxy-http -n gloo-system 8080:8080

Aborts

Use a RouteOption resource to abort all incoming requests to a specific route.

  1. Create a RouteOption custom resource to specify your fault injection rules. In the following example, 50% of all requests are rejected with a 503 HTTP response code.

    kubectl apply -n httpbin -f- <<EOF
    apiVersion: gateway.solo.io/v1
    kind: RouteOption
    metadata:
      name: faults
      namespace: httpbin
    spec:
      options:
        faults:
          abort:
            percentage: 50
            httpStatus: 503
    EOF
  2. Create an HTTPRoute resource for the httpbin app that references the RouteOption resource that you created.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1beta1
    kind: HTTPRoute
    metadata:
      name: httpbin-faults
      namespace: httpbin
    spec:
      parentRefs:
      - name: http
        namespace: gloo-system
      hostnames:
        - faults.example
      rules:
        - filters:
            - type: ExtensionRef
              extensionRef:
                group: gateway.solo.io
                kind: RouteOption
                name: faults
          backendRefs:
            - name: httpbin
              port: 8000
    EOF
  3. Send a few requests to the httpbin app on the faults.example domain. Verify that some requests succeed with a 200 HTTP response code and other requests are rejected with a 503 HTTP response code.

    curl -i http://$INGRESS_GW_ADDRESS:8080/status/200 -H "host: faults.example:8080"
    curl -i localhost:8080/status/200 -H "host: faults.example"

    Example output for a successful response:

    HTTP/1.1 200 OK
    access-control-allow-credentials: true
    access-control-allow-origin: *
    date: Tue, 23 Apr 2024 17:12:13 GMT
    x-envoy-upstream-service-time: 0
    server: envoy
    transfer-encoding: chunked

    Example output for a denied request:

    HTTP/1.1 503 Service Unavailable
    content-length: 18
    content-type: text/plain
    date: Tue, 23 Apr 2024 17:12:08 GMT
    server: envoy
  4. Optional: Remove the resources that you created.

    kubectl delete httproute httpbin-faults -n httpbin
    kubectl delete routeoption faults -n httpbin

Delays

Use a RouteOption resource to deny incoming requests to a specific route.

  1. Create a RouteOption custom resource to specify your fault injection rules. In the following example, 50% of all requests are delayed by 5 seconds.

    kubectl apply -n httpbin -f- <<EOF
    apiVersion: gateway.solo.io/v1
    kind: RouteOption
    metadata:
      name: faults
      namespace: httpbin
    spec:
      options:
        faults:
          delay:
            percentage: 50
            fixedDelay: '5s'
    EOF
  2. Create an HTTPRoute resource for the httpbin app that references the RouteOption resource that you created.

    kubectl apply -f- <<EOF
    apiVersion: gateway.networking.k8s.io/v1beta1
    kind: HTTPRoute
    metadata:
      name: httpbin-faults
      namespace: httpbin
    spec:
      parentRefs:
      - name: http
        namespace: gloo-system
      hostnames:
        - faults.example
      rules:
        - filters:
            - type: ExtensionRef
              extensionRef:
                group: gateway.solo.io
                kind: RouteOption
                name: faults
          backendRefs:
            - name: httpbin
              port: 8000
    EOF
  3. Send a few requests to the httpbin app on the faults.example domain. Verify that some requests succeed immediately and other requests are delayed by 5 seconds.

    curl -i http://$INGRESS_GW_ADDRESS:8080/status/200 -H "host: faults.example:8080"
    curl -i localhost:8080/status/200 -H "host: faults.example"

    Example output:

    HTTP/1.1 200 OK
    access-control-allow-credentials: true
    access-control-allow-origin: *
    date: Tue, 23 Apr 2024 17:18:51 GMT
    x-envoy-upstream-service-time: 0
    server: envoy
    transfer-encoding: chunked
  4. Optional: Remove the resources that you created.

    kubectl delete httproute httpbin-faults -n httpbin
    kubectl delete routeoption faults -n httpbin