How to configure HTTPS backends in envoy

Originally posted on my blog. Drop by for a better reading experience, including the highlighted source code.

Envoy is an extremely flexible reverse proxy, most known by its use in istio where it functions as an envelope in every job, routing the traffic and managing authorization.

That said, it’s totally fine to use envoy on its own; one case for such would be gRPC-Web. Despite gRPC being based on HTTP/2, the web browsers don’t expose enough of the HTTP insides to the JS runtime for the client code to talk gRPC directly, and thus there’s a need in proxying a web-safe gRPC-Web into the “native” gRPC. This is where envoy comes in.

Here’s a typical envoy configuration to serve as a gRPC-Web proxy:

# ...

static_resources:
  #...

  clusters:
  - name: echo_service
    connect_timeout: 0.25s
    type: logical_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts:
    - { socket_address: { address: node-server, port_value: 9090 }}

For every outgoing connection, envoy needs an entry in clusters, specifying the connection details. In the example above the cluster echo_service will be reachable at http://node-server:9090.

What if your backend talks HTTPS though? This is where the configuration gets interesting and somewhat cryptic.

#...
  clusters:
  - name: remote.example.com|443
    connect_timeout: 0.25s
    type: logical_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    load_assignment:
      cluster_name: remote.example.com|443
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: remote.example.com
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        sni: remote.example.com
        common_tls_context: 
          validation_context:
            match_subject_alt_names:
            - exact: "*.remote.example.com"
            trusted_ca:
              filename: /etc/ssl/certs/ca-certificates.crt

First, notice how the hosts is now deprecated and you need to specify the load_assignment configuration. It’s straightforward; the logical_dnsoption tells envoy to resolve the socket address.

I set the cluster name is set to remote.example.com|443. That bears no technical reason and I do that only to match the internal envoy’s reporting; i.e. it is customary but not required to name the clusters like that.

The transport_socket part tells envoy to use HTTPS (or rather—TLS). The crucial parts are the sni field which tells envoy which host to present for SNI validation (this should be your remote hostname in most of the cases) and the validation_context. Hilariously, it’s 2020 and envoy doesn’t verify the remote certificates by default, which means you must explicitly tell it to do that or face a potential MITM attack.

Beware of match_subject_alt_names being a string matcher. It means envoy won’t just behave like your browser; instead you need to include literally the expected SAN, e.g. if your remote has a wildcard certificate you must use that wildcard, not the actual domain.

Side note: if you’re using envoyproxy/envoy-alpine from Dockerhub, it doesn’t include the ca-certificates by default. Inherit from it and do something like:

FROM envoyproxy/envoy-alpine:latest

RUN apk --no-cache add ca-certificates

to make sure you have those certificates.

Final note. If your backend only talks HTTP/1.x but not HTTP/2, remove the http2_protocol_options flag and envoy will fall back talking the old HTTP.

Stay safe, verify your peer certificates, and use TLS. happy hacking!