Istio Sidecar Proxy Access Log

1. Istio Sidecar Proxy Access Log

This document examines the Sidecar Proxy Access Logs in various cases within an Istio environment.

1.1. Test Environment Setup

[Figure 1] shows the Istio Sidecar Proxy Access Log test environment. It consists of 2 Worker Nodes, each containing a shell Pod acting as a Client and a mock-server Pod acting as a Server. The shell Pod accesses the mock-server Pod through the configured Service, Destination Rule, and Virtual Service. For HTTP Protocol access, the curl command is used inside the shell Pod, and for gRPC Protocol access, the grpcurl command is used inside the shell Pod.

1.1.1. Kubernetes, Istio Environment Setup

# Create kubernetes cluster with kind
$ kind create cluster --config=- <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF

# Install istio
$ istioctl install --set profile=demo -y

# Enable sidecar injection to default namespace
$ kubectl label namespace default istio-injection=enabled

[Shell 1] Kubernetes, Istio Environment Setup

[Shell 1] shows the script for setting up the Kubernetes and Istio environment. A Kubernetes Cluster is created using kind, and Istio is installed. Then, Sidecar Injection is enabled for the default Namespace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    accessLogEncoding: TEXT
    accessLogFormat: |
      {
        "start_time": "%START_TIME%",
        "method": "%REQ(:METHOD)%",
        "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
        "protocol": "%PROTOCOL%",
        "response_code": "%RESPONSE_CODE%",
        "response_flags": "%RESPONSE_FLAGS%",
        "response_code_details": "%RESPONSE_CODE_DETAILS%",
        "connection_termination_details": "%CONNECTION_TERMINATION_DETAILS%",
        "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
        "bytes_received": "%BYTES_RECEIVED%",
        "bytes_sent": "%BYTES_SENT%",
        "duration": "%DURATION%",
        "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
        "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
        "user_agent": "%REQ(USER-AGENT)%",
        "request_id": "%REQ(X-REQUEST-ID)%",
        "authority": "%REQ(:AUTHORITY)%",
        "upstream_host": "%UPSTREAM_HOST%",
        "upstream_cluster": "%UPSTREAM_CLUSTER_RAW%",
        "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
        "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
        "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
        "requested_server_name": "%REQUESTED_SERVER_NAME%",
        "route_name": "%ROUTE_NAME%",
        "grpc_status": "%GRPC_STATUS%",
        "upstream_request_attempt_count": "%UPSTREAM_REQUEST_ATTEMPT_COUNT%",
        "request_duration": "%REQUEST_DURATION%",
        "response_duration": "%RESPONSE_DURATION%"
      }

[Text 1] Set Mesh Config

[Text 1] shows the Istio Mesh Config for changing the format of Istio Sidecar Proxy Access Log. The default format of Access Log is in plain text format which has poor readability, so the accessLogFormat field is used to change it to JSON format.

1.1.2. Workload Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
apiVersion: v1
kind: Pod
metadata:
  name: mock-server
  labels:
    app: mock-server
spec:
  containers:
  - name: mock-server
    image: ghcr.io/ssup2/mock-go-server:0.1.7
    ports:
    - containerPort: 8080
    - containerPort: 9090
    securityContext:
      capabilities:
        add: ["NET_ADMIN"]
---
apiVersion: v1
kind: Service
metadata:
  name: mock-server
spec:
  selector:
    app: mock-server
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: grpc
    port: 9090
    targetPort: 9090
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: mock-server
spec:
  hosts:
  - mock-server
  http:
  - timeout: 60s # default is disabled
    retries:
      attempts: 2                                                         # default value
      retryOn: "502,connect-failure,refused-stream,unavailable,cancelled" # add 502 to default value
    route:
    - destination:
        host: mock-server
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: mock-server
spec:
  host: mock-server
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1 # default value is 2^31-1
      http:
        http1MaxPendingRequests: 1   # default value is 2^31-1 (unlimited)
        maxConcurrentStreams: 1      # default value is 2^31-1 (unlimited)
    outlierDetection:
      consecutive5xxErrors: 5 # default value
      interval: 10s           # default value
      baseEjectionTime: 30s   # default value
      maxEjectionPercent: 100 # default value

[File 1] mock-server Pod Manifest

[File 1] shows the Manifest for the mock-server Workload. The mock-server Pod is created using the mock-server Image, opening port 8080 for HTTP service and port 9090 for gRPC service. In the Virtual Service, Timeout is set to 60s, and retry is configured with 2 retries (same as default) for a maximum of 3 request attempts. Also, the retry conditions are configured by adding the 502 Status Code to the 4 default error conditions: connect-failure, refused-stream, unavailable, cancelled.

Circuit Breaking is configured in the Destination Rule for testing. The outlierDetection field defines the criteria for determining abnormal status and is configured with default values. Circuit Breaking activates when 5 consecutive 5xx errors occur at 10-second intervals, with the Circuit Breaking duration set to 30 seconds. The connectionPool field specifies settings to limit the concurrent processing count of HTTP/GRPC requests, configured to allow only one request to be processed at a time.

There are two main methods to limit concurrent processing count: limiting based on maximum TCP Connections and limiting based on maximum concurrent HTTP/GRPC request processing count. The TCP Connection-based method uses the tcp.maxConnections field to limit the maximum number of TCP Connections. In [File 1], tcp.maxConnections is set to 1 to limit the maximum TCP Connections to 1, and http.http1MaxPendingRequests is set to 1 to limit the maximum pending requests to 1 before the TCP Connection becomes Ready.

For GRPC, multiple requests can be processed simultaneously on a single TCP Connection using HTTP/2’s Stream feature. Therefore, http.maxConcurrentStreams is set to 1 to force a maximum of 1 Stream per TCP Connection, easily triggering GRPC request Pending. If http.maxConcurrentStreams is not specified, unlimited Streams can be processed on a single TCP Connection, so GRPC request Pending does not occur.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: mock-server
spec:
  host: mock-server
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 1   # default value is 2^31-1 (unlimited)
    outlierDetection:
      consecutive5xxErrors: 5 # default value
      interval: 10s           # default value
      baseEjectionTime: 30s   # default value
      maxEjectionPercent: 100 # default value

[File 2] mock-server Destination Rule

The method to limit maximum concurrent HTTP/GRPC request processing count uses the http.http2MaxRequests field. In [File 2], http.http2MaxRequests is set to 1 to limit the maximum HTTP/GRPC request processing count to 1. Also, the remaining connectionPool fields are not set to prevent Request from being Pending. The Destination Rule set in [File 1] is used for most Cases, while [File 2]’s Destination Rule is used in some Circuit Breaking Cases.

Endpoint	Description
/status/{code}	Return specific HTTP status code
/bytes/{bytes}	Return specified number of bytes
/delay/{ms}	Delay response by milliseconds
/reset-before-response/{ms}	Server sends TCP RST before response after delay
/reset-after-response/{ms}	Server sends dummy data, then TCP RST after delay
/close-before-response/{ms}	Server closes connection before response after delay
/close-after-response/{ms}	Server sends dummy data, then closes connection after delay

[Table 1] mock-server HTTP Endpoints

Function	Description
/mock.MockService/Status	Return specific gRPC status code
/mock.MockService/Delay	Delay response by milliseconds
/mock.MockService/ResetBeforeResponse	Server sends TCP RST before response after delay
/mock.MockService/ResetAfterResponse	Server sends dummy data, then TCP RST after delay
/mock.MockService/CloseBeforeResponse	Server closes connection before response after delay
/mock.MockService/CloseAfterResponse	Server sends dummy data, then closes connection after delay

[Table 2] mock-server gRPC Endpoints

[Table 1] and [Table 2] show the behavior of the mock-server Workload’s HTTP Endpoints and gRPC Functions. The endpoints provided by mock-server are used to reproduce various cases.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: v1
kind: Pod
metadata:
  name: shell
  labels:
    app: shell
spec:
  containers:
  - name: shell
    image: nicolaka/netshoot
    command: ["sleep", "infinity"]
    securityContext:
      capabilities:
        add: ["NET_ADMIN"]

[File 3] shell Pod Manifest

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
syntax = "proto3";

package mock;

option go_package = "mock-go-server/proto";

service MockService {
  // Return specific status code
  rpc Status(StatusRequest) returns (StatusResponse);

  // Delay response
  rpc Delay(DelayRequest) returns (DelayResponse);

  // Server closes connection before response after delay
  rpc CloseBeforeResponse(CloseRequest) returns (Empty);

  // Server sends dummy data, then closes connection
  rpc CloseAfterResponse(CloseRequest) returns (stream CloseStreamResponse);

  // Server sends wrong protocol data after delay
  rpc WrongProtocol(WrongProtocolRequest) returns (Empty);

  // Server sends RST before response after delay
  rpc ResetBeforeResponse(ResetRequest) returns (Empty);

  // Server sends dummy data, then RST
  rpc ResetAfterResponse(ResetRequest) returns (stream ResetStreamResponse);
}

message CloseRequest {
  int32 milliseconds = 1;
}

message CloseStreamResponse {
  int32 sequence = 1;
  bytes data = 2;
}

message WrongProtocolRequest {
  int32 milliseconds = 1;
}

message ResetRequest {
  int32 milliseconds = 1;
}

message ResetStreamResponse {
  int32 sequence = 1;        // Message sequence number
  bytes data = 2;            // Payload data
}

message Empty {}

message StatusRequest {
  int32 code = 1;
}

message StatusResponse {
  int32 status_code = 1;
  string service = 2;
  string message = 3;
}

message DelayRequest {
  int32 milliseconds = 1;
}

message DelayResponse {
  string service = 1;
  int32 delayed_ms = 2;
  string message = 3;
}

[File 4] mock-server gRPC Service Definition

1
$ kubectl cp mock.proto shell:mock.proto

[Shell 2] Copy mock.proto to shell Pod

[File 3] shows the Manifest for the shell Pod. The shell Pod is created using the netshoot Image, with Network Admin privileges to enable the use of the iptables command. [File 4] shows the Proto file for calling the mock-server gRPC Service using the grpcurl command. [Shell 2] shows an example of copying the Proto file to the shell Pod.

1.2. HTTP Cases

1.2.1. OK Case

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
{"message":"OK","service":"mock-server","status_code":200}

[Shell 3] HTTP OK Case / curl Command

[Figure 2] shows the HTTP OK Case where a GET request is sent to the mock-server’s /status/200 Endpoint using the curl command from the shell Pod, and a 200 OK response is received. [Shell 3] shows an example of executing [Figure 2].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-14T15:04:12.558Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "59",
  "duration": "10",
  "upstream_service_time": "6",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "0432d33a-0ffb-94fc-98fc-b9b322d5eaa3",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.3:45178",
  "downstream_local_address": "10.96.191.168:8080",
  "downstream_remote_address": "10.244.2.3:34226",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "10"
}

[Text 2] HTTP OK Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-14T15:04:12.563Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "59",
  "duration": "4",
  "upstream_service_time": "3",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "0432d33a-0ffb-94fc-98fc-b9b322d5eaa3",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:46243",
  "downstream_local_address": "10.244.1.4:8080",
  "downstream_remote_address": "10.244.2.3:45178",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "4"
}

[Text 3] HTTP OK Case / mock-server Access Log

[Text 2] shows the Access Log of the shell Pod’s istio-proxy, and [Text 3] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /status/200 Endpoint and the 200 OK response.

1.2.2. Service Unavailable Case

[Figure 3] HTTP Service Unavailable Case

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
{"message":"Service Unavailable","service":"mock-server","status_code":503}

[Shell 4] HTTP Service Unavailable Case / curl Command

[Figure 3] shows the HTTP Service Unavailable Case where a GET request is sent to the mock-server’s /status/503 Endpoint using the curl command from the shell Pod, and a 503 Service Unavailable response is received. [Shell 4] shows an example of executing [Figure 3].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-28T12:47:37.317Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "56",
  "upstream_service_time": "52",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "af4ab845-948b-9669-8bae-384dc22cf9f7",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.11:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.6:39692",
  "downstream_local_address": "10.96.95.31:8080",
  "downstream_remote_address": "10.244.1.6:52850",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "56"
}

[Text 4] HTTP Service Unavailable Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-28T12:47:37.325Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "37",
  "upstream_service_time": "25",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "af4ab845-948b-9669-8bae-384dc22cf9f7",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.11:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:36269",
  "downstream_local_address": "10.244.2.11:8080",
  "downstream_remote_address": "10.244.1.6:39692",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "2",
  "response_duration": "36"
}

[Text 5] HTTP Service Unavailable Case / mock-server Access Log

[Text 4] shows the Access Log of the shell Pod’s istio-proxy, and [Text 5] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /status/503 Endpoint and the 503 Service Unavailable response.

1.2.3. Downstream TCP Close Case

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/delay/5000
^C

[Shell 5] Downstream TCP Close Case / curl Command

[Figure 4] shows the Downstream TCP Close Case where a GET request is sent to the mock-server’s /delay/5000 Endpoint using the curl command from the shell Pod, and the request is forcefully terminated using Ctrl+C before 5000ms passes. [Shell 5] shows an example of executing [Figure 4].

When the curl command is forcefully terminated during execution, the curl command internally terminates the Connection and sends a TCP FIN Flag to the shell Pod’s istio-proxy. The shell Pod’s istio-proxy that received the TCP FIN Flag aborts the request in progress and sends a TCP FIN Flag to the mock-server Pod, which is ultimately delivered to the mock-server Container. Afterward, when the mock-server Container sends the response after 5000ms, the mock-server Pod’s istio-proxy sends a TCP RST Flag to the mock-server Container because the Connection is already terminated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-01T11:29:33.615Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "2966",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "dfa53579-b29c-9787-9719-10e42ca2cf98",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.17:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.7:47142",
  "downstream_local_address": "10.96.188.135:8080",
  "downstream_remote_address": "10.244.1.7:40350",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 6] Downstream TCP Close Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-01T11:29:33.616Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "2972",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "dfa53579-b29c-9787-9719-10e42ca2cf98",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.17:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:34691",
  "downstream_local_address": "10.244.2.17:8080",
  "downstream_remote_address": "10.244.1.7:47142",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 7] Downstream TCP Close Case / mock-server Pod Access Log

[Text 6] shows the Access Log of the shell Pod’s istio-proxy, and [Text 7] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /delay/5000 Endpoint and response_code showing as 0. Also, response_flags showing as DC (DownstreamConnectionTermination) can be confirmed.

1.2.4. Downstream TCP RST Case

1
2
3
4
5
6
7
8
$ kubectl exec -it shell -- python3 -c '
import socket, struct, time
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("mock-server", 8080))
s.sendall(b"GET /delay/5000 HTTP/1.1\r\nHost: mock-server:8080\r\nUser-Agent: python-rst-client\r\n\r\n")
time.sleep(1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close()'

[Shell 6] Downstream TCP RST Case / python3 Command

[Shell 6] shows the Downstream TCP RST Case where a GET request is sent to the mock-server’s /delay/5000 Endpoint from the shell Pod, and the request is forcefully terminated by sending a TCP RST Flag instead of a TCP FIN Flag before 5000ms passes. Since the curl command cannot control the Socket’s SO_LINGER Option and therefore cannot send a TCP RST Flag, the python3 command is used to set the SO_LINGER Option to 0 and close the Socket 1000ms after sending the request, which sends a TCP RST Flag.

The shell Pod’s istio-proxy that received the TCP RST Flag does not forward the TCP RST Flag to the mock-server Pod as-is, but instead sends a TCP FIN Flag to terminate the Connection, identical to the Downstream TCP Close Case. Since istio-proxy manages the Downstream Connection and the Upstream Connection as separate TCP Connections, even if the Downstream Connection is abnormally terminated with a TCP RST Flag, the Upstream Connection is gracefully terminated with a TCP FIN Flag. The mock-server Pod’s istio-proxy that received the TCP FIN Flag also sends a TCP FIN Flag to the mock-server Container. Afterward, the mock-server Container sends the response after 5000ms, but since the Connection is already terminated, it receives a TCP RST Flag from the mock-server Pod’s istio-proxy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-26T14:44:50.149Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "992",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "python-rst-client",
  "request_id": "6829f238-8938-964e-b075-5acc45deea89",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.9:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.7:48142",
  "downstream_local_address": "10.96.221.63:8080",
  "downstream_remote_address": "10.244.2.7:52548",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 8] Downstream TCP RST Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-26T14:44:50.192Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "958",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "python-rst-client",
  "request_id": "6829f238-8938-964e-b075-5acc45deea89",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.9:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:55605",
  "downstream_local_address": "10.244.1.9:8080",
  "downstream_remote_address": "10.244.2.7:48142",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 9] Downstream TCP RST Case / mock-server Pod Access Log

[Text 8] shows the Access Log of the shell Pod’s istio-proxy, and [Text 9] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs show response_code as 0, response_flags as DC (DownstreamConnectionTermination), and response_code_details as downstream_remote_disconnect, identical to the Downstream TCP Close Case. In other words, it can be confirmed that istio-proxy does not distinguish between receiving a TCP FIN Flag and a TCP RST Flag from the Downstream in the Access Log.

1.2.5. Downstream TCP RST with Backpressure Case

[Figure 6] Downstream TCP RST with Backpressure Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ kubectl exec -i shell -- python3 - <<'EOF'
import socket, struct, time, subprocess

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("mock-server", 8080))
s.sendall(b"GET /bytes/50000000 HTTP/1.1\r\nHost: mock-server:8080\r\nUser-Agent: python-backpressure-rst\r\n\r\n")
time.sleep(4)   # do not read the response so the buffers on the path fill up
# show unread bytes stacked in istio-proxy's upstream socket
print(subprocess.run(["ss", "-tn", "dport", "=", ":8080"], capture_output=True, text=True).stdout)
# Close with SO_LINGER(on, 0) so the kernel sends TCP RST instead of FIN
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close()
EOF
State Recv-Q  Send-Q Local Address:Port  Peer Address:Port
ESTAB 124312  0         10.244.2.7:38012 10.96.221.63:8080
ESTAB 1868933 0         10.244.2.7:50072  10.244.1.10:8080

[Shell 7] Downstream TCP RST with Backpressure Case / python3 Command

[Shell 7] shows the Downstream TCP RST with Backpressure Case where a GET request is sent to the mock-server’s /bytes/50000000 Endpoint from the shell Pod to receive a 50MB response, and while receiving the response, the request is forcefully terminated by sending a TCP RST Flag after waiting 4000ms without reading the response.

When the Client does not read the response, the shell Pod’s istio-proxy keeps the data it could not deliver to the Client in its internal Buffer, and when the Buffer becomes full (High Watermark), Backpressure kicks in and the istio-proxy stops reading Upstream data. The data subsequently sent by the mock-server Pod accumulates in the Kernel Receive Buffer of the Upstream Socket since the istio-proxy no longer reads it. The ss command output in [Shell 7] shows about 1.8MB of unread data stacked in the Recv-Q of the istio-proxy’s Upstream Socket (10.244.1.10:8080).

In this state, the shell Pod’s istio-proxy that received the TCP RST Flag terminates the Upstream Connection, but unlike the Downstream TCP RST Case, it closes a Socket with unread data remaining in the Kernel Receive Buffer, so according to TCP rules, a TCP RST Flag is sent to the mock-server Pod instead of a TCP FIN Flag. In other words, the Flag that istio-proxy sends when terminating the Upstream Connection is determined by whether unread data exists in the Kernel Receive Buffer at the time of termination. The mock-server Pod’s istio-proxy that received the TCP RST Flag also sends a TCP RST Flag to the mock-server Container for the same reason. Due to Backpressure, the mock-server Pod’s istio-proxy was also not reading the data sent by the mock-server Container, so unread data also remains in the Kernel Receive Buffer of the Socket connected to the mock-server Container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-27T02:24:12.605Z",
  "method": "GET",
  "path": "/bytes/50000000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "1818624",
  "duration": "3982",
  "upstream_service_time": "60",
  "x_forwarded_for": "-",
  "user_agent": "python-backpressure-rst",
  "request_id": "ab01ee94-eacc-96f2-a3f1-291a830f1036",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.10:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.7:50072",
  "downstream_local_address": "10.96.221.63:8080",
  "downstream_remote_address": "10.244.2.7:38012",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "72"
}

[Text 10] Downstream TCP RST with Backpressure Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-27T02:24:12.633Z",
  "method": "GET",
  "path": "/bytes/50000000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "8752321",
  "duration": "3967",
  "upstream_service_time": "23",
  "x_forwarded_for": "-",
  "user_agent": "python-backpressure-rst",
  "request_id": "ab01ee94-eacc-96f2-a3f1-291a830f1036",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.10:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:49821",
  "downstream_local_address": "10.244.1.10:8080",
  "downstream_remote_address": "10.244.2.7:50072",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "43"
}

[Text 11] Downstream TCP RST with Backpressure Case / mock-server Pod Access Log

[Text 10] shows the Access Log of the shell Pod’s istio-proxy, and [Text 11] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs record response_flags as DC (DownstreamConnectionTermination) and response_code_details as downstream_remote_disconnect, identical to the Downstream TCP RST Case, and since the request was terminated while the response was being sent, response_code is recorded as 200. The bytes_sent field shows how much data each istio-proxy sent to its Downstream before the termination.

In other words, the Downstream TCP RST Case and the Downstream TCP RST with Backpressure Case cannot be distinguished in the Access Log, and the difference that istio-proxy sends a TCP RST Flag instead of a TCP FIN Flag to the Upstream can only be confirmed through a Packet Dump.

1.2.6. Upstream Request Retry Case

Since the retryOn Field of the Virtual Service in [File 1] includes the 502 Status Code, up to 2 retries are performed when a 502 Status Code response is received, so up to 3 requests are sent.

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/status/502
{"message":"Bad Gateway","service":"mock-server","status_code":502}

[Shell 8] Upstream Request Retry Case / curl Command

[Shell 8] shows the Upstream Request Retry Case where a GET request is sent to the mock-server’s /status/502 Endpoint using the curl command from the shell Pod. Since the mock-server returns a 502 Bad Gateway response for every request, the shell Pod’s istio-proxy delivers the last received 502 Bad Gateway response to the curl command after performing both retries.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-25T00:19:18.701Z",
  "method": "GET",
  "path": "/status/502",
  "protocol": "HTTP/1.1",
  "response_code": "502",
  "response_flags": "URX",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "68",
  "duration": "118",
  "upstream_service_time": "91",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.21.0",
  "request_id": "c0ace0c7-fe4b-9244-81cd-72cda95dee0b",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.9:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.7:41160",
  "downstream_local_address": "10.96.221.63:8080",
  "downstream_remote_address": "10.244.2.7:48340",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "118"
}

[Text 12] Upstream Request Retry Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-08-25T00:19:18.761Z",
  "method": "GET",
  "path": "/status/502",
  "protocol": "HTTP/1.1",
  "response_code": "502",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "68",
  "duration": "13",
  "upstream_service_time": "7",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.21.0",
  "request_id": "c0ace0c7-fe4b-9244-81cd-72cda95dee0b",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.9:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:51459",
  "downstream_local_address": "10.244.1.9:8080",
  "downstream_remote_address": "10.244.2.7:41132",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "13"
}
{
  "start_time": "2026-08-25T00:19:18.802Z",
  "method": "GET",
  "path": "/status/502",
  "protocol": "HTTP/1.1",
  "response_code": "502",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "68",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.21.0",
  "request_id": "c0ace0c7-fe4b-9244-81cd-72cda95dee0b",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.9:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:51459",
  "downstream_local_address": "10.244.1.9:8080",
  "downstream_remote_address": "10.244.2.7:41148",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2026-08-25T00:19:18.819Z",
  "method": "GET",
  "path": "/status/502",
  "protocol": "HTTP/1.1",
  "response_code": "502",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "68",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.21.0",
  "request_id": "c0ace0c7-fe4b-9244-81cd-72cda95dee0b",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.9:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:51459",
  "downstream_local_address": "10.244.1.9:8080",
  "downstream_remote_address": "10.244.2.7:41160",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}

[Text 13] Upstream Request Retry Case / mock-server Pod Access Log

[Text 12] shows the Access Log of the shell Pod’s istio-proxy, and [Text 13] shows the Access Log of the mock-server’s istio-proxy. In [Text 12], upstream_request_attempt_count is recorded as 3, which includes the first attempt and both retries, and response_flags is recorded as URX (UpstreamRetryLimitExceeded), indicating that the retry limit was exhausted.

In [Text 13], three logs with the same request_id are recorded. Since retries are performed by the istio-proxy of the shell Pod that serves as the client, the mock-server Pod’s istio-proxy handles each retry as a separate request, so upstream_request_attempt_count is recorded as 1 in every log.

1.2.7. Upstream TCP RST before Response Case

[Figure 8] Upstream TCP RST before Response Case

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/reset-before-response/1000
upstream connect error or disconnect/reset before headers. reset reason: connection termination

[Shell 9] Upstream TCP RST before Response Case / curl Command

[Figure 8] shows the Upstream TCP RST before Response Case where a GET request is sent to the mock-server’s /reset-before-response/1000 Endpoint using the curl command from the shell Pod, and after 1000ms, the mock-server Pod sends a TCP RST Flag to forcefully terminate the Connection. [Shell 9] shows an example of executing [Figure 8].

When the mock-server Pod’s istio-proxy receives a TCP RST Flag from the mock-server Container, it does not send a TCP RST Flag to the shell Pod but sends a 503 Service Unavailable response. Therefore, the shell Pod’s istio-proxy Access Log does not have response_flags and only shows the 503 Service Unavailable response.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-01T11:58:47.152Z",
  "method": "GET",
  "path": "/reset-before-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "95",
  "duration": "1077",
  "upstream_service_time": "1063",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "65cb201d-5b20-9321-8550-8749675883ee",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.17:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.7:49326",
  "downstream_local_address": "10.96.188.135:8080",
  "downstream_remote_address": "10.244.1.7:55326",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1075"
}

[Text 14] Upstream TCP RST before Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-01T11:58:47.167Z",
  "method": "GET",
  "path": "/reset-before-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "95",
  "duration": "1047",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "65cb201d-5b20-9321-8550-8749675883ee",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.17:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:33619",
  "downstream_local_address": "10.244.2.17:8080",
  "downstream_remote_address": "10.244.1.7:49326",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "1",
  "response_duration": "-"
}

[Text 15] Upstream TCP RST before Response Case / mock-server Pod Access Log

[Text 14] shows the Access Log of the shell Pod’s istio-proxy, and [Text 15] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /reset-before-response/1000 Endpoint and the 503 Service Unavailable response. Also, response_flags showing as UC (UpstreamConnectionTermination) can be confirmed, and response_code_details shows upstream_reset_before_response_started{connection_termination}, indicating that a TCP RST Flag was sent from the Upstream before the response started.

1.2.8. Upstream TCP RST after Response Case

[Figure 9] Upstream TCP RST after Response Case

1
2
3
$ kubectl exec -it shell -- curl -s mock-server:8080/reset-after-response/1000
curl: (18) transfer closed with outstanding read data remaining
dummy datacommand terminated with exit code 18

[Shell 10] Upstream TCP RST after Response Case / curl Command

[Figure 9] shows the Upstream TCP RST after Response Case where a GET request is sent to the mock-server’s /reset-after-response/1000 Endpoint using the curl command from the shell Pod, and after 1000ms, the mock-server Pod sends a partial response and then sends a TCP RST Flag to forcefully terminate the Connection. [Shell 10] shows an example of executing [Figure 9].

When the mock-server Pod’s istio-proxy receives a TCP RST Flag, it sends a TCP FIN Flag to the shell Pod to terminate the TCP Connection. Also, since it was an unexpected Connection termination, a TCP RST Flag is sent after the TCP RST Flag.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-01T12:47:45.064Z",
  "method": "GET",
  "path": "/reset-after-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "UPE",
  "response_code_details": "upstream_reset_after_response_started{protocol_error}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "10",
  "duration": "2023",
  "upstream_service_time": "1006",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "4b388e57-8df0-9198-b9a8-3f72f0865739",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.17:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.7:52250",
  "downstream_local_address": "10.96.188.135:8080",
  "downstream_remote_address": "10.244.1.7:59530",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1007"
}

[Text 16] Upstream TCP RST after Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-01T12:47:45.066Z",
  "method": "GET",
  "path": "/reset-after-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "UPE",
  "response_code_details": "upstream_reset_after_response_started{protocol_error}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "10",
  "duration": "1017",
  "upstream_service_time": "1003",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "4b388e57-8df0-9198-b9a8-3f72f0865739",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.17:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:44401",
  "downstream_local_address": "10.244.2.17:8080",
  "downstream_remote_address": "10.244.1.7:52250",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1004"
}

[Text 17] Upstream TCP RST after Response Case / mock-server Pod Access Log

[Text 16] shows the Access Log of the shell Pod’s istio-proxy, and [Text 17] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /reset-after-response/1000 Endpoint and the 200 OK response. Also, response_flags showing as UPE (UpstreamProtocolError) can be confirmed.

response_code_details shows upstream_reset_after_response_started{protocol_error}, indicating that a TCP RST Flag was sent from the Upstream after partial response transmission. The Protocol Error occurs because the TCP RST Flag was sent from the Upstream before the complete HTTP response was transmitted.

1.2.9. Upstream TCP Close before Response Case

[Figure 10] Upstream TCP Close before Response Case

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/close-before-response/1000
upstream connect error or disconnect/reset before headers. reset reason: connection termination

[Shell 11] Upstream TCP Close before Response Case / curl Command

[Figure 10] shows the Upstream TCP Close before Response Case where a GET request is sent to the mock-server’s /close-before-response/1000 Endpoint using the curl command from the shell Pod, and after 1000ms, the mock-server Pod forcefully terminates the Connection. [Shell 11] shows an example of executing [Figure 10].

When the mock-server Pod’s istio-proxy receives a TCP FIN Flag from the mock-server Container, it sends a 503 Service Unavailable response to the shell Pod to indicate that the request was abnormally terminated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T08:00:51.487Z",
  "method": "GET",
  "path": "/close-before-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "95",
  "duration": "1004",
  "upstream_service_time": "1003",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "4fa56d4c-a9ce-9fa3-b938-9c99162e5b74",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.22:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:45602",
  "downstream_local_address": "10.96.211.131:8080",
  "downstream_remote_address": "10.244.1.8:55556",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1004"
}

[Text 18] Upstream TCP Connection Close Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T08:00:51.488Z",
  "method": "GET",
  "path": "/close-before-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "95",
  "duration": "1002",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "4fa56d4c-a9ce-9fa3-b938-9c99162e5b74",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.22:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:48343",
  "downstream_local_address": "10.244.2.22:8080",
  "downstream_remote_address": "10.244.1.8:45602",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 19] Upstream TCP Connection Close Case / mock-server Pod Access Log

[Text 18] shows the Access Log of the shell Pod’s istio-proxy, and [Text 19] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /disconnect/1000 Endpoint and the 503 Service Unavailable response. Also, response_flags showing as UC (UpstreamConnectionTermination) can be confirmed.

response_code_details shows upstream_reset_before_response_started{connection_termination}, indicating that a TCP FIN Flag was sent from the Upstream before the response started. This is the same detail as when receiving a TCP RST Flag in [Figure 8], confirming that the mock-server Pod’s istio-proxy logs the same response_code_details when receiving either a TCP FIN Flag or TCP RST Flag before the response is sent.

1.2.10. Upstream TCP Close after Response Case

[Figure 11] Upstream TCP Close after Response Case

1
2
$ kubectl exec -it shell -- curl -s mock-server:8080/close-after-response/1000
dummy datacommand terminated with exit code 18

[Shell 12] Upstream TCP Close after Response Case / curl Command

[Figure 11] shows the Upstream TCP Close after Response Case where a GET request is sent to the mock-server’s /close-after-response/1000 Endpoint using the curl command from the shell Pod, and after 1000ms, the mock-server Pod sends a response and then forcefully terminates the Connection. [Shell 12] shows an example of executing [Figure 11].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T08:01:36.305Z",
  "method": "GET",
  "path": "/close-after-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "UPE",
  "response_code_details": "upstream_reset_after_response_started{protocol_error}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "10",
  "duration": "2110",
  "upstream_service_time": "1008",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "feab8e89-b2ac-9992-add8-0c67d8624427",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.22:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:52206",
  "downstream_local_address": "10.96.211.131:8080",
  "downstream_remote_address": "10.244.1.8:33028",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1009"
}

[Text 20] Upstream TCP Close after Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T08:01:36.307Z",
  "method": "GET",
  "path": "/close-after-response/1000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "UPE",
  "response_code_details": "upstream_reset_after_response_started{protocol_error}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "10",
  "duration": "1109",
  "upstream_service_time": "1002",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "feab8e89-b2ac-9992-add8-0c67d8624427",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.22:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:58855",
  "downstream_local_address": "10.244.2.22:8080",
  "downstream_remote_address": "10.244.1.8:52206",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1007"
}

[Text 21] Upstream TCP Close after Response Case / mock-server Pod Access Log

[Text 20] shows the Access Log of the shell Pod’s istio-proxy, and [Text 21] shows the Access Log of the mock-server’s istio-proxy. Both Access Logs confirm the access to the /close-after-response/1000 Endpoint and the 200 OK response. Also, response_flags showing as UPE (UpstreamProtocolError) can be confirmed.

response_code_details shows upstream_reset_after_response_started{protocol_error}, indicating that a Protocol Error occurred and the Connection was forcefully terminated after the response started. This is the same detail as when receiving a TCP RST Flag in [Figure 9], confirming that the mock-server Pod’s istio-proxy logs the same response_code_details when receiving either a TCP FIN Flag or TCP RST Flag after partial response transmission.

1.2.11. Circuit Breaking with Upstream Connection Pool Overflow Case

[Figure 12] Circuit Breaking with Upstream Connection Pool Overflow Case

1
2
3
4
5
6
$ kubectl exec shell -- curl -s mock-server:8080/delay/5000 &
$ kubectl exec shell -- curl -s mock-server:8080/delay/5000 &
$ kubectl exec shell -- curl -s mock-server:8080/delay/5000 &
upstream connect error or disconnect/reset before headers. reset reason: overflow
{"delayed_ms":5000,"message":"Response delayed by 5000ms","service":"mock-server"}
{"delayed_ms":5000,"message":"Response delayed by 5000ms","service":"mock-server"}

[Shell 13] Circuit Breaking with Upstream Connection Pool Overflow Case / curl Command

[Figure 12] shows the Case where 3 consecutive GET requests are sent to the mock-server’s /delay/5000 Endpoint using the curl command from the shell Pod, causing Upstream Connection Pool Overflow and triggering Circuit Breaking. [Shell 13] shows an example of executing [Figure 12].

Due to the Destination Rule in [File 1], the first request is immediately forwarded to the mock-server Pod and completes with a 200 OK response after waiting 5000ms. The second request is Pending because the first request is being processed and waits until the first request finishes before being forwarded to the mock-server Pod. Therefore, the time for the second request to be processed is 5000ms + 5000ms = 10000ms. The third request cannot even be Pending, so istio-proxy considers it Upstream Overflow and triggers Circuit Breaking, sending a 503 Service Unavailable response.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2025-12-22T16:08:03.507Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UO",
  "response_code_details": "upstream_reset_before_response_started{overflow}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "81",
  "duration": "2",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "4c299e03-e57f-9613-8817-6682e9deb675",
  "authority": "mock-server:8080",
  "upstream_host": "-",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:55486",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-22T16:08:02.442Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "83",
  "duration": "5011",
  "upstream_service_time": "5010",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "8d667b79-534b-9d95-a3a7-4b2fa17263e6",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:58132",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:55470",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5011"
}
{
  "start_time": "2025-12-22T16:08:02.847Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "83",
  "duration": "9615",
  "upstream_service_time": "9615",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "afe67ad1-54bd-9a82-b947-6fe65a1dfe69",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:58134",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:55472",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "9615"
}

[Text 22] Circuit Breaking with Upstream Connection Pool Overflow Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
{
  "start_time": "2025-12-22T16:08:02.443Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "83",
  "duration": "5008",
  "upstream_service_time": "5008",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "8d667b79-534b-9d95-a3a7-4b2fa17263e6",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:52515",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:58132",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5008"
}
{
  "start_time": "2025-12-22T16:08:07.456Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "83",
  "duration": "5004",
  "upstream_service_time": "5003",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "afe67ad1-54bd-9a82-b947-6fe65a1dfe69",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:49425",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:58134",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5004"
}

[Text 23] Circuit Breaking with Upstream Connection Pool Overflow Case / mock-server Pod Access Log

[Text 22] shows the Access Log of the shell Pod’s istio-proxy, and [Text 23] shows the Access Log of the mock-server’s istio-proxy. In the shell Pod’s istio-proxy Access Log, the first log recorded is for the third request which failed immediately due to Upstream Connection Pool Overflow. response_flags showing as UO (UpstreamOverflow) can be confirmed, and comparing start_time with other logs confirms it started last. The second log is for the first request, and the third log is for the second request. response_duration can be confirmed as 5000ms and 10000ms respectively.

In the mock-server Pod’s istio-proxy Access Log, only logs for the first and second requests exist, and response_duration can be confirmed as 5000ms for both. The third request was not forwarded to the mock-server Pod due to Upstream Connection Pool Overflow at the shell Pod’s istio-proxy, so there is no log for the third request in the mock-server Pod’s istio-proxy.

1.2.12. Circuit Breaking with Upstream Request Limit Overflow Case

[Figure 13] Circuit Breaking with Upstream Request Limit Overflow Case

1
2
3
4
5
6
$ kubectl exec shell -- curl -s mock-server:8080/delay/5000 &
$ kubectl exec shell -- curl -s mock-server:8080/delay/5000 &
upstream connect error or disconnect/reset before headers. reset reason: overflow
$ kubectl exec shell -- curl -s mock-server:8080/delay/5000 &
upstream connect error or disconnect/reset before headers. reset reason: overflow
{"delayed_ms":5000,"message":"Response delayed by 5000ms","service":"mock-server"}

[Shell 14] Circuit Breaking with Upstream Request Limit Overflow Case / curl Command

[Figure 13] shows the Case where 3 consecutive GET requests are sent to the mock-server’s /delay/5000 Endpoint using the curl command from the shell Pod, causing Upstream Request Limit Overflow. To reproduce this Case, the Destination Rule set in [File 2] must be applied. [Shell 14] shows an example of executing [Figure 13].

Due to the Destination Rule settings in [File 2], the maximum concurrent requests that can be processed is one and request Pending is not possible, so the second and third requests are not forwarded to the mock-server Pod due to Upstream Overflow.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-01-03T15:23:06.371Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UO",
  "response_code_details": "upstream_reset_before_response_started{overflow}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "81",
  "duration": "4",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "1f7ca280-0458-94b0-b520-f62a4d7655f7",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.18:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.188.135:8080",
  "downstream_remote_address": "10.244.1.7:47206",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-03T15:23:06.694Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UO",
  "response_code_details": "upstream_reset_before_response_started{overflow}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "81",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "e844139f-1dd6-91f4-9acd-f3511058cc52",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.18:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.188.135:8080",
  "downstream_remote_address": "10.244.1.7:47210",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-03T15:23:06.118Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "83",
  "duration": "5017",
  "upstream_service_time": "5016",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "99a50488-a899-9a8c-ab1b-07867b2ba1fd",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.18:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.7:56360",
  "downstream_local_address": "10.96.188.135:8080",
  "downstream_remote_address": "10.244.1.7:47198",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5017"
}

[Text 24] Circuit Breaking with Request Limit Upstream Overflow Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-03T15:23:06.121Z",
  "method": "GET",
  "path": "/delay/5000",
  "protocol": "HTTP/1.1",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "83",
  "duration": "5008",
  "upstream_service_time": "5007",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "99a50488-a899-9a8c-ab1b-07867b2ba1fd",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.18:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:51333",
  "downstream_local_address": "10.244.2.18:8080",
  "downstream_remote_address": "10.244.1.7:56360",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5007"
}

[Text 25] Circuit Breaking with Request Limit Upstream Overflow Case / mock-server Pod Access Log

[Text 24] shows the Access Log of the shell Pod’s istio-proxy, and [Text 25] shows the Access Log of the mock-server Pod’s istio-proxy. In the shell Pod’s istio-proxy Access Log, the first logs recorded are for the second and third requests which failed immediately due to Upstream Request Limit Overflow. The first log is for the second request, and the second log is for the third request. Both show response_flags as UO (UpstreamOverflow). The last log is for the first request, which was successfully forwarded to the mock-server Pod and processed.

1.2.13. Circuit Breaking with No Healthy Upstream Case

[Figure 14] Circuit Breaking with No Healthy Upstream Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
{"message":"Service Unavailable","service":"mock-server","status_code":503}
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
{"message":"Service Unavailable","service":"mock-server","status_code":503}
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
{"message":"Service Unavailable","service":"mock-server","status_code":503}
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
{"message":"Service Unavailable","service":"mock-server","status_code":503}
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
{"message":"Service Unavailable","service":"mock-server","status_code":503}
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
no healthy upstream
$ kubectl exec -it shell -- curl -s mock-server:8080/status/503
no healthy upstream
$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
no healthy upstream

[Shell 15] Circuit Breaking with No Healthy Upstream Case / curl Command

[Figure 14] shows the Case where 8 consecutive GET requests are sent to the mock-server’s /status/503 Endpoint using the curl command from the shell Pod, causing Circuit Breaking through No Healthy Upstream. [Shell 15] shows an example of executing [Figure 14].

Due to the Destination Rule in [File 1], Circuit Breaking activates when 5 consecutive 5XX Errors occur. Therefore, the first 5 requests from the shell Pod are all forwarded to the mock-server Pod, but the subsequent 3 requests are not forwarded to the mock-server Pod due to Circuit Breaking.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
{
  "start_time": "2025-12-22T12:23:20.109Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "42",
  "upstream_service_time": "37",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "464940f1-963a-90e4-8deb-172245eac437",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:48362",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:60748",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "42"
}
{
  "start_time": "2025-12-22T12:23:24.234Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "2",
  "upstream_service_time": "2",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "75817232-e306-94d0-89f7-095718dbe70d",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:48372",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:60750",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "2"
}
{
  "start_time": "2025-12-22T12:23:25.743Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "1",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "9de7cc1c-f93b-9cf3-9db8-53f29de86d1b",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:48372",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:60756",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2025-12-22T12:23:26.981Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "80ee43d3-af9a-9aa3-bfc1-a288fb7d64a1",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:48362",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:41660",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2025-12-22T12:23:28.384Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "1",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "411cedbf-1f1c-9d16-8ea1-6a10678fdd6d",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.3:48372",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:41676",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2025-12-22T12:23:29.590Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "f8dc8df5-a957-93d1-92aa-226110d9dd10",
  "authority": "mock-server:8080",
  "upstream_host": "-",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:41690",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-22T12:23:31.367Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "0ff441a9-9bfd-9c10-8911-5b6c079dd31a",
  "authority": "mock-server:8080",
  "upstream_host": "-",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:41702",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-22T12:23:40.187Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "c1becf45-cf49-9262-9df1-a74f63190b6b",
  "authority": "mock-server:8080",
  "upstream_host": "-",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:37202",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 26] Circuit Breaking with No Healthy Upstream Case / shell Pod Access Log

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
{
  "start_time": "2025-12-22T12:23:20.129Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "6",
  "upstream_service_time": "4",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "464940f1-963a-90e4-8deb-172245eac437",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:58085",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:48362",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "6"
}
{
  "start_time": "2025-12-22T12:23:24.236Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "75817232-e306-94d0-89f7-095718dbe70d",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:58085",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:48372",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2025-12-22T12:23:25.743Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "9de7cc1c-f93b-9cf3-9db8-53f29de86d1b",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:58085",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:48372",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2025-12-22T12:23:26.981Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "80ee43d3-af9a-9aa3-bfc1-a288fb7d64a1",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:58085",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:48362",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2025-12-22T12:23:28.384Z",
  "method": "GET",
  "path": "/status/503",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "76",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "411cedbf-1f1c-9d16-8ea1-6a10678fdd6d",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:58085",
  "downstream_local_address": "10.244.2.4:8080",
  "downstream_remote_address": "10.244.1.3:48372",
  "requested_server_name": "outbound_.8080_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}

[Text 27] Circuit Breaking with No Healthy Upstream Case / mock-server Pod Access Log

[Text 26] shows the Access Log of the shell Pod’s istio-proxy, and [Text 27] shows the Access Log of the mock-server Pod’s istio-proxy. In the shell Pod’s istio-proxy Access Log, only the last 3 requests show response_flags as UH (NoHealthyUpstream) and were not forwarded to the mock-server Pod. Also, in the mock-server Pod’s istio-proxy Access Log, only logs for the first 5 requests exist.

1.2.14. Upstream Connection Failure with Timeout Case

[Figure 15] Upstream Connection Failure with Timeout Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ SHELL_IP=$(kubectl get pod shell -o jsonpath='{.status.podIP}')
$ kubectl exec mock-server -c mock-server -- iptables -A INPUT -s ${SHELL_IP} -j DROP
# $ kubectl exec mock-server -c mock-server -- iptables -D INPUT 1 remove rule after case execution

$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout
$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
no healthy upstream
$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
no healthy upstream

[Shell 16] Upstream Connection Failure with Timeout Case / iptables & curl Command

[Figure 15] shows the Upstream Connection Failure with Timeout Case where a connection to the mock-server’s /status/200 Endpoint from the shell Pod using the curl command fails due to Timeout and is Retried. [Shell 16] shows an example of executing [Figure 15]. To trigger the Timeout, a rule is added using the iptables command to DROP traffic from the shell Pod’s IP Address, and then a request is sent using the curl command.

Due to the Virtual Service’s connect-failure setting in [File 1], 2 retries occur for a total of 3 requests. Therefore, the first request from the shell Pod performs 3 retries and then outputs a connection timeout error. The second request from the shell Pod triggers only 1 retry for a total of 2 requests, because Circuit Breaking activates when 5 consecutive 5XX Errors occur due to the Destination Rule in [File 1].

The first request’s 3 requests and the second request’s 2nd request, a total of 5 requests occurred and all failed due to Timeout, so it determines there is no Healthy Upstream and Circuit Breaking activates. Therefore, the second request’s 2nd retry is not sent to the mock-server Pod due to Circuit Breaking, and the second request outputs a no healthy upstream error. The third request immediately outputs a no healthy upstream error due to Circuit Breaking.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2025-12-21T07:17:42.331Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "URX,UF",
  "response_code_details": "upstream_reset_before_response_started{connection_timeout}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "114",
  "duration": "30066",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "78343cc2-d6f2-9a5d-8dff-84c7ea3596c3",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.12:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.225.216:8080",
  "downstream_remote_address": "10.244.2.5:35248",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-21T07:18:15.580Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "20062",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "575b956c-0e96-9471-a21f-0555763492ab",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.1.12:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.225.216:8080",
  "downstream_remote_address": "10.244.2.5:46700",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-21T07:18:38.162Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "24be226c-274c-9948-bb96-88df90492bdf",
  "authority": "mock-server:8080",
  "upstream_host": "-",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.225.216:8080",
  "downstream_remote_address": "10.244.2.5:33536",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 28] Upstream Connection Failure with Timeout Case / shell Pod Access Log

[Text 28] shows the Access Log of the shell Pod’s istio-proxy. Since requests from the shell Pod are not forwarded to the mock-server Pod by istio-proxy, nothing is logged in the mock-server Pod’s istio-proxy Access Log. The first request shows response_flags as URX (UpstreamRetryLimitExceeded) and UF (UpstreamConnectionFailure), and response_code_details shows upstream_reset_before_response_started{connection_timeout}, confirming a Connection Timeout occurred. upstream_request_attempt_count shows as 3.

The second and third requests show response_flags as UH (NoHealthyUpstream) due to Circuit Breaking, and response_code_details shows no_healthy_upstream. The second request shows upstream_request_attempt_count as 3. The third request shows upstream_request_attempt_count as 1.

1.2.15. Upstream Connection Failure with TCP Reset Case

[Figure 16] Upstream Connection Failure with TCP Reset Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ SHELL_IP=$(kubectl get pod shell -o jsonpath='{.status.podIP}')
$ kubectl exec mock-server -c mock-server -- iptables-legacy -A INPUT -p tcp -s ${SHELL_IP} -j REJECT --reject-with tcp-reset
# $ kubectl exec mock-server -c mock-server -- iptables-legacy -D INPUT 1 remove rule after case execution

$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
no healthy upstream
$ kubectl exec -it shell -- curl -s mock-server:8080/status/200
no healthy upstream

[Shell 17] Upstream Connection Failure with TCP Reset Case / iptables & curl Command

[Figure 16] shows the Upstream Connection Failure with TCP Reset Case where a connection to the mock-server’s /status/200 Endpoint from the shell Pod using the curl command fails due to TCP Reset and is Retried. [Shell 17] shows an example of executing [Figure 16]. To trigger TCP Reset, a rule is added using the iptables command to REJECT traffic from the shell Pod’s IP Address, and then a request is sent using the curl command. Except for the Connection Refused error message, it shows the same results as the Case where Retry is performed due to Timeout.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2025-12-22T17:09:54.276Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "URX,UF",
  "response_code_details": "upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "delayed_connect_error:_Connection_refused",
  "bytes_received": "0",
  "bytes_sent": "190",
  "duration": "63",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "6e0d7462-2323-9e73-8dcf-43701a368edb",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:36360",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-22T17:09:56.768Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "delayed_connect_error:_Connection_refused",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "17",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "af1888dc-b298-9c53-8d44-ed21a0da304f",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.4:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:36368",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2025-12-22T17:10:00.420Z",
  "method": "GET",
  "path": "/status/200",
  "protocol": "HTTP/1.1",
  "response_code": "503",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "19",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "9b901845-b436-95f3-89d1-868d9acd05ac",
  "authority": "mock-server:8080",
  "upstream_host": "-",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.90.250:8080",
  "downstream_remote_address": "10.244.1.3:36370",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 29] Upstream Connection Failure with TCP Reset Case / shell Pod Access Log

[Text 29] shows the Access Log of the shell Pod’s istio-proxy. Since requests from the shell Pod are not forwarded to the mock-server Pod by istio-proxy, nothing is logged in the mock-server Pod’s istio-proxy Access Log. response_code_details shows upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}, confirming that Remote Connection Failure and Delayed Connect Error occurred. Except for this part, it shows the same results as the Case where Retry is performed due to Timeout.

1.2.16. Upstream Request Timeout Case

[Figure 17] Upstream Request Timeout Case

1
$ kubectl exec -it shell -- curl -s mock-server:8080/delay/70000

[Shell 18] Upstream Request Timeout Case / curl Command

[Figure 17] shows the Upstream Request Timeout Case where a GET request is sent to the mock-server’s /delay/70000 Endpoint from the shell Pod using the curl command, but the mock-server Pod’s istio-proxy does not receive a response after waiting 60000ms and processes the Request as Timeout. [Shell 18] shows an example of executing [Figure 17].

Due to the Virtual Service in [File 1], requests sent to the mock-server Pod can wait up to 60000ms. However, the request sent to the mock-server Pod’s /delay/70000 Endpoint requires 70000ms, so a Timeout occurs. When a Timeout occurs, the mock-server Pod’s istio-proxy sends TCP FIN Flag and TCP RST Flag sequentially to terminate the connection with the mock-server Pod. It also sends a 504 Gateway Timeout response to the shell Container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-12T16:07:19.812Z",
  "method": "GET",
  "path": "/delay/70000",
  "protocol": "HTTP/1.1",
  "response_code": "504",
  "response_flags": "UT",
  "response_code_details": "response_timeout",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "24",
  "duration": "60022",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "da28cc83-9055-9ed5-8c0e-104bf3337a0c",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.23:8080",
  "upstream_cluster": "outbound|8080||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:37022",
  "downstream_local_address": "10.96.1.12:8080",
  "downstream_remote_address": "10.244.1.8:37910",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 30] Upstream Timeout Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-12T16:07:19.851Z",
  "method": "GET",
  "path": "/delay/70000",
  "protocol": "HTTP/1.1",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "59989",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "curl/8.14.1",
  "request_id": "da28cc83-9055-9ed5-8c0e-104bf3337a0c",
  "authority": "mock-server:8080",
  "upstream_host": "10.244.2.23:8080",
  "upstream_cluster": "inbound|8080||",
  "upstream_local_address": "127.0.0.6:52563",
  "downstream_local_address": "10.244.2.23:8080",
  "downstream_remote_address": "10.244.1.8:37022",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 31] Upstream Timeout Case / mock-server Pod Access Log

[Text 30] shows the Access Log of the shell Pod’s istio-proxy, and [Text 31] shows the Access Log of the mock-server Pod’s istio-proxy. The shell Pod’s istio-proxy shows response_flags as UT (UpstreamTimeout). The mock-server Pod’s istio-proxy shows response_flags as DC (DownstreamConnectionTermination).

1.3. GRPC Cases

1.3.1. OK Case

1
2
3
4
5
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
{
  "service": "mock-server",
  "message": "OK"
}

[Shell 19] OK Case / grpcurl Command

[Figure 18] shows the OK Case where the grpcurl command is used from the shell Pod to send a code: 0 request to the mock-server’s /mock.MockService/Status function and receive an OK response. [Shell 19] shows an example of executing [Figure 18].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-25T11:18:51.880Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "5",
  "bytes_sent": "22",
  "duration": "2",
  "upstream_service_time": "1",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "7eb9994b-2491-9e41-9094-7664914a3692",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.8:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.5:58590",
  "downstream_local_address": "10.96.186.69:9090",
  "downstream_remote_address": "10.244.1.5:53152",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "2"
}

[Text 32] OK Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-25T11:18:51.881Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "5",
  "bytes_sent": "22",
  "duration": "1",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "7eb9994b-2491-9e41-9094-7664914a3692",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.8:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:43691",
  "downstream_local_address": "10.244.2.8:9090",
  "downstream_remote_address": "10.244.1.5:58590",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1"
}

[Text 33] OK Case / mock-server Pod Access Log

[Text 32] shows the Access Log of the shell Pod’s istio-proxy, and [Text 33] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/Status function and grpc_status as OK.

1.3.2. Internal Case

1
2
3
4
5
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Internal
  Message: Simulated error with gRPC code 13 (Internal)
command terminated with exit code 77

[Shell 20] Internal Case / grpcurl Command

[Figure 19] shows the Internal Case where the grpcurl command is used from the shell Pod to send a code: 13 request to the mock-server’s /mock.MockService/Status function and receive an Internal response. [Shell 20] shows an example of executing [Figure 19].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-25T02:32:08.271Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "150",
  "upstream_service_time": "135",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "0ea3e5b2-6c30-9f3b-8c06-bc056c442043",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.7:42196",
  "downstream_local_address": "10.96.221.63:9090",
  "downstream_remote_address": "10.244.2.7:49548",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "13",
  "response_duration": "149"
}

[Text 34] Internal Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-25T02:32:08.341Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "57",
  "upstream_service_time": "36",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "0ea3e5b2-6c30-9f3b-8c06-bc056c442043",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:53621",
  "downstream_local_address": "10.244.1.9:9090",
  "downstream_remote_address": "10.244.2.7:42196",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "20",
  "response_duration": "57"
}

[Text 35] Internal Case / mock-server Pod Access Log

[Text 34] shows the Access Log of the shell Pod’s istio-proxy, and [Text 35] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/Status function and grpc_status as Internal. Also, response_code shows 200 OK, and when using gRPC, response_code always shows 200 OK regardless of the gRPC result. Since the INTERNAL (13) Status Code is not included in the retryOn Field of the Virtual Service in [File 1], no retry occurs, and upstream_request_attempt_count is recorded as 1.

1.3.3. Downstream TCP Close Case

1
2
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay
^C

[Shell 21] Downstream TCP Close Case / grpcurl Command

[Figure 20] shows the Downstream TCP Close Case where the grpcurl command is used from the shell Pod to send a milliseconds: 5000 request to the mock-server’s /mock.MockService/Delay function, and forcefully terminates the request using Ctrl+C before 5000ms passes. [Shell 21] shows an example of executing [Figure 20].

When the grpcurl command is forcefully terminated during execution, the grpcurl command internally terminates the Connection and sends a TCP FIN Flag to the shell Pod’s istio-proxy, and the shell Pod’s istio-proxy that received the TCP FIN Flag sends an HTTP/2 RST_STREAM Frame to the mock-server Pod, which is ultimately delivered to the mock-server Container to terminate the request. Unlike the HTTP/1.1 Protocol, the TCP Connection between Pods using the HTTP/2 Protocol is shared by multiple requests through Stream multiplexing, so the shell Pod’s istio-proxy does not terminate the TCP Connection between Pods and only terminates the Stream of the request through the HTTP/2 RST_STREAM Frame.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-05T14:41:20.286Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "778",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b62ed186-f4a0-9f38-b6a9-e5d351c626c9",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.19:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:58320",
  "downstream_local_address": "10.96.212.50:9090",
  "downstream_remote_address": "10.244.1.8:53602",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "8",
  "response_duration": "-"
}

[Text 36] Downstream TCP Close Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-05T14:41:20.300Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "0",
  "response_flags": "DR",
  "response_code_details": "http2.remote_reset",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "778",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b62ed186-f4a0-9f38-b6a9-e5d351c626c9",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.19:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:52919",
  "downstream_local_address": "10.244.2.19:9090",
  "downstream_remote_address": "10.244.1.8:58320",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "8",
  "response_duration": "-"
}

[Text 37] Downstream TCP Close Case / mock-server Pod Access Log

[Text 36] shows the Access Log of the shell Pod’s istio-proxy, and [Text 37] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/Delay function with response_code as 0 and grpc_status as -.

Also, the shell Pod’s istio-proxy shows response_flags as DC (DownstreamConnectionTermination) because it receives TCP FIN Flag from the grpcurl command, and the mock-server Pod’s istio-proxy shows response_flags as DR (DownstreamRemoteReset) because it receives HTTP/2 RST_STREAM Frame.

1.3.4. Downstream TCP RST Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ kubectl exec -i shell -- python3 - <<'EOF'
import socket, struct, time

def hdr(name, value):
    # HPACK literal header field without indexing, new name
    return b"\x00" + bytes([len(name)]) + name + bytes([len(value)]) + value

def frame(ftype, flags, stream_id, payload):
    return struct.pack(">I", len(payload))[1:] + bytes([ftype, flags]) + struct.pack(">I", stream_id) + payload

headers = b"".join([
    hdr(b":method", b"POST"),
    hdr(b":scheme", b"http"),
    hdr(b":path", b"/mock.MockService/Delay"),
    hdr(b":authority", b"mock-server:9090"),
    hdr(b"content-type", b"application/grpc"),
    hdr(b"te", b"trailers"),
    hdr(b"user-agent", b"python-rst-client"),
])
# DelayRequest{milliseconds: 5000} protobuf + gRPC length prefix
grpc_msg = b"\x00" + struct.pack(">I", 3) + b"\x08\x88\x27"

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("mock-server", 9090))
s.sendall(b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n")            # HTTP/2 preface
s.sendall(frame(0x4, 0x0, 0, b""))                        # empty SETTINGS
s.sendall(frame(0x1, 0x4, 1, headers))                    # HEADERS (END_HEADERS)
s.sendall(frame(0x0, 0x1, 1, grpc_msg))                   # DATA (END_STREAM)
time.sleep(1)
# Close with SO_LINGER(on, 0) so the kernel sends TCP RST instead of FIN
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
s.close()
EOF

[Shell 22] Downstream TCP RST Case / python3 Command

[Shell 22] shows the Downstream TCP RST Case where a milliseconds: 5000 request is sent to the mock-server’s /mock.MockService/Delay function from the shell Pod, and the request is forcefully terminated by sending a TCP RST Flag instead of a TCP FIN Flag before 5000ms passes. Since the grpcurl command cannot control the Socket’s SO_LINGER Option and therefore cannot send a TCP RST Flag, the python3 command is used to construct the HTTP/2 Frames and the gRPC Message directly to send the request, and 1000ms after sending the request, the SO_LINGER Option is set to 0 and the Socket is closed, which sends a TCP RST Flag.

The shell Pod’s istio-proxy that received the TCP RST Flag does not forward the TCP RST Flag to the mock-server Pod as-is, identical to the Downstream TCP Close Case, but sends an HTTP/2 RST_STREAM Frame to terminate only the Stream of the request. In other words, regardless of whether the Downstream Connection is gracefully terminated with a TCP FIN Flag or abnormally terminated with a TCP RST Flag, the TCP Connection between Pods is maintained and only the HTTP/2 RST_STREAM Frame is sent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-26T15:16:13.003Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "0",
  "response_flags": "DC",
  "response_code_details": "downstream_remote_disconnect",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "988",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "python-rst-client",
  "request_id": "7ccbd0bb-976a-904f-bd45-349ba1dce793",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.7:34766",
  "downstream_local_address": "10.96.221.63:9090",
  "downstream_remote_address": "10.244.2.7:33084",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "14",
  "response_duration": "-"
}

[Text 38] Downstream TCP RST Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-26T15:16:13.026Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "0",
  "response_flags": "DR",
  "response_code_details": "http2.remote_reset",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "979",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "python-rst-client",
  "request_id": "7ccbd0bb-976a-904f-bd45-349ba1dce793",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:34371",
  "downstream_local_address": "10.244.1.9:9090",
  "downstream_remote_address": "10.244.2.7:34766",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "13",
  "response_duration": "-"
}

[Text 39] Downstream TCP RST Case / mock-server Pod Access Log

[Text 38] shows the Access Log of the shell Pod’s istio-proxy, and [Text 39] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs are identical to the Downstream TCP Close Case: the shell Pod shows response_flags as DC (DownstreamConnectionTermination) with response_code_details as downstream_remote_disconnect, and the mock-server Pod shows response_flags as DR (DownstreamRemoteReset) with response_code_details as http2.remote_reset. In other words, just like the HTTP/1.1 Protocol, it can be confirmed that istio-proxy does not distinguish between receiving a TCP FIN Flag and a TCP RST Flag from the Downstream in the Access Log for the gRPC Protocol as well.

1.3.5. Upstream Request Retry Case

Since the retryOn Field of the Virtual Service in [File 1] includes the gRPC retry conditions unavailable and cancelled, up to 2 retries are performed when an UNAVAILABLE (14) Status Code response is received, so up to 3 requests are sent.

1
2
3
4
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 14}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: Simulated error with gRPC code 14 (Unavailable)

[Shell 23] Upstream Request Retry Case / grpcurl Command

[Shell 23] shows the Upstream Request Retry Case where the grpcurl command is used from the shell Pod to send a code: 14 (Unavailable) request to the mock-server’s /mock.MockService/Status function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-08-25T02:32:08.437Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "URX",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "56",
  "upstream_service_time": "56",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "01f901a0-01e7-978a-aecc-a81094ae5ed6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.2.7:42196",
  "downstream_local_address": "10.96.221.63:9090",
  "downstream_remote_address": "10.244.2.7:49562",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "56"
}

[Text 40] Upstream Request Retry Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-08-25T02:32:08.437Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "1",
  "upstream_service_time": "1",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "01f901a0-01e7-978a-aecc-a81094ae5ed6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:53621",
  "downstream_local_address": "10.244.1.9:9090",
  "downstream_remote_address": "10.244.2.7:42196",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1"
}
{
  "start_time": "2026-08-25T02:32:08.455Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "01f901a0-01e7-978a-aecc-a81094ae5ed6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:53621",
  "downstream_local_address": "10.244.1.9:9090",
  "downstream_remote_address": "10.244.2.7:42196",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2026-08-25T02:32:08.493Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "01f901a0-01e7-978a-aecc-a81094ae5ed6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.1.9:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:53621",
  "downstream_local_address": "10.244.1.9:9090",
  "downstream_remote_address": "10.244.2.7:42196",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}

[Text 41] Upstream Request Retry Case / mock-server Pod Access Log

[Text 40] shows the Access Log of the shell Pod’s istio-proxy. upstream_request_attempt_count is recorded as 3, which includes the first attempt and both retries, and response_flags is recorded as URX (UpstreamRetryLimitExceeded), indicating that the retry limit was exhausted.

In [Text 41], three logs with the same request_id are recorded. Since retries are performed by the istio-proxy of the shell Pod that serves as the client, the mock-server Pod’s istio-proxy handles each retry as a separate request, so upstream_request_attempt_count is recorded as 1 in every log.

1.3.6. Upstream TCP RST before Response Case

[Figure 23] Upstream TCP RST before Response Case

1
2
3
4
5
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 1000}' mock-server:9090 mock.MockService/ResetBeforeResponse
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: connection termination
command terminated with exit code 78

[Shell 24] Upstream TCP RST before Response Case / grpcurl Command

[Figure 23] shows the Upstream TCP RST before Response Case where the grpcurl command is used from the shell Pod to send a milliseconds: 1000 request to the mock-server’s /mock.MockService/ResetBeforeResponse function, and after waiting 1000ms, a TCP RST Flag is sent to forcefully terminate the Connection. [Shell 24] shows an example of executing [Figure 23].

The mock-server Pod’s istio-proxy that receives the TCP RST Flag returns an Unavailable status code to notify the shell Pod’s istio-proxy that the request was abnormally terminated. Due to the unavailable setting in the Virtual Service in [File 1], the shell Pod’s istio-proxy performs 2 retries for a total of 3 requests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-10T15:25:19.861Z",
  "method": "POST",
  "path": "/mock.MockService/ResetBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "URX",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "3066",
  "upstream_service_time": "3064",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b195727a-1004-9084-931d-ba8c01f7b1e3",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:55012",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:59464",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "1",
  "response_duration": "3065"
}

[Text 42] Upstream TCP RST before Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-01-10T15:25:19.863Z",
  "method": "POST",
  "path": "/mock.MockService/ResetBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "1005",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b195727a-1004-9084-931d-ba8c01f7b1e3",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:35179",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:55012",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-10T15:25:20.888Z",
  "method": "POST",
  "path": "/mock.MockService/ResetBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "1002",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b195727a-1004-9084-931d-ba8c01f7b1e3",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:53123",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:55012",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-10T15:25:21.922Z",
  "method": "POST",
  "path": "/mock.MockService/ResetBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "1003",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b195727a-1004-9084-931d-ba8c01f7b1e3",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:48095",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:55012",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 43] Upstream TCP RST before Response Case / mock-server Pod Access Log

[Text 42] shows the Access Log of the shell Pod’s istio-proxy, and [Text 43] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/ResetBeforeResponse function with response_code as 200 and grpc_status as Unavailable. Also, both Access Logs show response_flags as UC (UpstreamConnectionTermination).

Since the shell Pod’s istio-proxy sends 3 requests, upstream_request_attempt_count shows 3 in the shell Pod’s istio-proxy Access Log. Also, 3 Access Logs are left in the mock-server Pod’s istio-proxy.

1.3.7. Upstream TCP RST after Response Case

[Figure 24] Upstream TCP RST after Response Case

1
2
3
4
5
6
7
8
9
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 1000}' mock-server:9090 mock.MockService/ResetAfterResponse
kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 1000}' mock-server:9090 mock.MockService/ResetAfterResponse
{
  "data": "ZHVtbXkgZGF0YQ=="
}
ERROR:
  Code: Internal
  Message: stream terminated by RST_STREAM with error code: NO_ERROR
command terminated with exit code 77

[Shell 25] Upstream TCP RST after Response Case / grpcurl Command

[Figure 24] shows the Upstream TCP RST after Response Case where the grpcurl command is used from the shell Pod to send a milliseconds: 1000 request to the mock-server’s /mock.MockService/ResetAfterResponse function, and after 1000ms, the mock-server Pod sends a partial response and then sends a TCP RST Flag to forcefully terminate the Connection. [Shell 25] shows an example of executing [Figure 24].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T06:37:13.914Z",
  "method": "POST",
  "path": "/mock.MockService/ResetAfterResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UR",
  "response_code_details": "upstream_reset_after_response_started{remote_reset}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "17",
  "duration": "1216",
  "upstream_service_time": "1092",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "4d986174-88b0-97b1-88fb-46f493a880e5",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:35336",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:36028",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unknown",
  "upstream_request_attempt_count": "1",
  "request_duration": "9",
  "response_duration": "1101"
}

[Text 44] Upstream TCP RST after Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T06:37:13.949Z",
  "method": "POST",
  "path": "/mock.MockService/ResetAfterResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_after_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "17",
  "duration": "1167",
  "upstream_service_time": "1010",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "4d986174-88b0-97b1-88fb-46f493a880e5",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:42975",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:35336",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unknown",
  "upstream_request_attempt_count": "1",
  "request_duration": "55",
  "response_duration": "1065"
}

[Text 45] Upstream TCP RST after Response Case / mock-server Pod Access Log

[Text 44] shows the Access Log of the shell Pod’s istio-proxy, and [Text 45] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/ResetAfterResponse function with response_code as 200 and grpc_status as Unknown. The shell Pod’s istio-proxy Access Log shows response_flags as UR (UpstreamRemoteReset), and the mock-server Pod’s istio-proxy Access Log shows response_flags as UC (UpstreamConnectionTermination).

1.3.8. Upstream TCP Close before Response Case

[Figure 25] Upstream TCP Close before Response Case

1
2
3
4
5
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 1000}' mock-server:9090 mock.MockService/CloseBeforeResponse
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: connection termination
command terminated with exit code 78

[Shell 26] Upstream TCP Close before Response Case / grpcurl Command

[Figure 25] shows the Upstream TCP Close before Response Case where the grpcurl command is used from the shell Pod to send a milliseconds: 1000 request to the mock-server’s /mock.MockService/CloseBeforeResponse function, and after 1000ms, the mock-server Pod forcefully terminates the Connection. [Shell 26] shows an example of executing [Figure 25].

Except for the fact that the mock-server Container sends TCP FIN Flag, the same process as [Figure 23] which receives TCP RST Flag is performed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T11:55:39.852Z",
  "method": "POST",
  "path": "/mock.MockService/CloseBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "URX",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "3086",
  "upstream_service_time": "3071",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "196a9ef4-4383-9d02-9dbb-9752f082fba6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:45386",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:38082",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "1",
  "response_duration": "3073"
}

[Text 46] Upstream TCP Close before Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-01-11T11:55:39.855Z",
  "method": "POST",
  "path": "/mock.MockService/CloseBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "1006",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "196a9ef4-4383-9d02-9dbb-9752f082fba6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:60797",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:45386",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-11T11:55:40.874Z",
  "method": "POST",
  "path": "/mock.MockService/CloseBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "1003",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "196a9ef4-4383-9d02-9dbb-9752f082fba6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:45085",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:45386",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-11T11:55:41.921Z",
  "method": "POST",
  "path": "/mock.MockService/CloseBeforeResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "0",
  "duration": "1003",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "196a9ef4-4383-9d02-9dbb-9752f082fba6",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:58531",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:45386",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "-"
}

[Text 47] Upstream TCP Close before Response Case / mock-server Pod Access Log

[Text 46] shows the Access Log of the shell Pod’s istio-proxy, and [Text 47] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/CloseBeforeResponse function with response_code as 200 and grpc_status as Unavailable. The same process as [Figure 23] which receives TCP RST Flag is performed.

1.3.9. Upstream TCP Close after Response Case

[Figure 26] Upstream TCP Close after Response Case

1
2
3
4
5
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 1000}' mock-server:9090 mock.MockService/CloseAfterResponse
ERROR:
  Code: Internal
  Message: stream terminated by RST_STREAM with error code: NO_ERROR
command terminated with exit code 77

[Shell 27] Upstream TCP Close after Response Case / grpcurl Command

[Figure 26] shows the Upstream TCP Close after Response Case where the grpcurl command is used from the shell Pod to send a milliseconds: 1000 request to the mock-server’s /mock.MockService/CloseAfterResponse function, and after 1000ms, the mock-server Pod sends a partial response and then forcefully terminates the Connection. [Shell 27] shows an example of executing [Figure 26].

Except for the fact that the mock-server Container sends TCP FIN Flag, the same process as [Figure 24] which receives TCP RST Flag is performed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T12:12:56.184Z",
  "method": "POST",
  "path": "/mock.MockService/CloseAfterResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UR",
  "response_code_details": "upstream_reset_after_response_started{remote_reset}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "17",
  "duration": "1221",
  "upstream_service_time": "1091",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "5352fc42-1f26-953b-8934-54be55aac934",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:55444",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:45082",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unknown",
  "upstream_request_attempt_count": "1",
  "request_duration": "15",
  "response_duration": "1106"
}

[Text 48] Upstream TCP Close after Response Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-11T12:12:56.215Z",
  "method": "POST",
  "path": "/mock.MockService/CloseAfterResponse",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UC",
  "response_code_details": "upstream_reset_after_response_started{connection_termination}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "17",
  "duration": "1175",
  "upstream_service_time": "1013",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "5352fc42-1f26-953b-8934-54be55aac934",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:47473",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:55444",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Unknown",
  "upstream_request_attempt_count": "1",
  "request_duration": "61",
  "response_duration": "1074"
}

[Text 49] Upstream TCP Close before Response Case / mock-server Pod Access Log

[Text 48] shows the Access Log of the shell Pod’s istio-proxy, and [Text 49] shows the Access Log of the mock-server Pod’s istio-proxy. Both Access Logs show the access to the /mock.MockService/CloseAfterResponse function with response_code as 200 and grpc_status as Unknown. The same process as [Figure 24] which receives TCP RST Flag is performed.

1.3.10. Circuit Breaking with Upstream Connection Pool Overflow Case

[Figure 27] Circuit Breaking with Upstream Connection Pool Overflow Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay &
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay &
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay &
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: overflow
command terminated with exit code 78
{
  "service": "mock-server",
  "delayedMs": 5000,
  "message": "Response delayed by 5000ms"
}
{
  "service": "mock-server",
  "delayedMs": 5000,
  "message": "Response delayed by 5000ms"
}

[Shell 28] Circuit Breaking with Upstream Connection Pool Overflow Case / grpcurl Command

[Figure 27] shows the Case where grpcurl command is used from the shell Pod to send 3 consecutive milliseconds: 5000 requests to the mock-server’s /mock.MockService/Delay function, causing Upstream Connection Pool Overflow. [Shell 28] shows an example of executing [Figure 27]. Except for the fact that requests and responses occur via GRPC, the same process as described in [Figure 12] is performed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-01-11T16:00:42.913Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UO",
  "response_code_details": "upstream_reset_before_response_started{overflow}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "b3f85437-be22-9083-8193-972ac2f9d0bb",
  "authority": "mock-server:9090",
  "upstream_host": "-",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:52054",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}
{
  "start_time": "2026-01-11T16:00:42.393Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "49",
  "duration": "5018",
  "upstream_service_time": "5015",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "ab76aaaf-ab05-900c-8c45-162dff809b27",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:53896",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:52044",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5017"
}
{
  "start_time": "2026-01-11T16:00:42.636Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "49",
  "duration": "9784",
  "upstream_service_time": "9783",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "66defa5b-ae78-9fe0-8529-af5dbec52725",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:53896",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:52052",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "9783"
}

[Text 50] Circuit Breaking with Upstream Connection Pool Overflow Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
{
  "start_time": "2026-01-11T16:00:42.395Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "49",
  "duration": "5014",
  "upstream_service_time": "5012",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "ab76aaaf-ab05-900c-8c45-162dff809b27",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:49515",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:53896",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "1",
  "response_duration": "5013"
}
{
  "start_time": "2026-01-11T16:00:47.412Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "49",
  "duration": "5006",
  "upstream_service_time": "5004",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "66defa5b-ae78-9fe0-8529-af5dbec52725",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:49515",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:53896",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "5005"
}

[Text 51] Circuit Breaking with Upstream Connection Pool Overflow Case / mock-server Pod Access Log

[Text 50] shows the Access Log of the shell Pod’s istio-proxy, and [Text 51] shows the Access Log of the mock-server Pod’s istio-proxy. Except for the fact that requests and responses occur via GRPC, the same process as [Text 22] and [Text 23] is performed.

1.3.11. Circuit Breaking with Upstream Request Limit Overflow Case

[Figure 28] Circuit Breaking with Upstream Request Limit Overflow Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay &
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay &
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: overflow
command terminated with exit code 78
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 5000}' mock-server:9090 mock.MockService/Delay &
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: overflow
command terminated with exit code 78

[Shell 29] Circuit Breaking with Upstream Request Limit Overflow Case / grpcurl Command

[Figure 28] shows the Case where grpcurl command is used from the shell Pod to send 3 consecutive milliseconds: 5000 requests to the mock-server’s /mock.MockService/Delay function, causing Upstream Request Limit Overflow. To reproduce this Case, the Destination Rule configured in [File 2] must be applied. [Shell 29] shows an example of executing [Figure 28]. Except for the fact that requests and responses occur via GRPC, the same process as described in [Figure 13] is performed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2025-12-25T14:45:01.595Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UO",
  "response_code_details": "upstream_reset_before_response_started{overflow}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "82f56865-fa1b-9fbe-8367-205fcae1dafd",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.8:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.186.69:9090",
  "downstream_remote_address": "10.244.1.5:42196",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}
{
  "start_time": "2025-12-25T14:45:01.927Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UO",
  "response_code_details": "upstream_reset_before_response_started{overflow}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "0277cd35-3680-914d-9b1b-e402076d3838",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.8:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.186.69:9090",
  "downstream_remote_address": "10.244.1.5:42212",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}
{
  "start_time": "2025-12-25T14:45:01.235Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "49",
  "duration": "5032",
  "upstream_service_time": "5027",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "69957e0b-65a5-97fa-baba-dcc3943f6b67",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.8:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.5:46834",
  "downstream_local_address": "10.96.186.69:9090",
  "downstream_remote_address": "10.244.1.5:42190",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "3",
  "response_duration": "5030"
}

[Text 52] Circuit Breaking with Upstream Request Limit Overflow Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2025-12-25T14:45:01.241Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "8",
  "bytes_sent": "49",
  "duration": "5020",
  "upstream_service_time": "5013",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "69957e0b-65a5-97fa-baba-dcc3943f6b67",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.8:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:60721",
  "downstream_local_address": "10.244.2.8:9090",
  "downstream_remote_address": "10.244.1.5:46834",
  "requested_server_name": "outbound_.9090_._.mock-server.default.svc.cluster.local",
  "route_name": "default",
  "grpc_status": "OK",
  "upstream_request_attempt_count": "1",
  "request_duration": "4",
  "response_duration": "5018"
}

[Text 53] Circuit Breaking with Upstream Request Limit Overflow Case / mock-server Pod Access Log

[Text 52] shows the Access Log of the shell Pod’s istio-proxy, and [Text 53] shows the Access Log of the mock-server Pod’s istio-proxy. Except for the fact that requests and responses occur via GRPC, the same process as [Text 24] and [Text 25] is performed.

1.3.12. Circuit Breaking with No Healthy Upstream Case

[Figure 29] Circuit Breaking with No Healthy Upstream Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Internal
  Message: Simulated error with gRPC code 13 (Internal)
command terminated with exit code 77
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Internal
  Message: Simulated error with gRPC code 13 (Internal)
command terminated with exit code 77
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Internal
  Message: Simulated error with gRPC code 13 (Internal)
command terminated with exit code 77
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Internal
  Message: Simulated error with gRPC code 13 (Internal)
command terminated with exit code 77
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Internal
  Message: Simulated error with gRPC code 13 (Internal)
command terminated with exit code 77
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78
$ kubectl exec shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 13}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78

[Shell 30] Circuit Breaking with No Healthy Upstream Case / grpcurl Command

[Figure 29] shows the Case where grpcurl command is used from the shell Pod to send 8 consecutive code: 13 requests to the mock-server’s /mock.MockService/Status function, causing Circuit Breaking via No Healthy Upstream. [Shell 30] shows an example of executing [Figure 29].

According to the Destination Rule in [File 1], Circuit Breaking is activated when 5 consecutive 5XX Errors occur. Therefore, the first 5 requests from the shell Pod are all delivered to the mock-server Pod, but the subsequent 3 requests are not delivered to the mock-server Pod due to Circuit Breaking. Therefore, the response to the first 5 requests shows Internal, and the response to the subsequent 3 requests shows Unavailable.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
{
  "start_time": "2026-01-12T15:02:58.363Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "177",
  "upstream_service_time": "138",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "63a34f0f-2ba7-96fe-bf09-a9b557de9e7c",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:42296",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55756",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "38",
  "response_duration": "176"
}
{
  "start_time": "2026-01-12T15:02:59.115Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "1",
  "upstream_service_time": "1",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "dfa5d9a4-205f-9ed0-b8f9-a57503eef534",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:42296",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55772",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1"
}
{
  "start_time": "2026-01-12T15:03:00.019Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "2",
  "upstream_service_time": "1",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "3ccce983-77fe-901b-8cde-9015d2f04224",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:42310",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55782",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "2"
}
{
  "start_time": "2026-01-12T15:03:00.783Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "1",
  "upstream_service_time": "1",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "542a7709-ee30-91b3-8f94-5d98d38d7979",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:42296",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55784",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1"
}
{
  "start_time": "2026-01-12T15:03:01.586Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "1",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "fffa72a1-d8ec-97fa-adb7-8ac84397d5ef",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:42296",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55792",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1"
}
{
  "start_time": "2026-01-12T15:03:02.459Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "df1b083c-e98c-93dd-90b6-153fc4017e1e",
  "authority": "mock-server:9090",
  "upstream_host": "-",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55794",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}
{
  "start_time": "2026-01-12T15:03:03.131Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "6593b359-3b58-9dd2-bdb8-9b5024bd25fb",
  "authority": "mock-server:9090",
  "upstream_host": "-",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55804",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}
{
  "start_time": "2026-01-12T15:03:03.863Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "d73c7aa7-e2af-90fb-91db-c205335a979b",
  "authority": "mock-server:9090",
  "upstream_host": "-",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.211.131:9090",
  "downstream_remote_address": "10.244.1.8:55812",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}

[Text 54] Circuit Breaking Case / shell Pod Access Log

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
{
  "start_time": "2026-01-12T15:02:58.454Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "65",
  "upstream_service_time": "26",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "63a34f0f-2ba7-96fe-bf09-a9b557de9e7c",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:42649",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:42296",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "22",
  "response_duration": "49"
}
{
  "start_time": "2026-01-12T15:02:59.115Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "1",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "dfa5d9a4-205f-9ed0-b8f9-a57503eef534",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:42649",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:42296",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "1"
}
{
  "start_time": "2026-01-12T15:03:00.020Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "3ccce983-77fe-901b-8cde-9015d2f04224",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:45955",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:42310",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2026-01-12T15:03:00.783Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "542a7709-ee30-91b3-8f94-5d98d38d7979",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:42649",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:42296",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}
{
  "start_time": "2026-01-12T15:03:01.586Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "-",
  "response_code_details": "via_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "7",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "0",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "fffa72a1-d8ec-97fa-adb7-8ac84397d5ef",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.22:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:42649",
  "downstream_local_address": "10.244.2.22:9090",
  "downstream_remote_address": "10.244.1.8:42296",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "Internal",
  "upstream_request_attempt_count": "1",
  "request_duration": "0",
  "response_duration": "0"
}

[Text 55] Circuit Breaking Case / mock-server Pod Access Log

[Text 54] shows the Access Log of the shell Pod’s istio-proxy, and [Text 55] shows the Access Log of the mock-server Pod’s istio-proxy. In the shell Pod’s istio-proxy Access Log, only the last 3 requests show response_flags as UH (NoHealthyUpstream) along with confirmation that requests were not delivered to the mock-server Pod. Also, only the first 5 request logs are present in the mock-server Pod’s istio-proxy Access Log.

1.3.13. Upstream Connection Failure with Timeout Case

[Figure 30] Upstream Connection Failure with Timeout Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
$ SHELL_IP=$(kubectl get pod shell -o jsonpath='{.status.podIP}')
$ kubectl exec mock-server -c mock-server -- iptables -A INPUT -s ${SHELL_IP} -j DROP
# $ kubectl exec mock-server -c mock-server -- iptables -D INPUT 1 remove rule after case execution

$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection timeout
command terminated with exit code 78
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78

[Shell 31] Upstream Connection Failure with Timeout Case / iptables Command

[Figure 30] shows the Upstream Connection Failure with Timeout Case where grpcurl command is used from the shell Pod to send 3 consecutive code: 0 requests to the mock-server’s /mock.MockService/Status function, causing Retry due to Timeout. [Shell 31] shows an example of executing [Figure 30]. Except for the fact that requests and responses occur via GRPC, the same process as described in [Figure 15] is performed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-01-12T16:03:05.367Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "URX,UF",
  "response_code_details": "upstream_reset_before_response_started{connection_timeout}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "5",
  "bytes_sent": "0",
  "duration": "30022",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "d0bfefd5-7d33-91e5-99b9-20be930cb454",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.23:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:38130",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "1",
  "response_duration": "-"
}
{
  "start_time": "2026-01-12T16:03:36.696Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "5",
  "bytes_sent": "0",
  "duration": "20027",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "298c1ca4-1d65-96fe-b954-e9940ad05219",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.23:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:33064",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-12T16:03:57.696Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "024b3626-a1a7-9a48-a0ce-7b51fb0cb045",
  "authority": "mock-server:9090",
  "upstream_host": "-",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:56904",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}

[Text 56] Upstream Connection Failure with Timeout Case / shell Pod Access Log

[Text 56] shows the Access Log of the shell Pod’s istio-proxy. Except for the fact that requests and responses occur via GRPC, the same process as [Text 28] is performed.

1.3.14. Upstream Connection Failure with TCP Reset Case

[Figure 31] Upstream Connection Failure with TCP Reset Case

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
$ SHELL_IP=$(kubectl get pod shell -o jsonpath='{.status.podIP}')
$ kubectl exec mock-server -c mock-server -- iptables-legacy -A INPUT -p tcp -s ${SHELL_IP} -j REJECT --reject-with tcp-reset
# $ kubectl exec mock-server -c mock-server -- iptables-legacy -D INPUT 1 remove rule after case execution

$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
command terminated with exit code 78
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"code": 0}' mock-server:9090 mock.MockService/Status
ERROR:
  Code: Unavailable
  Message: no healthy upstream
command terminated with exit code 78

[Shell 32] Upstream Connection Failure with TCP Reset Case / iptables Command

[Figure 31] shows the Upstream Connection Failure with TCP Reset Case where grpcurl command is used from the shell Pod to send 3 consecutive code: 0 requests to the mock-server’s /mock.MockService/Status function, causing Retry due to TCP Reset. [Shell 32] shows an example of executing [Figure 31]. Except for the fact that requests and responses occur via GRPC, the same process as described in [Figure 16] is performed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
{
  "start_time": "2026-01-12T15:38:06.556Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "URX,UF",
  "response_code_details": "upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "delayed_connect_error:_Connection_refused",
  "bytes_received": "5",
  "bytes_sent": "0",
  "duration": "47",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "33dbdad0-498c-946a-9cca-4011993ab77d",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.23:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:35730",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-12T15:38:07.473Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "delayed_connect_error:_Connection_refused",
  "bytes_received": "5",
  "bytes_sent": "0",
  "duration": "37",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "febde591-cafd-9ad9-b567-85f31bdee0ce",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.23:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:35734",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "3",
  "request_duration": "0",
  "response_duration": "-"
}
{
  "start_time": "2026-01-12T15:38:08.939Z",
  "method": "POST",
  "path": "/mock.MockService/Status",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UH",
  "response_code_details": "no_healthy_upstream",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "3965c57d-d452-9572-b1f2-22f39c36c446",
  "authority": "mock-server:9090",
  "upstream_host": "-",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "-",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:35738",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "-",
  "response_duration": "-"
}

[Text 57] Upstream Connection Failure with TCP Reset Case / shell Pod Access Log

[Text 57] shows the Access Log of the shell Pod’s istio-proxy. Except for the fact that requests and responses occur via GRPC, the same process as [Text 29] is performed.

1.3.15. Upstream Request Timeout Case

[Figure 32] Upstream Request Timeout Case

1
$ kubectl exec -it shell -- grpcurl -plaintext -proto mock.proto -d '{"milliseconds": 70000}' mock-server:9090 mock.MockService/Delay

[Shell 33] Upstream Request Timeout Case / grpcurl Command

[Figure 32] shows the Upstream Request Timeout Case where the grpcurl command is used from the shell Pod to send a milliseconds: 70000 request to the mock-server’s /mock.MockService/Delay function, but the mock-server Pod’s istio-proxy does not receive a response after waiting 60000ms and processes the Request as Timeout.

Due to the Virtual Service in [File 1], requests sent to the mock-server Pod can wait up to 60000ms. However, the request sent to the mock-server Pod’s /mock.MockService/Delay function with milliseconds: 70000 requires 70000ms, so a Timeout occurs. The mock-server Pod’s istio-proxy sends an HTTP/2 RST_STREAM Frame to terminate the connection with the mock-server Pod when a Timeout occurs. It also returns an Unavailable status code to notify the shell Pod’s istio-proxy that the request was abnormally terminated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-12T16:10:41.589Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "200",
  "response_flags": "UT",
  "response_code_details": "response_timeout",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "9",
  "bytes_sent": "0",
  "duration": "60015",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "1dfff112-112f-90e4-a6fe-c247011b9d94",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.23:9090",
  "upstream_cluster": "outbound|9090||mock-server.default.svc.cluster.local",
  "upstream_local_address": "10.244.1.8:33502",
  "downstream_local_address": "10.96.1.12:9090",
  "downstream_remote_address": "10.244.1.8:48540",
  "requested_server_name": "-",
  "route_name": "-",
  "grpc_status": "Unavailable",
  "upstream_request_attempt_count": "1",
  "request_duration": "16",
  "response_duration": "-"
}

[Text 58] Upstream Request Timeout Case / shell Pod Access Log

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "start_time": "2026-01-12T16:10:41.644Z",
  "method": "POST",
  "path": "/mock.MockService/Delay",
  "protocol": "HTTP/2",
  "response_code": "0",
  "response_flags": "DR",
  "response_code_details": "http2.remote_reset",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "9",
  "bytes_sent": "0",
  "duration": "59975",
  "upstream_service_time": "-",
  "x_forwarded_for": "-",
  "user_agent": "grpcurl/v1.9.3 grpc-go/1.61.0",
  "request_id": "1dfff112-112f-90e4-a6fe-c247011b9d94",
  "authority": "mock-server:9090",
  "upstream_host": "10.244.2.23:9090",
  "upstream_cluster": "inbound|9090||",
  "upstream_local_address": "127.0.0.6:35027",
  "downstream_local_address": "10.244.2.23:9090",
  "downstream_remote_address": "10.244.1.8:33502",
  "requested_server_name": "-",
  "route_name": "default",
  "grpc_status": "-",
  "upstream_request_attempt_count": "1",
  "request_duration": "19",
  "response_duration": "-"
}

[Text 59] Upstream Request Timeout Case / mock-server Pod Access Log

[Text 58] shows the Access Log of the shell Pod’s istio-proxy, and [Text 59] shows the Access Log of the mock-server Pod’s istio-proxy. The shell Pod’s istio-proxy shows response_flags as UT (UpstreamTimeout). The mock-server Pod’s istio-proxy shows response_flags as DR (DownstreamRemoteReset).

2. References

Istio Access Log : https://istio.io/latest/docs/tasks/observability/logs/access-log/
Enovy Access Log : https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#default-format-string

Istio Sidecar Proxy Jaeger Architecture