TLDR Shashank is having trouble enabling mtls with Istio in their openobserve deployment. Hengfei suggested the issue could be with how Istio proxies etcd communication. Prabhat confirmed ongoing istio-related problems and West detailed issues in strict ISTIO mode.
Yep, i am not sure how to resolve it. it looks istio proxied etcd communication.
etcd use port: 2379
We have a known problem around istio. Need more time to study and find a resolution.
More information on this issue : Quarrier and Ingester Services are failing in ISTIO strict mode , rest of the services are fine with mTLS
`curl -u
2023-11-22T15:59:52+05:30 2023-11-22T10:29:52.153248Z ERROR openobserve::service::db::file_list::broadcast: [broadcast] send event to node[4cab14ff-283b-43ba-afea-5b636f0bfd20] failed: status: Unavailable, message: "upstream connect error or disconnect/reset before headers. reset reason: connection termination", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "server": "envoy", "date": "Wed, 22 Nov 2023 10:29:51 GMT"} }, retrying...
Shashank
Tue, 21 Nov 2023 06:09:45 UTCHi Team, I have deployed openobserve in HA mode. I have security use case where I need to enable mtls(using istio). When I do so calls like: ```
```
starts failing with status code 503.
Now as per this istio doc: . I have added grpc port in following services: (compactor, ingester, querier, router), but no luck with that !!
```ports:
- name: grpc
port: 5081
protocol: TCP
targetPort: grpc```
I am continuously getting below error:
```[2023-11-20T22:20:13Z DEBUG tonic::codec::decode] decoder inner stream error: Status { code: Internal, message: "h2 protocol error: error reading a body from connection: stream error received: not a result of an error", source: Some(hyper::Error(Body, Error { kind: Reset(StreamId(13), NO_ERROR, Remote) })) }
[2023-11-20T22:20:13Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/nodes/, get message error: grpc request error: status: Internal, message: "h2 protocol error: error reading a body from connection: stream error received: not a result of an error", details: [], metadata: MetadataMap { headers: {} }
[2023-11-20T22:20:13Z DEBUG tower::balance::p2c::service] updating from discover
[2023-11-20T22:20:13Z DEBUG tower::buffer::worker] service.ready=true message=processing request
[2023-11-20T22:20:13Z DEBUG hyper::proto::h2::client] client request body error: error writing a body to connection: send stream capacity unexpectedly closed
[2023-11-20T22:20:13Z DEBUG hyper::proto::h2::client] client request body error: error writing a body to connection: send stream capacity unexpectedly closed
[2023-11-20T22:20:13Z DEBUG tonic::codec::decode] decoder inner stream error: Status { code: Internal, message: "h2 protocol error: error reading a body from connection: stream error received: not a result of an error", source: Some(hyper::Error(Body, Error { kind: Reset(StreamId(23), NO_ERROR, Remote) })) }
[2023-11-20T22:20:13Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/user/, get message error: grpc request error: status: Internal, message: "h2 protocol error: error reading a body from connection: stream error received: not a result of an error", details: [], metadata: MetadataMap { headers: {} }```
Can someone pls helm me here ?