Connectivity Testing Issues and Solutions with OpenObserve Router
TLDR Dylan experienced connectivity issues while testing OpenObserve router. Suggested by Prabhat regarding etcd issues and possibility of Istio service mesh causing connectivity problems. Reinstallation and increasing memory were also suggested but issue remained unresolved.
Prabhat
Thu, 16 Nov 2023 19:17:12 UTCWhat version of OpenObserve are you using?
Dylan
Thu, 16 Nov 2023 19:17:55 UTC`0.7.0` ```- name: openobserve-{{ .Environment.Name }} chart: openobserve/openobserve version: 0.7.0 labels: service-name: openobserve is-elk: true namespace: {{ .Environment.Name }} values: - ./openobserve/values/{{ .Environment.Name }}.yaml```
Prabhat
Thu, 16 Nov 2023 19:18:35 UTCbrb
Prabhat
Thu, 16 Nov 2023 19:28:17 UTC```[2023-11-16T18:45:07Z ERROR openobserve::service::router]
Prabhat
Thu, 16 Nov 2023 19:28:27 UTCThis basically indicates issue with etcd
Prabhat
Thu, 16 Nov 2023 19:29:10 UTCAre you already ingesting lot of data ?
Dylan
Thu, 16 Nov 2023 19:29:58 UTC> Are you already ingesting lot of data ?
Not yet, just sending test curls atm. Once the logstash output is sent to the router there will be a good amount of data ingested
gotcha, I do have etcd-2 failing, a team member suggested looking into
Prabhat
Thu, 16 Nov 2023 19:30:29 UTCyeah, etcd is turning out to be a problem
Prabhat
Thu, 16 Nov 2023 19:30:37 UTCmaintaining it is hard
Prabhat
Thu, 16 Nov 2023 19:30:46 UTCWe will look at replacing it in long term.
Dylan
Thu, 16 Nov 2023 19:31:13 UTChere's the error from etcd-2
``` k --context=development -n openobserve logs openobserve-development-etcd-2
etcd 19:30:23.36
etcd 19:30:23.36 Welcome to the Bitnami etcd container
etcd 19:30:23.36 Subscribe to project updates by watching
Prabhat
Thu, 16 Nov 2023 19:58:33 UTCtry deleting the etcd pod along with its pvc
Prabhat
Thu, 16 Nov 2023 19:58:36 UTCthat should help
Prabhat
Thu, 16 Nov 2023 19:58:42 UTCits not able to join the cluster
Prabhat
Thu, 16 Nov 2023 19:58:54 UTCyou have only 1 etcd pod up and running right now. right?
Prabhat
Thu, 16 Nov 2023 20:00:10 UTCso what seems to have happened is that etcd info of the crashing pod is out of sync with the cluster for whatever be the reason
Prabhat
Thu, 16 Nov 2023 20:00:30 UTCrecreating the pod along with the associated pvc should rectify that
Dylan
Thu, 16 Nov 2023 20:01:17 UTCThe ns looks like this, only 1/2 etcd-2 seems to be failing. I'll try deleting the etcd pod and pvc like you suggested ```NAME READY STATUS RESTARTS AGE devops-shell-20231116172852 1/1 Running 0 150m openobserve-development-alertmanager-5d6679958d-dq6hk 2/2 Running 3 (46h ago) 46h openobserve-development-compactor-7cb5cf5bb-txs9v 2/2 Running 2 (46h ago) 46h openobserve-development-etcd-0 2/2 Running 0 47h openobserve-development-etcd-1 2/2 Running 0 47h openobserve-development-etcd-2 1/2 CrashLoopBackOff 552 (3m35s ago) 46h openobserve-development-ingester-0 2/2 Running 2 (46h ago) 46h openobserve-development-querier-6b446774bb-vg4cj 2/2 Running 2 (46h ago) 46h openobserve-development-router-8669cfc86d-wr45p 2/2 Running 15 (73m ago) 46h```
Prabhat
Thu, 16 Nov 2023 20:01:47 UTCso much better. only 1 pod failing
Prabhat
Thu, 16 Nov 2023 20:02:08 UTCdelete the etcd-2 pod along with the pvc
Prabhat
Thu, 16 Nov 2023 20:02:50 UTC`552 restarts` :eyes:
Dylan
Thu, 16 Nov 2023 20:02:59 UTC:laughing:
Prabhat
Thu, 16 Nov 2023 20:04:20 UTCLet's solve the etcd problem first and then we can look at the connectvity problem
Dylan
Thu, 16 Nov 2023 20:12:07 UTCalright pd and pvc were terminated and the new one's are booting up
Dylan
Thu, 16 Nov 2023 20:15:36 UTCquite interesting, now etcd-0 has that same error. etcd-1 and etcd-0 terminated and restarted automatically
``` k --context=development -n openobserve logs openobserve-development-etcd-0
etcd 20:14:53.64
etcd 20:14:53.64 Welcome to the Bitnami etcd container
etcd 20:14:53.64 Subscribe to project updates by watching
Dylan
Thu, 16 Nov 2023 20:16:05 UTC:sigh: :laughing:
Dylan
Thu, 16 Nov 2023 20:16:48 UTC``` k --context=development get pods -n openobserve -w 1 ↵ droberts@RM-NB-DROB2 NAME READY STATUS RESTARTS AGE devops-shell-20231116172852 1/1 Running 0 167m openobserve-development-alertmanager-5d6679958d-dq6hk 2/2 Running 3 (47h ago) 47h openobserve-development-compactor-7cb5cf5bb-txs9v 2/2 Running 2 (47h ago) 47h openobserve-development-etcd-0 1/2 CrashLoopBackOff 4 (58s ago) 2m49s openobserve-development-etcd-1 2/2 Running 1 (3m57s ago) 4m13s openobserve-development-etcd-2 2/2 Running 0 5m43s openobserve-development-ingester-0 2/2 Running 2 (47h ago) 47h openobserve-development-querier-6b446774bb-vg4cj 2/2 Running 2 (47h ago) 47h openobserve-development-router-8669cfc86d-wr45p 2/2 Running 15 (90m ago) 47h```
Prabhat
Thu, 16 Nov 2023 20:16:49 UTCYou did not delete etcd-0 though. It happened on its own after you tried fixing etcd-2. is that right?
Dylan
Thu, 16 Nov 2023 20:16:55 UTCyep
Prabhat
Thu, 16 Nov 2023 20:18:05 UTCTry doing the same with etcd-0
Dylan
Thu, 16 Nov 2023 20:18:11 UTC:+1:
Prabhat
Thu, 16 Nov 2023 20:18:12 UTCdelete pvc and pod
Dylan
Thu, 16 Nov 2023 20:20:19 UTCalright we're in the rebooting process
Dylan
Thu, 16 Nov 2023 20:21:30 UTClooking good atm ```AME READY STATUS RESTARTS AGE devops-shell-20231116172852 1/1 Running 0 172m openobserve-development-alertmanager-5d6679958d-dq6hk 2/2 Running 3 (47h ago) 47h openobserve-development-compactor-7cb5cf5bb-txs9v 2/2 Running 2 (47h ago) 47h openobserve-development-etcd-0 2/2 Running 0 95s openobserve-development-etcd-1 2/2 Running 1 (8m32s ago) 8m48s openobserve-development-etcd-2 2/2 Running 0 10m openobserve-development-ingester-0 2/2 Running 2 (47h ago) 47h openobserve-development-querier-6b446774bb-vg4cj 2/2 Running 2 (47h ago) 47h openobserve-development-router-8669cfc86d-wr45p 2/2 Running 15 (95m ago) 47h```
Prabhat
Thu, 16 Nov 2023 20:22:33 UTCNow let's run `curl`
Dylan
Thu, 16 Nov 2023 20:23:16 UTCyep looking like the same output, OOMKILLED status for the router
Dylan
Thu, 16 Nov 2023 20:23:21 UTClemme check the logs
Prabhat
Thu, 16 Nov 2023 20:23:35 UTCthen give it more memory
Prabhat
Thu, 16 Nov 2023 20:24:18 UTCthat should fix it
Prabhat
Thu, 16 Nov 2023 20:24:33 UTCwhat is the current limit ?
Prabhat
Thu, 16 Nov 2023 20:24:50 UTCfor router
Dylan
Thu, 16 Nov 2023 20:26:50 UTCpretty low amount req ``` Limits: cpu: 2 memory: 1Gi Requests: cpu: 10m memory: 40Mi```
Prabhat
Thu, 16 Nov 2023 20:27:38 UTCthat is generous enough limit
Prabhat
Thu, 16 Nov 2023 20:28:21 UTCIs it still getting OOM killed?
Dylan
Thu, 16 Nov 2023 20:29:29 UTCno, happily running, and like previously
```Works
curl -X POST "
Prabhat
Thu, 16 Nov 2023 20:32:01 UTC`helm -n openobserve ls` what is the output of this ?
Prabhat
Thu, 16 Nov 2023 20:32:23 UTCand `kubectl -n openobserve get svc`
Dylan
Thu, 16 Nov 2023 20:32:55 UTClocally it is ```AME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION openobserve-development openobserve 2 2023-11-14 16:15:51.923371 -0500 EST failed openobserve-0.7.0 v0.7.0 kubectl -n openobserve get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE openobserve-development-alertmanager ClusterIP 172.20.53.221 <none> 5080/TCP 47h openobserve-development-compactor ClusterIP 172.20.16.64 <none> 5080/TCP 47h openobserve-development-etcd ClusterIP 172.20.110.205 <none> 2379/TCP,2380/TCP 47h openobserve-development-etcd-headless ClusterIP None <none> 2379/TCP,2380/TCP 47h openobserve-development-ingester ClusterIP 172.20.196.128 <none> 5080/TCP 47h openobserve-development-querier ClusterIP 172.20.75.176 <none> 5080/TCP 47h openobserve-development-router ClusterIP 172.20.167.149 <none> 5080/TCP 47h```
Dylan
Thu, 16 Nov 2023 20:33:19 UTC`status=failed` :hmm:
Prabhat
Thu, 16 Nov 2023 20:34:02 UTC> `status=failed` :hmm: of what ?
Prabhat
Thu, 16 Nov 2023 20:34:08 UTCah got it
Prabhat
Thu, 16 Nov 2023 20:34:15 UTCI see it
Prabhat
Thu, 16 Nov 2023 20:34:23 UTCmaybe you wnat to reinstall from scratch
Prabhat
Thu, 16 Nov 2023 20:34:41 UTCsomething is wrong with your installation
Prabhat
Thu, 16 Nov 2023 20:34:53 UTCprobably will be a lot easier to just reinstall
Dylan
Thu, 16 Nov 2023 20:35:01 UTCgotcha, will do
Dylan
Thu, 16 Nov 2023 20:53:10 UTCI reinstalled and now seeing
```NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
openobserve-development openobserve 1 2023-11-16 15:43:56.23242 -0500 EST deployed openobserve-0.7.0 v0.7.0```
``` k --context=development get pods -n openobserve -w 1 ↵ droberts@RM-NB-DROB2
NAME READY STATUS RESTARTS AGE
devops-shell-20231116172852 1/1 Running 0 3h21m
openobserve-development-alertmanager-5d6679958d-xp2xs 2/2 Running 2 (6m32s ago) 6m40s
openobserve-development-compactor-7cb5cf5bb-l89mp 2/2 Running 2 (6m32s ago) 6m40s
openobserve-development-etcd-0 2/2 Running 0 6m40s
openobserve-development-etcd-1 2/2 Running 0 6m40s
openobserve-development-etcd-2 2/2 Running 0 6m40s
openobserve-development-ingester-0 2/2 Running 2 (6m28s ago) 6m40s
openobserve-development-querier-6b446774bb-x7g89 2/2 Running 2 (6m32s ago) 6m40s
openobserve-development-router-8669cfc86d-gtddg 2/2 Running 3 (3m25s ago) 6m40s```
but unfortunately I'm seeing the same behavior,
```#works
curl -X POST "
Dylan
Thu, 16 Nov 2023 20:53:23 UTC:confused:
Prabhat
Thu, 16 Nov 2023 20:54:07 UTC2 things
Prabhat
Thu, 16 Nov 2023 20:54:16 UTC1. Your installation is good now
Prabhat
Thu, 16 Nov 2023 20:54:27 UTC2. you seem to have 2 containers for each pod.
Prabhat
Thu, 16 Nov 2023 20:54:33 UTCare you using a service mesh?
Dylan
Thu, 16 Nov 2023 20:55:11 UTCyes istio
Prabhat
Thu, 16 Nov 2023 20:55:25 UTCistio might have something to do with connectivity
Dylan
Thu, 16 Nov 2023 20:55:33 UTChmm gotcha
Prabhat
Thu, 16 Nov 2023 20:55:59 UTCit intercepts everything
Prabhat
Thu, 16 Nov 2023 20:56:10 UTCI have not tested with istio or any other service mesh for now
Dylan
Thu, 16 Nov 2023 20:56:41 UTCgotcha, but why would the router die? It looks like it's receiving something
Prabhat
Thu, 16 Nov 2023 20:58:17 UTCrouter has trouble connecting to etcd
Prabhat
Thu, 16 Nov 2023 20:58:23 UTC`[2023-11-16T20:47:12Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/nodes/, get message error: grpc request error: status: Unknown, message: "h2 protocol error: error reading a body from connection: stream closed because of a broken pipe", details: [], metadata: MetadataMap { headers: {} }`
Prabhat
Thu, 16 Nov 2023 20:58:50 UTCCan you disable istio for this namespace to make sure that that is not what is causing the issue
Prabhat
Thu, 16 Nov 2023 20:59:02 UTCI have a very strong feeling that istio is the trouble here
Dylan
Thu, 16 Nov 2023 20:59:20 UTCGotcha, I'll need to check in w/my team on that will get back to you
Dylan
Thu, 16 Nov 2023 22:19:27 UTCI haven't heard back from my team but I'd just like to thank you Prabhat for all the help in the past few days, very cool project and I'm excited to use it as a potential replacement for our ELK stack!
Dylan
Thu, 16 Nov 2023 22:20:11 UTCand just as a note I'll be on vacation for the next 2 weeks, if you don't hear from me thats why :slightly_smiling_face:
Prabhat
Thu, 16 Nov 2023 22:25:10 UTCThank you Dylan and enjoy your vacation. Happy thanksgiving.
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Dylan
Thu, 16 Nov 2023 18:50:14 UTCHey guys! I'm doing some connectivity testing from our development namespace to our openobserve router which is in an openobserve ns, getting some strange behavior. Wonder if you guys have any ideas? I'm able to successfully curl test data from a pod in a seperate development ns to the router with `curl -X POST "" -H "Content-Type: application/json" -H "Authorization: Basic XXX" -d '[{"level":"info","job":"test","log":"test message for openobserve"}]'`
however when I try to use `openobserve-development-router.openobserve.svc.cluster.local`
I get
```curl -X POST "openobserve-development-router.openobserve.svc.cluster.local:5080/api/default/default/_json" -H "Authorization: Basic XXX" -H "Content-Type: application/json" -d '[{"level":"info","job":"test","log":"test message for openobserve"}]'
curl: (52) Empty reply from server```
the router starts logging a ton of grpc errors
```[2023-11-16T18:45:07Z ERROR openobserve::service::router] : Failed to connect to host: Internal error: connector has been disconnected
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.950588
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.943231
[2023-11-16T18:45:07Z ERROR openobserve::service::router] : Failed to connect to host: Internal error: connector has been disconnected
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.928565
[2023-11-16T18:45:07Z ERROR openobserve::service::router] : Failed to connect to host: Internal error: connector has been disconnected
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.915907
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 14.004568
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.919026
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 14.138667
[2023-11-16T18:45:07Z ERROR openobserve::service::router] : Failed to connect to host: Internal error: connector has been disconnected
[2023-11-16T18:45:07Z ERROR openobserve::service::router] : Failed to connect to host: Internal error: connector has been disconnected
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.976337
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.907410
[2023-11-16T18:45:07Z INFO actix_web::middleware::logger] 127.0.0.6 "POST /api/default/default/_json HTTP/1.1" 503 58 "-" "-" "curl/7.68.0" 13.945222
[2023-11-16T18:45:08Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/user/, error: grpc request error: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }
[2023-11-16T18:45:08Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/nodes/, error: grpc request error: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }
[2023-11-16T18:45:09Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/user/, error: grpc request error: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }
[2023-11-16T18:45:09Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/nodes/, error: grpc request error: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }
[2023-11-16T18:45:10Z ERROR openobserve::common::infra::db::etcd] watching prefix: /zinc/observe/user/, error: grpc request error: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }```
and the pod goes into a OOM status and restarts.
running nslookup openobserve-development-router.openobserve.svc.cluster.local gives
```Server: 172.20.0.10
Address: 172.20.0.10#53
Name: openobserve-development-router.openobserve.svc.cluster.local
Address: 172.20.167.149```
and using that address seems to work fine
``` curl -X POST "" -H "Content-Type: application/json" -H "Authorization: Basic XXX" -d '[{"level":"info","job":"test","log":"test message for openobserve"}]'
{"code":200,"status":[{"name":"default","successful":1,"failed":0}]}```
I'm also seeing a etcd-2 connectivity issue that is likely it's own issue: `openobserve-development-etcd-2 1/2 CrashLoopBackOff 531 (59s ago) 44h`
Essentially I want to point the logstash output to `openobserve-development-router.openobserve.svc.cluster.local` to start getting real data from our microservices, but I'm not confident if the curl is failing
```output: |-
if [fields][type] == "application-logs" {
elasticsearch {
hosts => [""]
user => ""
password => "${}"
index => "application-logs-%{+YYYY.MM.dd}"
ilm_enabled => false
manage_template => false
pool_max => 65536
}
http {
url => [""]
format => "json"
http_method => "post"
content_type => "application/json"
headers => ["Authorization", "Basic XXX"]
mapping => {
"@timestamp" => "%{[@timestamp]}"
"source" => "%{[source]}"
"tags" => "%{[tags]}"
"logdate" => "%{[logdate]}"
"level" => "%{[level]}"
"thread" => "%{[thread]}"
"class" => "%{[class]}"
"line" => "%{[line]}"
"msg" => "%{[msg]}"
"server" => "%{[fields][service]}"
"log_type" => "%{[fields][logType]}"
"host_name" => "%{[host][name]}"
}
}
}```