Troubleshooting Fluentd Authentication Errors in OpenObserve

Jonathan

Hello! I'm using `fluentd` with openobserve, using the fluent.conf file provided in the ingestion section of openobserve: ```<source> @type forward port 24224 bind 0.0.0.0 </source> <match **> @type http endpoint content_type json json_array true <auth> method basic username ********** password *************** </auth> </match>``` I'm running fluentd and openobserve as containers in a dockerfile alongside nginx as a reverse proxy docker-compose.yml: (only the relevant bits) ```proxy: image: build: context: nginx dockerfile: Dockerfile ports: - 8181:80 volumes: - nginx_logs:/var/log/nginx:z env_file: - ./.env.dev.local deploy: replicas: 1 update_config: parallelism: 2 delay: 10s restart_policy: condition: on-failure placement: constraints: [ node.labels.service == proxy ] links: - fluentd logging: driver: "fluentd" options: fluentd-address: 127.0.0.1:24224 fluentd-async-connect: "true" tag: nginx.access depends_on: - landing - api - db - rabbitmq - fluentd networks: - frontend - api openobserve: image: public.ecr.aws/zinclabs/openobserve:latest ports: - 5080:5080 volumes: - openobserve:/data:z environment: ZO_DATA_DIR: /data ZO_ROOT_USER_EMAIL: ZO_ROOT_USER_PASSWORD: ************** ZO_DATA_STREAM_DIR: /data networks: - api fluentd: image: build: context: logging/fluentd dockerfile: Dockerfile volumes: - ./logging/fluentd/conf:/fluentd/etc:z # - nginx_logs:/var/log/nginx:z ports: - 24224:24224 - 24224:24224/udp depends_on: - openobserve networks: - api``` I am getting 401 errors when trying to post logs to openobserve and I'm noticing the following in the `fluentd` logs: ```fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: gem 'fluent-plugin-out-http' version '1.3.4' fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: gem 'fluentd' version '1.3.2' fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: adding match pattern="**" type="http" fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: adding match pattern="nginx.*" type="stdout" fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: adding source type="forward" fluentd_1 | 2023-07-04 14:21:34 +0000 [warn]: parameter 'content_type' in <match **> fluentd_1 | @type http fluentd_1 | endpoint_url "" fluentd_1 | content_type json fluentd_1 | json_array true fluentd_1 | <auth> fluentd_1 | method basic fluentd_1 | username fluentd_1 | password Ihopenobs1171 fluentd_1 | </auth> fluentd_1 | </match> is not used. fluentd_1 | 2023-07-04 14:21:34 +0000 [warn]: parameter 'json_array' in <match **> fluentd_1 | @type http fluentd_1 | endpoint_url "" fluentd_1 | content_type json fluentd_1 | json_array true fluentd_1 | <auth> fluentd_1 | method basic fluentd_1 | username ************ fluentd_1 | password **************** fluentd_1 | </auth> fluentd_1 | </match> is not used. fluentd_1 | 2023-07-04 14:21:34 +0000 [warn]: section <auth> is not used in <match **> of http plugin fluentd_1 | 2023-07-04 14:21:34 +0000 [warn]: section <auth> is not used in <match **> of http plugin fluentd_1 | 2023-07-04 14:21:34 +0000 [warn]: section <auth> is not used in <match **> of http plugin fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: #0 starting fluentd worker pid=18 ppid=7 worker=0 fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0" fluentd_1 | 2023-07-04 14:21:34 +0000 [info]: #0 fluentd worker is now running worker=0 fluentd_1 | 2023-07-04 14:21:34 +0000 [warn]: #0 failed to POST (401 Unauthorized )``` Could someone suggest what the problem might be here? Thanks!

Jonathan

Specifically, the `section <auth> is not used in <match **> of http plugin` bit has caught my eye, but I cannot find any information on this anywhere else.

Jonathan

oh, and this is my dockerfile: ```FROM fluent/fluentd:latest RUN ["gem", "install", "fluent-plugin-out-http"]```

Prabhat

You seem to be using `endpoint_url""`

Prabhat

I do not see that in the spec at

Jonathan

ah, that's something I was trying

Prabhat

ohk

Jonathan

that's the ip of the host running all the docker containers

Jonathan

but I would like to use: `endpoint_url""`

Prabhat

btw, any specific reason to use fluentd and not fliutntbit?

Prabhat

I meant `endpoint_url` as opposed to `endpoint`

Jonathan

...the docs on fluentbit seemed more complicated

Jonathan

ah, when I run with `endpoint` I get a different error, hold on

Jonathan

```config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'endpoint_url' parameter is required```

Jonathan

I'm happy to switch from one service to another, but it wasn't as intuitive to use fluent-bit as it was to use fluentd, so I figured I could make fluentd work

Prabhat

ohk

Prabhat

```config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'endpoint_url' parameter is required``` totally does not match documentation

Jonathan

agreed

Prabhat

I will have to try this one. We don't really use fluentd much and neither recommend.

Jonathan

If you have an example of a docker-compose file with fluent-bit, I'm happy to try that. At this point I just need logs going into openobserve

Prabhat

No docker-compose, just kubernetes for now.

Prabhat

Jonathan

ok, I'll try fumbling around with that some more since fluentd isn't working

Jonathan

I really appreciate your time Prabhat. thanks for the help

Prabhat

Prabhat

looks like there is a docker compose example for fluentbit that you could use

Jonathan

I must have missed it. thanks!