#general

Docker Swarm Cluster Logs with OpenObserve

TLDR arun asked how to collect logs from Docker services; Prabhat suggested using Vector or Fluentbit. arun found a ready-made solution and successfully implemented Vector.

5mo

Join the chat

Jun 15, 2023 (6 months ago)

Photo of md5-0e0e2259002665b2bb3f1e87eae4b39e

arun

01:46 PM

Hi! I just setup openobserve on my docker swarm cluster, looking to migrate from ES stack. What component do i need to start getting logs from my docker services?

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

01:50 PM

You will need a log forwarder that can tail the log files created by docker and send them to OPenObserve

01:50

Prabhat

01:50 PM

You could use fluentbit or vector for this

01:51

Prabhat

01:51 PM

Check vector file config source - https://vector.dev/docs/reference/configuration/sources/file/

01:52

Prabhat

01:52 PM

or fluentbit tail input - https://docs.fluentbit.io/manual/pipeline/inputs/tail

01:52

Prabhat

01:52 PM

docker log files are generally present in /var/log/containers

arun

01:55 PM

Hi, thanks for the answers. I checked and there are no log files in that directory, im not sure how logstash does it but from a user perspective, the docker logs can be shown via the docker service logs <service name> command. I will check fluentbit and vector to see if they offer anything

01:56

arun

01:56 PM

im guessing fluentbit is like filebit, which ingests files/logs from the filesystem?

01:57

arun

01:57 PM

hmm, maybe this could be of use https://vector.dev/docs/reference/configuration/sources/docker_logs/

Prabhat

01:57 PM

yeah, you got a readymade one. Good searcher

arun

01:58 PM

i'll try and see, thanks for your help!

Photo of md5-6fe3c7a15dc56f7a510aea6c129a7486

Craig

08:32 PM

Can Elastic Agent send to OpenObserve?
https://www.elastic.co/elastic-agent

Elastic Agent is a process of processes which runs Filebeat, Metricbeat

Prabhat

11:27 PM

We haven't tested elastic agent. Filebeat is supported though.

Craig

11:29 PM

If you are curious about the internals of elastic-agent, and how it incorporates filebeat:

https://www.youtube.com/watch?v=xwdCuhN2uTM

Prabhat

11:30 PM

Thanks. Will take a look.

Craig

11:30 PM

elastic agent is one complicated thing, but for people stuck on elastic…..

Jun 16, 2023 (5 months ago)

Photo of md5-540a8e08ce1c199c4efaeb0388742259

Gaby

12:08 AM

👍 for using Vector, works out of the box with OpenObserve :-)

Jun 19, 2023 (5 months ago)

arun

07:05 AM

just wanted to report that it worked! i used vector and i am not ingesting the logs from docker. since i run vector as a docker service in docker swarm i had to mount /var/run/docker.sock:/var/run/docker.sock inside

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 404 threads (74% resolved)

Join Our Community

Similar Threads

Elastic Fleet/Agents Reporting to ZincObserve

Joe inquired about using Elastic Fleet/agents with ZincObserve. Prabhat found it possible, but faced issues related to mutable data in ES indexes. Further investigation required.

9mo

Troubleshooting Openobserve Issue On Linux Machine

Sushma encountered difficulties viewing pods and creating multiple streams in Openobserve. Prabhat and Hengfei suggested upgrading the software, adjusting SQL syntax, and amending the configuration setup. Despite progress, the issue remains partially unresolved.

Setting up Fluentd with OpenObserve for Kubernetes Logging

vasanth sought help for integrating Fluentd with OpenObserve. Prabhat recommended using FluentBit or Vector instead and provided a tutorial, which resolved vasanth's issue successfully.

6mo

OpenObserve issues with FluentBit and Dashboard

Alejandro experienced issues with FluentBit losing connection with OpenObserve and discarding logs, and an error when saving a chart on the OpenObserve dashboard. Prabhat could not identify the cause of record loss. However, potential solutions were suggested to save the dashboard with a string-type filter instead of integer one.

2mo

Querying Local Storage Files and Ingesting Logs with OpenObserve

Rinshad asks about querying local storage files and ingesting logs. Prabhat suggests using parquet file format and tools like vector and fluentbit. Mark mentions Sublime's plugin for editing parquet files.

5mo