OpenObserve syslog and forwarding with fluentbit/vector
TLDR Toto asked about setting up OpenObserve to ingest logs using syslog in cloud tenant. Prabhat recommended setting up a local fluentbit or vector server and forwarding syslog, as syslog is not directly supported due to lack of authentication. They also provided help with vector configuration.
Prabhat
Fri, 09 Jun 2023 07:38:15 UTCsyslog is not supported directly in cloud
Prabhat
Fri, 09 Jun 2023 07:38:42 UTCYou must use fluentbit/vector as a local server and then forward using it if you want to use cloud for sending syslog
Prabhat
Fri, 09 Jun 2023 07:38:55 UTCelse you must host opensobserve on your own
Prabhat
Fri, 09 Jun 2023 07:39:16 UTCthe reason for this is that syslog protocol does not support authentication
Prabhat
Fri, 09 Jun 2023 07:39:32 UTCand we cannot open something on internet that has no authentication
Toto
Fri, 09 Jun 2023 07:41:16 UTCThanks
Prabhat
Fri, 09 Jun 2023 07:41:37 UTCare you planing to send logs directly from network devices?
Prabhat
Fri, 09 Jun 2023 07:41:48 UTCor from your servers?
Toto
Fri, 09 Jun 2023 07:46:36 UTCfrom network devices, opnsense
Prabhat
Fri, 09 Jun 2023 07:49:10 UTCyou should setup a local fluentbit that can act as a syslog server and can forward logs to openObserve cloud
Prabhat
Fri, 09 Jun 2023 07:49:21 UTCHere are the docs -
Prabhat
Fri, 09 Jun 2023 07:49:38 UTCLet us know if that helps, we can help you configure it
Prabhat
Fri, 09 Jun 2023 07:50:05 UTCYour config file would look like
Toto
Fri, 09 Jun 2023 07:50:05 UTCyup thank you. I got that from your initial reply above
Toto
Fri, 09 Jun 2023 07:50:24 UTChowever I'm working with vector instead of fluent bit .. any preference here?
Prabhat
Fri, 09 Jun 2023 07:50:39 UTCnah, both are good.
Toto
Fri, 09 Jun 2023 07:54:31 UTCwhile configuring vector to receive syslog msgs, it requires configuring a socket. I'm trying to understand what's used for.
Toto
Fri, 09 Jun 2023 07:54:38 UTC```path = "/path/to/socket"```
Toto
Fri, 09 Jun 2023 07:55:25 UTCis this socket a temporary file to process the msgs before sending them out ?
Prabhat
Fri, 09 Jun 2023 07:58:16 UTCpath is the unix domain socket that will be used to receive syslog messages
Prabhat
Fri, 09 Jun 2023 07:58:23 UTCpossible values are
Prabhat
Fri, 09 Jun 2023 07:58:45 UTC```/dev/log or /var/run/syslog.sock```
Prabhat
Fri, 09 Jun 2023 07:59:17 UTCyou need the path parameter only if you are using tcp
Prabhat
Fri, 09 Jun 2023 07:59:31 UTCif you set the mode to UDP then you don't need it
Prabhat
Fri, 09 Jun 2023 07:59:48 UTCI suggest you go ahead with mode "udp"
Toto
Fri, 09 Jun 2023 08:01:56 UTCOk, will test that, and let u know, thanks
Toto
Fri, 09 Jun 2023 08:46:40 UTCSo I managed to ship the logs. How can I search for all message containing a certain string ?
Prabhat
Fri, 09 Jun 2023 10:55:53 UTCCheck the syntax guide button.
Prabhat
Fri, 09 Jun 2023 10:56:15 UTCabove the query editor
Prabhat
Fri, 09 Jun 2023 10:56:41 UTCfor full text search on a specific field you may need to enable it in the stream details
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Toto
Fri, 09 Jun 2023 07:37:44 UTCHey guys, trying to test OpenObserve to ingest logs using syslog as documented here