OpenObserve syslog and forwarding with fluentbit/vector

TLDR Toto asked about setting up OpenObserve to ingest logs using syslog in cloud tenant. Prabhat recommended setting up a local fluentbit or vector server and forwarding syslog, as syslog is not directly supported due to lack of authentication. They also provided help with vector configuration.

Photo of Toto
Toto
Fri, 09 Jun 2023 07:37:44 UTC

Hey guys, trying to test OpenObserve to ingest logs using syslog as documented here However, when I log in to my cloud tenant, I don't see the syslog option. Any thoughts? Thanks

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:38:15 UTC

syslog is not supported directly in cloud

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:38:42 UTC

You must use fluentbit/vector as a local server and then forward using it if you want to use cloud for sending syslog

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:38:55 UTC

else you must host opensobserve on your own

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:39:16 UTC

the reason for this is that syslog protocol does not support authentication

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:39:32 UTC

and we cannot open something on internet that has no authentication

Photo of Toto
Toto
Fri, 09 Jun 2023 07:41:16 UTC

Thanks

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:41:37 UTC

are you planing to send logs directly from network devices?

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:41:48 UTC

or from your servers?

Photo of Toto
Toto
Fri, 09 Jun 2023 07:46:36 UTC

from network devices, opnsense

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:49:10 UTC

you should setup a local fluentbit that can act as a syslog server and can forward logs to openObserve cloud

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:49:21 UTC

Here are the docs -

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:49:38 UTC

Let us know if that helps, we can help you configure it

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:50:05 UTC

Your config file would look like

Photo of Toto
Toto
Fri, 09 Jun 2023 07:50:05 UTC

yup thank you. I got that from your initial reply above

Photo of Toto
Toto
Fri, 09 Jun 2023 07:50:24 UTC

however I'm working with vector instead of fluent bit .. any preference here?

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:50:39 UTC

nah, both are good.

Photo of Toto
Toto
Fri, 09 Jun 2023 07:54:31 UTC

while configuring vector to receive syslog msgs, it requires configuring a socket. I'm trying to understand what's used for.

Photo of Toto
Toto
Fri, 09 Jun 2023 07:54:38 UTC

```path = "/path/to/socket"```

Photo of Toto
Toto
Fri, 09 Jun 2023 07:55:25 UTC

is this socket a temporary file to process the msgs before sending them out ?

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:58:16 UTC

path is the unix domain socket that will be used to receive syslog messages

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:58:23 UTC

possible values are

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:58:45 UTC

```/dev/log or /var/run/syslog.sock```

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:59:17 UTC

you need the path parameter only if you are using tcp

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:59:31 UTC

if you set the mode to UDP then you don't need it

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 07:59:48 UTC

I suggest you go ahead with mode "udp"

Photo of Toto
Toto
Fri, 09 Jun 2023 08:01:56 UTC

Ok, will test that, and let u know, thanks

Photo of Toto
Toto
Fri, 09 Jun 2023 08:46:40 UTC

So I managed to ship the logs. How can I search for all message containing a certain string ?

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 10:55:53 UTC

Check the syntax guide button.

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 10:56:15 UTC

above the query editor

Photo of Prabhat
Prabhat
Fri, 09 Jun 2023 10:56:41 UTC

for full text search on a specific field you may need to enable it in the stream details