Forwarding Google Cloud Logging to Own Cloud Service
TLDR Florian sought help forwarding Google Cloud logging to their cloud service. Prabhat suggested exporting logs via pub-sub to OpenObserve, with authentication needing improvement. After considering alternatives, they agreed to use a URL with a shared key.
Prabhat
Fri, 16 Jun 2023 12:22:38 UTCthe recommended mechanism on GCP for exporting logs is via pub sub
Prabhat
Fri, 16 Jun 2023 12:23:00 UTClogs -> log router -> pub-sub-> OpenObserve
Prabhat
Fri, 16 Jun 2023 12:23:22 UTCthere one missing item on authentication at OpenObserve for google cloud
Prabhat
Fri, 16 Jun 2023 12:24:13 UTCpub-sub sends a JWT that needs to be validated by OpenObserve to amke sure that the request is coming from the expected source
Prabhat
Fri, 16 Jun 2023 12:24:35 UTCWe will need to build this on OpenObserve cloud as well as open source.
Prabhat
Fri, 16 Jun 2023 12:24:52 UTCWe have done something similar for AWS using kinesis firehose.
Prabhat
Fri, 16 Jun 2023 12:26:25 UTCI am trying to find the keys that can be used for validating the JWT from GCP docs
Florian
Fri, 16 Jun 2023 12:28:57 UTCNo worries I am not in a hurry.
Looking at DD they worked around this by using a URL located API key
Prabhat
Fri, 16 Jun 2023 12:29:30 UTCAh, neat trick. a shared key.
Prabhat
Fri, 16 Jun 2023 12:29:34 UTCthat could work too
Prabhat
Fri, 16 Jun 2023 12:30:15 UTCI would rather prefer the JWT as its more secure than the shared secret but let's figure out the complexity
Florian
Fri, 16 Jun 2023 12:30:29 UTCYeah, the google way is rather complex but also more secure :smile:
Prabhat
Fri, 16 Jun 2023 12:55:34 UTCI checked. We will take the easy path
Prabhat
Fri, 16 Jun 2023 12:55:49 UTCgive a URL with the shared key
Florian
Fri, 16 Jun 2023 13:34:19 UTCPerfect, I will happily try it :smile:
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
Florian
Fri, 16 Jun 2023 12:21:51 UTCHi, do you have an idea how to forward google cloud logging to your cloud service? (I think the same process would work with OSS as well)