OpenObserve

OpenObserve

#openobserve

Discussing OpenObserve for Application Schema and Weight-Based Searches

TLDR Murugesan inquired about using zplane for application schema in OpenObserve. Prabhat explained its limitations for weight-based searches and offered a technical discussion to explore potential solutions.

Powered by Struct AI

1

19

5mo

Jun 26, 2023 (5 months ago)

Photo of md5-894e524356146cb346abbb1f595710f0

Murugesan

03:41 AM

I am looking into seeing how to use zplane to add application schema since each application of ours would write a specific index schema. Any help is appreciated

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

04:54 AM

You do not need to define schema specifically in OpenObserve (In fact as of now schema is created automatically upon data ingestion and you cannot create it manually)

04:55

Prabhat

04:55 AM

If you are looking for full text search on specific fields then you can enable full text search using stream details UI

04:55

Prabhat

04:55 AM

Image 1 for

04:56

Prabhat

04:56 AM

match_all function searches all the fields that have full text search enabled.

04:57

Prabhat

04:57 AM

OPenObserve does not build inverted indexes like elasticsearch and relies on brute force search if required on specific fields

04:58

Prabhat

04:58 AM

How do you plan to use OpenObserve?

05:04

Prabhat

05:04 AM

or how do you use elasticsearch today?

Photo of md5-894e524356146cb346abbb1f595710f0

Murugesan

05:04 AM

Thanks Prabhat. I see that by default, OpenObserve creates a default organization.

Use case is as follows:
• we wanted to allow search of certain components of our product to be searched.
◦ For e.g, if you have product like component A, component B, component C, we wanted to ingest all of the the component data, but when we search by specific text, we wanted to literally have the search return based on weights (how close the search string to the actual data that is stored in the ingestion)
• The search should return in sub ms response

05:05

Murugesan

05:05 AM

I like the whole HA architecture of OpenObserve

1

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

05:08 AM

OpenObserve has not been built to provide weight based searches which is a characteristic of general purpose inverted index based text search that you see in elasticsearch or Zincsearch. OpenObserve is built specifically for log search where it does exact searches over large amounts of data.

05:09

Prabhat

05:09 AM

what is your data volume?

Photo of md5-894e524356146cb346abbb1f595710f0

Murugesan

05:13 AM

Our typical data volume would is 10s of GB/day per customer for some of our initial customers that we are seeing

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

05:14 AM

I would have pointed you to typesense/meilisearch but your data volume is more apt for OpenObserve.

05:14

Prabhat

05:14 AM

Let me DM you

Photo of md5-894e524356146cb346abbb1f595710f0

Murugesan

07:00 PM

Prabhat, if Uno was able to fork out of openobserve and attempt to add tf-idf and contribute back, would that solution be ok for OpenObserve?

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

07:11 PM

Theoretically that should be ok. However before you embark on this journey, we should have a technical discussion(s) on what is the best way to achieve it, since we will need to maintain it for longer term.

Photo of md5-894e524356146cb346abbb1f595710f0

Murugesan

08:04 PM

Prabhat, I totally agree with starting with a technical discussion. We would love to have a white-boarding session with you at our Palo Alto Office at your convenience if you are willing to visit or else we can meet at a suitable place and we can co-ordinate

Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5

Prabhat

10:15 PM

Yeah sure. I am planning to visit Palo Alto in next 2 weeks. Let me DM u

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (73% resolved)

Join Our Community

Similar Threads

Understanding and Integrating with OpenObserve

刘希晨 sought clarification on the difference between ZincSearch, ZincObserve, and OpenObserve. Prabhat responded and actively engaged in a long discussion regarding an integration for 刘希晨's microservice framework. Hengfei shared that the issue 刘希晨 faced with traces upload was acknowledged and fixed.

2

Setting up Fluentd with OpenObserve for Kubernetes Logging

vasanth sought help for integrating Fluentd with OpenObserve. Prabhat recommended using FluentBit or Vector instead and provided a tutorial, which resolved vasanth's issue successfully.

3

ZincSearch vs Kibana and OpenObserve Improvements

Gaby is looking to move away from ElasticSearch, but finds OpenObserve lacks features he needs. Prabhat and Kirtan discuss potential improvements and consider user feedback valuable.

7

Guidance on Using ZincSearch and API Calls for OpenObserve Cloud

YULL was initially confused about making API calls to OpenObserve Cloud while working on a technical test involving ZincSearch. Prabhat clarified the distinction between ZincSearch and OpenObserve cloud and advised on how to proceed.

2

Connecting ZincSearch to Grafana

Ross needs help connecting zincsearch to grafana. Prabhat provides insights and instead recommends OpenObserve, detailing its benefits over zincsearch and its compatibility with grafana. They also discuss licensing and deployment issues.

2

Powered By