#openobserve

Discussing OpenObserve for Application Schema and Weight-Based Searches

TLDR Murugesan inquired about using zplane for application schema in OpenObserve. Prabhat explained its limitations for weight-based searches and offered a technical discussion to explore potential solutions.

Powered by Struct AI

1

19
5mo
Solved
Join the chat
Jun 26, 2023 (5 months ago)
Murugesan
Photo of md5-894e524356146cb346abbb1f595710f0
Murugesan
03:41 AM
I am looking into seeing how to use zplane to add application schema since each application of ours would write a specific index schema. Any help is appreciated
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:54 AM
You do not need to define schema specifically in OpenObserve (In fact as of now schema is created automatically upon data ingestion and you cannot create it manually)
04:55
Prabhat
04:55 AM
If you are looking for full text search on specific fields then you can enable full text search using stream details UI
04:55
Prabhat
04:55 AM
Image 1 for
04:56
Prabhat
04:56 AM
match_all function searches all the fields that have full text search enabled.
04:57
Prabhat
04:57 AM
OPenObserve does not build inverted indexes like elasticsearch and relies on brute force search if required on specific fields
04:58
Prabhat
04:58 AM
How do you plan to use OpenObserve?
05:04
Prabhat
05:04 AM
or how do you use elasticsearch today?
Murugesan
Photo of md5-894e524356146cb346abbb1f595710f0
Murugesan
05:04 AM
Thanks Prabhat. I see that by default, OpenObserve creates a default organization.

Use case is as follows:
• we wanted to allow search of certain components of our product to be searched.
◦ For e.g, if you have product like component A, component B, component C, we wanted to ingest all of the the component data, but when we search by specific text, we wanted to literally have the search return based on weights (how close the search string to the actual data that is stored in the ingestion)
• The search should return in sub ms response
05:05
Murugesan
05:05 AM
I like the whole HA architecture of OpenObserve

1

Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:08 AM
OpenObserve has not been built to provide weight based searches which is a characteristic of general purpose inverted index based text search that you see in elasticsearch or Zincsearch. OpenObserve is built specifically for log search where it does exact searches over large amounts of data.
05:09
Prabhat
05:09 AM
what is your data volume?
Murugesan
Photo of md5-894e524356146cb346abbb1f595710f0
Murugesan
05:13 AM
Our typical data volume would is 10s of GB/day per customer for some of our initial customers that we are seeing
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:14 AM
I would have pointed you to typesense/meilisearch but your data volume is more apt for OpenObserve.
05:14
Prabhat
05:14 AM
Let me DM you
Murugesan
Photo of md5-894e524356146cb346abbb1f595710f0
Murugesan
07:00 PM
Prabhat, if Uno was able to fork out of openobserve and attempt to add tf-idf and contribute back, would that solution be ok for OpenObserve?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
07:11 PM
Theoretically that should be ok. However before you embark on this journey, we should have a technical discussion(s) on what is the best way to achieve it, since we will need to maintain it for longer term.
Murugesan
Photo of md5-894e524356146cb346abbb1f595710f0
Murugesan
08:04 PM
Prabhat, I totally agree with starting with a technical discussion. We would love to have a white-boarding session with you at our Palo Alto Office at your convenience if you are willing to visit or else we can meet at a suitable place and we can co-ordinate
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
10:15 PM
Yeah sure. I am planning to visit Palo Alto in next 2 weeks. Let me DM u

OpenObserve

OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI

Indexed 406 threads (73% resolved)

Join Our Community