Synchronizing MySQL table with OpenObserve
TLDR dlonghi asked how to synchronize a MySQL table with OpenObserve. Prabhat advised using the same `log_id` field, and addressed a potential duplicity concern, though deletion is still not available. A feature request was created for this issue.
Prabhat
Mon, 03 Jul 2023 14:02:40 UTCOpenObserve does not have an equivalent special `_id` field . You can however insert the same `log_id` or `_id` field in OpenObserve and it will gladly accept it and you can continue using it. The only difference compared to ES would be that if you send 2 records with same value for `_id` field OpenObserve will not update the first record, but will create 2 records.
dlonghi
Mon, 03 Jul 2023 14:06:22 UTCThanks. The goal for this use case is to avoid the duplicates. Can documents be deleted from OO ? I looking at the docs but couldn't find info about that.
Prabhat
Mon, 03 Jul 2023 14:07:28 UTCDeletion is not a functionality available right now. We have a feature request for enabling deduplication but that is not yet implemented.
Prabhat
Mon, 03 Jul 2023 14:07:59 UTCDon't you have deduplicated data already in mysql
Prabhat
Mon, 03 Jul 2023 14:08:36 UTCIF log)id is primary key or has UNIQUE index then it should be deduplicated already
dlonghi
Mon, 03 Jul 2023 14:18:29 UTCYes the data is unique dedup. The ingesting script ir poor though : / no control of what data was already indexed.. no error handling... If it fails, run again and the Elasticsearch upsert and _id would handle the dupes. Certainly this ingesting script should be worked on : )
Prabhat
Mon, 03 Jul 2023 14:48:59 UTCAh got it. Valid use case. Created this issue to track it -
Prabhat
Mon, 03 Jul 2023 14:49:53 UTCYou can still query using SQL mode and use DISTINCT and you will get deduplicate values today.
Prabhat
Mon, 03 Jul 2023 14:49:57 UTCif that helps
dlonghi
Mon, 03 Jul 2023 18:30:56 UTCThank you Mr Prabhat.
OpenObserve
Indexed 419 threads
OpenObserve is an open-source, petabyte-scale observability platform for the cloud native realm, offering a 10x cost reduction and 140x less storage use compared to competitors like Elasticsearch or Splunk. Built in Rust for exceptional performance, it offers comprehensive features like logs, metrics, traces, dashboards, and more | Knowledge Base powered by Struct.AI
View in App
dlonghi
Mon, 03 Jul 2023 11:57:30 UTCHow could we sync a MySQL table with OpenObserve ? It is a log table where an old php app writes to. It has 3 fields `log_id`, `timestamp`, `message`; In Elastisearch we have been doing this kind of sync using an UPSERT operation with ES doc `_id` being equal to the table's `log_id`.