TLDR dlonghi asked how to synchronize a MySQL table with OpenObserve. Prabhat advised using the same `log_id` field, and addressed a potential duplicity concern, though deletion is still not available. A feature request was created for this issue.
OpenObserve does not have an equivalent special `_id` field . You can however insert the same `log_id` or `_id` field in OpenObserve and it will gladly accept it and you can continue using it. The only difference compared to ES would be that if you send 2 records with same value for `_id` field OpenObserve will not update the first record, but will create 2 records.
Thanks. The goal for this use case is to avoid the duplicates. Can documents be deleted from OO ? I looking at the docs but couldn't find info about that.
Deletion is not a functionality available right now. We have a feature request for enabling deduplication but that is not yet implemented.
Don't you have deduplicated data already in mysql
IF log)id is primary key or has UNIQUE index then it should be deduplicated already
Yes the data is unique dedup. The ingesting script ir poor though : / no control of what data was already indexed.. no error handling... If it fails, run again and the Elasticsearch upsert and _id would handle the dupes. Certainly this ingesting script should be worked on : )
Ah got it. Valid use case. Created this issue to track it -
You can still query using SQL mode and use DISTINCT and you will get deduplicate values today.
if that helps
Thank you Mr Prabhat.
dlonghi
Mon, 03 Jul 2023 11:57:30 UTCHow could we sync a MySQL table with OpenObserve ? It is a log table where an old php app writes to. It has 3 fields `log_id`, `timestamp`, `message`; In Elastisearch we have been doing this kind of sync using an UPSERT operation with ES doc `_id` being equal to the table's `log_id`.