TLDR Sa had issues with querying logs in ZO on Kubernetes and creating users. Hengfei provided solutions for both problems, including updating with a dev version and setting memory cache values.
where are you stuck?
getting any error
please share more details
I cannot query log at here. The request is error.
already have data
can you check querier pod log
and share what error does querier gives
yes. this is the querier log.
There is no error trace in querier.
checking..will get back to you
this not error. just shows that memory cache is full, drop something and then can cache new file.
and at the end, the query give result with 114s
Do you try to query the all data? 88GB?
i guess, your node have 8GB memory.
can you go to steams page
and share the details of stream
I only query logs from the last 15 mins
I have 1 pod for querier and the node for this pod is 16GB Ram
the logs shows it not only for 15 minutes. can you try search agian, and check what is in the query logs. it will give the query detail.
Do you have alerts?
What is the alert ?
past 15 mins
the log output
when I perform query
no result
What is your deploy values for query? you set memory cache is 100MB?
or, you eanbeld `ZO_MEMORY_CACHE_CACHE_LATEST_FILES`
I'm using default value from helm charts
can you go to steams menu -> click
a side panel will open
please share screen shot of it
```[2023-05-23T11:03:44Z INFO zincobserve::service::search::grpc::storage] [TRACE] storage->search: load files 339 done [2023-05-23T11:03:44Z INFO tracing::span] service:search:storage:group_and_calc_files_size; [2023-05-23T11:03:44Z INFO tracing::span] service:file_list:calculate_files_size; [2023-05-23T11:03:44Z INFO zincobserve::service::search::grpc::storage] [TRACE] storage->search: load files 339, scan_size 2986565933``` as this logs, for 15 minutes, it need load 339 parquet files, default our configure is 32MB per file. it should be 10GB, the but scan_size give us 2.8GB. and the log ```[2023-05-23T11:03:44Z INFO zincobserve::infra::cache::file_data] [TRACE] File cache is full 105101150/104857600, can't cache 721844 bytes``` tell use the total memory cache size is: 104857600 = 100MB
did you add some resource limit for the querier pod?
this is the stream captured. Ashish
Hengfei there is no limit for querier pod
Don't know why ZO need to download all log contents from S3. I think the log content can paginate loading when perform query
it should be. but the logs shows something not as we expect. like i said, the memory cache was set only 100MB.
it should only load the files in the 15 minutes.
can you try add some config: ```ZO_MEMORY_CACHE_ENABLED=true ZO_MEMORY_CACHE_CACHE_LATEST_FILES=false ZO_MEMORY_CACHE_MAX_SIZE=4096```
those config means: enable memory cache for query, and set max memory cache is 4GB
I will try to set these values
there is no loading log any more
can you share the logs of ingester, to check the ingest is working
oh. my agent is stopped. I will restart now
this is the error right ?
this is an error, which version do you use? 0.4.3?
yes. the latest version
i will give you a dev version for fix this issue.
Can you try this image: ```public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-dd87ab7-amd64```
yes. let's me try
upgrade the querier ?
you use helm, right? you can upgrade all pods.
the error still there
Ha...
Thanks, let me check
ok
```public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-2e0d7a7-amd64```
Can you try this version?
yes
try to query the last 2 days but there is no result
last 2 days maybe have a lot of data, need more resource.
can you try query last 5 minutes, 15 minutes, 1 hours.
Does it have result?
first, let's confirm the query can work. then let's talk about query all data.
I have to change to another node agent. because the log of current node is too large (2GB/h)
I will inform you the result later
2GB/h no problem, in our own test cluster have more than 2TB data, every hour increase 10GB logs
Yes, you can use small data size to test.
Pushing log from other machine with lower log size
It's working now.
One more bug, I cannot create a new user .
there is no button on User page
So, the next problem will be when you search more data, it maybe timeout.
Do you use the root user login? and what is your organanition?
I will try to push more data over time (maybe 2 days) then try to query the last 2 days.
yes. I logged in with the root role in default organization
can you give a screenshot for users page.
here you are.
there is no button create user.
Do you use the version 0.4.3 or the dev version yesterday i given you?
only querier pod is using your version. other components is 0.4.3
no one reported this issue can't create user. i saw you already create 2 users, earlier you can, but now, you can't?
yes.
now, I cannot create more user.
Can you open the console of chrome, to check if there are some js error report?
Or, try to logout and login again?
I opened the console and there is no js error.
I logged out and logged in but no different
using incognito tab but no different
Okay, this is a bug, i confirm.
Can you help create an issue on github? we will fix it soon.
Bug about create user ?
yes
thanks
we fixed the add user issue, please try this tag: ```public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-56a4c15-amd64```
Sa
Tue, 23 May 2023 10:10:29 UTCI tried to setup ZO on kubernetes with helm charts. The installation is successful and I can push log using fluentbit too. But, now I cannot query log in the dashboard.