ZO Kubernetes Issues: Query Log Error and Adding Users

Sa

I tried to setup ZO on kubernetes with helm charts. The installation is successful and I can push log using fluentbit too. But, now I cannot query log in the dashboard.

Ashish

where are you stuck?

Ashish

getting any error

Ashish

please share more details

Sa

I cannot query log at here. The request is error.

Sa

already have data

Ashish

can you check querier pod log

Ashish

and share what error does querier gives

Sa

yes. this is the querier log.

Sa

There is no error trace in querier.

Ashish

checking..will get back to you

Hengfei

this not error. just shows that memory cache is full, drop something and then can cache new file.

Hengfei

and at the end, the query give result with 114s

Hengfei

Do you try to query the all data? 88GB?

Hengfei

i guess, your node have 8GB memory.

Ashish

can you go to steams page

Ashish

and share the details of stream

Sa

I only query logs from the last 15 mins

Sa

I have 1 pod for querier and the node for this pod is 16GB Ram

Hengfei

the logs shows it not only for 15 minutes. can you try search agian, and check what is in the query logs. it will give the query detail.

Hengfei

Do you have alerts?

Sa

What is the alert ?

Sa

past 15 mins

Sa

Sa

the log output

Sa

when I perform query

Sa

no result

Hengfei

What is your deploy values for query? you set memory cache is 100MB?

Hengfei

or, you eanbeld `ZO_MEMORY_CACHE_CACHE_LATEST_FILES`

Sa

I'm using default value from helm charts

Sa

this helm chart value

Ashish

can you go to steams menu -> click

Ashish

a side panel will open

Ashish

please share screen shot of it

Hengfei

```[2023-05-23T11:03:44Z INFO zincobserve::service::search::grpc::storage] [TRACE] storage->search: load files 339 done [2023-05-23T11:03:44Z INFO tracing::span] service:search:storage:group_and_calc_files_size; [2023-05-23T11:03:44Z INFO tracing::span] service:file_list:calculate_files_size; [2023-05-23T11:03:44Z INFO zincobserve::service::search::grpc::storage] [TRACE] storage->search: load files 339, scan_size 2986565933``` as this logs, for 15 minutes, it need load 339 parquet files, default our configure is 32MB per file. it should be 10GB, the but scan_size give us 2.8GB. and the log ```[2023-05-23T11:03:44Z INFO zincobserve::infra::cache::file_data] [TRACE] File cache is full 105101150/104857600, can't cache 721844 bytes``` tell use the total memory cache size is: 104857600 = 100MB

Hengfei

did you add some resource limit for the querier pod?

Sa

this is the stream captured. Ashish

Sa

Hengfei there is no limit for querier pod

Sa

Don't know why ZO need to download all log contents from S3. I think the log content can paginate loading when perform query

Hengfei

it should be. but the logs shows something not as we expect. like i said, the memory cache was set only 100MB.

Hengfei

it should only load the files in the 15 minutes.

Hengfei

can you try add some config: ```ZO_MEMORY_CACHE_ENABLED=true ZO_MEMORY_CACHE_CACHE_LATEST_FILES=false ZO_MEMORY_CACHE_MAX_SIZE=4096```

Hengfei

those config means: enable memory cache for query, and set max memory cache is 4GB

Sa

I will try to set these values

Sa

there is no loading log any more

Hengfei

can you share the logs of ingester, to check the ingest is working

Sa

oh. my agent is stopped. I will restart now

Sa

Sa

this is the error right ?

Hengfei

this is an error, which version do you use? 0.4.3?

Sa

yes. the latest version

Hengfei

i will give you a dev version for fix this issue.

Hengfei

Can you try this image: ```public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-dd87ab7-amd64```

Sa

yes. let's me try

Sa

upgrade the querier ?

Hengfei

you use helm, right? you can upgrade all pods.

Sa

the error still there

Hengfei

Ha...

Hengfei

Thanks, let me check

Sa

ok

Hengfei

```public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-2e0d7a7-amd64```

Hengfei

Can you try this version?

Sa

yes

Sa

try to query the last 2 days but there is no result

Hengfei

last 2 days maybe have a lot of data, need more resource.

Hengfei

can you try query last 5 minutes, 15 minutes, 1 hours.

Hengfei

Does it have result?

Hengfei

first, let's confirm the query can work. then let's talk about query all data.

Sa

I have to change to another node agent. because the log of current node is too large (2GB/h)

Sa

I will inform you the result later

Hengfei

2GB/h no problem, in our own test cluster have more than 2TB data, every hour increase 10GB logs

Hengfei

Yes, you can use small data size to test.

Sa

Pushing log from other machine with lower log size

Sa

It's working now.

Sa

One more bug, I cannot create a new user .

Sa

there is no button on User page

Hengfei

So, the next problem will be when you search more data, it maybe timeout.

Hengfei

Do you use the root user login? and what is your organanition?

Sa

I will try to push more data over time (maybe 2 days) then try to query the last 2 days.

Sa

yes. I logged in with the root role in default organization

Hengfei

can you give a screenshot for users page.

Sa

here you are.

Sa

there is no button create user.

Hengfei

Do you use the version 0.4.3 or the dev version yesterday i given you?

Sa

only querier pod is using your version. other components is 0.4.3

Hengfei

no one reported this issue can't create user. i saw you already create 2 users, earlier you can, but now, you can't?

Sa

yes.

Sa

now, I cannot create more user.

Hengfei

Can you open the console of chrome, to check if there are some js error report?

Hengfei

Or, try to logout and login again?

Sa

I opened the console and there is no js error.

Sa

I logged out and logged in but no different

Sa

using incognito tab but no different

Hengfei

Okay, this is a bug, i confirm.

Hengfei

Can you help create an issue on github? we will fix it soon.

Sa

Bug about create user ?

Hengfei

yes

Sa

here you are

Hengfei

thanks

Hengfei

we fixed the add user issue, please try this tag: ```public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-56a4c15-amd64```