Pushing Logs to S3 and Installing ZincObserve with Helm Chart

sparrow

Hello Team, I was exploring the product and able to install and and use it properly but I can’t find any way around by which i can push my logs to s3 bucket using fluent bit could you please help me

Hengfei

you can follow this document to use s3

sparrow

Yes i’ve gone through this doc, but where exactly i have to mention in fluent-bit so that it will select s3 as a storage type, because by default it is setting up as disk storage

sparrow

as you can see all index is using disk as a storage

Hengfei

oh sorry, you use zincSearch for logs?

Hengfei

zincSearch has some problems for support s3. we suggest use zincObserve for logs.

sparrow

do we have any helm chart for zincObserve?

Hengfei

there is some introduce for the difference between zincSearch and zincObserve

Hengfei

Prabhat are we ready for helm?

Hengfei

i think we are working for helm, should be ready this week.

sparrow

If not helm then how I can install zincObserve?

Hengfei

single node, right?

Hengfei

Hengfei

you can use k8s deploy

sparrow

Okay got it, Thanks. Do zincObserve also use telemetry to fetch data?

Hengfei

Yep, you can use `ZO_TELEMETRY=false` to disable it

Hengfei

just report some usage for us to analytics how many user. no private data.

Hengfei

you also can disable it.

Hengfei

Hengfei

there is a quick start video.

sparrow

but i can see it’s fetching `querystring, match`

sparrow

In our logs there so many data, we can’t just let this to be available for others

Hengfei

this is zincSearch, it report this, because we have old api & new api, we want to know if we should delete old api.

Hengfei

Yes of course, you can disable it with the environment.

Hengfei

after you disable it, it will report nothing.

sparrow

but in our production account we have blocked all the outbounds rules, in this case it would fail

Hengfei

i didnot get your point. which will fail? you already disable it.

sparrow

is there anyway by which I can confirm the telemetry is disabled or not?

Hengfei

we provider an environment for disable it. if you want to confirm that, you can disconnect internet, itf it still report something, you will see network error.

Hengfei

you can try this with keep enable it. you will see the error.

sparrow

After installing zincObserve from where sould i get the credential to login?

sparrow

Also I’m not able to exec into the zincobserve pod neither with sh nor with bash

Hengfei

for production image, no, we use micro image. if you want login into the container, you can dev image.

Hengfei

where sould i get the credential to login?

Hengfei

you can see the deployment, it set the root user.

Hengfei

and in the document, it also mentioned the user name & password.

Prabhat

Helm chart should be ready this week

sparrow

What is this error?

sparrow

Hengfei

Hengfei

i have no idea, haven’t met this

Hengfei

Have you ingest some data?

sparrow

yes

Hengfei

How

sparrow

at first i was able to see the logs, but suddenly getting this error

sparrow

Using fluent-bit

sparrow

You can see

Hengfei

Yep

Hengfei

can you give some logs about the error in container for zincobserve

Prabhat

helm charts for s3 is now available -

sparrow

Hello Hengfei I’ve a update on the above error

Hengfei

Which error? maybe i didn’t get it.

sparrow

The above one that i reported

sparrow

sparrow

this one

Hengfei

Okay, if you have more logs about it on the server side?

Hengfei

i am not sure the problem.

Hengfei

how do you install zincobserve and the configure variables.

sparrow

Actually i’m getting this error when I’m trying to run some query and it’s not returning anything

sparrow

I used this manifest ```apiVersion: v1 kind: Service metadata: name: zincobserve spec: clusterIP: None selector: app: zincobserve ports: - name: http port: 5080 targetPort: 5080 --- # create statefulset apiVersion: apps/v1 kind: StatefulSet metadata: name: zincobserve labels: name: zincobserve spec: serviceName: zincobserve replicas: 1 selector: matchLabels: name: zincobserve app: zincobserve template: metadata: labels: name: zincobserve app: zincobserve spec: securityContext: fsGroup: 2000 runAsUser: 10000 runAsGroup: 3000 runAsNonRoot: true # terminationGracePeriodSeconds: 0 containers: - name: zincobserve image: public.ecr.aws/zinclabs/zincobserve:latest env: - name: ZO_ROOT_USER_EMAIL value: - name: ZO_ROOT_USER_PASSWORD value: Complexpass#123 - name: ZO_DATA_DIR value: /data - name: ZO_TELEMETRY value: "false" - name: ZO_S3_SERVER_URL value: default - name: ZO_S3_REGION_NAME value: - name: ZO_S3_ACCESS_KEY value: - name: ZO_S3_SECRET_KEY value: - name: ZO_S3_BUCKET_NAME value: # command: ["/bin/bash", "-c", "while true; do sleep 1; done"] imagePullPolicy: Always resources: limits: cpu: 1000m memory: 2048Mi requests: cpu: 256m memory: 50Mi ports: - containerPort: 5080 name: http volumeMounts: - name: data mountPath: /data volumeClaimTemplates: - metadata: name: data spec: accessModes: - ReadWriteOnce # storageClassName: default # NOTE: You can increase the storage size resources: requests: storage: 10Gi```

Hengfei

okay, can you give some errors in the container

Hengfei

to check what happens with zincobserve

Hengfei

can you send me the logs of the container

Hengfei

Or, can you delete the deploy and try again, to see it there still have this error, if you can reproduce the error, it will be better.

Hengfei

let’s talk in private. i will ping you.