No logs in Kibana logs app

[Vaishali_Gupta]

I have configured the EFK stack and logs are visible in kibana under the discover button when i create index pattern fluentd*.
But same logs are not visible in logs app when I make custom settings.

PFA the configuration done via UI for logs app source -

And the sample log available under Discover button -

Not sure whats going wrong here.

Also, checking Kibana logs, it shows this -

{"type":"error","@timestamp":"2020-02-26T10:39:02Z","tags":["connection","client","error"],"pid":1,"level":"error","error":{"message":"139989745084288:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n","name":"Error","stack":"Error: 139989745084288:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n"},"message":"139989745084288:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 46\n"}

And API response being
{
"data": null,
"errors": [{
"message": "[illegal_argument_exception] Fielddata is not supported on field [_version] of type [_version]",
"locations": [{
"line": 4,
"column": 5
}],
"path": ["source", "logEntriesAround"]
}]
}

[mattkime (Matthew Kime)]

Hello @Vaishali_Gupta

The Kibana log you posted potentially indicates a connection issue.

Can you post a har file of the failed request? That way I can see full information from the browser perspective regarding the failure.

It might be helpful to have the Kibana log lines proceeding the failure you've posted.

[Vaishali_Gupta]

Hi @mattkime, I could get logs in Kibana app. But still would like to understand one thing. Can we configure multiple sources in xpack section of logs app i.e. something like -

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: {{ template "platform.fullname" . }}
spec:
  version: {{ .Values.kibana.version }}
  count: {{ .Values.kibana.nodeCount }}
  podTemplate:
    metadata:
      annotations:
        {{- toYaml .Values.kibana.podAnnotations | nindent 8 }}
    tolerations:
      {{- toYaml .Values.kibana.tolerations | nindent 6 }}
  elasticsearchRef:
    name: {{ template "platform.fullname" . }}
  http:
    service:
      spec:
        type: LoadBalancer
  config:
    xpack:
      infra:
        sources:
          default:
            logAlias: fluent-bit-default*
            fields:
              timestamp: time
              tiebreaker: _score
              container: kubernetes.docker_id
              host: kubernetes.host
              pod: kubernetes.pod_id
          sourceA:
            logAlias: fluent-bit-sourceA*
            fields:
              timestamp: time
              tiebreaker: _score
              container: kubernetes.docker_id
              host: kubernetes.host
              pod: kubernetes.pod_id
          sourceB:
            logAlias: fluent-bit-sourceB*
            fields:
              timestamp: time
              tiebreaker: _score
              container: kubernetes.docker_id
              host: kubernetes.host
              pod: kubernetes.pod_id

If not, is there any way to configure multiple sources in logs app just like multiple index-patterns in discover option of Kibana?

[mattkime (Matthew Kime)]

What is your goal in configuring multiple sources? I think you can only define the default, but you can define multiple log aliases in comma delimited format - fluent-bit-default*,fluent-bit-sourceA*,fluent-bit-sourceB*

[Vaishali_Gupta]

I have different indices in elastic search for each user i.e. fluent-bit-sourceA for userA and fluent-bit-sourceB index for userB. I want to configure different sources in logs app for different users so that users are not filtering on their end based on index pattern. If in "Discover" app, we can define multiple index patterns, "Logs" app should also be somehow configurable with multiple sources I believe.

[mattkime (Matthew Kime)]

I'm not sure if this will meet your needs but you can set Metrics settings on a per space basis and then have different users use different spaces.

[Vaishali_Gupta]

Can this be setup while bringing up kibana? I mean how do i configure multiple spaces in kibana.yml. Can you share details around that?

[mattkime (Matthew Kime)]

Apologies for the slow response.

Here are the docs on setting up spaces - https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html

You would configure the Metrics app from within each space, in Kibana, not via kibana.yml.

[Vaishali_Gupta]

Are there some public apis or client libraries exposed to create these spaces and configure logs app and other things in a spaces?

[Vaishali_Gupta]

@mattkime Also if there is some go-client to configure spaces in kibana and configure logs app that you could point me to, that would be great.

I can see this - https://github.com/elastic/go-elasticsearch. Could not find any kibana specific library.
Not sure if it can configure spaces in kibana and logs app.

[mattkime (Matthew Kime)]

Spaces configuration is typically done through the graphical user interface. If you need to create a large number of them - and I'd be curious to how many spaces you'd need to configure - it may be possible to automate it via a custom kibana plugin.

[Vaishali_Gupta]

Hi, we need to onboard different clients to our platform and make sure no client can see logs of other client. As far as the number of spaces are concerned, lets take a ballpark number as 1000. I am doing it using Kibana APIs. But given the APIs are experimental, little doubtful about using those. Also, how many spaces can kibana handle in general?

[mattkime (Matthew Kime)]

there's a configurable limit to the number of spaces - https://www.elastic.co/guide/en/kibana/current/spaces-settings-kb.html

[Vaishali_Gupta]

So do you think that using APIs is a go to solution for this given the apis are experimental?

[mattkime (Matthew Kime)]

Its the only option as far as I can tell.