Enabling Slow Logging

Elastic Stack Elasticsearch
1

ES, LS, Kibana Version 5.4.0

I'm attempting to enable slow logging, as some queries are taking >10 minutes to return the last 24 hours of data. In order to remedy this, I'm attempting to enable slow logging, but unable to get the logs to populate with data. I've followed the documentation but am still not getting slow logging data.

What I've done:
Placed logging parameters on each index, the nginx-* index for example:

PUT /nginx-*/_settings
     {
        "index.search.slowlog.threshold.query.warn": "1s",
"index.search.slowlog.threshold.query.info": "1s",
"index.search.slowlog.threshold.query.debug": "1s",
"index.search.slowlog.threshold.query.trace": "1ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "10ms",
"index.search.slowlog.threshold.fetch.debug": "10ms",
"index.search.slowlog.threshold.fetch.trace": "20ms"
    }

This adds what appear to be proper config for slow logging, as can be seen from querying today's index:

GET /_cluster/settings

{
  "persistent": {},
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "cluster_concurrent_rebalance": "4",
          "node_concurrent_recoveries": "4",
          "disk": {
            "watermark": {
              "low": "95%",
              "high": "95%"
            }
          }
        }
      },
      "info": {
        "update": {
          "interval": "1m"
        }
      }
    },
    "logger": {
      "index": {
        "indexing": {
          "slowlog": "WARN"
        },
        "search": {
          "slowlog": "DEBUG"
        }
      }
    }
  }
}

And the index settings:

GET /nginx-2017.07.10/_settings

{
  "nginx-2017.07.10": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        },
        "search": {
          "slowlog": {
            "threshold": {
              "fetch": {
                "warn": "1s",
                "trace": "20ms",
                "debug": "10ms",
                "info": "10ms"
              },
              "query": {
                "warn": "1s",
                "trace": "1ms",
                "debug": "1s",
                "info": "1s"
              }
            }
          }
        },
        "indexing": {
          "slowlog": {
            "threshold": {
              "index": {
                "trace": "0s",
                "info": "5s"
              }
            }
          }
        },
        "number_of_shards": "3",
        "provided_name": "nginx-2017.07.10",
        "creation_date": "1499644778210",
        "number_of_replicas": "1",
        "uuid": "IR3V-tEZQfODxPGKFjZ4KQ",
        "version": {
          "created": "5040099"
        }
      }
    }
  }
}

I even also restarted ES. What config am I missing?

2

there is no need to change the log levels in the cluster update settings... the loglevel are just misused as thresholds for slow logs.

Does your dedicated slow logfile simply remain empty?

3

That's correct, all slow logs are completely empty.

4

Anyone have any ideas on something I may be missing? I have queries that are taking >5 minutes.

image.png1011×107 17.4 KB

I have verified slow searches should be getting indexed:

GET /nginx-2017.07.10/_settings

{
  "nginx-2017.07.17": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        },
        "search": {
          "slowlog": {
            "threshold": {
              "fetch": {
                "warn": "1ms",
                "trace": "20ms",
                "debug": "10ms",
                "info": "10ms"
              },
              "query": {
                "warn": "1ms",
                "trace": "1ms",
                "debug": "1ms",
                "info": "1ms"
              }
            }
          }
        },
        "number_of_shards": "3",
        "provided_name": "nginx-2017.07.17",
        "creation_date": "1500249568478",
        "number_of_replicas": "1",
        "uuid": "ldnxrEqVQmKnSTp0bGHsSw",
        "version": {
          "created": "5040099"
        }
      }
    }
  }
}

Are there any other settings I need to update in order to enable slow logging?

5

It appears as though slow search logs are working. I was looking for them on the client node which also runs Kibana, however it appears that the slow logs are populated on a per-node basis, logged locally on whichever node performs a slow search.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.