How to prevent writing user credentials into .watcher-history index?

[Martin_Becker]

I have a watch which checks the request_cache. Therefore I need to query the elasticsearch API using basic auth for the watch which looks as follows:

 PUT _xpack/watcher/watch/request-cache-watcher
     {
       "trigger": {
         "schedule": {
           "interval": "1h"
         }
       },
       "input": {
         "http": {
           "request": {
             "host": "localhost",
             "port": 9200,
             "path": "/_nodes/stats/indices/request_cache",
             "params": {
               "human": "true"
             },
             "auth": {
               "basic": {
                 "username": "request_cache_watch",
                 "password": "somepassword"
...

Reading the data from the index ".watcher-history-*" I was surprised to see username and password in the resulting answer to my query. Is there a way to prevent writing this sensitive data into the index? If not what would be the most secure way to watch the request cache?
I use Elasticsearch and Kibana 5.0 btw.

Thank you.

[jetnet]

You can use the "search" input type instead of the "http" one and get the same (I hope) stats from the .monitoring-es indices, e.g.:

  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".monitoring-es-6-*" /* would date patterns work here? */
        ],
        "types": [],
        "body": {
  "size": 0, 
  "query": {"match_all": {}},
  "aggs": {
    "hits": {
      "sum": {
        "field": "index_stats.total.query_cache.hit_count"
      }
    },
    "miss": {
      "sum": {
        "field": "index_stats.total.query_cache.miss_count"
      }
    }    
  }
}

[Martin_Becker]

Yes, this is correct and this index holds the data I am interested in. Thank you. I will investigate the contents of the monitoring index. But nonetheless I find it odd to read user credentials in the .watcher-history* index. I hope there will be a better solution than storing passwords in clear text.

[spinscale (Alexander Reelsen)]

So, running this snippet does not store the password in the watch history

PUT _xpack/watcher/watch/my_watch
{
  "input": {
    "http": {
      "request": {
        "host": "test.de",
        "port": 80,
        "path": "/",
        "auth": {
          "basic": {
            "username": "foo",
            "password": "bar"
          }
        }
      }
    }
  },
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "actions": {
    "logging": {
      "logging": {
        "text": "{{ctx}}"
      }
    }
  }
}

# search for the watch history, which does not contain the above mentioned password
GET .watcher-history-*/_search
{
  "query": {
    "match": {
      "watch_id": "my_watch"
    }
  }
}

Can you provide a full reproduction of your watch and provide the version of Elasticsearch you are using? Wondering about that behaviour.

Thanks!

--Alex

[Martin_Becker]

Yes, sure. Here you go:

Setting the watch:

PUT _xpack/watcher/watch/request-cache-watcher
{
  "trigger": {
    "schedule": {
      "interval": "5s"
    }
  },
  "input": {
    "http": {
      "request": {
        "host": "localhost",
        "port": 9200,
        "path": "/_nodes/stats/indices/request_cache",
        "params": {
          "human": "true"
        },
        "auth": {
          "basic": {
            "username": "watcher",
            "password": "asdf"
          }
        }
      }
    }
  }
}

...and query the watcher-history:

GET .watcher-history-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "watch_id": "request-cache-watcher"
          }
        }
      ]
    }
  },
  "size" : 25
}

The result (snippet):

{
    "_index": ".watcher-history-1-2017.10.19",
    "_type": "watch_record",
    "_id": "request-cache-watcher_4-2017-10-19T08:07:53.484Z",
    "_score": 0,
    "_source": {
      "watch_id": "request-cache-watcher",
      "state": "executed",
      "trigger_event": {
        "type": "schedule",
        "triggered_time": "2017-10-19T08:07:53.484Z",
        "schedule": {
          "scheduled_time": "2017-10-19T08:07:53.265Z"
        }
      },
      "input": {
        "http": {
          "request": {
            "scheme": "http",
            "host": "localhost",
            "port": 9200,
            "method": "get",
            "path": "/_nodes/stats/indices/request_cache",
            "params": {
              "human": "true"
            },
            "headers": {},
            "auth": {
              "basic": {
                "username": "watcher",
                "password": "asdf"
              }
            }
          }
        }
      },
      "condition": {
        "always": {}
      },
      "result": {
        "execution_time": "2017-10-19T08:07:53.484Z",
        "execution_duration": 2,
        "input": {
          "type": "http",
          "status": "success",
          "payload": {
            "_headers": {
              "content-length": [
                "396"
              ],
              "content-type": [
                "application/json; charset=UTF-8"
              ]
            }, ...

I use Elasticsearch version 5.0.0 and Kibana 5.0.0.

[spinscale (Alexander Reelsen)]

hey,

this has been fixed from 5.1 onwards.

--Alex