Logstash with elasticsearch input

Elastic Stack Logstash
1

Hi Masters,
I'm trying to use Logsash with "elasticsearch" input for taking specific events from winlogbeat Index and write them to another Index,
I have a problem with the "query" on the input section,
when I'm trying to query something except this:
query => '{ "query": { "query_string": { "query": "*" } } }'
im getting EROORs or Logstsash failed to start.

Examples of failures:
query => '{ "query": { "query_string": { "event_id" : "3" } } }'
query => '{ "query": { "query_string": { "event.id": 3 } } }'
query => '{"query": {"query_string": {"fields": ["event.id","content"],"query": "4625 OR 4624 OR 3"}}}'
query => '{ "query": { "match": { "event.code": "4625" } } }'
query => '{ "query": { "match": { "statuscode": 200 } }, "sort": [ "_doc" ] }'
query => '{ "query": { "match": { "event.id": 4625 } } }'
query => '{ "query": { "match": { "event.code": 4625 } }, "sort": [ "_doc" ] }'
query => '{ "query": {"query_string": {"query": "event.code:"4625"","fields": ["event.code"]}}}'
query => '{"query": {"bool":{"should":[{"term":{"host.raw":"host 1"}},{"term":{"host.raw":"host 2"}}], "must_not":{"term":{"code":"123"}}}}}'
query => '{"query": {"bool":{"should":[{"term":{"event.id":"4624"}},{"term":{"event.id":"4625"}}], "must_not":{"term":{"event.id":"123"}}}}}'

conf file:

input {
      elasticsearch {
        hosts=>["elk01:9200"]
		index => "winlogbeat-2021.04.09-000453"
        #WORK#query => '{ "query": { "query_string": { "query": "*" } } }'
        query => '{ "query": { "query_string": { "event_id" : "3" } } }'
		size => 500
        scroll => "5m"
        docinfo => true
		ca_file => '/etc/logstash3/config/certs/ca.crt'
		ssl => true
        user => 'user'
        password => '123456'
      }
    }
output {
      elasticsearch {
        index => "winlogbeat-2021.04.09-000453-copy"
		hosts => ["https://elk01:9200"]
		cacert => '/etc/logstash3/config/certs/ca.crt'
        user => 'user'
        password => '123456'
      }
    }

any help will very appreciated

2

Hi Kfiro,

Could you share some logs when you put in "Examples of failures" to see what does the error causing your logstash fail to start?

I tried on one of the query in demo site, it works.

query => '{ "query": { "match": { "event.code": "4625" } } }'

in demo site:

GET .kibana-event-log-7.10.0-000002/_search/
{
  "query": {
    "match": {
      "event.code": "4625"
    }
  }
}

Result

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}
3

Hi Kavierkoo,
thanks for the reply,

i deleted all logs and start logsash again with this conf file:
i added the logs output bellow,

input {
      elasticsearch {
        hosts=>["elk01:9200"]
		index => "winlogbeat-2021.04.09-000453"
		query => '{ "query": { "match": { "event.code": "4625" } } }'
		ca_file => '/etc/logstash3/config/certs/ca.crt'
		ssl => true
        user => 'elastic'
        password => '123456'
      }
    }
output {
      elasticsearch {
        index => "winlogbeat-2021.04.09-000453-copy"
		hosts => ["https://elk01:9200"]
		cacert => '/etc/logstash3/config/certs/ca.crt'
        user => 'elastic'
        password => '123456'
      }
    }

logstash logs output after crashing:

4

Hi Kfir,

For logstash constantly stopping after start,
Based on the logs, it seems that this is your only pipeline, logstash will start and stop after it finishes it's job as nothing else require it to do.

Just to cross check, could you also share your pipeline.yml

Can you do a search on index "winlogbeat-2021.04.09-000453-copy" to see if "event.code 4625” successfully indexed into this index?

If no, can you do a GET search on Devtools with the same query?this is to check if there's any logs with event.code 4625 also to check if syntax of the query is correct.

5

Thanks Kavierkoo,
i had confused and i didn't see the events,
it works like a charm,
do you know how can i run it continuously on the last 5 minutes every time?

6

You can take a look at "schedule" setting for elasticsearch input plugin