How to save the data obtained by using the http_poller input plug-in of logstash to elasticsearch?

Elastic Stack Logstash
1

Hi
:slightly_smiling_face:

I use version 7.12.0.

  1. I used http_poller to call the SQL API of elasticsearch and got some statistics.
    input {
    http_poller {
    urls => {
    item => {
    method => post
    url => "http://localhost:9205/_sql?format=csv"
    body => '{"query": "SELECT \u0027test\u0027 AS data_type, time, sum(count) AS count FROM test group by time"}'
    headers => {
    "content-type" => "application/json"
    }
    }
    }
    codec => "plain"
    schedule => { cron => "*/2 * * * * *"}
    }
    }
    It can get data
    data_type,time,count
    test,2021-08-10,1
    test,2021-08-11,2

  2. I want to match each line in the filter and save it to elasticsearch. My configuration is like this, but it doesn't work.

filter {
grok {
match => { "message" => "test,%{TIMESTAMP:time},%{NUMBER:count}" }
add_field => {
"time" => "%{time}"
"count" => "%{count}"
}
}
mutate {
remove_field => ["@timestamp", "@version"]
}
}

3、The following is the output configuration.

elasticsearch {
ecs_compatibility => disabled
action => "update"
doc_as_upsert => true
hosts => ["localhost:9205"]
index => "demo"
document_id => "%{time}"
}

2

This is not a valid GROK pattern.

You should use:
test,%{YEAR}-%{MONTHNUM}-%{MONTHDAY},%{NUMBER:count}

Then you can combine YEAR, MONTHNUM and MONTHDAY into a single field and combine it into the @timestamp field using the
date filter

date {
match => [ "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}", "yyyy-MM-dd" ]
}

3

Or use a custom pattern

grok {
    pattern_definitions => { "TIMESTAMP" => "{YEAR}-%{MONTHNUM}-%{MONTHDAY}" }
    match => { "message" => "test,%{TIMESTAMP:time},%{NUMBER:count}" }
}

The add_field option is not needed.

4

Sorry, I got the data format wrong.The data format is JSON.

{"message":"data_type,time,count\r\ntest,2021-08-10,1.0\r\n","@timestamp":"2021-08-25T02:11:38.563Z","@version":"1"}

Although I wanted to set it to TXT format at first, it will make mistakes.

Content-Type header [text/plain; charset=ISO-8859-1] is not supported

企业微信截图_20210825102858
企业微信截图_20210825102858868×367 25.3 KB

5

I tried all the above methods. Here are the results.

{"message":"data_type,time,count\r\ntest,2021-08-10,1.0\r\n","tags":["_grokparsefailure"]}

6

To achieve this summation, I tried using the aggregate filter. However, I can't get the results correctly. Can you help me?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.