# How to save the data obtained by using the http\_poller input plug-in of logstash to elasticsearch? **URL:** \ **Category:** Logstash\ **Created:** [August 24, 2021, 7:16am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304 "2021-08-24T07:16:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![zhyp](https://avatars.discourse-cdn.com/v4/letter/z/91b2a8/32.png) [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Post date:** [August 24, 2021, 7:16am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/1 "2021-08-24T07:16:42Z")
Hi 🙂 I use version 7.12.0. 1. I used http\_poller to call the SQL API of elasticsearch and got some statistics. input { http\_poller { urls =\> { item =\> { method =\> post url =\> "[http://localhost:9205/\_sql?format=csv](http://localhost:9205/_sql?format=csv)" body =\> '{"query": "SELECT \u0027test\u0027 AS data\_type, time, sum(count) AS count FROM test group by time"}' headers =\> { "content-type" =\> "application/json" } } } codec =\> "plain" schedule =\> { cron =\> "\*/2 \* \* \* \* \*"} } } It can get data data\_type,time,count test,2021-08-10,1 test,2021-08-11,2 2. I want to match each line in the filter and save it to elasticsearch. My configuration is like this, but it doesn't work. filter { grok { match =\> { "message" =\> "test,%{TIMESTAMP:time},%{NUMBER:count}" } add\_field =\> { "time" =\> "%{time}" "count" =\> "%{count}" } } mutate { remove\_field =\> ["@timestamp", "@version"] } } 3、The following is the output configuration. elasticsearch { ecs\_compatibility =\> disabled action =\> "update" doc\_as\_upsert =\> true hosts =\> ["localhost:9205"] index =\> "demo" document\_id =\> "%{time}" } ---
**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\ **Post date:** [August 24, 2021, 6:23pm UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/2 "2021-08-24T18:23:05Z")
> [@zhyp](#): > > %{TIMESTAMP:time} This is not a valid GROK pattern. You should use: `test,%{YEAR}-%{MONTHNUM}-%{MONTHDAY},%{NUMBER:count}` Then you can combine YEAR, MONTHNUM and MONTHDAY into a single field and combine it into the @timestamp field using the `date` filter > date { > match =\> ["%{YEAR}-%{MONTHNUM}-%{MONTHDAY}", "yyyy-MM-dd"] > } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 24, 2021, 6:31pm UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/3 "2021-08-24T18:31:57Z")
Or use a custom pattern ``` grok { pattern_definitions => { "TIMESTAMP" => "{YEAR}-%{MONTHNUM}-%{MONTHDAY}" } match => { "message" => "test,%{TIMESTAMP:time},%{NUMBER:count}" } } ``` The add\_field option is not needed. ---
**Author:** ![zhyp](https://avatars.discourse-cdn.com/v4/letter/z/91b2a8/32.png) [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Post date:** [August 25, 2021, 2:29am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/4 "2021-08-25T02:29:57Z")
Sorry, I got the data format wrong.The data format is JSON. {"message":"data\_type,time,count\r\ntest,2021-08-10,1.0\r\n","@timestamp":"2021-08-25T02:11:38.563Z","@version":"1"} Although I wanted to set it to TXT format at first, it will make mistakes. Content-Type header [text/plain; charset=ISO-8859-1] is not supported ![企业微信截图_20210825102858](https://us1.discourse-cdn.com/elastic/original/3X/c/e/cecfd3db8e4f47beaec98604fbf45c0e0feacec7.png) ---
**Author:** ![zhyp](https://avatars.discourse-cdn.com/v4/letter/z/91b2a8/32.png) [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Post date:** [August 25, 2021, 2:59am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/5 "2021-08-25T02:59:25Z")
I tried all the above methods. Here are the results. {"message":"data\_type,time,count\r\ntest,2021-08-10,1.0\r\n","tags":["\_grokparsefailure"]} ---
**Author:** ![zhyp](https://avatars.discourse-cdn.com/v4/letter/z/91b2a8/32.png) [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Post date:** [August 25, 2021, 3:06am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/6 "2021-08-25T03:06:01Z")
To achieve this summation, I tried using the aggregate filter. However, I can't get the results correctly. Can you help me? > [@Logstash aggregate filter sum incorrect](https://discuss.elastic.co/t/logstash-aggregate-filter-sum-incorrect/282199): > > Hi, slightly_smiling_face Here is my es data.I can query them correctly. {"count":1,"time":"2021-08-10T00:15:00.000+08:00"} {"count":2,"time":"2021-08-10T00:30:00.000+08:00"} I want to sum the count by day. Here are the results I want. {"count":3,"time":"2021-08-10T00:00:00.000+08:00"} Here is the filter configuration.I use version 7.12.0.I refer to the official website example.[Aggregate filter plugin | Logstash Reference [7.12] | Elastic](https://www.elastic.co/guide/en/logstash/7.12/plugins-filters-aggregate.html#plugins-filters-aggregate-example5) filter { ruby { code =\> "event.set('date', e… ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 22, 2021, 3:07am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304/7 "2021-09-22T03:07:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.