# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=534 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 535 --- ## [Match all query not working elastic search](https://discuss.elastic.co/t/match-all-query-not-working-elastic-search/184507)
**Author:** [@Smack\_Alpha](https://discuss.elastic.co/u/Smack_Alpha)\ **Replies:** 0\ **Last updated:** [June 6, 2019, 7:14am UTC](https://discuss.elastic.co/t/match-all-query-not-working-elastic-search/184507 "2019-06-06T07:14:32Z")
I am writing a python program to get the data from the elastic search indices. I want to get the data based on the match query which i indicated is upto 25. I want the first 25 data's. The data's in my index is 10842. Bu… --- ## [Putting server time in the request http poller body logstash](https://discuss.elastic.co/t/putting-server-time-in-the-request-http-poller-body-logstash/184010)
**Author:** [@Hiven](https://discuss.elastic.co/u/Hiven)\ **Replies:** 4\ **Last updated:** [June 6, 2019, 6:59am UTC](https://discuss.elastic.co/t/putting-server-time-in-the-request-http-poller-body-logstash/184010 "2019-06-06T06:59:15Z")
Hello, I have a logstash confg with uses http poller. The big problem I have is that, I need the current date sent in the body so the API knows what data to return. It doesn't work with relative dates, it needs the exac… --- ## [Error while running multiple pipelines in logstash on centOS 7](https://discuss.elastic.co/t/error-while-running-multiple-pipelines-in-logstash-on-centos-7/182423)
**Author:** [@Vicky\_S](https://discuss.elastic.co/u/Vicky_S)\ **Replies:** 1\ **Last updated:** [June 6, 2019, 6:38am UTC](https://discuss.elastic.co/t/error-while-running-multiple-pipelines-in-logstash-on-centos-7/182423 "2019-06-06T06:38:48Z")
Hi, Iam running multiple pipelines on centOS 7. I'm moving data from mySQl to elasticsearch. One or two pipelines seem to work fine but when I run more than three pipelines some pipelines get aborted showing error messag… --- ## [Capturing only first group in Logstash regex](https://discuss.elastic.co/t/capturing-only-first-group-in-logstash-regex/184214)
**Author:** [@f4fe5bb60e3686943680](https://discuss.elastic.co/u/f4fe5bb60e3686943680)\ **Replies:** 3\ **Last updated:** [June 6, 2019, 6:18am UTC](https://discuss.elastic.co/t/capturing-only-first-group-in-logstash-regex/184214 "2019-06-06T06:18:05Z")
Hello everyone! Can someone explain me next please. I have log: "bool":{"must":\[{"bool":{"should":\[{"codesearch\_score\_query":{"query\_id":"9baec4c4-d910-4023-899e-5a580bfbd3a3","query":{"term":{"content":"copy"}}}},{"… --- ## [Declaring variables in output.conf and using those variable values in conditional statements](https://discuss.elastic.co/t/declaring-variables-in-output-conf-and-using-those-variable-values-in-conditional-statements/184140)
**Author:** [@Navin\_Krishnamurthi](https://discuss.elastic.co/u/Navin_Krishnamurthi)\ **Replies:** 2\ **Last updated:** [June 6, 2019, 4:55am UTC](https://discuss.elastic.co/t/declaring-variables-in-output-conf-and-using-those-variable-values-in-conditional-statements/184140 "2019-06-06T04:55:08Z")
Hi, I have following configuration in the output.conf output { if ((\[column1\] == "valc11" and \[column2\] == "valc12") or (\[column3\] == "valc13" and \[column1\] == "valc11") or (\[column2\] == "valc12" and \[column3\] == "va… --- ## [Duplicate and forward logs to a new instance of Elasticsearch](https://discuss.elastic.co/t/duplicate-and-forward-logs-to-a-new-instance-of-elasticsearch/182819)
**Author:** [@Camilo\_Diaz](https://discuss.elastic.co/u/Camilo_Diaz)\ **Replies:** 6\ **Last updated:** [June 5, 2019, 11:10pm UTC](https://discuss.elastic.co/t/duplicate-and-forward-logs-to-a-new-instance-of-elasticsearch/182819 "2019-06-05T23:10:42Z")
Hi, I am fairly new to ELK and I am seeking some advice. I have inherited an ELK stack which is processing logs from multiple sources and I am looking into forwarding some of those logs to another instance of Elasticsea… --- ## [Use of grok and regular expressions](https://discuss.elastic.co/t/use-of-grok-and-regular-expressions/184255)
**Author:** [@Camilo\_Franco](https://discuss.elastic.co/u/Camilo_Franco)\ **Replies:** 6\ **Last updated:** [June 5, 2019, 9:55pm UTC](https://discuss.elastic.co/t/use-of-grok-and-regular-expressions/184255 "2019-06-05T21:55:54Z")
Hello I'm sending an asterisk log from filebeat to logstash, The format of the logs is as follows. \[May 19 14:57:19\] NOTICE \[8583\] chan\_sip.c: Registration from '\' failed for '91.214.44.144: 27… --- ## [Logstash HTTP plugin dynamic URL](https://discuss.elastic.co/t/logstash-http-plugin-dynamic-url/184451)
**Author:** [@hive](https://discuss.elastic.co/u/hive)\ **Replies:** 1\ **Last updated:** [June 5, 2019, 9:15pm UTC](https://discuss.elastic.co/t/logstash-http-plugin-dynamic-url/184451 "2019-06-05T21:15:19Z")
Dynamic URL with below does not work. When I replace the "oid" with a constant it works fine. Can I have values changed in URL? Thanks output{ http { format=\>"json" http\_method=\>"post" url=\>"http://localhost:9200/… --- ## [My bind9 dns logs dont look like your dns logs, is it filebeat? logstash?](https://discuss.elastic.co/t/my-bind9-dns-logs-dont-look-like-your-dns-logs-is-it-filebeat-logstash/184433)
**Author:** [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\ **Replies:** 0\ **Last updated:** [June 5, 2019, 5:47pm UTC](https://discuss.elastic.co/t/my-bind9-dns-logs-dont-look-like-your-dns-logs-is-it-filebeat-logstash/184433 "2019-06-05T17:47:52Z")
hi, i'm trying to pull in bind9 dns logs into elasticsearch.. my workflow is this: filebeat transfers bind9 logs to logstash logstash dumps them into a rabbitmq queue another logstash instance on another machine pulls… --- ## [S3 to elastic](https://discuss.elastic.co/t/s3-to-elastic/184388)
**Author:** [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\ **Replies:** 3\ **Last updated:** [June 5, 2019, 6:00pm UTC](https://discuss.elastic.co/t/s3-to-elastic/184388 "2019-06-05T18:00:43Z")
Hi , I am trying to push log files from some devices to s3 And from s3 to elasticsearch using logstash and visualize in kibana. Now I am doing it via ec2 instance from aws. But when I POST \[{"SessionID":"8686720203… --- ## [Elasticsearch output plugin .. with script not working](https://discuss.elastic.co/t/elasticsearch-output-plugin-with-script-not-working/184431)
**Author:** [@hive](https://discuss.elastic.co/u/hive)\ **Replies:** 0\ **Last updated:** [June 5, 2019, 5:41pm UTC](https://discuss.elastic.co/t/elasticsearch-output-plugin-with-script-not-working/184431 "2019-06-05T17:41:02Z")
Below UPADTE using script works fine using REST client http://localhost:9200/my/associate/54630935\_105471/\_update?pretty { "upsert": { "positions": \[ \] }, "script": { "params": { "pos": { "eff\_date\_end": "4000-… --- ## [Normalize the logs in logstash](https://discuss.elastic.co/t/normalize-the-logs-in-logstash/184427)
**Author:** [@farida](https://discuss.elastic.co/u/farida)\ **Replies:** 0\ **Last updated:** [June 5, 2019, 5:28pm UTC](https://discuss.elastic.co/t/normalize-the-logs-in-logstash/184427 "2019-06-05T17:28:07Z")
Hi, I want to know if we can normalize the different logs in the configuration of logstash or we are obliged to create a parser in python to normalize them? --- ## [Logstash Index Name as "%{\[@metadata\]\[beat\]}-%{\[@metadata\]\[version\]}"](https://discuss.elastic.co/t/logstash-index-name-as-metadata-beat-metadata-version/184225)
**Author:** [@robscott27](https://discuss.elastic.co/u/robscott27)\ **Replies:** 1\ **Last updated:** [June 5, 2019, 5:19pm UTC](https://discuss.elastic.co/t/logstash-index-name-as-metadata-beat-metadata-version/184225 "2019-06-05T17:19:25Z")
How can I get the index name for netflow to be set correctly upon creation? Some background: Prior to using Netflow, I've been using Logstash to send beats data into Elasticsearch - Filebeat, Auditbeat, Metricbeat, etc… --- ## [Date filter, match, target and index name](https://discuss.elastic.co/t/date-filter-match-target-and-index-name/184400)
**Author:** [@arikfre](https://discuss.elastic.co/u/arikfre)\ **Replies:** 2\ **Last updated:** [June 5, 2019, 5:07pm UTC](https://discuss.elastic.co/t/date-filter-match-target-and-index-name/184400 "2019-06-05T17:07:41Z")
Hi, I have jdbc input that selects from MySQL. One of the fields is called report\_date from type datetime. filter { date { match =\> \[ "report\_date", "yyyy-MM-dd HH:mm:ss" \] target =\> "report\_date" } } output { e… --- ## [Multiple translate filters on the same event field in logstash](https://discuss.elastic.co/t/multiple-translate-filters-on-the-same-event-field-in-logstash/184047)
**Author:** [@aamirg](https://discuss.elastic.co/u/aamirg)\ **Replies:** 2\ **Last updated:** [June 5, 2019, 4:33pm UTC](https://discuss.elastic.co/t/multiple-translate-filters-on-the-same-event-field-in-logstash/184047 "2019-06-05T16:33:27Z")
I am using the following translate filters in my logstash configuration file. I have two separate YAML files that serve as the lookup dictionaries. The logstash event field referenced is the same for both filters but the… --- ## [JSON codec polluting logs with ERROR messages if input is plain string](https://discuss.elastic.co/t/json-codec-polluting-logs-with-error-messages-if-input-is-plain-string/184356)
**Author:** [@Ivan\_Smurygin](https://discuss.elastic.co/u/Ivan_Smurygin)\ **Replies:** 5\ **Last updated:** [June 5, 2019, 3:46pm UTC](https://discuss.elastic.co/t/json-codec-polluting-logs-with-error-messages-if-input-is-plain-string/184356 "2019-06-05T15:46:54Z")
Hello. I have a question about Generally we use the codec and it works fine. Our application has logs like { "profile": "dev", "logging-context": "default", "hostname": "xxx", "app": "xxx", "@time… --- ## [Time outside range and possible syntax error with logstash-influxdb plugin](https://discuss.elastic.co/t/time-outside-range-and-possible-syntax-error-with-logstash-influxdb-plugin/184404)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 0\ **Last updated:** [June 5, 2019, 3:07pm UTC](https://discuss.elastic.co/t/time-outside-range-and-possible-syntax-error-with-logstash-influxdb-plugin/184404 "2019-06-05T15:07:17Z")
Hi, I have been trying to send data from logstash to influxdb through output inlfluxdb plugin and i see parsing error and time out of range error Non recoverable exception while writing to InfluxDB {:exception=\>#\ **Author:** [@BrunoSdeMenezes](https://discuss.elastic.co/u/BrunoSdeMenezes)\ **Replies:** 3\ **Last updated:** [June 5, 2019, 1:41pm UTC](https://discuss.elastic.co/t/create-an-index-with-aggregations/183570 "2019-06-05T13:41:40Z") Hi, I need to create a new index from a previous index that exists but only with aggregations and some metrics. For example: I have a field called "status\_contrato" which is a string, and need to bring the percentile of… --- ## [Adding additional parameters to jms .yml file](https://discuss.elastic.co/t/adding-additional-parameters-to-jms-yml-file/184348)
**Author:** [@karolsputo](https://discuss.elastic.co/u/karolsputo)\ **Replies:** 0\ **Last updated:** [June 5, 2019, 12:00pm UTC](https://discuss.elastic.co/t/adding-additional-parameters-to-jms-yml-file/184348 "2019-06-05T12:00:10Z")
Hi, I'm trying to access Solace from Logstash. I've set up a connection, but on Solace side I see that I'm connecting to the wrong VPN (default one). To establish a working connection I need to pass additional parameter… --- ## [Not add field if jdbc\_static plugin lookup fail](https://discuss.elastic.co/t/not-add-field-if-jdbc-static-plugin-lookup-fail/184081)
**Author:** [@Izek](https://discuss.elastic.co/u/Izek)\ **Replies:** 3\ **Last updated:** [June 5, 2019, 10:55am UTC](https://discuss.elastic.co/t/not-add-field-if-jdbc-static-plugin-lookup-fail/184081 "2019-06-05T10:55:10Z")
I have a jdbc\_static config as below. How can I not add field if lookup failed or don't have matched record. Can I specify if else statement for add fields? jdbc\_static { loaders =\> \[ { id =\> "host… --- ## [Logstash and Grok Matching](https://discuss.elastic.co/t/logstash-and-grok-matching/183987)
**Author:** [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\ **Replies:** 10\ **Last updated:** [June 5, 2019, 9:53am UTC](https://discuss.elastic.co/t/logstash-and-grok-matching/183987 "2019-06-05T09:53:04Z")
Hi My question is regarding Logstash and Grok. If I have two filters in two different files filter { if \[message\] =~ Regex grok { match =\> \[PATTERNA\] } filter { if \[message\] =~ Regex grok { match =\> \[PATTERNB\] } If … --- ## [Jdbc\_static and mssql server](https://discuss.elastic.co/t/jdbc-static-and-mssql-server/184170)
**Author:** [@Sam\_Kisada](https://discuss.elastic.co/u/Sam_Kisada)\ **Replies:** 1\ **Last updated:** [June 5, 2019, 9:00am UTC](https://discuss.elastic.co/t/jdbc-static-and-mssql-server/184170 "2019-06-05T09:00:21Z")
Hi! I trying to get some fields through jdbc\_static: if "grokked" not in \[tags\] { grok { match =\> \[ "message", "{%{DATESTAMP\_EVENTLOG:Date},%{INT:ID}:%{GREEDYDATA:GUID} \] add\_tag =\> \[ "grokked"\] tag\_on\_f… --- ## [Logstash tiwwter](https://discuss.elastic.co/t/logstash-tiwwter/184252)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 1\ **Last updated:** [June 5, 2019, 8:41am UTC](https://discuss.elastic.co/t/logstash-tiwwter/184252 "2019-06-05T08:41:11Z")
i am isuning logstash to have some marketing data from tweeter, but i saw taht when i stop the command "logstash -f tweeter.conf" all the data is removed and i cant see anytinhg in kibana dicover, what is the solution t… --- ## [ES output plugin- Update Action with 7.x](https://discuss.elastic.co/t/es-output-plugin-update-action-with-7-x/184188)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 2\ **Last updated:** [June 5, 2019, 8:36am UTC](https://discuss.elastic.co/t/es-output-plugin-update-action-with-7-x/184188 "2019-06-05T08:36:38Z")
Hi All I am using latest version of 7.x LS, and using ES output plugin, i am in a state to use upsert, that is update action. Can someone confirm that update action earlier was delete and recreate records and now impro… --- ## [Logstash communication fails to ES with x-pack with user logstash\_internal](https://discuss.elastic.co/t/logstash-communication-fails-to-es-with-x-pack-with-user-logstash-internal/184182)
**Author:** [@dsmteam](https://discuss.elastic.co/u/dsmteam)\ **Replies:** 2\ **Last updated:** [June 5, 2019, 8:17am UTC](https://discuss.elastic.co/t/logstash-communication-fails-to-es-with-x-pack-with-user-logstash-internal/184182 "2019-06-05T08:17:04Z")
I'm trying to setup latest ELK stack with x-pack but i'm failing on the last step which is communication between logstash and ES I have put username/password in my output.conf file for logstash and it's properly created… --- ## [Refer to multiple templates in logstash elastic output?](https://discuss.elastic.co/t/refer-to-multiple-templates-in-logstash-elastic-output/184267)
**Author:** [@ankh](https://discuss.elastic.co/u/ankh)\ **Replies:** 0\ **Last updated:** [June 5, 2019, 2:14am UTC](https://discuss.elastic.co/t/refer-to-multiple-templates-in-logstash-elastic-output/184267 "2019-06-05T02:14:43Z")
We specify a template file to use in our logstash elastic output definition in our conf files. We currently have one template file per index, and each contains some common field definitions. We'd like to simplify this an… --- ## [Logstach for facebook](https://discuss.elastic.co/t/logstach-for-facebook/184254)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 2\ **Last updated:** [June 5, 2019, 12:47am UTC](https://discuss.elastic.co/t/logstach-for-facebook/184254 "2019-06-05T00:47:24Z")
i try to connect logstash to facebook, is there any script ".conf" to do that, i see an example on github but i dont want to read data from only a signle page, i want to collect from all pages without specifying a single… --- ## [Sprintf +FORMAT Field Reference](https://discuss.elastic.co/t/sprintf-format-field-reference/184226)
**Author:** [@josh\_jacques](https://discuss.elastic.co/u/josh_jacques)\ **Replies:** 2\ **Last updated:** [June 4, 2019, 8:46pm UTC](https://discuss.elastic.co/t/sprintf-format-field-reference/184226 "2019-06-04T20:46:23Z")
Under the sprintf section of Accessing Event Data and Fields in the Configuration in the logstash reference. There is mention of being able to use a +FORMAT to format the @timestamp..in their example they mention using … --- ## [Using 7.1 Can't set @timestamp from message](https://discuss.elastic.co/t/using-7-1-cant-set-timestamp-from-message/184205)
**Author:** [@pgervais](https://discuss.elastic.co/u/pgervais)\ **Replies:** 2\ **Last updated:** [June 4, 2019, 7:56pm UTC](https://discuss.elastic.co/t/using-7-1-cant-set-timestamp-from-message/184205 "2019-06-04T19:56:34Z")
I have a simple logstash definition file as shown below. I can parse my message date properly with no dateparse error shown. I cannot set the @timestamp with the ObsDate from my message input stream. As well, the ruby… --- ## [Twitter client error - undefined method](https://discuss.elastic.co/t/twitter-client-error-undefined-method/184206)
**Author:** [@mickey\_lubs](https://discuss.elastic.co/u/mickey_lubs)\ **Replies:** 3\ **Last updated:** [June 4, 2019, 6:56pm UTC](https://discuss.elastic.co/t/twitter-client-error-undefined-method/184206 "2019-06-04T18:56:56Z")
Hi, I have been facing issues using the "locations" parameter in logstash twitter inputs. When i define my longitude and latitude it fails logstash with below error: This is how i define it: locations =\> " -26.195246,… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=533) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=535)