# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=820 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 821 --- ## [Elasticsearch query to pull the record based on max timestamp](https://discuss.elastic.co/t/elasticsearch-query-to-pull-the-record-based-on-max-timestamp/120131)
**Author:** [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Replies:** 5\ **Last updated:** [February 16, 2018, 1:58pm UTC](https://discuss.elastic.co/t/elasticsearch-query-to-pull-the-record-based-on-max-timestamp/120131 "2018-02-16T13:58:35Z")
Hi, I have 4 records in the elasticsearch DB for the ddocname "CNT1882742" for the below query Logstash elasticsearch query: query =\> '{ "query": {"match": { "ddocname" :"CNT1882742"} }, "sort": \["@timestamp"\] }' Quer… --- ## [Dead Letter Queue Pipeline is not creating Index](https://discuss.elastic.co/t/dead-letter-queue-pipeline-is-not-creating-index/120189)
**Author:** [@manya12](https://discuss.elastic.co/u/manya12)\ **Replies:** 0\ **Last updated:** [February 16, 2018, 1:56pm UTC](https://discuss.elastic.co/t/dead-letter-queue-pipeline-is-not-creating-index/120189 "2018-02-16T13:56:32Z")
Hey @RobBavey I am trying to implement dead letter queue in Logstash. I have made required changes in my logstash.yml and can see logs in my path.data/dead\_letter\_queue/main/.log file. I am running logstash through doc… --- ## [NetFlow not received in Elasticsearch/Kibana](https://discuss.elastic.co/t/netflow-not-received-in-elasticsearch-kibana/120185)
**Author:** [@gittinfunky](https://discuss.elastic.co/u/gittinfunky)\ **Replies:** 0\ **Last updated:** [February 16, 2018, 1:48pm UTC](https://discuss.elastic.co/t/netflow-not-received-in-elasticsearch-kibana/120185 "2018-02-16T13:48:30Z")
I a not receiving any NetFlow information in my ELK Stack and dont know why. ELK Stack setup (version 6.2.1) Cisco 2811 router generating NetFlow info logstash.conf file located in /etc/logstash/conf.d input { udp … --- ## [Datadog Metric Output Plugin](https://discuss.elastic.co/t/datadog-metric-output-plugin/120181)
**Author:** [@vtorre](https://discuss.elastic.co/u/vtorre)\ **Replies:** 0\ **Last updated:** [February 16, 2018, 1:38pm UTC](https://discuss.elastic.co/t/datadog-metric-output-plugin/120181 "2018-02-16T13:38:42Z")
Hi, I'm trying to parse log files with losgash and create metrics in Datadog. I want to create "counter" or "count" Datadog metric types increasing by one for each log line parsed. output { datadog\_metrics { api\_… --- ## [Text file without line split](https://discuss.elastic.co/t/text-file-without-line-split/120168)
**Author:** [@neerajsql](https://discuss.elastic.co/u/neerajsql)\ **Replies:** 0\ **Last updated:** [February 16, 2018, 12:36pm UTC](https://discuss.elastic.co/t/text-file-without-line-split/120168 "2018-02-16T12:36:03Z")
Experts, Which filter should I use for below sample log to get data in 4 column like Date,Port,Status,Message. I am facing problem to split first then separate. 2016-07-13 11:00:00,000 \[http-bio-80-exec-3846\] INFO com… --- ## [Multiple \_grokparsefaillure issue](https://discuss.elastic.co/t/multiple-grokparsefaillure-issue/120104)
**Author:** [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Replies:** 1\ **Last updated:** [February 16, 2018, 11:55am UTC](https://discuss.elastic.co/t/multiple-grokparsefaillure-issue/120104 "2018-02-16T11:55:14Z")
I am pulling access logs from 2 machines (BPM00, BPM01) via filebeat as log shipper everything was fine until yesterday the logs. In below image I have applied filters avaeragetime, path, source, host. In this the hos… --- ## [Filebeat to Logstash Error publishing events](https://discuss.elastic.co/t/filebeat-to-logstash-error-publishing-events/120038)
**Author:** [@pavanssetty](https://discuss.elastic.co/u/pavanssetty)\ **Replies:** 1\ **Last updated:** [February 16, 2018, 9:55am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-error-publishing-events/120038 "2018-02-16T09:55:35Z")
I'm running filebeat and logstash on windows server 2012. I am constantly seeing below error messages in the Filebeat log. I guess my filebeat is not able to publish all records to logstash as I can see some records in … --- ## [Not able to query using the elastic search filter](https://discuss.elastic.co/t/not-able-to-query-using-the-elastic-search-filter/120023)
**Author:** [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Replies:** 2\ **Last updated:** [February 16, 2018, 9:28am UTC](https://discuss.elastic.co/t/not-able-to-query-using-the-elastic-search-filter/120023 "2018-02-16T09:28:23Z")
Hi, I have the below entries for the record "ddocname": "CNT1882742" and I have to extract the ddoctitle of the same using the elastic search filter. { "\_index": "test-m-docs", "\_type": "data", "\_id": "AWF0dLEQ\_An67… --- ## [How to prevent logging at logstash for /var/log/messages](https://discuss.elastic.co/t/how-to-prevent-logging-at-logstash-for-var-log-messages/117950)
**Author:** [@david\_k](https://discuss.elastic.co/u/david_k)\ **Replies:** 1\ **Last updated:** [February 16, 2018, 8:48am UTC](https://discuss.elastic.co/t/how-to-prevent-logging-at-logstash-for-var-log-messages/117950 "2018-02-16T08:48:59Z")
Hi all logstash data is written to / var / log / messages. This causes the server capacity to continue to grow. Is there a way to prevent this data from being written to / var / log / messages? Thank you --- ## [Logstash-input-zeromq build fails](https://discuss.elastic.co/t/logstash-input-zeromq-build-fails/120123)
**Author:** [@pmr](https://discuss.elastic.co/u/pmr)\ **Replies:** 0\ **Last updated:** [February 16, 2018, 8:38am UTC](https://discuss.elastic.co/t/logstash-input-zeromq-build-fails/120123 "2018-02-16T08:38:34Z")
https://travis-ci.org/logstash-plugins/logstash-input-zeromq Also the plugin is not listed in the current (6.2.1) version of logstash --- ## [Append data to each line](https://discuss.elastic.co/t/append-data-to-each-line/119936)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 9\ **Last updated:** [February 16, 2018, 7:41am UTC](https://discuss.elastic.co/t/append-data-to-each-line/119936 "2018-02-16T07:41:56Z")
Ingesting XML data which appears like: \\\\\\ \\\\\\ \\\\\\ \\\ **Author:** [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\ **Replies:** 3\ **Last updated:** [February 16, 2018, 7:41am UTC](https://discuss.elastic.co/t/skip-header/119828 "2018-02-16T07:41:39Z") =============================================================================================== DATE TIME | COMPONENT |SVR| THREAD NAME | MESSAGE 14/02/2018 13:11:34:722|MessagingRepository |IN… --- ## [How to read XML format within a .log file](https://discuss.elastic.co/t/how-to-read-xml-format-within-a-log-file/119999)
**Author:** [@garcesdavid](https://discuss.elastic.co/u/garcesdavid)\ **Replies:** 3\ **Last updated:** [February 16, 2018, 7:24am UTC](https://discuss.elastic.co/t/how-to-read-xml-format-within-a-log-file/119999 "2018-02-16T07:24:37Z")
Hi all :slight\_smile: I have an app that writes XML format in a .log file per transaction it receive. So basically, this is the XML format: \ \
**Author:** [@shahidg](https://discuss.elastic.co/u/shahidg)\ **Replies:** 3\ **Last updated:** [February 16, 2018, 7:22am UTC](https://discuss.elastic.co/t/how-logstash-come-to-know-about-new-entries-in-log-file/119983 "2018-02-16T07:22:42Z") I am thinking to use logstash, but I have a question that how log stash come to know about the latest entries in log file ? --- ## [Setting patterns\_dir for grok relatively?](https://discuss.elastic.co/t/setting-patterns-dir-for-grok-relatively/117471)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 3\ **Last updated:** [February 15, 2018, 11:53pm UTC](https://discuss.elastic.co/t/setting-patterns-dir-for-grok-relatively/117471 "2018-02-15T23:53:38Z")
Hi, can the patterns\_dir of grok command be set recursively? How? Can I set up a default folder / file for patterns, so that there is no need to set them in each grok filter? Thanks, Andreas --- ## [Logstash reffdering to old index for the mapping](https://discuss.elastic.co/t/logstash-reffdering-to-old-index-for-the-mapping/120079)
**Author:** [@nsiva1](https://discuss.elastic.co/u/nsiva1)\ **Replies:** 0\ **Last updated:** [February 15, 2018, 9:53pm UTC](https://discuss.elastic.co/t/logstash-reffdering-to-old-index-for-the-mapping/120079 "2018-02-15T21:53:49Z")
I am facing an issue with existing mapping. In few occations mapping type some times referring current settings and some times referring to old mapping due to that data is spilling over so the graphs are getting floating… --- ## [Specify the type when making the config for the file](https://discuss.elastic.co/t/specify-the-type-when-making-the-config-for-the-file/120072)
**Author:** [@Pororo](https://discuss.elastic.co/u/Pororo)\ **Replies:** 0\ **Last updated:** [February 15, 2018, 9:26pm UTC](https://discuss.elastic.co/t/specify-the-type-when-making-the-config-for-the-file/120072 "2018-02-15T21:26:20Z")
Hi, For now, my json file has one attribute called "time" and I did not set the json filter in config file. I directly imported the data into elasticsearch. After that, I checked the mapping of the index I made for thos… --- ## [Log Encoding message incorrect](https://discuss.elastic.co/t/log-encoding-message-incorrect/119546)
**Author:** [@shri](https://discuss.elastic.co/u/shri)\ **Replies:** 8\ **Last updated:** [February 15, 2018, 8:04pm UTC](https://discuss.elastic.co/t/log-encoding-message-incorrect/119546 "2018-02-15T20:04:55Z")
Hello, Im new to elk. I'm using multiple inputs i) tcp ii) beats in my logstash. i) For tcp , using the spring pattern with in the code itself and simply default value of logstash server IP as below And able to see… --- ## [Monitoring Logstash running from docker container](https://discuss.elastic.co/t/monitoring-logstash-running-from-docker-container/120049)
**Author:** [@weili](https://discuss.elastic.co/u/weili)\ **Replies:** 0\ **Last updated:** [February 15, 2018, 6:05pm UTC](https://discuss.elastic.co/t/monitoring-logstash-running-from-docker-container/120049 "2018-02-15T18:05:00Z")
Logstash 5 supports node stats monitoring https://www.elastic.co/guide/en/logstash/5.0/node-stats-api.html However, I am having trouble making it work with logstash running from container. I use docker compose to star… --- ## [How to remove additional comma in part of message](https://discuss.elastic.co/t/how-to-remove-additional-comma-in-part-of-message/119887)
**Author:** [@Lingfang\_Zhu](https://discuss.elastic.co/u/Lingfang_Zhu)\ **Replies:** 3\ **Last updated:** [February 15, 2018, 5:28pm UTC](https://discuss.elastic.co/t/how-to-remove-additional-comma-in-part-of-message/119887 "2018-02-15T17:28:13Z")
I am trying to use csv filter to separate log message to several fields including message, date, time, module..., they are separated by comma. But when it is a error message, the first part contains some commas, which ma… --- ## [Logstash Filter for XML file](https://discuss.elastic.co/t/logstash-filter-for-xml-file/119956)
**Author:** [@charan.gandra](https://discuss.elastic.co/u/charan.gandra)\ **Replies:** 5\ **Last updated:** [February 15, 2018, 2:38pm UTC](https://discuss.elastic.co/t/logstash-filter-for-xml-file/119956 "2018-02-15T14:38:18Z")
Hi, I am looking to parse multiple XML files and can't find a suitable filter in Logstash for the same. I've around 50 XML files in different sub folders but all should have same structure as it is an output of pnp4nagi… --- ## [Help filtering JSON, multiple data formats for records](https://discuss.elastic.co/t/help-filtering-json-multiple-data-formats-for-records/119560)
**Author:** [@jpbruckler](https://discuss.elastic.co/u/jpbruckler)\ **Replies:** 6\ **Last updated:** [February 15, 2018, 1:59pm UTC](https://discuss.elastic.co/t/help-filtering-json-multiple-data-formats-for-records/119560 "2018-02-15T13:59:54Z")
Full disclosure: I'm a complete noob when it comes to the Elastic Stack. I've hooked up Carbon Black (CB) Response Event Forwarder to logstash/elasticsearch and I'm having some issues with data. I think it's down to th… --- ## [Logstash translate filter](https://discuss.elastic.co/t/logstash-translate-filter/119923)
**Author:** [@Harsh\_Jain](https://discuss.elastic.co/u/Harsh_Jain)\ **Replies:** 1\ **Last updated:** [February 15, 2018, 1:56pm UTC](https://discuss.elastic.co/t/logstash-translate-filter/119923 "2018-02-15T13:56:52Z")
I'm new to the Elastic stack and Logstash. I have a new Logstash instance that is accepting logs from beats sending to Elasticsearch. I am trying to apply translate filter based on regex keys. Here the key point to note … --- ## [Logstash elasticsearch to csv pipeline issue - returns all the documents irrespective of the "size" parameter in the query](https://discuss.elastic.co/t/logstash-elasticsearch-to-csv-pipeline-issue-returns-all-the-documents-irrespective-of-the-size-parameter-in-the-query/119723)
**Author:** [@somkhil](https://discuss.elastic.co/u/somkhil)\ **Replies:** 1\ **Last updated:** [February 15, 2018, 11:43am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-to-csv-pipeline-issue-returns-all-the-documents-irrespective-of-the-size-parameter-in-the-query/119723 "2018-02-15T11:43:28Z")
logstash elasticsearch to csv pipeline returns all the documents irrespective of the "size" parameter in the query. I have set up elasticsearch to csv pipeline using logstash and it pulls all the docs into output file s… --- ## [Disable retry for some elasticsearch outputs](https://discuss.elastic.co/t/disable-retry-for-some-elasticsearch-outputs/119941)
**Author:** [@devjr](https://discuss.elastic.co/u/devjr)\ **Replies:** 0\ **Last updated:** [February 15, 2018, 8:36am UTC](https://discuss.elastic.co/t/disable-retry-for-some-elasticsearch-outputs/119941 "2018-02-15T08:36:17Z")
Hello, I have a logstash cluster which get logs from redis and then send them to several elasticsearch instances. One of them isn't under my management. My problem is when this one is faulty (retrying failed action wit… --- ## [Options in place of redis](https://discuss.elastic.co/t/options-in-place-of-redis/119603)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 3\ **Last updated:** [February 15, 2018, 10:58am UTC](https://discuss.elastic.co/t/options-in-place-of-redis/119603 "2018-02-15T10:58:33Z")
Hello can anyone suggest what all plugins I can use in place of redis to pull data from a server and how. Regards Gaurav --- ## [Create a custom id](https://discuss.elastic.co/t/create-a-custom-id/119951)
**Author:** [@Nayaz\_JH](https://discuss.elastic.co/u/Nayaz_JH)\ **Replies:** 11\ **Last updated:** [February 15, 2018, 10:19am UTC](https://discuss.elastic.co/t/create-a-custom-id/119951 "2018-02-15T10:19:05Z")
"\_index": "access\_log", "\_type": "access\_logs", "\_id": "0udbmGEBevLW4jI4We7N", "\_score": 1, "\_source": { "@timestamp": "2018-02-15T07:26:18.252Z", "path": "V:/cc/7.7/mcs\_7\_indonesia\_dev/mpower/out/log/MessagingBrok… --- ## [Set custom type in logstash 6.0.0](https://discuss.elastic.co/t/set-custom-type-in-logstash-6-0-0/119808)
**Author:** [@Abhishek\_Wahi](https://discuss.elastic.co/u/Abhishek_Wahi)\ **Replies:** 2\ **Last updated:** [February 15, 2018, 8:53am UTC](https://discuss.elastic.co/t/set-custom-type-in-logstash-6-0-0/119808 "2018-02-15T08:53:26Z")
Hi All, I have a index in es 6.0.0 with type name as 'realtime'. I am loading some data from mysql in es using logstash. But it is giving me some deprecated warning for type. If i remove my entry for document type in o… --- ## [A path as default for an env variable in config not detected](https://discuss.elastic.co/t/a-path-as-default-for-an-env-variable-in-config-not-detected/119926)
**Author:** [@Shaoranlaos](https://discuss.elastic.co/u/Shaoranlaos)\ **Replies:** 2\ **Last updated:** [February 15, 2018, 9:06am UTC](https://discuss.elastic.co/t/a-path-as-default-for-an-env-variable-in-config-not-detected/119926 "2018-02-15T09:06:50Z")
Hallo, how do i correctly define a default value for an env variable substituion in a pipeline configuration? I have tried with %{fdc.path.tool\_cluster\_dict:/etc/logstash/tool\_cluster\_dict.json} but he says the file do… --- ## [Grok WARN without using it](https://discuss.elastic.co/t/grok-warn-without-using-it/117334)
**Author:** [@Robert\_D](https://discuss.elastic.co/u/Robert_D)\ **Replies:** 6\ **Last updated:** [February 15, 2018, 8:03am UTC](https://discuss.elastic.co/t/grok-warn-without-using-it/117334 "2018-02-15T08:03:10Z")
Filebeat config: filebeat.prospectors: - paths: - /\*.log tags: \["internal", "log"\] json.message\_key: msg json.add\_error\_key: true harvester\_limit: 500 close\_inactive: 5m - paths: - /other/\*.log tags… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=819) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=821)