# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=699 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 700 --- ## [How to ingest different set of logs into single index](https://discuss.elastic.co/t/how-to-ingest-different-set-of-logs-into-single-index/144702)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 3\ **Last updated:** [August 20, 2018, 12:36pm UTC](https://discuss.elastic.co/t/how-to-ingest-different-set-of-logs-into-single-index/144702 "2018-08-20T12:36:09Z")
Hi Team, I am working on a project, where i have multiple set of log files(apache,syslogs & java logs) coming from different set of sources, i was successful in creating separate set of indices and combined one's where… --- ## [Logstash for Adding a New Field](https://discuss.elastic.co/t/logstash-for-adding-a-new-field/145129)
**Author:** [@Priyaranjan\_Mudliar](https://discuss.elastic.co/u/Priyaranjan_Mudliar)\ **Replies:** 1\ **Last updated:** [August 20, 2018, 12:29pm UTC](https://discuss.elastic.co/t/logstash-for-adding-a-new-field/145129 "2018-08-20T12:29:01Z")
Can i somehow get only the filename in my "source" field instead of the whole path of the file. Please help. Thanks --- ## [Secure submission of filebeat from remote](https://discuss.elastic.co/t/secure-submission-of-filebeat-from-remote/145005)
**Author:** [@syfy323](https://discuss.elastic.co/u/syfy323)\ **Replies:** 2\ **Last updated:** [August 20, 2018, 11:59am UTC](https://discuss.elastic.co/t/secure-submission-of-filebeat-from-remote/145005 "2018-08-20T11:59:18Z")
Hi! I would like to set up filebeat and logstash for multiple networks (~100). As we are four admins who need the logs, we only plan a small deployment (1x elasticsearch, 1x logstash and 1x kibana - nodes). This deplo… --- ## [How to process a large csv file and add custom Python code to transform few columns to index into ElasticSearch?](https://discuss.elastic.co/t/how-to-process-a-large-csv-file-and-add-custom-python-code-to-transform-few-columns-to-index-into-elasticsearch/145133)
**Author:** [@Abhishek\_Shivkumar](https://discuss.elastic.co/u/Abhishek_Shivkumar)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 10:18am UTC](https://discuss.elastic.co/t/how-to-process-a-large-csv-file-and-add-custom-python-code-to-transform-few-columns-to-index-into-elasticsearch/145133 "2018-08-20T10:18:22Z")
Hi, I have a large csv file with a million rows and ~50 columns. I want to add an extra two columns and the values in these two columns is based on analysing the context on the existing few columns using some kind of te… --- ## [POST method http\_poller](https://discuss.elastic.co/t/post-method-http-poller/141940)
**Author:** [@m.sereda](https://discuss.elastic.co/u/m.sereda)\ **Replies:** 53\ **Last updated:** [August 3, 2018, 10:33am UTC](https://discuss.elastic.co/t/post-method-http-poller/141940 "2018-08-03T10:33:40Z")
Trying to make a POST method to login but smth is going wrong. To make a login I have to fill "login" and "password" poles which is in JSON. Where is a mistake? input { http\_poller { urls =\> { soap\_request … --- ## [Cant's set log\_time to @timestamp](https://discuss.elastic.co/t/cants-set-log-time-to-timestamp/144811)
**Author:** [@Haitianisgood](https://discuss.elastic.co/u/Haitianisgood)\ **Replies:** 2\ **Last updated:** [August 20, 2018, 8:09am UTC](https://discuss.elastic.co/t/cants-set-log-time-to-timestamp/144811 "2018-08-20T08:09:30Z")
env： logstash 6.2 ES: 6.2 I touch two config files in /etc/logstash/conf.d , eg：a.conf b.conf a.conf input from kafka, setting type is 'aa' b.conf input from file, setting type is 'bb' Both output are same ES Ques… --- ## [Logstash 6.2 multiple input confused](https://discuss.elastic.co/t/logstash-6-2-multiple-input-confused/144636)
**Author:** [@Haitianisgood](https://discuss.elastic.co/u/Haitianisgood)\ **Replies:** 8\ **Last updated:** [August 20, 2018, 8:06am UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-input-confused/144636 "2018-08-20T08:06:04Z")
env：logstash 6.2 I touch two config files in /etc/logstash/conf.d , eg：a.conf b.conf a.conf input from kafka, setting type is 'aa' b.conf input from file, setting type is 'bb' Both output are same ES The question … --- ## [Some questions of logstash 6](https://discuss.elastic.co/t/some-questions-of-logstash-6/144857)
**Author:** [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\ **Replies:** 1\ **Last updated:** [August 20, 2018, 7:44am UTC](https://discuss.elastic.co/t/some-questions-of-logstash-6/144857 "2018-08-20T07:44:28Z")
Dear Elastic, I have some questions of Logstash to you. Could you please answer my questions? Best Regards Robin ##################### logstash server sepc ##################### Cpu:4 core Mem: 16g Java heap: 8g … --- ## [How may I get only ERROR related logs from my stack trace?](https://discuss.elastic.co/t/how-may-i-get-only-error-related-logs-from-my-stack-trace/144329)
**Author:** [@Vyankatesh\_S\_Repal](https://discuss.elastic.co/u/Vyankatesh_S_Repal)\ **Replies:** 4\ **Last updated:** [August 20, 2018, 7:11am UTC](https://discuss.elastic.co/t/how-may-i-get-only-error-related-logs-from-my-stack-trace/144329 "2018-08-20T07:11:31Z")
Currently I am developing an application where I need to retrieve only "ERROR" related logs and put them to elasticsearch index. Here is my sample log: 2018-07-05 19:37:15,888 \[6\] ERROR \ - Error while Proc… --- ## [Not able to parse xml](https://discuss.elastic.co/t/not-able-to-parse-xml/144673)
**Author:** [@pavan\_padaki](https://discuss.elastic.co/u/pavan_padaki)\ **Replies:** 3\ **Last updated:** [August 20, 2018, 7:10am UTC](https://discuss.elastic.co/t/not-able-to-parse-xml/144673 "2018-08-20T07:10:30Z")
logstash-6.3.1\\bin\\logstash -f xml\_config.conf Ignoring the 'pipelines.yml' file because modules or command line options are specified Starting Logstash {"logstash.version"=\>"6.3.1"} \[logstash.outputs.elasticsearch\] Y… --- ## [XML has a varying schema...Unable to give Xpath](https://discuss.elastic.co/t/xml-has-a-varying-schema-unable-to-give-xpath/145086)
**Author:** [@dinesh1](https://discuss.elastic.co/u/dinesh1)\ **Replies:** 1\ **Last updated:** [August 20, 2018, 7:02am UTC](https://discuss.elastic.co/t/xml-has-a-varying-schema-unable-to-give-xpath/145086 "2018-08-20T07:02:58Z")
Hi My XML logs has a varying schema and it's failed to define XPath expressions. How to parse XML logs without Xpath...? --- ## [Logstash does not make indices in elasticsearch](https://discuss.elastic.co/t/logstash-does-not-make-indices-in-elasticsearch/145091)
**Author:** [@wonki](https://discuss.elastic.co/u/wonki)\ **Replies:** 7\ **Last updated:** [August 20, 2018, 7:00am UTC](https://discuss.elastic.co/t/logstash-does-not-make-indices-in-elasticsearch/145091 "2018-08-20T07:00:54Z")
Hello, i got in trouble. Logstash does not make indeces in elasticsearch. Here is my conf file. -- logstash.conf -- input{ file { path =\> "/home/wonki/access\_log" start\_position =\> "beginning" sincedb\_path =\> "/de… --- ## [Docker logstash container not listening on port 5044](https://discuss.elastic.co/t/docker-logstash-container-not-listening-on-port-5044/145046)
**Author:** [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\ **Replies:** 1\ **Last updated:** [August 20, 2018, 6:43am UTC](https://discuss.elastic.co/t/docker-logstash-container-not-listening-on-port-5044/145046 "2018-08-20T06:43:46Z")
container of logstash runs. but ports not open. When i run docker port logstash it yields no output. when i run docker port elasticsearch or kibana it does yield results. When i telnet hostname 5044 or telnet hostname… --- ## [Windows Event Trace logs](https://discuss.elastic.co/t/windows-event-trace-logs/145079)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 0\ **Last updated:** [August 20, 2018, 2:49am UTC](https://discuss.elastic.co/t/windows-event-trace-logs/145079 "2018-08-20T02:49:15Z")
Does logstash or #beats:filebeat can retrieve files of Windows Performance Recorder or Windows Event Trace Logs (ETL Files)?? Thank you. --- ## [Logstash - Parse XML attributes](https://discuss.elastic.co/t/logstash-parse-xml-attributes/145029)
**Author:** [@Ju\_La](https://discuss.elastic.co/u/Ju_La)\ **Replies:** 3\ **Last updated:** [August 19, 2018, 9:45pm UTC](https://discuss.elastic.co/t/logstash-parse-xml-attributes/145029 "2018-08-19T21:45:45Z")
I am trying to parse and send field names to ElasticSearch and am running into issue with getting the correct fields lined up. Here is some sample XML \ \\ \… --- ## [Not able to loop and read key value pairs in logstash](https://discuss.elastic.co/t/not-able-to-loop-and-read-key-value-pairs-in-logstash/144338)
**Author:** [@Sahaya\_Agnel](https://discuss.elastic.co/u/Sahaya_Agnel)\ **Replies:** 4\ **Last updated:** [August 19, 2018, 4:22pm UTC](https://discuss.elastic.co/t/not-able-to-loop-and-read-key-value-pairs-in-logstash/144338 "2018-08-19T16:22:10Z")
Hi, Below is the JSON content that i want to process and read all the 'amount' and sum those values based on the business logic in the Logstash config file. Can you please help ? "ClientSalaryData": { "person\_salary\_i… --- ## [Logstash TCP Socket 'Ready' but not writeable](https://discuss.elastic.co/t/logstash-tcp-socket-ready-but-not-writeable/145060)
**Author:** [@rcoundon](https://discuss.elastic.co/u/rcoundon)\ **Replies:** 0\ **Last updated:** [August 19, 2018, 3:19pm UTC](https://discuss.elastic.co/t/logstash-tcp-socket-ready-but-not-writeable/145060 "2018-08-19T15:19:38Z")
I'm writing a JavaScript extension to run in nodejs for Winston to add Logstash as a transport over TCP. The way I'm testing at the moment is running ELK within a set of 3 Docker containers and taking down the Logstash … --- ## [Convert logstash date sting](https://discuss.elastic.co/t/convert-logstash-date-sting/145039)
**Author:** [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\ **Replies:** 1\ **Last updated:** [August 19, 2018, 2:19pm UTC](https://discuss.elastic.co/t/convert-logstash-date-sting/145039 "2018-08-19T14:19:44Z")
Hello, I am not blocked on a problem of conversion of string in date timestamp. Indeed, I have a filebeat that reads a json file already made and sends it to logstaash. and at the logstash level, I wanted to convert th… --- ## [Logstash http output not ignoring ssl](https://discuss.elastic.co/t/logstash-http-output-not-ignoring-ssl/144984)
**Author:** [@saching](https://discuss.elastic.co/u/saching)\ **Replies:** 0\ **Last updated:** [August 18, 2018, 2:09am UTC](https://discuss.elastic.co/t/logstash-http-output-not-ignoring-ssl/144984 "2018-08-18T02:09:35Z")
logstash version 5.6.10 output { http { ssl\_certificate\_validation =\> false url =\> "https://some-test-domain.com" format =\> "json" content\_type =\> "application/json" http\_method =\> "post" headers =\> \["Authorizati… --- ## [Ruby script to quickly parse URL for easier filtering or use in visualizations](https://discuss.elastic.co/t/ruby-script-to-quickly-parse-url-for-easier-filtering-or-use-in-visualizations/145003)
**Author:** [@mbreest](https://discuss.elastic.co/u/mbreest)\ **Replies:** 0\ **Last updated:** [August 18, 2018, 11:40am UTC](https://discuss.elastic.co/t/ruby-script-to-quickly-parse-url-for-easier-filtering-or-use-in-visualizations/145003 "2018-08-18T11:40:48Z")
Hi folks, just wanted to share the Ruby script that we use for parsing the URLs the we receive in our CDN logs to do further analysis on it. It is a mix of what you can do with Grok plugin (https://www.elastic.co/guide/… --- ## [Performance considerations Grok/plugins vs Ruby script](https://discuss.elastic.co/t/performance-considerations-grok-plugins-vs-ruby-script/145002)
**Author:** [@mbreest](https://discuss.elastic.co/u/mbreest)\ **Replies:** 0\ **Last updated:** [August 18, 2018, 11:25am UTC](https://discuss.elastic.co/t/performance-considerations-grok-plugins-vs-ruby-script/145002 "2018-08-18T11:25:24Z")
Hi folks, does anyone have experience regarding the performance of grok/plugins vs. Ruby scripts (https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) and can share some insights on that. I did no… --- ## [Grok filter help!](https://discuss.elastic.co/t/grok-filter-help/140233)
**Author:** [@Inammathe\_Inna](https://discuss.elastic.co/u/Inammathe_Inna)\ **Replies:** 4\ **Last updated:** [August 18, 2018, 4:55am UTC](https://discuss.elastic.co/t/grok-filter-help/140233 "2018-08-18T04:55:44Z")
Hi, I am having a hard time getting my logstash pipeline to process messages efficiently. I am running at 100% CPU utilization (4 CPUs and 4 pipeline workers) and only processing about 20-30 documents a minute (measured… --- ## [Kibana Client Request timed out with Logstash Netflow module](https://discuss.elastic.co/t/kibana-client-request-timed-out-with-logstash-netflow-module/144255)
**Author:** [@ccutmt](https://discuss.elastic.co/u/ccutmt)\ **Replies:** 4\ **Last updated:** [August 17, 2018, 10:09pm UTC](https://discuss.elastic.co/t/kibana-client-request-timed-out-with-logstash-netflow-module/144255 "2018-08-17T22:09:11Z")
Hi, I've been trying to connect Logstash Netflow module with Kibana. I followed the documentation in the link. https://www.elastic.co/guide/en/logstash/6.x/netflow-module.html My whole scenario is that I have elasticse… --- ## [Table name with a multiple /](https://discuss.elastic.co/t/table-name-with-a-multiple/144934)
**Author:** [@Jitendrasingh](https://discuss.elastic.co/u/Jitendrasingh)\ **Replies:** 2\ **Last updated:** [August 17, 2018, 8:45pm UTC](https://discuss.elastic.co/t/table-name-with-a-multiple/144934 "2018-08-17T20:45:25Z")
Hey Gurus, I've a table with the name having / in its name. I tried using the escape sequence and it still throws and unidentified table name. The table name is /BI0/PPS and schema is ER3. Its an oracle 12c table. The s… --- ## [Netflow - Flow data reaching server, Logstash zero output](https://discuss.elastic.co/t/netflow-flow-data-reaching-server-logstash-zero-output/144774)
**Author:** [@taking-care-of-busin](https://discuss.elastic.co/u/taking-care-of-busin)\ **Replies:** 6\ **Last updated:** [August 17, 2018, 7:09pm UTC](https://discuss.elastic.co/t/netflow-flow-data-reaching-server-logstash-zero-output/144774 "2018-08-17T19:09:21Z")
Hello, I have installed an ELK Stack on a Ubuntu 18.04 LXC/Container. I am sending Netflow v9 flows from a Cisco ASR9000 node to Logstash (end purpose is to use Elastiflow), and the flows are getting to the server (con… --- ## [Log4j2 time based log deletion not working](https://discuss.elastic.co/t/log4j2-time-based-log-deletion-not-working/144924)
**Author:** [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Replies:** 0\ **Last updated:** [August 17, 2018, 3:16pm UTC](https://discuss.elastic.co/t/log4j2-time-based-log-deletion-not-working/144924 "2018-08-17T15:16:09Z")
I am using the default log4j2 supplied with Logstash 6.3.2 installed from the Debian repo. status = error name = LogstashPropertiesConfig appender.console.type = Console appender.console.name = plain\_console appender.c… --- ## [Locating Pipeline Error](https://discuss.elastic.co/t/locating-pipeline-error/142610)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [August 17, 2018, 2:25pm UTC](https://discuss.elastic.co/t/locating-pipeline-error/142610 "2018-08-17T14:25:37Z")
I have about 60 config files for a single pipeline and I receive the below error during Logstash startup....is there a way to pinpoint which file the error is located in? \[2018-08-01T10:27:13,736\]\[ERROR\]\[logstash.agent … --- ## [Logstash Error parsing csv java.lang.ArrayIndexOutOfBoundsException](https://discuss.elastic.co/t/logstash-error-parsing-csv-java-lang-arrayindexoutofboundsexception/144900)
**Author:** [@kennymarx0](https://discuss.elastic.co/u/kennymarx0)\ **Replies:** 4\ **Last updated:** [August 17, 2018, 2:16pm UTC](https://discuss.elastic.co/t/logstash-error-parsing-csv-java-lang-arrayindexoutofboundsexception/144900 "2018-08-17T14:16:38Z")
when i use logstash to import data from csv file to elasticsearch, it has error like below and there are no data imported to es. how to fix it? \[2018-08-15T23:23:07,540\]\[WARN \]\[logstash.filters.csv \] Error parsing… --- ## [S3 Output Plugin separate json strings](https://discuss.elastic.co/t/s3-output-plugin-separate-json-strings/144910)
**Author:** [@John\_Ryan](https://discuss.elastic.co/u/John_Ryan)\ **Replies:** 0\ **Last updated:** [August 17, 2018, 1:22pm UTC](https://discuss.elastic.co/t/s3-output-plugin-separate-json-strings/144910 "2018-08-17T13:22:39Z")
The s3 output plugin is sending the files to S3. In the files are a group of json strings. I need to parse these and I would like to know if it is possible for the files to have a separator written between each json stri… --- ## [Csv field data as count](https://discuss.elastic.co/t/csv-field-data-as-count/144740)
**Author:** [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Replies:** 6\ **Last updated:** [August 17, 2018, 1:06pm UTC](https://discuss.elastic.co/t/csv-field-data-as-count/144740 "2018-08-17T13:06:50Z")
Hi I have a csv field as below with a numeric value the logstash config is as below everything is indexing fine but when i try to create the visualization, the value on y-axis is showing the count as (1) instead o… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=698) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=700)