# What is the purpose of type field in Input section **URL:** \ **Category:** Logstash\ **Created:** [January 21, 2016, 11:00pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828 "2016-01-21T23:00:37Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 21, 2016, 11:00pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/1 "2016-01-21T23:00:38Z")
Hello All, Whats the use of "type =\> " in Input section of Logstash ? Anyways I will be using grok filter in filter section to filter incoming messages ? Thanks, gaurav ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2016, 7:00am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/2 "2016-01-22T07:00:35Z")
The `type` option sets the value of the field with the same name. If you only ingest a single kind of log (and never will do anything else) you don't have to worry about it, but in all likelihood you'll eventually want to process different kinds of logs and then the `type` field will be a good way of distinguishing them. ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 22, 2016, 6:58pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/3 "2016-01-22T18:58:12Z")
Thanks. Problem is : I have data center where i have let us for e.g 3 networking providers. A B C. All of them has slightly similar syslog format. (some of them has extra spaces some of them has not, some of them has program name and severity in that some of them are not) How can I set type filed in Input section prior seeing their actual syslog message ? Ideally it should be like : input { type=\> A type=\> B type=\> C .... } Thats why I dont understand the purpose of type field in input section. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 22, 2016, 7:31pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/4 "2016-01-22T19:31:54Z")
How are you sending the logs to LS - TCP, file, syslog? ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 22, 2016, 8:06pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/5 "2016-01-22T20:06:08Z")
I doing POC here. So I ask logstash to do this : input { tcp { port =\> 5000 } udp { port =\> 5000 } } Then I do telnet localhost 5000 from same host machine and then insert manually syslog message which I have. In production, I will make sure all network devices to publish syslog messages on some port and then logstash will listen to that port so that It will have continuous stream of syslog messages. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 23, 2016, 4:19pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/6 "2016-01-23T16:19:38Z")
I don't see why you'd have to use different types for different kinds of syslog messages. When you're searching for messages in Kibana, why should your query be affected by the vendor of the device producing the events you're interested in? ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 24, 2016, 5:55am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/7 "2016-01-24T05:55:53Z")
Thanks. My requirement is : Say, I have 3 Networking routers- A B C which are producing syslog messages. Their syslog messages come to logstash instance at any time and at any order. How can I search in Kibana , if I want to know how many messages from Vendor A I have got during certain time frame. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 24, 2016, 5:28pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/8 "2016-01-24T17:28:59Z")
You could indeed use different types or this, I just suspect there are better criteria. Wouldn't e.g. the hostname be a more interesting condition? Anyway, to set different types depending on what the message looks like, something like this would work: ```auto filter { if [message] =~ /regexp that matches vendor A/ { mutate { replace => { "type" => "syslog_A" } } } } ``` Another way which I'd probably use is having multiple grok filters that match each vendor's messages—I'm just assuming they have different formats—to avoid regexp duplication and add a vendor-specific tag that you later on can translate into a change of the type field. ```auto filter { grok { match => ... add_tag => ["syslog_A"] } if "_grokparsefailure" not in [tags] { grok { match => ... add_tag => ["syslog_B"] } } ... if "syslog_A" in [tags] { mutate { replace => { "type" => "syslog_A" } remove_tag => ["syslog_A"] } } } ``` ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 24, 2016, 8:28pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/9 "2016-01-24T20:28:14Z")
Excellent. Thanks a lot for valuable suggestion. Appreciate your help. ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 27, 2016, 10:13pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/10 "2016-01-27T22:13:32Z")
How Can I multiple match =\> statements within grok filter ? e.g: First line : match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp}\s+%{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } One second line I have to reuse what I parsed in first line. I want to do again like this : match =\> {"syslog\_hostname"=\> some REGEX} How can I do that in Logstash config. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 28, 2016, 6:55am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/11 "2016-01-28T06:55:48Z")
If you want to try multiple grok expressions on a field and break on the first match: ```auto grok { match => { "fieldname" => ["expression1", "expression2"] } } ``` If you want to match against different fields it might work to have multiple `match` options in the same filter, but I'd use two consecutive filters. ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 28, 2016, 6:59pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/12 "2016-01-28T18:59:45Z")
Thanks for reply. Problem is simple. 1: I need to parse hostname from syslog string. I am able to do that with match expression. 2: With parsed hostname, I need to apply regex to find out which vendor it is. Assume that we name sysloghost name in certain way so that it contains vendor name. for eg. hostname = region.city.vendor\_name.company.com How can I parse and get vendor\_name in this case ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 28, 2016, 7:02pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/13 "2016-01-28T19:02:00Z")
Just use a separate grok filter for that, somewhere after the current filter that extracts the `hostname` field. ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 29, 2016, 2:18am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/14 "2016-01-29T02:18:59Z")
I did this and I am NOT able to see MONTH, DAY, and TIME in my output of logstash. filter { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp}\s+%{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } ``` match => { "syslog_timestamp" => "%{MONTH:month} +%{MONTHDAY:day} %{TIME:time}" } remove_field => ["@version", "host", "message", "port"] } date { target => "@timestamp" match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] timezone => "UTC" } ``` } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 29, 2016, 6:42am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/15 "2016-01-29T06:42:34Z")
Did you try my suggestion of having two consecutive filters instead of two `match` options in the same filter? ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [January 29, 2016, 9:28pm UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/16 "2016-01-29T21:28:46Z")
Thanks It worked. Could you please take your valuable time and try to reply to my question here ? > [@Syslog date without year](https://discuss.elastic.co/t/syslog-date-without-year/29834/9): > > I am following this thread : [http://stackoverflow.com/questions/29384754/syslog-timestamp-without-year](http://stackoverflow.com/questions/29384754/syslog-timestamp-without-year) My requirement is : If syslog message comes with year in timestamp, I should match in date filter and take that year as time and put it in elasticsearch, If there is not year in syslog timestamp, then I need to have some default year / some default value which I will know in future, How can I do that ? My date filter is : date { target =\> "@timestamp" match =\> [ "syslog\_timestamp", "MMM … I appreciate your help. ---
**Author:** ![gaurav1424](https://avatars.discourse-cdn.com/v4/letter/g/97f17d/32.png) [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)\ **Post date:** [February 2, 2016, 12:46am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/17 "2016-02-02T00:46:08Z")
Thanks. I tried this but I think there is problem here. Let us say, If I pass syslog\_B type log first, Then it goes to first grok block and then as It doe not match , It puts \_grokparsefailure into tags. and then it goes to following if block and then I get match. So for syslog\_B type message, I always get two tags one is failure one and one is syslog\_B one. How do I avoid this ? I can do this by following your first option but , second one more elegant and right 🙂 Thanks, Gaurav ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 2, 2016, 7:05am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/18 "2016-02-02T07:05:14Z")
If you don't want the `_grokparsefailure` tag you can remove it in each grok filter with `remove_tag`. That option is only triggered when the grok is successful, just like `add_tag`. Then you won't get a `_grokparsefailure` tag in the end if one of the grok expressions matched, but if none of them matched the tag will be there. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:13am UTC](https://discuss.elastic.co/t/what-is-the-purpose-of-type-field-in-input-section/39828/19 "2017-07-06T05:13:28Z")