# Couple of issues with logstash input file :(

**URL:** <https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585>\
**Category:** Logstash\
**Created:** [May 9, 2016, 8:48pm UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585 "2016-05-09T20:48:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [May 9, 2016, 8:48pm UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/1 "2016-05-09T20:48:47Z")

</div>

Hello.  
Got a couple of questions relating to how the input file really works.  
I would like to brand (change type) for certain inputs, but the system doesn't seem to be picking up my 'type' comment. I also am having troubles with my tweets in that the system just puts them in 1 files (messages).  
Below is my config. Is there anything wrong with the config that you guys can see?

---

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [May 9, 2016, 8:50pm UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/2 "2016-05-09T20:50:39Z")

</div>

Sorry didn't attach config file.

root@log01:~# cat /etc/logstash/conf.d/02-logstash-input.conf  
input {  
twitter {  
consumer\_key =\> “key"  
consumer\_secret =\> “secrett"  
oauth\_token =\> “token"  
oauth\_token\_secret =\> “token\_secret"  
keywords =\> ["elk","logstash","elasticsearch","kibana"]  
type =\> tweet  
}  
beats {  
port =\> 5044  
ssl =\> true  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder-log01.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder-log01.key"  
type =\> beats  
}  
tcp {  
port =\> 5514  
type =\> syslog5514  
}  
tcp {  
port =\> 5515  
type =\> syslog5515  
}  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 9, 2016, 9:08pm UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/3 "2016-05-09T21:08:57Z")

</div>

It's supposed to be a string - [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-type](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-type) - you shoudl wrap it in quotes.

---

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [May 9, 2016, 11:11pm UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/4 "2016-05-09T23:11:17Z")

</div>

HI, thanks.  
Did that, but still in my Kibana screen, it shows type as:

%{[@metadata][type]}

It was doing the same before.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 10, 2016, 6:26am UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/5 "2016-05-10T06:26:38Z")

</div>

The `[@metadata][type]` field is set by Beats. The `type` option in Logstash options ends up in the `type` field.

---

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [May 10, 2016, 8:59pm UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/6 "2016-05-10T20:59:47Z")

</div>

Ahh yes I found that in my LS Output file:

30-elasticsearch-output.conf  
output {  
elasticsearch {  
hosts =\> ["els-node2:9200","els-node3:9200","els-node4:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
user =\> user  
password =\> password  
}  
}

Do you know what it should be?  
This is a direct copy from this site for when I built the System up under Logstash configure part.

> **[How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 16.04 |...](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04)**
>
> In this tutorial, we will go over the installation of the Elasticsearch ELK Stack on Ubuntu 16.04 (that is, Elasticsearch 2.3.x, Logstash 2.3.x, and Kibana 4.5.x). We will also show you how to configure it to gather and visualize the syslogs of your...

Cheers.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 11, 2016, 6:06am UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/7 "2016-05-11T06:06:04Z")

</div>

There is no definite right or wrong here. It's up to you how you want to organize and set the document types. If you're setting a good type value in all your beats you can use that, but if you have other inputs you'll have to set the type in the Logstash configuration. Something like

```nohighlight
filter {
  if [@metadata][type] {
    mutate {
      replace => {
        "type" => "%{[@metadata][type]}"
      }
    }
  }
}

```

might be useful to change the `type` into the beats's type if it's set.

---

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [June 4, 2016, 11:42am UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/8 "2016-06-04T11:42:28Z")

</div>

Hi, thanks for this. I have managed to split my logs by type and index so when I look at my indexes via 'elasticsearch \_head' I can see a seperate indexes for twitter, syslog5514 and syslog 5515'.

For some reason now most of my records are getting displayed in the correct newly created type/index, but others are still coming through with the old format (which I do not want).

Below is an extract of my input/filter/output config along with a couple examples of what I get when I search kibana showing a good format and a bad format.

Any ideas why some logs are correctly re-written, but others are not?

Thanks in advance.

## **------ Input file ------** --

input {

input {  
twitter {  
consumer\_key =\> "key"  
consumer\_secret =\> "secret"  
oauth\_token =\> "token"  
oauth\_token\_secret =\> "token-secret"  
keywords =\> ["elk","logstash","elasticsearch","kibana"]  
type =\> "twitter-log02"  
}  
beats {  
port =\> 5044  
type =\> "beats-log02"  
}  
tcp {  
port =\> 5514  
type =\> "syslog5514-log02"  
}  
tcp {  
port =\> 5515  
type =\> "syslog5515-log02"

**## ------ Filter file --------**

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

**## ------ Output file --------**

output {  
if [type] == "twitter-log02" {  
elasticsearch {  
hosts =\> ["els03:9200","els02:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "twitter-%{+YYYY.MM.dd}"  
document\_type =\> "twitter-log02"  
}  
} if [type] == "syslog5514-log02" {  
elasticsearch {  
hosts =\> ["els03:9200","els02:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "syslog5514-%{+YYYY.MM.dd}"  
document\_type =\> "syslog5514-log02"  
}  
} if [type] == "syslog5515-log02" {  
elasticsearch {  
hosts =\> ["els03:9200","els02:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "syslog5515-%{+YYYY.MM.dd}"  
document\_type =\> "syslog5515-log02"  
}  
} else {  
elasticsearch {  
hosts =\> ["els03:9200","els02:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}  
}

**----------------- end of output file.**

**Right format (\_index and \_type are looking good).**

{  
**"\_index": "syslog5514-2016.06.04",**  
**"\_type": "syslog5514-log01**",  
"\_id": "AVUbMu6b4mUVQoJOugLT",  
"\_score": null,  
"\_source": {  
"message": "\<191\>2289: 002129: Jun 4 21:36:09.475 AEST: IP ARP: rcvd req src 192.168.10.65 9410.3e4e.f6d1, dst 192.168.10.14 Vlan10",  
"@version": "1",  
"@timestamp": "2016-06-04T11:36:10.494Z",  
"host": "192.168.10.252",  
"port": 60624,  
"type": "syslog5514"  
},  
"fields": {  
"@timestamp": [  
1465040170494  
]  
},  
"highlight": {  
"host": [  
"@kibana-highlighted-field@192.168.10.252@/kibana-highlighted-field@"  
],  
"type": [  
"@kibana-highlighted-field@syslog5514@/kibana-highlighted-field@"  
]  
},  
"sort": [  
1465040170494  
]  
}

**And in the wrong format**

{  
**"\_index": "%{[@metadata][beat]}-2016.06.04",**  
**"\_type": "%{[@metadata][type]}**",  
"\_id": "AVUbMu9Do0XXlD820rpF",  
"\_score": null,  
"\_source": {  
"message": "\<191\>2290: 002130: Jun 4 21:36:09.475 AEST: IP ARP: ignored gratuitous arp src 192.168.10.65 9410.3e4e.f6d1, dst 192.168.10.14 e8ba.7099.f541, interface Vlan10",  
"@version": "1",  
"@timestamp": "2016-06-04T11:36:10.494Z",  
"host": "192.168.10.252",  
"port": 60624,  
"type": "syslog5514"  
},  
"fields": {  
"@timestamp": [  
1465040170494  
]  
},  
"highlight": {  
"host": [  
"@kibana-highlighted-field@192.168.10.252@/kibana-highlighted-field@"  
],  
"type": [  
"@kibana-highlighted-field@syslog5514@/kibana-highlighted-field@"  
]  
},  
"sort": [  
1465040170494  
]  
}

---

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [June 5, 2016, 5:33am UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/11 "2016-06-05T05:33:14Z")

</div>

Screen print, may help to show the problem I'm having.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/6/6e1bda071ebd3caff69a1cf839c622559cd92b64.jpg)  
 ![](https://us1.discourse-cdn.com/elastic/original/2X/9/94fb157238ba479dc8f756228bd47c22ed0087d3.jpg)

---

<div class="post-metadata">

**Author:** ![jamesl](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jamesl](https://discuss.elastic.co/u/jamesl)\
**Post date:** [June 8, 2016, 5:00am UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/12 "2016-06-08T05:00:22Z")

</div>

Hi. This is fixed !  
Had to modify my input filters a bit, was missing the 'else if' ...

e.g.

} else if [type] == "syslog5514-log01" {  
elasticsearch {  
hosts =\> ["els02:9200","els03:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "syslog5514-%{+YYYY.MM.dd}"  
document\_type =\> "syslog5514-log01"  
}  
} else if [type] == "syslog5515-log01" {  
elasticsearch {  
hosts =\> ["els02:9200","els03:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "syslog5515-%{+YYYY.MM.dd}"  
document\_type =\> "syslog5515-log01"  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:54am UTC](https://discuss.elastic.co/t/couple-of-issues-with-logstash-input-file/49585/13 "2017-07-06T04:54:15Z")

</div>