# What is the difference between stdin and file in logstash

**URL:** <https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910>\
**Category:** Logstash\
**Created:** [November 23, 2017, 3:10pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910 "2017-11-23T15:10:23Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![yishain11](https://avatars.discourse-cdn.com/v4/letter/y/df705f/32.png) [@yishain11](https://discuss.elastic.co/u/yishain11)\
**Post date:** [November 23, 2017, 3:10pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/1 "2017-11-23T15:10:23Z")

</div>

Hi  
I have two configurations, one with the input as stdin, and one using file path.  
both inputs are JSON structured text files.  
The filters are the same (using JSON encode, json and split), the output is the same.  
When I use the stdin config and enter the json file (minified) everything works fine, but when I use the file input, logstash reads every line as text and not as JSON.  
Why is there any difference?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 23, 2017, 3:48pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/2 "2017-11-23T15:48:14Z")

</div>

Please show your configuration instead of describing it.

---

<div class="post-metadata">

**Author:** ![yishain11](https://avatars.discourse-cdn.com/v4/letter/y/df705f/32.png) [@yishain11](https://discuss.elastic.co/u/yishain11)\
**Post date:** [November 26, 2017, 7:27am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/3 "2017-11-26T07:27:22Z")

</div>

This is th one with stdin:

```
input {
        stdin {}
}

filter {
  split{
    field => "changedFieldList"

        }
  json {
    source => "message"
    target => "parsedJSON"
       }

         }
output { stdout { codec => rubydebug }}

```

This is the one with the file load (via filebeat):

```
input {
  beats {
    port => 5044
  }
// another config:
file{
path => path/to/file/*
}
}

filter{
   split{
     field => "changedFieldList"
        }
   json{
        source => "message"
        target => "parsedJson"
      }
      }

output {
  elasticsearch {
    hosts => ["http://127.0.0.1:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"

  }
}
```

---

<div class="post-metadata">

**Author:** ![yishain11](https://avatars.discourse-cdn.com/v4/letter/y/df705f/32.png) [@yishain11](https://discuss.elastic.co/u/yishain11)\
**Post date:** [November 26, 2017, 9:32am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/4 "2017-11-26T09:32:00Z")

</div>

Additional Info:

This is the result when I use stdin and insert the json file manualy:

```
"parsedJSON" => {
    "DBSHDataChangeInfo" => "Archive Log Set archiveLogSet.108058603.108058399 Info:Thread# Sequence# FirstScn LastScn ",
         "documentsList" => [
        [0] {
               "objectSchemaName" => "YISHAIN",
                "commandSequence" => "0",
                    "commandType" => "UPDATE",
               "commandCommitScn" => "108058600",
              "changedFieldsList" => [
                [0] {
                       "fieldType" => "NUMBER",
                      "fieldValue" => "17",
                    "fieldChanged" => "N",
                         "fieldId" => "CUSTOMER_ID"
                },
                [1] {
                       "fieldType" => "VARCHAR2",
                      "fieldValue" => "Daniel", 
                         ...

```

This is the result when the file is loaded using file:

```
{
          "path" => "/path/to/folder",
    "@timestamp" => 2017-11-26T12:27:00.684Z,
    "parsedJSON" => nil,
      "@version" => "1",
          "host" => "qa8",
       "message" => "",
          "tags" => [
        [0] "_split_type_failure"
    ]
}
{
      "@version" => "1",
          "host" => "qa8",
          "path" => "path" => "/path/to/folder",
    "@timestamp" => 2017-11-26T12:27:00.787Z,
       "message" => " {",
          "tags" => [
        [0] "_split_type_failure",
        [1] "_jsonparsefailure"
    ]
}
...

```

As you can see, the stdin method process the json file fine, while the file method get everything as text. Why is that happening? How can I make the file method work?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 26, 2017, 8:01pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/5 "2017-11-26T20:01:07Z")

</div>

It looks like it's a multiline JSON file (i.e. the top-level JSON object it split over multiple lines). Are you really passing the exact same file via stdin?

---

<div class="post-metadata">

**Author:** ![yishain11](https://avatars.discourse-cdn.com/v4/letter/y/df705f/32.png) [@yishain11](https://discuss.elastic.co/u/yishain11)\
**Post date:** [November 27, 2017, 7:43am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/6 "2017-11-27T07:43:00Z")

</div>

yes I did

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 27, 2017, 9:08am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/7 "2017-11-27T09:08:37Z")

</div>

And what does the input file look like? I find it very hard to believe that the configuration you've shown us actually works as you describe.

---

<div class="post-metadata">

**Author:** ![yishain11](https://avatars.discourse-cdn.com/v4/letter/y/df705f/32.png) [@yishain11](https://discuss.elastic.co/u/yishain11)\
**Post date:** [November 27, 2017, 9:16am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/8 "2017-11-27T09:16:28Z")

</div>

```
{
	"ChangeInfo": "Archive Log Set archiveLogSet.108058603.108058399 Info:Thread# Sequence# FirstScn LastScn ",
	"documentsList": [{
		"commandScn": "108058599",
		"commandCommitScn": "108058600",
		"commandSequence": "0",
		"commandType": "UPDATE",
		"commandTimestamp": "2017-08-22 14:37:53+03:000",
		"objectDBName": "DEV2",
		"objectSchemaName": "YISHAIN",
		"objectId": "CUSTOMERS",
		"changedFieldsList": [{
			"fieldId": "CUSTOMER_ID",
			"fieldType": "NUMBER",
			"fieldValue": "17",
			"fieldChanged": "N"
		}, {
			"fieldId": "CUSTOMER_FIRST_NAME",
			"fieldType": "VARCHAR2",
			"fieldValue": "Daniel",
			"fieldChanged": "N"
		}, {
			"fieldId": "CUSTOMER_LAST_NAME",
			"fieldType": "VARCHAR2",
			"fieldValue": "Washington",
			"fieldChanged": "N"
		}, {
			"fieldId": "CUSTOMER_COUNTRY",
			"fieldType": "VARCHAR2",
			"fieldValue": "France",
			"fieldChanged": "N"
		}, {
			"fieldId": "CUSTOMER_CITY",
			"fieldType": "VARCHAR2",
			"fieldValue": "La Roche-sur-Yon",
			"fieldChanged": "N"
		}, {
			"fieldId": "CUSTOMER_STREET",
			"fieldType": "VARCHAR2",
			"fieldValue": "43 Maple Pass",
			"fieldChanged": "Y"
		}, {
			"fieldId": "CUSTOMER_ZIPCODE",
			"fieldType": "VARCHAR2",
			"fieldValue": "85928 CEDEX 9",
			"fieldChanged": "N"
		}, {
			"fieldId": "PHONE_NUMBER",
			"fieldType": "VARCHAR2",
			"fieldValue": "33-(414)618-3273",
			"fieldChanged": "N"
		}, {
			"fieldId": "EMAIL",
			"fieldType": "VARCHAR2",
			"fieldValue": "dwashingtong@tiny.cc",
			"fieldChanged": "N"
		}, {
			"fieldId": "GENDER",
			"fieldType": "VARCHAR2",
			"fieldValue": "Male",
			"fieldChanged": "N"
		}, {
			"fieldId": "DATE_OF_BIRTH",
			"fieldType": "DATE",
			"fieldValue": "1968-04-01 00:00:00+03:000",
			"fieldChanged": "N"
		}, {
			"fieldId": "CREDIT_LIMIT",
			"fieldType": "NUMBER",
			"fieldValue": "14161",
			"fieldChanged": "N"
		}, {
			"fieldId": "INCOME_LEVEL",
			"fieldType": "NUMBER",
			"fieldValue": "67035",
			"fieldChanged": "N"
		}, {
			"fieldId": "COMMENTS",
			"fieldType": "VARCHAR2",
			"fieldValue": "stuff",
			"fieldChanged": "N"
		}, {
			"fieldId": "LAST_UPDTAE",
			"fieldType": "DATE",
			"fieldValue": "2016-01-12 00:00:00+03:000",
			"fieldChanged": "N"
		}],
		"conditionFieldsList": [{
			"fieldId": "CUSTOMER_ID",
			"fieldType": "NUMBER",
			"fieldValue": "17"
		}, {
			"fieldId": "CUSTOMER_FIRST_NAME",
			"fieldType": "VARCHAR2",
			"fieldValue": "Daniel"
		}, {
			"fieldId": "CUSTOMER_LAST_NAME",
			"fieldType": "VARCHAR2",
			"fieldValue": "Washington"
		}, {
			"fieldId": "CUSTOMER_COUNTRY",
			"fieldType": "VARCHAR2",
			"fieldValue": "France"
		}, {
			"fieldId": "CUSTOMER_CITY",
			"fieldType": "VARCHAR2",
			"fieldValue": "La Roche-sur-Yon"
		}, {
			"fieldId": "CUSTOMER_STREET",
			"fieldType": "VARCHAR2",
			"fieldValue": "5 Maple Pass"
		}, {
			"fieldId": "CUSTOMER_ZIPCODE",
			"fieldType": "VARCHAR2",
			"fieldValue": "85928 CEDEX 9"
		}, {
			"fieldId": "PHONE_NUMBER",
			"fieldType": "VARCHAR2",
			"fieldValue": "33-(414)618-3273"
		}, {
			"fieldId": "EMAIL",
			"fieldType": "VARCHAR2",
			"fieldValue": "dwashingtong@tiny.cc"
		}, {
			"fieldId": "GENDER",
			"fieldType": "VARCHAR2",
			"fieldValue": "Male"
		}, {
			"fieldId": "DATE_OF_BIRTH",
			"fieldType": "DATE",
			"fieldValue": "1968-04-01 00:00:00+03:000"
		}, {
			"fieldId": "CREDIT_LIMIT",
			"fieldType": "NUMBER",
			"fieldValue": "14161"
		}, {
			"fieldId": "INCOME_LEVEL",
			"fieldType": "NUMBER",
			"fieldValue": "67035"
		}, {
			"fieldId": "COMMENTS",
			"fieldType": "VARCHAR2",
			"fieldValue": "stff"
		}, {
			"fieldId": "LAST_UPDTAE",
			"fieldType": "DATE",
			"fieldValue": "2016-01-12 00:00:00+03:000"
		}]
	}]
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 27, 2017, 12:52pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/9 "2017-11-27T12:52:34Z")

</div>

There are three reasons why the configuration you posted won't work with either the stdin or the file inputs:

- The split filter is listed before the JSON filter so the input won't have been deserialized into fields that you can split.
- You're splitting the `changedFieldList` field but there is no such field.
- The stdin input will, just like the file input, read its input line by line. If your JSON objects span over multiple lines you need to use a multiline codec.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2017, 12:52pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-stdin-and-file-in-logstash/108910/10 "2017-12-25T12:52:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.