Beginner issue - testing output to stdout

Elastic Stack Logstash
1

All im basically trying to parse a json file to pick out 2 fields
field 1 is name
feild 2 is product
my ls.conf

input {
file {
type => "mefile"
path => "/export/home/me/me.json"
sincedb_path => "/dev/null"
}
}
filter{
json{
source => "name"
target => "thename"
source => "product"
target => "theproduct"
}
}
output {
stdout { }
}

I want to output to go to the screen so I can verify the data is what I think. All I get is "pipeline Main started".
I've tried removing the filter, expecting the file to be sent to stdout - but still get pipeline main started.

final expectation is to only send name and product to ES.

HELP!

2

Also, Ive spent about 2 hours playing with this :slight_smile:

3

Logstash is tailing the input file. Set start_position => beginning for the file input. You should also familiarize yourself with its ignore_older option.

json{
source => "name"
target => "thename"
source => "product"
target => "theproduct"
}

You're misunderstanding how the filter works. The source option names the field that should be parsed. In this case each line of the input file will be stored in the message field. The json filter will always deserialize the whole JSON message. If you don't want all fields to tag along you can add a prune filter afterwards that removes all fields except the ones you want to keep.

4

Ok thank you. How can I get the output to stdout so I can see what's going on?

Thanks again

5

You already have a stdout output so you're all set. Almost; I recommend you use the rubydebug codec for your stdout output so that you see exactly what each event contains:

output {
  codec => rubydebug
}
6

Very nice thank you!

Can you give a hint or more about pulling just a few fields out of the message?

7

If I add this filter its even sweeter! very nice

filter{
json{
source => "message"
}


here are the feilds of interest...
     "@version" => "1",
        "@timestamp" => "2016-08-18T23:31:48.155Z",
              "path" => "..d...d",
              "host" => "hostname",
              "type" => "PUREPATH",
              "name" => " _BrowserType_SPLT_BT",
       "application" => "UCFE2 Application",
        "purePathId" => "PT=478227;PA=-727720757;PS=1336977021",
         "startTime" => "2016-08-06 00:52:04.757-0400",
        "dimensions" => {
        "nameBrowserType_SPLT_MSR" => "IE"
    },
            "failed" => false,
      "responseTime" => 901.0671997070312,
          "duration" => 901.067138671875,
           "cpuTime" => 790.1187744140625,
          "execTime" => 901.0671709775925,
    "suspensionTime" => 18.81926155090332,
          "waitTime" => 11.579818453107563


say I only need 3 (prune) ???
8

say I only need 3 (prune) ???

Add a prune filter that sets the whitelist_names option to a list of the fields you want to keep. Keep in mind that the strings in the last are interpreted as regular expressions.

9

Ok I will try that today, thanks you have been immensely helpful to get me going