Matching/Finding certain text using logstash filter

Elastic Stack Logstash
1

I have a text with [INFO or ERROR or Warn] concatenated with few other texts, which i am able to get under greedy data, But I want to split those Log Level values under new field.

Sample Text: 2016 Apr 26 15:44:40:603 GMT +0530 BW.Service-1-Service Info [BW-User] - Job-8309-2 [Logger/LogToLocal.process/Log]: Processed Service_01 with BusinessObjectId:1111 and ConversationId:11111 and MessageId:1111
If reg ex is the solution, kindly suggest with example.
Help or suggestion would be very helpful.

2

Use a grok filter to extract new fields from text in existing fields. Show us what you have so far. Presumably you already have a grok filter to parse the line. In the example above, is "Info" right after "BW.Service-1-Service" the string you want to extract?

3

Hi Magnus
Thanks for the reply,

Yes it is right after that. I am now able parse the text like below using the grok filter.

Text: 2016 Apr 26 15:44:40:603 GMT +0530 BW.Service-1-Service Info [BW-User] - Job-8309-2 [Logger/LogToLocal.process/Log]: Processed Service_01 with BusinessObjectId:1111 and ConversationId:11111 and MessageId:1111

%{YEAR} %{MONTH} %{MONTHDAY} %{TIME} GMT +%{INT} %{PROG:program} %{WORD:loglevel} [%{USER:auth}] %{GREEDYDATA:BwLog1}

Getting matched and able to see the data in kibana.

But in Kibana Visualization, these tags PROG:program is not showing up for filtering.. Same is available in Discover section but not in Visualization.

4

But in Kibana Visualization, these tags PROG:program is not showing up for filtering.. Same is available in Discover section but not in Visualization.

Sorry, I don't quite understand what the problem is. Maybe a screenshot would make it easier for you to explain what the problem.

5

Below is the Discover and Visualization /pie chart sections screen shots.

Program section getting displayed under discover is not getting displayed in visualization.

Capture1.PNG843×656 31.1 KB

Capture2.PNG1061×633 28.8 KB

6

Hi Magnus,
I found the solution, By Refreshing the settings section able to get the field in visualization section.

Can you give brief explain for what is Analysed Field and Indexed Field.

Thanks in advance.

7

See the description of the index attribute for fields for a definition of these terms: https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#_index_2

See https://www.elastic.co/guide/en/elasticsearch/guide/current/analysis-intro.html for a description of what analysis is all about.

8

Hi Magnus,

Everything was working fine. Suddenly facing few issues.

We have 4 different machine running logstash, and shipping data to elasticsearch running in different machine on same network. Issue are as below,

  1. In kibana, not able to view data from 3 machines, only one machine data is visible on kibana.
  2. In Kibana when i do monitoring particular field, and tried to sort that, receiving error as
    Failed to execute [org.elasticsearch.action.search.SearchRequest@74a9c9f4] lastShard [true],nested: IllegalStateException[Field data loading is forbidden on [jobid]];,Caused by: java.lang.IllegalStateException: Field data loading is forbidden on [jobid]
9

Please start new threads for your new problems instead of reviving old threads that discuss something else.