Atlassian access logs Index not getting created or data not sent / visible in Opensearch

Elastic Stack Logstash
1

I'm trying to use Logstash to send Atlassian access logs to opensearch.

I'm absolutely new to the topic but can successfully send other logs and view them.

It's the jira access logs that I cannot make work.
Having tried a lot in vain, I'm hoping to get an answer in the community. Please help...

My conf file for the jira access log is below:

input {
  file {
    path => "/path/to/jira/application/logs/access_log.%{+YYYY-MM-dd}"
    codec => multiline {
      pattern => "^\s"
      what => "previous"
      }
    }
}

filter {
    grok {
            match => { "message" => "^%{IP:ip}\s+%{WORD:userid}\s+%{USER:user}\s+\[%{HTTPDATE:timestamp}\]\s+%{GREEDYDATA:message}" }
    }
    date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}

output {
  opensearch {
    hosts => ["https://nn.nnn.nn.nn:9200"]
    ssl_certificate_verification => false
    index => "jira-access-log-%{+YYYY.MM.dd}"
    user => "logstash"
    password => "somepassword"
  }
}

The example log lines are below:

NNN.NNN.NNN.NN 123x456x789 - [26/Aug/2023:12:53:21 +0200] "GET /rest/capabilities/navigation?lang=de-DE HTTP/1.0" 200 1117 8 "-" "Confluence-7.19.12 (18035)" "-"
NNN.NNN.NNN.NN 901x2345x67890 some.string.text [26/Aug/2023:12:55:13 +0200] "PUT /rest/api/latest/issue/DUMMYIS-1234- HTTP/1.0" 400 123 13 "-" "Atlassian HttpClient 0.23.0 / Atlassian JIRA Rest Java Client-4.0.3-sc (0) / Default" "username"

I tried the filter plugin to use grok (and it itself seems to work because I have tried it for at least one log line example shown above. I added the date but I'm not sure if it's right/needed...

I also tried to use dissect but I am not sure if it worked...

filter {
  dissect {
    mapping => { "message" => "%{ip} %{id} %{user} [%{[@metadata][timestamp]} %{timezone}] %{message}" }
  }
  date {
    match => [ "[@metadata][timestamp]", "dd-MMM-yyyy:HH:mm:ss" ]   #is this correct?
    target => "@timestamp"
  }
}

Logstash starts without errors and the pipeline is successfully created.

However I'm not sure if I did it right because I cannot create an index pattern in opensearch..
I get: "The index pattern you've entered doesn't match any indices."

So I'm not sure if the index is even getting created / data being sent. And whether I need to do anything else in opensearch itself...?

I'd really appreciate it if someone could please help me figure this out? Thanks so much,
Dan

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

Welcome to the community.

This is a single line log, no need for the multiline codec.
Also you can improve grok with more fields if you like.

input {
  file {
   path => "/path/to/jira/application/logs/access_log.*"
   start_position => beginning
   #sincedb_path => "/dev/null" # if you like add comment to track record about lines in the default sincedb or /dev/null to keep in memory(useful for the testing)
   }
}
4

Thanks Rios, I'll remove the multiline codec.

Is the current grok somehow preventing the index creation?
How would I know the index is being being created?
I don't see it in opensearch's Dev Tools using "GET /_cat/indices"
Thanks

5

And also if I put nothing in the filter:

filter {
}

I still don't see the index in opensearch, so I can't create any index pattern for it...

6

Are you sure that logstash is reading the log file correctly?

You need to troubleshoot this first, remove the opensearch output and add a file output or stdout output to see at least if logstash is reading and processing the logs.

Also, how are you running Logstash? As a service? Can you start it again to get fresh logs and share the logs here?

7

If you not set sincedb_path, LS will keep record about files which are read.
Try with for test:

input {
  file {
   path => "/path/to/jira/application/logs/access_log.*"
   start_position => beginning
   sincedb_path => "/dev/null" 
   }
}
8

Hi leandrojmp, I am running LS as a systemd service. So, yes it would be good to see the output going into a file.

However would I be able to redirect output to a file on the same server on which LS is running?

I ask because I can't ssh login to the target opensearch server for now.

So, would the snip below work?

output {
  stdout { 
     codec =>  "json_lines" 
     path => "/tmp/checklogstash.tmp"
   }
}

Also which logs do you want me to share? The actual access logs that LS will work on or the logstash-plain.log ?

10

Thanks for your suggestion Rios, i will try it out.

11

ok so yes the output is to a file on the same server. I just tried it with some test data on a test server... I will check the access log output if it works soon and goes into a file.

12

Hi leandrojmp,
When I use a file output to see at least if LS is reading and processing the logs, the file does not get created on my production server. It does however get created in my test server.
I have rw permissions, there is space on the disk.
My full config is here including the input, filter and output:

input {
  file {
    path => "/path/to/jira/acces_log.%{+YYYY-MM-dd}"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
    grok {
            match => { "message" => "^%{IP:ip}\s+%{WORD:userid}\s+%{USER:user}|-\s+\[%{HTTPDATE:timestamp}\]\s+%{GREEDYDATA:message}" }
    }
    date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}

output {
                file {
                path => "/path/to/logstash.out"
                }
}

If I can see the output file and it's content I would know if the input and filter is working before checking why an index is not being created....
Please help..
Thanks

13

Also if I only have an input and output ( with file ) i.e without any filter section, I still don't see an output file...

14

You didn't share any logs, please restart logstash, get the logs and share then, this will show if there is any errors or warnings.

What is this path? When you run Logstash as a service it will run as the logstash user, so you need to make sure that the logstash user can in fact read those logs, check the path permissions.

15

The file input does not sprintf the path option, so I don't think this does what you want. I cannot figure out what it does do. It doesn'i tail /path/to/jira/acces_log.%{+YYYY-MM-dd}, nor does it tail /path/to/jira/acces_log.2023-08-27. Even trace level logging does not make it clear what it is doing.

1 Like
16

Hi Badger, thanks for replying.

My input file is and path is below:

The input describes which log I want to process and I have listed it below too.

These are essentially jira access log files that get rotated daily.

input {
  file {
    path => "/path/to/the/access_log.%{+YYYY-MM-dd}"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
ls -l /path/to/the/access_log.2023-08-27
-rw-r----- 1 <user> <group> 1593933 Aug 27 18:19 /path/to/the/access_log.2023-08-27

If the file cannot be read or opened it would make sense why there's no output, but I doubt it unless I'm wrong.

17

Just noticed it now after Badger comment.

This path => "/path/to/the/access_log.%{+YYYY-MM-dd}" does not work, you need to use a glob.

Something like this> path => "/path/to/the/access_log.*, using %{+YYYY-MM-dd}, will not work.

Try this:

input {
  file {
    path => "/path/to/the/access_log.*"
    start_position => "beginning"
    sincedb_path => "/dev/null" # only use this when testing, when it works remove this line
  }
}

Does the logstash user have permission in this path and this file? If the logstash user does not have permission, logstash won't be able to read the files, you are redacting both the path and the user/group, so it is not possible to know. You need to check the permissions.

Also, you didn't share any logs, so it is not possible to know if there is any issue while logstash is starting or not.

18

The file input does not expand the date reference to the current date. Even if it did then it would be a one time event and logstash would not re-expand it each day.

The standard approach for this would be to use path => "/path/to/the/access_log.* and let the file input track old files and rollovers in the sincedb.

1 Like
19

Hi leandrojmp and Badger,

Something worked!

I changed the file input path as you both suggested, to

input {
  file {
    path => "/path/to/the/access_log.*"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

But there are a couple of problems now:
(1)
I have strange message in the logstash file like:


[2023-08-27T19:01:15,622][WARN ][logstash.codecs.plain    ][jira-access-pipeline][15ed61b624792bc0909a82bf7de16475dec0fbb231fbff95fef42ce52bfa2802] Received an event that has a different character encoding than you configured. {:text=>"\\xDB\\r\\xF6\\xF0\\xC1\\t\\u0000\\u001C\\x9A<\\xB1w[T\\u0010x#FRT\\xB0L\\xF0ͦ+\\xC8\\xE9\\xB3\\xF9z\\xF1...etc.etc.etc 
 
....etc..\\u0005^\\xB7D\\x82\\xFD\\xA5W\\xE4\\u0001\\u0010\\u0000\\u0017\\xE3۩\\xE6\\x80\\xF2z|\\u0013\\xE9EUDEUQUDQUDUEUEUF(\\x8A\\xAB\\u0011DEE\\u0015EE\\u0015UU\\u0014UTQ\\u0015\\u0014UEUUUEUTQb\\xACQUUUE\\u0011UQEEU\\u0011PUQX\\xAA*\\xAA1QAUUUTEX\\xAA\\x8A\\x88\\xA8\\xAA1c\\u0010b\\xAA\\xAA\\xA8\\xA0\\xA8\\x8A\\b\\xAA**\\xAA\\xA8\\x8A\\xACb\\xA8\\xA8\\xC4ETU\\x8A\\xAD8\\u0000", :expected_charset=>"UTF-8"}
[2023-08-27T19:01:15,953][WARN ][logstash.codecs.plain    ][jira-access-pipeline][15ed61b624792bc0909a82bf7de16475dec0fbb231fbff95fef42ce52bfa2802] Received an event that has a different character encoding than you configured. {:text=>"\\xCE\\xF6\\x86Q\\x8....etc...xFA\\u0019\\xBFBG\\a\\xA4\\x82\\x90R", :expected_charset=>"UTF-8"}

(2)
I can see the index being created in opensearch - or atleast I when I go to create index pattern, I can see the correct index name, But after I choose select a primary time field for use with the global time filter as @timestamp, I get an error during creating the index pattern which says:

Error

No Reason Phrase

So I can't create the index. So near :slight_smile: yet...
please help with this..
Thank you.

20

Also, leandrojmp, I will removed the sincedb line like you suggested...

I will try a new grok or dissect filter because I am getting a grokparsefailure like:

tags"=>["_grokparsefailure"], "message"=>"B?\\xAD*7\\x9D$n\\xB4\\xAB\\xDFۥ/D\\xED\\x90\\xE5a\\u0000I\\xE5\\b\\x91\\u0001]\\xDD\\u0010\\xA4\\xED-f\\u0015\\xA2\\x9E\\xBBߓf֮\\xD983!\\xDD Dw......etc....etc.....99\\xC2GuT\\xF1\\u0017", "host"=>{"name"=>"myhostname.com"}}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [jira-access-log-2023.08.27] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}

21

And, I noticed it's also 'processing' the zipped rotated files because I used '*' in the path.
Does it unzip these before processing?

Since my logs are rotated, I have

access_log.2023-08-27 and I also have
access_log.2023-08-26.bz2
access_log.2023-08-25.bz2 etc

If it's reading compressed input, it might not grok...
Waiting for your thoughts, Thanks again.