Argthtjtr

I started using and configuring ELK stack a month ago as a personal project in the IT company I'm working in.

Without any training or a coding background, my Logstash works good enough to accept Logs from Cisco ASR5K on StarOS and some IOS devices, send them to Elasticsearch and also store them in RAW format.

I'm trying now to tune up a little bit the fields that I've created for this config and I'm encountering some difficulties, so I really hope that you could help me. Any feedback will be welcomed.

Ok, so my problem is the following:
I have a log in RAW format

<171>Nov 13 18:11:03 evlogd: [local-60sec3.378] [diameter 92004 error] [8/0/6001 <diactrl:0> flow_id_mgmt.c:743] [software internal system syslog] Flow ID sync with facility 87000 instance 377 failed 82649 times

And I have applied the following match

%{ASR_TIMESTAMP:log_date} %{ASR_TASK:task} [%{ASR_FACILITY:facility}] [%{ASR_INSTANCE:instance_info}] [%{ASR_MGR:mgr}] [%{ASR_LEVEL:level}] %{ASR_RAW:message}

With the following custom pattern:

CISCOTIMESTAMPTZ %{CISCOTIMESTAMP}( %{TZ})?

NEXUSTIMESTAMP %{YEAR} %{MONTH} %{MONTHDAY} %{TIME}( %{TZ})?

ASR_TIMESTAMP %{MONTH} %{MONTHDAY} %{TIME}

ASR_TASK %{DATA}

ASR_FACILITY %{DATA}

ASR_EVENT_ID %{DATA}

ASR_LEVEL %{DATA}

ASR_EVENT_LEVEL %{WORD}

ASR_INSTANCE %{DATA}

ASR_CONTEXT %{DATA}

ASR_EVENT %{WORD}

ASR_TYPE %{DATA}

ASR_MGR %{DATA}

ASR_RAW %{GREEDYDATA}

ASR_CALLID %{DATA}

And everything works fine (well, at least for me). The logs are matching so that's a good start.

{

         "log_date" => "Nov 15 13:20:07",

--

         "hostname" => "MM-MME",

      "fingerprint" => "9414227242b545335cb679d9905dddf6b8976cf2",

             "task" => "evlogd:",

            "level" => "software internal system syslog",

             "host" => "X.X.X.X",

              "mgr" => "8/0/6001 <diactrl:0> flow_id_mgmt.c:743",

          "message" => "Flow ID sync with facility 87000 instance 75 failed 83718 timesu0000",

       "@timestamp" => 2018-11-15T11:20:07.439Z,

    "instance_info" => "diameter 92004 error",

         "facility" => "local-60sec7.199",

             "type" => "syslog-cisco"

}

What I want to to, is to delete the Number from the "instance_info" field to be "diameter error"

Why? I want to create a visualization pie chart with the data from this field, and the numbers are absolutely irrelevant for me. More than that, it screws my piechart by creating thousands of:

sessmgr 10033 trace

sessmgr 10036 trace

sctp 87304 error

sgsn-gtpc 116043 debug

sessmgr 12902 debug

sgs 173002 debug

nas 153002 debug

sgs 173001 debug

sgsn-gtpc 116004 debug

sessmgr 12093 error

sgsn-gtpc 116003 debug

sgsn-gtpc 116094 info

sgsn-gtpc 116095 info

sgsn-gtpc 116120 debug

sgsn-gtpc 116122 debug

sgsn-gtpc 116057 debug

sessmgr 10177 debug

sessmgr 10207 info

sessmgr 10285 debug

aaamgr 36861 debug

sessmgr 10004 info

sessmgr 10738 trace

sessmgr 12952 info

aaa-client 6002 debug

aaa-client 6003 debug

And there should be roughly 15-20 results like:

sgsn-gtpc info/debug etc

sessmgr info/debug/error etc

aaamgr info/debug/error etc

I tried based on the following action plan:
Split the "instance_info" intro 3 fields:

ASR_INSTANCE %{WORD}

ASR_INSTANCE_NR %{NUMBER}

ASR_INSTANCE_LVL %{WORD}

And create a new match:

%{ASR_TIMESTAMP:log_date} %{ASR_TASK:task} [%{ASR_FACILITY:facility}] [%{ASR_INSTANCE:instance_info}%{SPACE}%{ASR_INSTANCE_NR:nr}%{SPACE}%{ASR_INSTANCE_LVL:Level}] [%{ASR_MGR:mgr}] [%{ASR_LEVEL:level}] %{ASR_RAW:message}

Which works (tested in an online GROK debugger):

"instance_info": [

    [

      "diameter"

    ]

  ],

  "nr": [

    [

      "92004"

    ]

  ],

  "Level": [

    [

      "error"

Now I want to delete the "nr" field and combine ["instance_info", "Level"] into one single field. And to be honest I have no idea how.
I tried with the mutate plugin but I lack the knowledge to understand how and where to insert it.

Kindly ask you to help me in this matter or to suggest a different way.
Config file: 01-cisco.conf

I know for most of you, my config file might look very creepy, but that's the best I could do, so please don't judge my frankenstein.conf file.
Any kind of help and suggestions for my issue, or for any line in the config will be most welcomed!

You are nearly there. All you have to do is use mutate/replace to combine the data you need and mutate/remove_filed to remove the unnecessary fields.

You need to change the ASR_INSTANCE_LVL field name not to be level, since you already have level field. Lets say you would call it asr_level.

Add the relevant mutate at the end of you conf file.

mutate {

   replace => [ "instance_info", "%{instance_info} %{asr_level}" ]

}

 mutate {

   remove_field => ["asr_level", "nr"]

}

You can look at this google forum and https://discuss.elastic.co/t/want-to-combine-field-values-into-new-field-value/28564 for more details.

Good Luck