ES mapping for GELF input data not correct (user fields)

Description

Dear all,

I've checked the new version, today. Many thanks for that excellent work. It fixes a lot.

By checking the new mappings for GELF data as an input, I found an issue with the field names. GELF differentiate between standard (manadatory) fileds an so called user fields. Die difference is quite simple to see in my example below:

{
"version": "1.0",
"host": "localhost",
"short_message": "Tue Jul 23 21:16:05.112 [initandlisten] ...",
"full_message": "Tue Jul 23 21:16:05.112 [initandlisten] ...",
"timestamp": 1291899928.412,
"level": 2,
"facility": "MongoDB",
"file": "C:\lx\mongodb\logs\mongodb.log",
"line": -1,
"_user_1": "my value 1",
"_user_2": "my value 2",
...
}

All user fields start with an underscore.

The new mapping introduced with version 1.2.0 remaps all GELF field >>without<< an underscore for user fields. It would be very helpful if the naming convention for GELF is used for logstash as well.

Many thanks for help,
regards

Juergen Adams

Activity

Show:
Juergen Adams
September 7, 2013, 8:13 AM

Hi all,

found the discussion adressing the new logstash event schema with https://logstash.jira.com/browse/LOGSTASH-675. Unsure, if it is possible the change the schema again.

The issue I've with the new schema is that i've to rewite all my programs for that mapping used for Graylog2 in the past. Anyway, I'll implement an abstraction layer for my programs to remap the schema Graylog2/Logstash in ES and vice versa. It should not that big deal to do this.

From my side, you can close that ticket or leave it open for further discussion.

Regards,
Juergen

Richard Pijnenburg
September 13, 2013, 8:47 AM

Hi,

Im afraid the schema won't be changed.
Is there anything we can do to make it easier?

Juergen Adams
September 23, 2013, 10:02 AM

Hi Richart,,

sorry for delayed answer. I was on a longer business trip.

Thanks for your answer. I found a way to indicate what mappings are related to the GELF protocols resp. GELF as a source. Logstash offers with the gelf input to add additional mappings. This way it is easy to "mark" the documents/messages with that additional mapping as send via GELF:

  1. Optional/hash. Add any number of key/value pairs that will result in

  2. additional mappings with the ES database for all messages. This may

  3. help with processing/filtering events later eg. by the web interface

  4. or using Query DSL request with ES
    add_field => [ "input", "gelf" ]

That means independently from the resulting mappings in ES by Logstash, adding an additional field clearly identifies the source.

Thanks a lot,
Juergen

Philippe Weber
August 4, 2014, 10:34 AM

Reporter solved is issues with event schema transition

Fixed

Assignee

Logstash Developers

Reporter

Juergen Adams

Affects versions

Configure