Ability to define data types for custom elasticsearch fields

Description

It seems that any custom fields defined in logstash today (e.g. via grok) will end up as strings by default in elastic search. Some ES capabilities such as statistical facets naturally only work on numeric data types. It would be nice if in the grok filter you could define an optional data type and if present, logstash would use this when creating the ES index.

For now, as a work around, you can define an ES template for logstash and include your field data types in there. When logstash then creates a new index the custom fields will have the correct data type. If you define a new custom field in your logstash config you just need to make sure to update your ES logstash template if you want the value to be stored as something other than a string.

More on the available ES data types here: http://www.elasticsearch.org/guide/reference/mapping/core-types.html

Activity

Show:
Jordan Sissel
February 24, 2012, 11:34 PM

You can achieve this one of two ways right now in logstash:

  • using grok's type conversion

  • or using the mutate filter's convert feature

In Grok, if your pattern is named %{FOO}, you can force it to be an integer by giving it a name and a type: %{FOO:fooname:int}

Grok supports 'int' and 'float'. One gives you a whole number, one gives you a decimal number.

The second way is the mutate filter; http://logstash.net/docs/1.1.0/filters/mutate#setting_convert – example

Ideally, the right place to do this is in the grok pattern definition. I'll accept patches to any grok patterns you need to update but will otherwise put doing this on my todo list.

Avishai Ish-Shalom
February 25, 2012, 7:11 PM

Elasticsearch support many more types and mapping options. The above methods do not allow using these, e.g. the IP type.

Jordan Sissel
February 25, 2012, 7:46 PM

That's quite true. For those situations, you'll need to set custom mappings in elasticsearch.

To use this feature well with logstash, you can use the 'index templates' feature which lets you define a template matching "logstash*" index names which have mappings you define.

Check it out here:

Philippe Weber
June 17, 2014, 4:45 AM

Assignee

Unassigned

Reporter

D

Affects versions

Configure