Ability to define data types for custom elasticsearch fields
Activity
Philippe Weber June 17, 2014 at 4:45 AM
Grok is not related to Elasticsearch, ES templates are the way to go.
Jordan Sissel February 25, 2012 at 7:46 PM
That's quite true. For those situations, you'll need to set custom mappings in elasticsearch.
To use this feature well with logstash, you can use the 'index templates' feature which lets you define a template matching "logstash*" index names which have mappings you define.
Check it out here:
Avishai Ish-Shalom February 25, 2012 at 7:11 PM
Elasticsearch support many more types and mapping options. The above methods do not allow using these, e.g. the IP type.
Jordan Sissel February 24, 2012 at 11:34 PM
You can achieve this one of two ways right now in logstash:
using grok's type conversion
or using the mutate filter's convert feature
In Grok, if your pattern is named %{FOO}, you can force it to be an integer by giving it a name and a type: %{FOO:fooname:int}
Grok supports 'int' and 'float'. One gives you a whole number, one gives you a decimal number.
The second way is the mutate filter; http://logstash.net/docs/1.1.0/filters/mutate#setting_convert – example
Ideally, the right place to do this is in the grok pattern definition. I'll accept patches to any grok patterns you need to update but will otherwise put doing this on my todo list.
It seems that any custom fields defined in logstash today (e.g. via grok) will end up as strings by default in elastic search. Some ES capabilities such as statistical facets naturally only work on numeric data types. It would be nice if in the grok filter you could define an optional data type and if present, logstash would use this when creating the ES index.
For now, as a work around, you can define an ES template for logstash and include your field data types in there. When logstash then creates a new index the custom fields will have the correct data type. If you define a new custom field in your logstash config you just need to make sure to update your ES logstash template if you want the value to be stored as something other than a string.
More on the available ES data types here: http://www.elasticsearch.org/guide/reference/mapping/core-types.html