I'm trying to output to S3 6 month of data coming from an elasticsearch.
I have a backlog of 1 week for an index named logstash-* and a backlog of 6 month for an other index.
If I configure my elasticearch input for logstash-* it works but for another-* it doesn't.
We can see errors on elasticsearch with this info :
[2013-10-09 17:43:12,075][WARN ][http.netty ] [i-338c3651] Caught exception while handling client http traffic, closing connection [id: 0x9c6588c8, /10.45.10.177:40916 => /10.45.10.92:9200]
org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException: An HTTP line is larger than 4096 bytes.
If I do a tcpdump I can see the scroll_id in the request is huge which could explain the 4096 bytes limit.
Is there anyway to post the scroll_id rather than to set it in the URL.
My problem seems similar to that one :
What do you think?
I guess it can be a big problem for people having a lot of data to migrate from Logstash 1.1 to 1.2 as the logic will be the same (use elasticsearch input)