amazon web services - Logstash & Elasticsearch issues -

i need pro figure issue es & logstash.

i'm using elasticsearch 1.1.0 , logstash 1.4.0 push logs es & kibana .

my servers located in aws ( master -> v.core (4) , ecu (8) , ram (15 gb) ) node same .

my logstash configuration :

i'm taking log files s3 , put locally in server after logstash take them , push them es cluster . logstash + kibana + es master (located in 1 server) files 12m size , have more 20,000 files.

my es configuration (master)

cluster.name: my-cluster-name node.name: my-node-name node.master: true node.data: true path.data: /path_to_data/data path.logs: /path_to_logs/logs 

es configuration (node)

cluster.name: my-cluster-name (same name) node.name: my-node-name (another name) node.master: false (this node not master) node.data: true path.data: /path_to_data/data path.logs: /path_to_logs/logs 

for checking cluster status :

http://master_ip:9200/_cluster/health 

this result :

{   "cluster_name": "es-cluster-onetagv2",   "status": "green",   "timed_out": false,   "number_of_nodes": 2,   "number_of_data_nodes": 2,   "active_primary_shards": 5,   "active_shards": 10,   "relocating_shards": 0,   "initializing_shards": 0,   "unassigned_shards": 0 } 

my java version ( don't know if important ) :

java version "1.7.0_51" openjdk runtime environment (amzn-2.4.4.1.36.amzn1-x86_64 u51-b02) openjdk 64-bit server vm (build 24.45-b08, mixed mode)

my issue i'm trying push more 400 million hits 1 day , , can push in 24 hours 60 million. i'm behind.

i can see es taking 100% cpu usage. don't know if problem.

maybe can guide me i'm doing wrong , how can push big logs es fast.

split input file , give each file different name multiple inputs work on it, helps logstash read multiple files parallelly different nodes.

please increase number of es datanodes 4, , 2 es masters. can achieved running 2 logstash instances( provides 2 masters , 2 datanodes) , 2 elasticsearch nodes(provides 2 data nodes).