amazon web services - Logstash & Elasticsearch issues -

-

i need pro figure issue es & logstash.

i'm using elasticsearch 1.1.0 , logstash 1.4.0 push logs es & kibana .

my servers located in aws ( master -> v.core (4) , ecu (8) , ram (15 gb) ) node same .

my logstash configuration : enter image description here

i'm taking log files s3 , put locally in server after logstash take them , push them es cluster . logstash + kibana + es master (located in 1 server) files 12m size , have more 20,000 files.

my es configuration (master)

cluster.name: my-cluster-name node.name: my-node-name node.master: true node.data: true path.data: /path_to_data/data path.logs: /path_to_logs/logs

es configuration (node)

cluster.name: my-cluster-name (same name) node.name: my-node-name (another name) node.master: false (this node not master) node.data: true path.data: /path_to_data/data path.logs: /path_to_logs/logs

for checking cluster status : 

http://master_ip:9200/_cluster/health

this result : 

{   "cluster_name": "es-cluster-onetagv2",   "status": "green",   "timed_out": false,   "number_of_nodes": 2,   "number_of_data_nodes": 2,   "active_primary_shards": 5,   "active_shards": 10,   "relocating_shards": 0,   "initializing_shards": 0,   "unassigned_shards": 0 }

my java version ( don't know if important ) :

java version "1.7.0_51" openjdk runtime environment (amzn-2.4.4.1.36.amzn1-x86_64 u51-b02) openjdk 64-bit server vm (build 24.45-b08, mixed mode)

my issue i'm trying push more 400 million hits 1 day , , can push in 24 hours 60 million. i'm behind.

i can see es taking 100% cpu usage. don't know if problem.

maybe can guide me i'm doing wrong , how can push big logs es fast.

split input file , give each file different name multiple inputs work on it, helps logstash read multiple files parallelly different nodes.

please increase number of es datanodes 4, , 2 es masters. can achieved running 2 logstash instances( provides 2 masters , 2 datanodes) , 2 elasticsearch nodes(provides 2 data nodes).


Comments

Post a Comment

Popular posts from this blog

c# - Unity IoC Lifetime per HttpRequest for UserStore -

-
i'm trying clean default implementation of accountcontroller.cs comes out of box in new mvc5/owin security implementation. have modified constructor this: private usermanager<applicationuser> usermanager; public accountcontroller(usermanager<applicationuser> usermanager) { this.usermanager = usermanager; } also, have created lifetime manager unity looks this: public class httpcontextlifetimemanager<t> : lifetimemanager, idisposable { private httpcontextbase _context = null; public httpcontextlifetimemanager() { _context = new httpcontextwrapper(httpcontext.current); } public httpcontextlifetimemanager(httpcontextbase context) { if (context == null) throw new argumentnullexception("context"); _context = context; } public void dispose() { this.removevalue(); } public override object get...
Read more

Change the color of an oval at click in Java AWT -

-
i have draw ovals in java, , @ click change color. beginning tried change color after 20 ms, doesn't work. my code is: public class mycomponentnew extends frame { public graphics2d g2d; public mycomponentnew(string title) { super(title); setsize(400, 550); } @override public void paint(graphics g) { this.g2d = (graphics2d) g; this.g2d.setcolor(color.red); this.g2d.filloval(10, 55, 50, 100); } public void changecolor () { this.g2d.setcolor(color.blue); this.g2d.filloval(10, 55, 50, 100); } } and in class main method have: mycomponentnew m; m = new mycomponentnew("fereastra cu baloane"); m.setvisible(true); m.addwindowlistener(new windowadapter() { @override public void windowclosing(windowevent we) { system.exit(0); } }); try { thread.sleep(20); } catch(interruptedexception e) {} m.changecolor(); the color of oval remains red. ...
Read more

I am trying to solve the error message 'incompatible ranks 0 and 1 in assignment' in a fortran 95 program. -

-
i writing program in fortran find velocity of parachuting person in relation time. keep getting error can't fix. new programming , appreciated. the error is v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) 1 error: incompatible ranks 0 , 1 in assignment @ (1) and program is program para integer :: real :: v(11) !velocity real :: q !initial velocity real :: h !time step real :: c !drag coefficient real :: m !mass ! gravity equal 32 ft/s^2 write (*,*)'enter time step' read(*,*)h write(*,*)'enter initial velocity' read(*,*)q write(*,*)'enter drag coefficient' read(*,*)c write(*,*)'enter mass' read(*,*)m i=1,10 ! 1 10, 1 being interval. end v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) q=v(1) end program you cannot use [] normal parenthesis in expressions. array constructor, [ items ] means array items elements. end do should after line.
Read more