apache pig - Hadoop: Pig error -

-

i absolute beginner in hadoop, , doing simple testing, however, not find error messages informative.

i have set hadoop environment in single-node mode on centos 6.4 vm 4gb of ram available.

i trying run simple pig script on 500mb csv file. have 2 500mb files, on first one, script successful. on second one, same size, different data (a lot more rows), error when execution reaches 60%.

this (very simple) pig script use:

records = load 'trans2013.csv' using pigstorage(',') (podracun_v_breme,datum_transakcije,znesek_transakcije,oznaka_valute_transakcije,racun_v_dobro,naziv_prejemnika,maticna_stevilka,davcna_stevilka,sifra_pu,zr_sns_oe,namen); transaction_recs = group records all; tot_trans = foreach transaction_recs generate sum(records.znesek_transakcije); store tot_trans '/user/root/totaltransactions';

this error in terminal:

2014-04-06 10:28:29,147 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 64% complete 2014-04-06 10:28:30,240 [main] warn org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - ooops! job has failed! specify -stop_on_failure if want pig stop on failure. 2014-04-06 10:28:30,241 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - job job_1396637732046_0008 has failed! stop running dependent jobs 2014-04-06 10:28:30,241 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 100% complete 2014-04-06 10:28:30,460 [main] error org.apache.pig.tools.pigstats.simplepigstats - error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

2014-04-06 10:28:30,461 [main] error org.apache.pig.tools.pigstats.pigstatsutil - 1 map reduce job(s) failed! 2014-04-06 10:28:30,463 [main] info org.apache.pig.tools.pigstats.simplepigstats - script statistics:

hadoopversion pigversion userid startedat finishedat features 2.0.6-alpha 0.11.1 root 2014-04-06 10:25:49 2014-04-06 10:28:30 group_by

failed!

failed jobs: jobid alias feature message outputs job_1396637732046_0008 records,tot_trans,transaction_recs group_by,combiner message: job failed! /user/root/totaltransactions,

input(s): failed read data "hdfs://localhost:8020/user/root/trans2013.csv"

output(s): failed produce result in "/user/root/totaltransactions"

counters: total records written : 0 total bytes written : 0 spillable memory manager spill count : 0 total bags proactively spilled: 0 total records proactively spilled: 0

job dag: job_1396637732046_0008

2014-04-06 10:28:30,463 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - failed! 2014-04-06 10:28:30,491 [main] error org.apache.pig.tools.grunt.gruntparser - error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

details @ logfile: /root/pig_1396797945352.log

here error log:

backend error message --------------------- attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

pig stack trace --------------- error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

org.apache.pig.backend.executionengine.execexception: error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

at org.apache.pig.backend.hadoop.executionengine.mapreducelayer.launcher.geterrormessages(launcher.java:217) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.launcher.getstats(launcher.java:149) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher.launchpig(mapreducelauncher.java:400) @ org.apache.pig.pigserver.launchplan(pigserver.java:1264) @ org.apache.pig.pigserver.executecompiledlogicalplan(pigserver.java:1249) @ org.apache.pig.pigserver.execute(pigserver.java:1239) @ org.apache.pig.pigserver.executebatch(pigserver.java:333) @ org.apache.pig.tools.grunt.gruntparser.executebatch(gruntparser.java:137) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:198) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:170) @ org.apache.pig.tools.grunt.grunt.exec(grunt.java:84) @ org.apache.pig.main.run(main.java:604) @ org.apache.pig.main.main(main.java:157) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:606) @ org.apache.hadoop.util.runjar.main(runjar.java:212)

so ... tried running pig script mapreduce option:

pig -x mapreduce script.pig

it still failed, @ least produced meaningful error. seems had remove header csv file, because pig used header row data. seems happens when working floating point numbers - if using same script integers, header row ignored.

so it. first removed header file, running script against - worked.


Comments

Post a Comment

Popular posts from this blog

c# - Unity IoC Lifetime per HttpRequest for UserStore -

-
i'm trying clean default implementation of accountcontroller.cs comes out of box in new mvc5/owin security implementation. have modified constructor this: private usermanager<applicationuser> usermanager; public accountcontroller(usermanager<applicationuser> usermanager) { this.usermanager = usermanager; } also, have created lifetime manager unity looks this: public class httpcontextlifetimemanager<t> : lifetimemanager, idisposable { private httpcontextbase _context = null; public httpcontextlifetimemanager() { _context = new httpcontextwrapper(httpcontext.current); } public httpcontextlifetimemanager(httpcontextbase context) { if (context == null) throw new argumentnullexception("context"); _context = context; } public void dispose() { this.removevalue(); } public override object get...
Read more

Change the color of an oval at click in Java AWT -

-
i have draw ovals in java, , @ click change color. beginning tried change color after 20 ms, doesn't work. my code is: public class mycomponentnew extends frame { public graphics2d g2d; public mycomponentnew(string title) { super(title); setsize(400, 550); } @override public void paint(graphics g) { this.g2d = (graphics2d) g; this.g2d.setcolor(color.red); this.g2d.filloval(10, 55, 50, 100); } public void changecolor () { this.g2d.setcolor(color.blue); this.g2d.filloval(10, 55, 50, 100); } } and in class main method have: mycomponentnew m; m = new mycomponentnew("fereastra cu baloane"); m.setvisible(true); m.addwindowlistener(new windowadapter() { @override public void windowclosing(windowevent we) { system.exit(0); } }); try { thread.sleep(20); } catch(interruptedexception e) {} m.changecolor(); the color of oval remains red. ...
Read more

I am trying to solve the error message 'incompatible ranks 0 and 1 in assignment' in a fortran 95 program. -

-
i writing program in fortran find velocity of parachuting person in relation time. keep getting error can't fix. new programming , appreciated. the error is v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) 1 error: incompatible ranks 0 , 1 in assignment @ (1) and program is program para integer :: real :: v(11) !velocity real :: q !initial velocity real :: h !time step real :: c !drag coefficient real :: m !mass ! gravity equal 32 ft/s^2 write (*,*)'enter time step' read(*,*)h write(*,*)'enter initial velocity' read(*,*)q write(*,*)'enter drag coefficient' read(*,*)c write(*,*)'enter mass' read(*,*)m i=1,10 ! 1 10, 1 being interval. end v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) q=v(1) end program you cannot use [] normal parenthesis in expressions. array constructor, [ items ] means array items elements. end do should after line.
Read more