apache pig - Hadoop: Pig error -

-

i absolute beginner in hadoop, , doing simple testing, however, not find error messages informative.

i have set hadoop environment in single-node mode on centos 6.4 vm 4gb of ram available.

i trying run simple pig script on 500mb csv file. have 2 500mb files, on first one, script successful. on second one, same size, different data (a lot more rows), error when execution reaches 60%.

this (very simple) pig script use:

records = load 'trans2013.csv' using pigstorage(',') (podracun_v_breme,datum_transakcije,znesek_transakcije,oznaka_valute_transakcije,racun_v_dobro,naziv_prejemnika,maticna_stevilka,davcna_stevilka,sifra_pu,zr_sns_oe,namen); transaction_recs = group records all; tot_trans = foreach transaction_recs generate sum(records.znesek_transakcije); store tot_trans '/user/root/totaltransactions';

this error in terminal:

2014-04-06 10:28:29,147 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 64% complete 2014-04-06 10:28:30,240 [main] warn org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - ooops! job has failed! specify -stop_on_failure if want pig stop on failure. 2014-04-06 10:28:30,241 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - job job_1396637732046_0008 has failed! stop running dependent jobs 2014-04-06 10:28:30,241 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 100% complete 2014-04-06 10:28:30,460 [main] error org.apache.pig.tools.pigstats.simplepigstats - error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

2014-04-06 10:28:30,461 [main] error org.apache.pig.tools.pigstats.pigstatsutil - 1 map reduce job(s) failed! 2014-04-06 10:28:30,463 [main] info org.apache.pig.tools.pigstats.simplepigstats - script statistics:

hadoopversion pigversion userid startedat finishedat features 2.0.6-alpha 0.11.1 root 2014-04-06 10:25:49 2014-04-06 10:28:30 group_by

failed!

failed jobs: jobid alias feature message outputs job_1396637732046_0008 records,tot_trans,transaction_recs group_by,combiner message: job failed! /user/root/totaltransactions,

input(s): failed read data "hdfs://localhost:8020/user/root/trans2013.csv"

output(s): failed produce result in "/user/root/totaltransactions"

counters: total records written : 0 total bytes written : 0 spillable memory manager spill count : 0 total bags proactively spilled: 0 total records proactively spilled: 0

job dag: job_1396637732046_0008

2014-04-06 10:28:30,463 [main] info org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - failed! 2014-04-06 10:28:30,491 [main] error org.apache.pig.tools.grunt.gruntparser - error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

details @ logfile: /root/pig_1396797945352.log

here error log:

backend error message --------------------- attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

pig stack trace --------------- error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

org.apache.pig.backend.executionengine.execexception: error 2997: unable recreate exception backed error: attemptid:attempt_1396637732046_0008_m_000001_0 info:container killed applicationmaster.

at org.apache.pig.backend.hadoop.executionengine.mapreducelayer.launcher.geterrormessages(launcher.java:217) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.launcher.getstats(launcher.java:149) @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher.launchpig(mapreducelauncher.java:400) @ org.apache.pig.pigserver.launchplan(pigserver.java:1264) @ org.apache.pig.pigserver.executecompiledlogicalplan(pigserver.java:1249) @ org.apache.pig.pigserver.execute(pigserver.java:1239) @ org.apache.pig.pigserver.executebatch(pigserver.java:333) @ org.apache.pig.tools.grunt.gruntparser.executebatch(gruntparser.java:137) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:198) @ org.apache.pig.tools.grunt.gruntparser.parsestoponerror(gruntparser.java:170) @ org.apache.pig.tools.grunt.grunt.exec(grunt.java:84) @ org.apache.pig.main.run(main.java:604) @ org.apache.pig.main.main(main.java:157) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:606) @ org.apache.hadoop.util.runjar.main(runjar.java:212)

so ... tried running pig script mapreduce option:

pig -x mapreduce script.pig

it still failed, @ least produced meaningful error. seems had remove header csv file, because pig used header row data. seems happens when working floating point numbers - if using same script integers, header row ignored.

so it. first removed header file, running script against - worked.


Comments

Post a Comment

Popular posts from this blog

PHPMotion implementation - URL based videos (Hosted on separate location) -

-
i have implement video collection website users of specific isp. site hosted on public web-server movies hosted on local isp server. user of isp + general visitors can see available movies; isp's users able play them. player on site point url of local machine ( http://192.x.x.x/movies/mymovie.mov ) private class (192.x.x.x) accessible isp user. question: possible phpmotion script? or need custom work. if possible, can please suggest available mods? phpmotion require modifications able handle this. uploading, conversion, thumbnail generation, , playback pretty static in phpmotion. really though, rather modding entire upload , conversion process, if created quick custom form add entries phpmotion database (maybe include thumbnail upload), tweaked playback paths, should go on phpmotion side of things. assume had videos in format player handle (such flv or mp4).
Read more

c# - Unity IoC Lifetime per HttpRequest for UserStore -

-
i'm trying clean default implementation of accountcontroller.cs comes out of box in new mvc5/owin security implementation. have modified constructor this: private usermanager<applicationuser> usermanager; public accountcontroller(usermanager<applicationuser> usermanager) { this.usermanager = usermanager; } also, have created lifetime manager unity looks this: public class httpcontextlifetimemanager<t> : lifetimemanager, idisposable { private httpcontextbase _context = null; public httpcontextlifetimemanager() { _context = new httpcontextwrapper(httpcontext.current); } public httpcontextlifetimemanager(httpcontextbase context) { if (context == null) throw new argumentnullexception("context"); _context = context; } public void dispose() { this.removevalue(); } public override object get
Read more

I am trying to solve the error message 'incompatible ranks 0 and 1 in assignment' in a fortran 95 program. -

-
i writing program in fortran find velocity of parachuting person in relation time. keep getting error can't fix. new programming , appreciated. the error is v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) 1 error: incompatible ranks 0 , 1 in assignment @ (1) and program is program para integer :: real :: v(11) !velocity real :: q !initial velocity real :: h !time step real :: c !drag coefficient real :: m !mass ! gravity equal 32 ft/s^2 write (*,*)'enter time step' read(*,*)h write(*,*)'enter initial velocity' read(*,*)q write(*,*)'enter drag coefficient' read(*,*)c write(*,*)'enter mass' read(*,*)m i=1,10 ! 1 10, 1 being interval. end v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) q=v(1) end program you cannot use [] normal parenthesis in expressions. array constructor, [ items ] means array items elements. end do should after line.
Read more