Python Hadoop streaming with import package that are not installed on data nodes -

-

i tried import scikit image in python hadoop streaming, , i've tried out existing posts on stackoverflow here , here, none of them solve problem.

the real question is, if distributed using -file zip/mod file packaged scikit-image folder, how python scripts running on data nodes know how extract packages , import code? note i've installed python scikit-image on name node , , able run local experiments.

my script trivial: classic word count example python streaming, "import skimage" in mapper.py. http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python

my command :

hadoop jar hadoop-streaming.jar \ -file mapper.py    -mapper mapper.py \ -file reducer.py   -reducer reducer.py \ -file ./skimage.mod \ -input /user/text/* \ -output /user/textoutput/

screen printouts:

packagejobjar: [mapper.py, reducer.py, ./skimage.zip] [/usr/lib/gphd/hadoop-mapreduce-2.0.2_alpha_gphd_2_0_1_0/hadoop-streaming-2.0.2-alpha-gphd-2.0.1.0.jar] /tmp/streamjob6159562120374599467.jar tmpdir=null 14/04/04 18:00:02 info service.abstractservice: service:org.apache.hadoop.yarn.client.yarnclientimpl inited. 14/04/04 18:00:02 info service.abstractservice: service:org.apache.hadoop.yarn.client.yarnclientimpl started. 14/04/04 18:00:03 info service.abstractservice: service:org.apache.hadoop.yarn.client.yarnclientimpl inited. 14/04/04 18:00:03 info service.abstractservice: service:org.apache.hadoop.yarn.client.yarnclientimpl started. 14/04/04 18:00:03 warn snappy.loadsnappy: snappy native library not loaded 14/04/04 18:00:03 info mapred.fileinputformat: total input paths process : 1 14/04/04 18:00:03 info mapreduce.jobsubmitter: number of splits:2 14/04/04 18:00:03 warn conf.configuration: mapred.jar deprecated. instead, use mapreduce.job.jar 14/04/04 18:00:03 warn conf.configuration: mapred.cache.files deprecated. instead, use mapreduce.job.cache.files 14/04/04 18:00:03 warn conf.configuration: mapred.output.value.class deprecated. instead, use mapreduce.job.output.value.class 14/04/04 18:00:03 warn conf.configuration: mapred.mapoutput.value.class deprecated. instead, use mapreduce.map.output.value.class 14/04/04 18:00:03 warn conf.configuration: mapred.job.name deprecated. instead, use mapreduce.job.name 14/04/04 18:00:03 warn conf.configuration: mapred.input.dir deprecated. instead, use mapreduce.input.fileinputformat.inputdir 14/04/04 18:00:03 warn conf.configuration: mapred.output.dir deprecated. instead, use mapreduce.output.fileoutputformat.outputdir 14/04/04 18:00:03 warn conf.configuration: mapred.map.tasks deprecated. instead, use mapreduce.job.maps 14/04/04 18:00:03 warn conf.configuration: mapred.cache.files.timestamps deprecated. instead, use mapreduce.job.cache.files.timestamps 14/04/04 18:00:03 warn conf.configuration: mapred.output.key.class deprecated. instead, use mapreduce.job.output.key.class 14/04/04 18:00:03 warn conf.configuration: mapred.mapoutput.key.class deprecated. instead, use mapreduce.map.output.key.class 14/04/04 18:00:03 warn conf.configuration: mapred.working.dir deprecated. instead, use mapreduce.job.working.dir 14/04/04 18:00:03 info mapreduce.jobsubmitter: submitting tokens job: job_1384839777050_0106 14/04/04 18:00:04 info client.yarnclientimpl: submitted application application_1384839777050_0106 resourcemanager @ hdm3.gphd.local/172.28.9.252:8032 14/04/04 18:00:04 info mapreduce.job: url track job: http://hdm3.gphd.local:8088/proxy/application_1384839777050_0106/ 14/04/04 18:00:04 info mapreduce.job: running job: job_1384839777050_0106 14/04/04 18:00:08 info mapreduce.job: job job_1384839777050_0106 running in uber mode : false 14/04/04 18:00:08 info mapreduce.job:  map 0% reduce 0% 14/04/04 18:00:12 info mapreduce.job: task id : attempt_1384839777050_0106_m_000001_0, status : failed error: java.lang.runtimeexception: pipemapred.waitoutputthreads(): subprocess failed code 1     @ org.apache.hadoop.streaming.pipemapred.waitoutputthreads(pipemapred.java:320)     @ org.apache.hadoop.streaming.pipemapred.mapredfinished(pipemapred.java:533)     @ org.apache.hadoop.streaming.pipemapper.close(pipemapper.java:130)

i checked error log in hadoop job, it's complaining cannot find "import skimage" means it's not picked data nodes.

have tried zipimport solution?

here example: hadoop: how include third party library in python mapreduce


Comments

Post a Comment

Popular posts from this blog

PHPMotion implementation - URL based videos (Hosted on separate location) -

-
i have implement video collection website users of specific isp. site hosted on public web-server movies hosted on local isp server. user of isp + general visitors can see available movies; isp's users able play them. player on site point url of local machine ( http://192.x.x.x/movies/mymovie.mov ) private class (192.x.x.x) accessible isp user. question: possible phpmotion script? or need custom work. if possible, can please suggest available mods? phpmotion require modifications able handle this. uploading, conversion, thumbnail generation, , playback pretty static in phpmotion. really though, rather modding entire upload , conversion process, if created quick custom form add entries phpmotion database (maybe include thumbnail upload), tweaked playback paths, should go on phpmotion side of things. assume had videos in format player handle (such flv or mp4).
Read more

javascript - Using Windows Media Player as video fallback for video tag -

-
getting video play cross platform using example video.js bit of fuss. in days when video should embedded use embed windows media player, work out of box on windows. since it's today old ie on windows doesn't support video tag experimented using windows media player fallback, , work well. what wonder here is, why haven't else solution? have missed something? code small, there's no flash files, cross domain issues or need of server working. works. so before develop further , add custom control , subtitles support wonder say. the code looks this: <video id="myvideo" style="width:640px; height: 360px;" autoplay data-wimpsource="http://esd.volvocars.com/dc/cdn/video/birds-on-a-wire.mp4"> <source src="http://esd.volvocars.com/dc/cdn/video/birds-on-a-wire.mp4" type="video/mp4" /> <source src="http://esd.volvocars.com/dc/cdn/video/birds-on-a-wire.webm" type="video/webm" />
Read more

c# - Unity IoC Lifetime per HttpRequest for UserStore -

-
i'm trying clean default implementation of accountcontroller.cs comes out of box in new mvc5/owin security implementation. have modified constructor this: private usermanager<applicationuser> usermanager; public accountcontroller(usermanager<applicationuser> usermanager) { this.usermanager = usermanager; } also, have created lifetime manager unity looks this: public class httpcontextlifetimemanager<t> : lifetimemanager, idisposable { private httpcontextbase _context = null; public httpcontextlifetimemanager() { _context = new httpcontextwrapper(httpcontext.current); } public httpcontextlifetimemanager(httpcontextbase context) { if (context == null) throw new argumentnullexception("context"); _context = context; } public void dispose() { this.removevalue(); } public override object get
Read more