r - Aggregating duplicate rows by taking sum -

-

following on questions:
1. identifying whether set of variables uniquely identifies each row of data or not;
2. tagging rows duplicates in terms of given set of variables,
aggregate/consolidate duplicate rows in terms of given set of variables, taking sum.

solution 1:

there guidance on how here, when there large number of levels of variables form index, ddply method recommended there slow, in case trying tag duplicates given set of variables. 

# values of (f1, f2, f3, f4) uniquely identify observations dfunique = expand.grid(f1 = factor(1:16),                        f2 = factor(1:41),                        f3 = factor(1:2),                        f4 = factor(1:104))  # sample rows , rbind them dfdup = rbind(dfunique, dfunique[sample(1:nrow(dfunique), 100), ])  # dummy data  dfdup$data = rnorm(nrow(dfdup))  # aggregate duplicate rows taking sum dfdupagg = ddply(dfdup, .(f1, f2, f3, f4), summarise, data = sum(data))

solution 2:

the second solution use data.table, , following advice here, do

# data.table solution indexvars = paste0('f', 1:4, sep = '') dtdup = data.table(dfdup, key = indexvars) dtdupagg = dtdup[, list(data = sum(data)), = key(dtdup)]

i have couple of questions:
1. there way make ddply version faster?
2. data.table correct? want check since new data.table.

regarding data.table solution, don't need set key aggregation operations. can directly do:

indexvars = paste0('f', 1:4, sep = '') dtdup <- as.data.table(dfdup) ## faster data.table(.) dtdupagg = dtdup[, list(data = sum(data)), = c(indexvars)]

data.table version 1.9.2+ implements function setdt enables conversion of data.frames data.tables by reference (which means, there no copy , therefore takes no time in conversion, useful on large data.frames).

so, instead of doing:

dtdup <- as.data.table(dfdup) dtdup[...]

you do:

## data.table v1.9.2+ setdt(dfdup) ## faster as.data.table(.) dfdup[...]   ## dfdup data.table, converted reference

on first question, plyr not known speed. check why plyr slow? (and many informative comments there) more info.

perhaps maybe interested in dplyr, orders of magnitude faster plyr, still slower data.table, imho. here's equivalent dplyr version:

dfdup %.% group_by(f1, f2, f3, f4) %.% summarise(data = sum(data))

here's benchmark between data.table , dplyr on data (all timings minimum of 3 consecutive runs):

## data.table v1.9.2+ system.time(ans1 <- dtdup[, list(data=sum(data)), by=c(indexvars)]) #  user  system elapsed  # 0.049   0.009   0.057   ## dplyr (commit ~1360 github) system.time(ans2 <- dfdup %.% group_by(f1, f2, f3, f4) %.% summarise(data = sum(data))) #  user  system elapsed  # 0.374   0.013   0.389

i don't have patience run plyr version (stopped after 93 seconds of first run). can see dplyr faster plyr, ~7x times slower data.table here.

check if results equal sure:

all.equal(as.data.frame(ans1[order(f1,f2,f3,f4)]),            as.data.frame(ans2)) # [1] true

hth


Comments

Post a Comment

Popular posts from this blog

c# - Unity IoC Lifetime per HttpRequest for UserStore -

-
i'm trying clean default implementation of accountcontroller.cs comes out of box in new mvc5/owin security implementation. have modified constructor this: private usermanager<applicationuser> usermanager; public accountcontroller(usermanager<applicationuser> usermanager) { this.usermanager = usermanager; } also, have created lifetime manager unity looks this: public class httpcontextlifetimemanager<t> : lifetimemanager, idisposable { private httpcontextbase _context = null; public httpcontextlifetimemanager() { _context = new httpcontextwrapper(httpcontext.current); } public httpcontextlifetimemanager(httpcontextbase context) { if (context == null) throw new argumentnullexception("context"); _context = context; } public void dispose() { this.removevalue(); } public override object get
Read more

I am trying to solve the error message 'incompatible ranks 0 and 1 in assignment' in a fortran 95 program. -

-
i writing program in fortran find velocity of parachuting person in relation time. keep getting error can't fix. new programming , appreciated. the error is v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) 1 error: incompatible ranks 0 , 1 in assignment @ (1) and program is program para integer :: real :: v(11) !velocity real :: q !initial velocity real :: h !time step real :: c !drag coefficient real :: m !mass ! gravity equal 32 ft/s^2 write (*,*)'enter time step' read(*,*)h write(*,*)'enter initial velocity' read(*,*)q write(*,*)'enter drag coefficient' read(*,*)c write(*,*)'enter mass' read(*,*)m i=1,10 ! 1 10, 1 being interval. end v(i+1)=v(i)+[32-((c*v(i)*v(i))/m)]*(h) q=v(1) end program you cannot use [] normal parenthesis in expressions. array constructor, [ items ] means array items elements. end do should after line.
Read more

PHPMotion implementation - URL based videos (Hosted on separate location) -

-
i have implement video collection website users of specific isp. site hosted on public web-server movies hosted on local isp server. user of isp + general visitors can see available movies; isp's users able play them. player on site point url of local machine ( http://192.x.x.x/movies/mymovie.mov ) private class (192.x.x.x) accessible isp user. question: possible phpmotion script? or need custom work. if possible, can please suggest available mods? phpmotion require modifications able handle this. uploading, conversion, thumbnail generation, , playback pretty static in phpmotion. really though, rather modding entire upload , conversion process, if created quick custom form add entries phpmotion database (maybe include thumbnail upload), tweaked playback paths, should go on phpmotion side of things. assume had videos in format player handle (such flv or mp4).
Read more