Thursday, March 09, 2006

Re: st: memory required for -merge-

It seems possible that if there is no match at all that the data set would be much larger than you expect. One way to examine what's happening would be to merge using only part of the data set.

Fred

At 09:39 PM 3/8/2006, you wrote: >Hi all, > >Is there a rule for the amount of memory required for -merge-? I keep >getting a "no room to add more variables due to width" error on the >same merge. I am attempting to -merge- on only one variable >(placefip), and 67% of memory is free before I attempt the -merge- >(i.e., while the "master" dataset is loaded). The "using dataset is >small, and I've got the memory set quite high (900m). So, I can't >understand why I am running into memory problems. > >This is the info on the "master" dataset: > >. des > >Contains data from ../data/geog/geog_pooled.dta > obs: 20,873,141 >vars: 4 8 Mar 2006 21:32 >size: 333,970,256 (66.5% of memory free) (_dta has notes) >------------------------------------------------------------------------ >------- > storage display value >variable name type format label variable label >------------------------------------------------------------------------ >------- >cityres long %05.0f * City of residence >msarfip int %04.0f * PMSA/MSA of residence >(FIPS) >year int %4.0g >placefip long %07.0f * Place (city) of >residence (FIPS) > * indicated variables >have notes >------------------------------------------------------------------------ >------- > >And this is the info on the "using" dataset: > >. des using ../data/geog/geog_cc > >Contains data > obs: 25,150 8 Mar 2006 22:21 >vars: 3 >size: 1,207,200 >------------------------------------------------------------------------ >------- > storage display value >variable name type format label variable label >------------------------------------------------------------------------ >------- >macci byte %1.0g Center city >placefip long %07.0f >place_name str39 %39s >------------------------------------------------------------------------ >------- >Sorted by: placefip > > This is the memory usage I see after the error, when the "master" >dataset is loaded: > >. memory > bytes >-------------------------------------------------------------------- >Details of set memory usage > overhead (pointers) 83,492,564 8.38% > data 250,477,692 25.14% > ---------------------------- > data + overhead 333,970,256 33.53% > free 662,176,944 66.47% > ---------------------------- > Total allocated 996,147,200 100.00% >-------------------------------------------------------------------- >Other memory usage > set maxvar usage 1,848,666 > set matsize usage 1,315,200 > programs, saved results, etc. 41,128 > --------------- > Total 3,204,994 >------------------------------------------------------- >Grand total 999,352,194 > >Any ideas? > >-- >Danielle H Ferry > > >* >* For searches and help try: >* http://www.stata.com/support/faqs/res/findit.html >* http://www.stata.com/support/statalist/faq >* http://www.ats.ucla.edu/stat/stata/

Fred Wolfe National Data Bank for Rheumatic Diseases Wichita, Kansas Tel (316) 263-2125 Fax (316) 263-0761 fwolfe@arthritis-research.org

* * For searches and help try: * http://www.stata.com/support/faqs/res/findit.html * http://www.stata.com/support/statalist/faq * http://www.ats.ucla.edu/stat/stata/


Tag:


Links to this post:

Create a Link



<< Home

This page is powered by Blogger. Isn't yours?