Home Forums Main Forums SAS Forum Teach you write SAS macro: How to split data by every 1000 records?

  • Teach you write SAS macro: How to split data by every 1000 records?

     Datura updated 3 years, 9 months ago 1 Member · 1 Post
  • Datura

    Member
    February 2, 2021 at 6:26 pm

    Question: I have a big data set with 100,000 records, I want to break it into 100 datasets by every 1000 records. How to do it?

    Step 1: Fulfill the task with standard code

     Data Z1  Z2  Z3.... Z100;
    Set AAA;
    If _N_=1-1000 then output Z1; --- I=1
    If _N_=1001-2000 then output Z2; --- I=2
    If _N_=2001-3000 then output Z3; --- I=3
    ..........
    If _N_=99001-100000 then output Z100 --- I=100
    Run;

    Step 2: Convert Macro Codes

    %macro cover();
    %local I;
    data
    %do I=1 %to 100;
    BBB&I
    %end;
    ;
    set AAA;

    %do I=1 %to 100;
    if _n_/1000<=&I and _n_/1000>&I-1 then output BBB&I;
    %end;
    run;
    %mend;
    %cover;

    Improved Method I:

     %macro subgroup;
    %local I;
    %do I=1 %to 100;
    data Z&I;
    set AAA;
    if _N_ LE &I*1000 and _N_ GT (&I-1)*1000 then output;
    else delete;
    run;
    %end;
    %mend;
    %subgroup;

    Improved Method II:
    This is much better than Method I in efficiency, because SAS only reads in 1000 records each loop, other than reading in all the observations as in Method I. Options mprint mlogic;

     %macro subgroup;
    %local I;
    %do I=1 %to 100;
    %let A=%eval((&I-1)*1000+1);
    %let B=%eval(&I*1000);
    %put A=*&A* B=*&B*;

     data Z&I;
    set AAA (firstobs=&A obs=&B);
    run;
    %end;
    %mend;
    %subgroup;

    Improved Method III:

     Data BBB;
    Set AAA;
    Order=_N_;
    run;

    %macro subgroup;
    %local I;
    %do I=1 %to 100;
    data Z&I (drop=Order);
    set BBB;
    where Order LE &I*1000 and Order GT (&I-1)*1000;
    run;
    %end;
    %mend;
    %subgroup;

    WHERE statement will do subsetting at compile time. This method is also more efficient than Method I. Bingo!

Log in to reply.

Original Post
0 of 0 posts June 2018
Now