Getting data into memory

Buffers seem to be the first step, specifically the bufsize and bufno system options.

These tell SAS® how much data is read into memory: bufsize sets the size of a page in the file cache and bufno sets the number of pages to be read in one go.

The product gives the memory requirement: bufsize x bufno.

After extensive testing, see Buffer size and CPU time for details, it appears that using the full 2GB produces the best results, but is a very greedy way of accomplishing a good result. It seems that for our set up a memory allocation of 256MB produces results that are nearly as good, and the best settings for this are achieved with bufsize = 256M and bufno = 1024.

Incidentally, 256MB seems to be a useful number to know as it works well as the sortsize setting for sorting.

The idea behind the testing was the SAS technical note Achieving Better I/O Throughput Using SGIO.

Technorati Tags: buffers, bufno, bufsize, data, memory, performance, SAS