min_free_kbytes changes the page reclaim thresholds. When this number is increased the system starts reclaiming memory earlier, when its lowered it starts reclaiming memory later. In adition, this controls the number of free pages the system maintains.
You don't want to make this value too low: the kernel frequently requires memory to be readily available, so a low value would force applications to sleep while kswapd tries to free up some memory, effectively causing some contention issues. On the other hand, a high value would limit the effectiveness of the page cache and would also wake up the kswapd daemon more frequently. This could hurt system performance.
The default min_free_kbytes is calculated at boot time based on the number of pages of physical RAM available on the system. Default values under 2.6.18 (used for Red Hat Enterprise Linux 5) are:
RAM min_free_kbytes 3171 * 16MB: 512k 3172 * 32MB: 724k 3173 * 64MB: 1024k 3174 * 128MB: 1448k 3175 * 256MB: 2048k 3176 * 512MB: 2896k 3177 * 1024MB: 4096k 3178 * 2048MB: 5792k 3179 * 4096MB: 8192k 3180 * 8192MB: 11584k 3181 * 16384MB: 16384k
The algorithm that computes those values is essentially:
min_free_kbytes = sqrt(lowmem_kbytes * 16)
For some systems, this algorithm could produce default values that are too low compared to the stress which will be imposed on the memory subsystem. It might thus be important to multiply the values by a factor of two or four, depending on how much memory we can afford to keep aside for the kernel.
Note that these values used to be smaller under kernel 2.6.9 and earlier. Four times smaller, more precisely.
In order to determine when kswapd should start freeing unused pages of memory, min_free_kbytes affects three different kernel variables:
pages_high sets the point after which kswapd will stop trying to free up unused memory and go back to sleep.
pages_low is used to wake up kswapd in a situation where we are running out of memory. Whenever the system reaches this watermark, kswapd starts freeing up unused pages of memory in order to let the kernel allocate more memory.
pages_min is the lowest threshold. Bellow that value, one page of RAM has to be freed before an application can allocate another page. The kernel, however, can make use of the memory available bellow this limit. This is essentially the value that min_free_kbytes corresponds to.