[nylug-talk] LOWMEM vs. HIGHMEM performance advantage?

[Eric Moore]

Ruben Safir <ruben at mrbrklyn.com> writes:

> On Thu, Apr 17, 2008 at 09:17:45PM -0400, Eric Moore wrote:
>> Ruben Safir <ruben at mrbrklyn.com> writes:
>> 
>> > On Thu, Apr 17, 2008 at 08:08:16AM -0700, Peter C. Norton wrote:
>> >> On Thu, Apr 17, 2008 at 10:30:55AM -0400, jh wrote:
>> >> > I have an application that appears to slow down once it starts using 
>> >> > about 900 megs of memory.
>> >> > 
>> >> > Interestingly, from dmesg:
>> >> > 
>> >> > [42949372.960000] 1150MB HIGHMEM available.
>> >> > [42949372.960000] 896MB LOWMEM available.
>> >> > 
>> >> > Is HIGHMEM slower, or are there circumstances or kernel configs such 
>> >> > that I could be seeing such a slowdown once memory usage goes above a 
>> >> > certain level?
>> >> 
>> >> Are you using a 2.4 kernel? I thought the split was moved much closer
>> >> to the 4gb line in 2.6.
>> >> 
>> >
>> > Why do we still have this ancient DOS ritual?
>> >
>> > Ruben
>> 
>> Because our computers still have finite word sizes.
>> 
>
> Thanks Eric.  I almost know how that relates, but can you explain that
> in more detail?

Sure.  The basic opcodes on 32 bit machines allow one to get the nth
byte from ram, where n is a 32 bit (unsigned) integer.  
2^32 =  4,294,967,296 or 4 gigabytes.  So without doing anything
tricky, you can access up to 4GB.  There's some indirection in there
(you're not actually addressing memory directly, there's multiple
different mappings of logical address to physical address), but
basically, you still need some way to refer to a given byte, and we
use 32-bit integers.  The kernel then splits this virtual address
space into 2 segments, one for it's own use, and one for the process
to use.  By default, these are (or were) 2 GB and 2 GB in size.  To
use "highmem" the compiler emits additional instructions that cause
the processor to change what virtual->physical mapping the process
uses for certain data.  So if you have code something like:

int
array_plus_equals (int array_reference, int value)
{
   static int array[1000000000];

   return array[array_reference] += value;
}

the compiler thinks "hey, I need to put array in himem", then emits
assembly that does something like (some of these actually being system
calls to the kernel):

X switch to special-highmem-memory-map
* reference byte array + array_reference * sizeof(int);
* add value to it, put that in a register
* store that register to the above memory location
X switch back to original-lowmem-memory-map
* return the value in the register

the X marked steps are kernel system calls, and take a while, but
as long as you're using 32 bit numbers to look things up in memory, if
you want to use more than 4 GB (ignoring the kernel/userspace split
for a moment), you need some code that says "from now on when I access
location 0xdeadbeef, I mean something different than I did before".

On a 64 bit system, you can address up to 1 exabyte (one billion
gigabytes) before having to use multiple memory maps in a given
process).

> Ruben
-- 
Eric