[nylug-talk] LOWMEM vs. HIGHMEM performance advantage?

[Alex Pilosov]

On Fri, 18 Apr 2008, Bryan J. Smith wrote:

> My simple answer (top post) ...
> 
> If you have a program that eats more than 1GiB, you should be running an
> OS built for x86-64/IA-32e on AMD64/EM64T.
will strongly disagree. 

x86-64 (i will call it x64 and spare me) - is *bad* idea *unless* you
*have* to. because pointers are 64-bit, lots of structures are much larger
in size, so your binaries and data are much larger in size, so you blow
l2/l3 cache much faster, and use up more memory for same work.

in empirical tests, bloat is anywhere between 25% to 50% depending on 
application.

also, the border is not 1GB. Border is 3GB. 3GB apps work fine with 3/1 
split.


> If you have a program that is constantly sucking up over 2GiB and is
> very I/O bound, you should really consider a x86-64 implementation like
> AMD64.  This is less of a consideration 2007+ though, as Intel's raw
> performance and MCH in newer EM64T processors mitigates a great majority
> of this (despite "bounce buffers" and the like).  I.e., it's not like it
> was per-2007.  IBM's X4 architecture for Intel is even more impressive
> in latency and throughput than Intel's reference as well (30%+ faster).
Kinda apples and oranges. X4 is for large number of cores (32 cores, 8
sockets), large amounts of ram (1TB). However, the X4 servers are xeon
7xxx series which are fail compared to 5xxx series.

<lots of very informative but not necessarily relevant info snipped>

> Physical platform/interconnect (always a big limitation) v.
>   processor addressing/paging registers (with legacy issues) v.
>     OS/kernel memory models (what is supports) v.
>       user-space program memory model (what is uses)
> 
> Those are 4 factors that are not easy to explain on their own, and even
> more complex when you start talking about X + Y but not Z
> combinations.  ;)
The above is excellent description "why this is complicated" :)

> But yes, there are issues with addressing >32-bit in PCI on _some_
> hardware platforms.
I'd like to make a point that it is device and driver specific.

> E.g., due to lack of hardware enforced I/O memory mapping, OSes on most
> EM64T hardware platforms (Intel's IA-32e processors) _always_ "bounce
> buffer" memory mapped I/O under 32-bit.  Yes, this includes when running
> a processor in 52-bit PAE / 48-bit flat "Long Mode" (aka "x86-64"
> 64-bit).  That is a significant performance hit when something is I/O
> bound.
There is hardware and drivers that do not require bounce-buffers on x64,
has been so since a few years ago.

In fact, your higher-performance hardware is likely to already work fine
without bounce buffers. (most scsi controllers and e1000 cards understand
this).

> AMD processors with their fully x86-64 command set (which is a superset
> of IA-32"e" for EM64T) have an I/O MMU.  Unfortunately AMD's hardware
> approach is far more complicated, and there are more errata as a result.
> It's been a headache for AMD, much like it was most RISC/UNIX vendors
> years ago, but they've made it through most of it.  At the same time,
> Intel is just beginning it's journey.**  ;)
Yeah, AMD has hw bounce buffers. Which intel said are not important. Intel 
was right. Suck it, amd boy.

> **NOTE:  Intel's new designs are adopting some and running into some of
> the same issues all over again, things AMD solved long ago.  I.e.,
> TLB-related errata not only plagued AMD Processor 10h B2 stepping (new,
> true 48-bit/256TiB physical addressing processors -- prior only being
> 40-bit/1TiB), but a very under-reported and major set of errata (and a
> seemingly rolling set ;) on Intel Xeon 5000 series G0 steppings (new
> 38-bit/256GiB physical addressing processors -- prior only being
> 36-bit/64GiB capable).  I can't comment any more, but there are now
> (finally) a number of public documents on the Internet related to these
> issues (although they don't go too deep).
Suck it, amd boy.

some rakes gotta be stepped on in every implementation.

-alex