[nylug-talk] Send a set really fast

[Yusuke Shinyama]

On Fri, 4 May 2007 12:36:09 -0500, Michael Bacarella <mbac at netgraft.com> wrote:
> 
> If I have to do any unmarshalling at all I've already lost.
> Even if I had the most efficient representation, populating a
> big list (not even a set) is *expensive*
> 
> This code:
> 	d = []
> 	for i in xrange(0,1000000):
> 		d.append(i)
> 
> takes whole seconds.  Unmarshalling anything will only make it
> slower.
> 
> I need to be able to load a BLOB and have Python instantly
> recognize it as a set.

>>> import array
>>> a=array.array('I',range(1000000))
>>> len(a)
1000000

Encoding this to a bytestring takes 0.2 second:
>>> s=a.tostring()
>>> len(s)
4000000

Decoding the bytes and construct a set object takes less than 1 second total.
>>> b=array.array('I')
>>> b.fromstring(s)
>>> c=set(b)
>>> len(c)
1000000

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 1
model name      : Intel(R) Pentium(R) 4 CPU 1.80GHz
stepping        : 2
cpu MHz         : 1794.222
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips        : 3578.26

Yusuke