[nylug-talk] Send a set really fast

[Jay Sulzberger]

On Fri, 4 May 2007, Michael Bacarella <mbac at netgraft.com> wrote:

> Peter:
>> You may want to see if the operations available in the numpy module
>> include a fast matrix pickle/unpickle pair along with the faster
>> matrix operations. String splitting involves a lot of memory
>> operations that are better handled by a proper matrix type since the
>> general string case doesn't seem to be optimized for what you're
>> doing.
>
> N.J. Thomas:
>> Have you thought about packing the integers into binary strings? (i.e.
>> the struct module)
>>
>> I think this would work well if your data sits within certain boundaries
>> (i.e. all the integers in the set are no bigger than 100, etc.) That way,
>> you can pack it into some efficient binary string and have A and B
>> marshal the data before it is sent and have Z just unmarshal it.
>
> If I have to do any unmarshalling at all I've already lost.
> Even if I had the most efficient representation, populating a
> big list (not even a set) is *expensive*
>
> This code:
> 	d = []
> 	for i in xrange(0,1000000):
> 		d.append(i)
>
> takes whole seconds.  Unmarshalling anything will only make it
> slower.
>
> I need to be able to load a BLOB and have Python instantly
> recognize it as a set.

I do not know how d.append(i)" does what it does, but the name
suggests much "consing", and much walking of cons chains if you
do the intersection in the most straighforward way.

Does the Python programming system have decent socket stuff?
Decent arrays?

Does the Python system have a compiler?

oo--JS.