[nylug-talk] Send a set really fast

[Michael Bacarella]

Peter:
> You may want to see if the operations available in the numpy module
> include a fast matrix pickle/unpickle pair along with the faster
> matrix operations. String splitting involves a lot of memory
> operations that are better handled by a proper matrix type since the
> general string case doesn't seem to be optimized for what you're
> doing.

N.J. Thomas:
> Have you thought about packing the integers into binary strings? (i.e.
> the struct module)
> 
> I think this would work well if your data sits within certain boundaries
> (i.e. all the integers in the set are no bigger than 100, etc.) That way,
> you can pack it into some efficient binary string and have A and B
> marshal the data before it is sent and have Z just unmarshal it.

If I have to do any unmarshalling at all I've already lost.
Even if I had the most efficient representation, populating a
big list (not even a set) is *expensive*

This code:
	d = []
	for i in xrange(0,1000000):
		d.append(i)

takes whole seconds.  Unmarshalling anything will only make it
slower.

I need to be able to load a BLOB and have Python instantly
recognize it as a set.