I am processing a lot of data. HDF5 with compression filter is helpful for doing such thing, without compression, I have 50GB data for storage; HDF5 makes it 7GB.
Looking forward to do "import h5py" in pythonesta.
Here is the github for HDF5
https://github.com/h5py/h5py
The module is partially written in Cython, which has to be compiled to native code. This isn't possible on Pythonista, so the only way to use it on Pythonista would be for omz to add it to the app. (Which is unlikely to happen anytime soon, if at all.)
If you're processing large amounts of data you'd be better off doing it on a regular computer anyway, it will probably be faster than on your iDevice and you have the freedom of installing whatever modules you want.
My 2 cents worth would be, the more compression the longer processing time required to manipulate the data. If you had the free space on your idevice, to have the data compressed minimally , process it then achieve a higher compression for the transfer to the target, as I assume the idevice is not the final target of 50gb of data
Sorry, I know I gave a very simplistic answer above. But I have done a lot of data processing, but the old days. But as so many Things have changed with processor power, bandwidth etc... It's really worth to check where the biggest bottle neck is in the process. Can be a surprising out come, then again maybe not. Eg, I can't believe the speed Of SQLite sometimes. Then you tweak a few pragmas as well as employing transactions can be so quick. Some of the tweaks are not compliant for power failure, rollback etc... But with bulk processing you can normally live with this. normally it's simple, it's a big pass or total failure. A rollback for example can be catastrophic with huge datasets. Better just to start the process again
I don't if my comments are useful or not. I also love data processing related problems. Does not make me good at them just because I like them though.
@dgelessus @Phuket2 thank you for your guys responses^_^, I am clear about the pythonesta ability in data analysis now. And I will drive my RMBP for major heavy work. And make my iDevices only work with simplified dataset from raw data.