Forum Archive

If there will not is there some way they can be loaded on the ipad if they are not fully integrated into Pythonista.

No and no, unfortunately. Pandas and SciPy are requested very often, but especially SciPy is not that easy to port, and it's not a very high priority for @omz right now. There's also no way to install the libraries yourself. Due to iOS sandboxing and code signing requirements, apps can only load native libraries that are included with iOS or the app. This means it's impossible to install Python libraries that require native code, such as Pandas or SciPy.

Drink!!!

As you can see, Pandas/SciPy are requested very often. :)

@jserenson Hi, to perform some calculations with scipy or pandas and Pythonista, you could use a script ('sage_interface.py') that uses a remote Python environment useful for math with Python (SageMathCell). One limit is: you need to be online with your device and if a lot of people are using the server at the same time, your calculation could take a long time even if it is pretty simple (the server is busy). The main code that passes a python string to SageMathCell is by Andrey Novoseltsev et al. and can be found here.

If you are interested, try to read the simple procedure here to use this script with Pythonista (and all Python environment with websocket-client library installed).

A full scipy script in general could be not executed directly in this way, because plot can not be shown easly in Pythonista console. But with some code adjustments you can use all scipy libraries provided by SageMath.

Some interesting resources are:

• list of available math libraries in SageMathCell

• a nice guide about SageMathCell

As a test, if you want, you could try to use the wrench version of 'sage_interface.py' written by JonB (see the external thread previously indicated) or the 'sage_interface.py' function version on the scipy example found at https://www.scipy.org/getting-started.html.
The example code of the scipy Getting Started should be adapted a bit if you want to plot the solution with Pythonista, but if you work mainly with arrays and numbers, the above solution ('sage_interface.py' wrench or function version) works well enough.

Bye
Matteo

I hope my post will not be treated as an ad. actually I want use these pkgs in Pythonista too, but I can't wait so I implemented it by myself, both scipy, pandas, scikit-learn and other analytic/machine learning pkgs have been integrated into my app Analyser: https://itunes.apple.com/app/id1083042861?mt=8
it's hard to migrate these complex pkgs to iOS to meet Apple's requirement. Pythonista is a great app, I asked omz to buy those pkgs to put it into Pythonista if he could pay a reasonable price, but has denied my proposal, ^_^

@Analyser, this post seems a little open ended. Eg, i am left wondering is it the price or a maintenance issue, eg. Even if you have offered for free he still may not have been willing to include them due to extra responsibilities he would have to take on.

@Analyser Hi, thank you for the info, your app seems very good to do math with Python, but in my opinion it is a bit expensive, sorry for this observation. Consider that, even if in your Analyser you added scipy and other powerful math libraries, user can't use/install other not-pure python libraries that depends on scipy and/or pandas. It is, in my opinion, the biggest problem with Apple policy about development apps (IDE) in the store: in other words, most of the development apps in the store are "static", that is: no way to increase the power of the app by user (adding external libraries for examples if user needs them) if the app developer decides to not increase the power.
If you tried to allow user to test your app for a short time, maybe he could decide if buy or not your powerful app, but it is my opinion (no problem with you, I know that you must pay Apple to develop and publish your apps: in fact I hate the Apple choice to force user to pay when he wants develop an app (also for hobby) with XCode, that is available only for Mac; I prefer much more an "open" way , that is the use of a full IDE inside the idevice with which I can write scripts for my purposes).
However thanks again for your work on Analyser!
Regards
Matteo

@Matteo: Thanks for your suggestions and support. yes, user cannot install pkgs that contains compiled codes. I'm willing to include user requested module in the next release if the module can be migrated to iOS.

@Phuket2: If I finish the migration of pyobjc omz may give a glance to my proposal, pyobjc can do anything(inter-operation with iOS) that Pythonista can do.

Analyser looks quite interesting. Personally, I would not want to give up the (many) superb features of Pythonista. I hope that @omz will (someday) find a way to add these much requested modules. (no drink for this one :-)

@Analyser, ok cool. I think my post may have looked a bit mean or cold. Was not meant to be. I think i was rushing. I was just trying to point out as a developer (which i have not been for so many years it does not matter), but on the surface some decisions look like they should be no brainers. But I am sure you are aware when you add any third party dependencies to your app/development you have to ask yourself many questions about the now and the future. I dont pretend to understand what would be involved here. I just received know sometimes the smallest things can have a huge domino effect. And to be fair that could be in a positive way also.
Anyway, for me I have only a single motivation. That is that Pythonista is the best it can be for all its users.

push
add pandas and scipy and i will instantly buy the app :D

With pandas not being integrated, I was always interested in using a remote environment of python. Running and editing code with Pythonista is so much more fun than any other method, I just wish I could use it more.

It is possible to use the sagemath servers for doing remote work. @Matteo has some scripts for interacting with sagecell
https://forum.omz-software.com/topic/4200/data-analysis-workflow/17

@ericbaranowski Hi, I have no more ideas about how to improve sage_interface, but if you have some hints to improve it post here your ideas!

sage_interface has the following features:

1. You must be online.
2. Max 64 kb of input scripts.
3. Max 2 minutes for calculation for any single script (must be tested, I’m not sure, Andrey the author said me that user could have also 2 hours for a full math session, but these limits could change if Andrey will decide to change them).
4. About max 4-5 Gb of Ram available (info by Andrey).
5. No way to install/compile your personal libraries (not-pure, like last version of scipy for example or any other library you want).
6. sage_interface: a)sends to server a script, b)waits for server output, c)processes the output string simply saving each string in a Pythonista variable you decide, with image and numpy array capabilities. a) and b) are available with the original script by sagemathcell authors (see here), while c) is written by me with the help of @JonB and @ccc. I use a lot also the wrench version by JonB because when I want to test a script that in future I will use often, I find very useful to test it by touching a single key in the same way I touch the built-in key run of Pythonista (Pythonista is very customizable if you know how to customize it ;-)). Wrench version has a powerful feature about error checking in the script send to server, so you can easily find any error returned by server. My function version has a very basic error checking feature, but I use it because Pythonista allows you to open several files and to surf easily among them.

For the second limit above, I say that if you must solve a very big linear equations system, for example, you could create a txt file with the big system as a sequence of numbers/coefficents (A and b for the classical Ax=b), save it in your dropbox (or any online storage service), use the server to open remotely your file with data input text (A and b) in order to save content in variables of the server python environment, and finally run your script for linear solver sending the solver script to server and using the variables for input data created previously.

My only target now about sage_interface is to write a series of standard scripts (standard for me, but they can be adapted by user) that send to sagemathcell server any script for calculation in the following numeric math fields (that I use often, for work or for hobby):
1. PDE solver
2. ODE solver
3. Big linear equations system solver (sparse, dense)
4. Non linear equations system solver
5. Optimization (linear or non linear – w/o contraints fitting data with plots).

Another thing is that with sage_interface you can import to python environment of the server a remote pure python library with httpimport by John Torakis, but it works obviously with pure python libraries, no way for now to use it with c-fortran based python libraries. See here.

I asked to Jonh Torakis if he knows about some python libraries that allow user to import any other library (pure or not pure python) as a full folder with compiled c-fortran code (for example the last scipy library compiled in the same os environment the server uses, for sagemathcell is linux), but he doesn’t know. Httpimport is great for pure-python libraries and you can use it if you want to maintain clean your site-packages folder in Pythonista (but if you use often a library then you would prefer to install it instead of remotely importing it every time you need).

Does someone know if a native (Linux/Windows/Mac) linker-loader using HTTP/S (or networking in general) exist (citing John in his answer to my question)?

Regards
Bye

Sorry if I insist, but I'm yet interested to the question I posted a few days ago:

does someone know if a native (Linux/Windows/Mac) linker-loader using HTTP/S (or networking in general) exist?

In other words:
1. Let's say I have two computers (A and B) with the same OS;
2. In computer A I have a full working python distribution with a properly installed scipy version;
3. in computer B I have an other full working python distribution (same python version of computer A) but I haven't the scipy library and I can/want not install it;
4. I copy/paste the full scipy folder of computer A in a remote service (github, dropbox, icloud, etc...); the full scipy folder contains not only pure-python scripts but also compiled files like .dll or .lib. or any kind of file required by scipy. Let's suppose that the full scipy folder is fully portable (it should, because some times ago I was able to compile a scipy version in my pc, then I copied the scipy folder created in site-packages and I pasted it in a different python distribution in a different pc with same OS: without compilation in the second pc, I was able to use scipy compiled in the first pc);
5. I know the HTTPS link of the full scipy folder stored in the remote service (as shared link);

Question: does exist nowadays a way to use the existing python environment of computer B (with Internet connection) to import only in RAM memory of B the full scipy folder with HTTPS link (of point 5) in order to use it with python, as if the full scipy folder was stored in the hard drive of B?

I only ask if any of you are aware of the existence of a tool like this and where I can find it.

Thank you for any hint
Regards

Drink!!!

@Matteo For pure Python modules that wouldn't be too difficult, you could write an import hook that tries to download the requested module from a server and then loads that. With native modules that's not as easy - I don't know of any OS that lets you load a native library from RAM. For example, the Unix dlopen function only takes a file path. You could download the library to a local file and load that, but if you can do that, you could install the library permanently as well.

In any case, this would be of no use on iOS anyway. Because of iOS app sandboxing, an app is only allowed to load native libraries if they are code-signed by Apple or the app developer. This is why you cannot for example compile SciPy for iOS on a Mac and copy over the compiled libraries. Even if you code-signed them with your own iOS development certificate, iOS would refuse to let Pythonista load them, because the library was signed by a different developer than the app.

@dgelessus I have been thinking about this recently... Does iOS actually prevent remapping RW memory to RX?

If so, How does CFUNCTYPE callbacks work under the hood?

I have not tried this yet, but it looks interesting..
https://csl.name/post/python-jit/

Of course, even if that works, the tricky bit would be writing one's own loader in python.

In thinking about this more, though not looking at the source, CFUNCTYPE almost certainly works by having an executable stack, or else by having a WX page full of trampoline functions. So it seems like it should be possible to use the same method to call object code in memory.... I can see it maybe working for statically linked code, but a loader would be ugly for something like pandas.

@dgelessus and @JonB

Hi, thanks guys for reply, I would like to clarify that I don’t want to import in Pythonista any precompiled not-pure python libraries (it is impossible for now), but to import them in a computer (named for example “X”) with a working python environment, and I’m interested in doing that with the remote computer provided by SageCellServer, that is a virtual machine with Linux OS; if user can create his/her own server with a home computer linked to Internet, it is possible to use it instead of the SageMathCell one, solving the limits related to impossibility to install own libraries.

Sorry but I try to explain what I have in mind with other words as below.

I’d like to use Pythonista with my idevice:

• to send some scripts (like the library “httpimport” by John Torakis, but it works only for pure-python libraries without any dll, lib, etc… files), to the remote server (computer “X”) (these scripts are the main reason of my question: Do they exist? Has someone created them, what do you know?);

• after the remote server has received these scripts, it should run them in order to import only in the RAM (not on hard disk) of computer “X” a full not-pure python library properly compiled with other computer with suitable compilers, same OS, etc… of the remote one (“X”); let’s suppose this library I want to import only in RAM of computer “X” is stored in a folder of my dropbox and I have its shared link.

• after the importing, I’d like to use the remote python interpreter of SageMathCell (or other own available remote computer) using Pythonista like a tool that sends python code to server and receives output from that server.

I know that a OS (windows, linux, mac, other…) needs to have an hard disk in order to work (is it true for all? I’m not sure). I think that nowadays no tools exist that allow user to decide if to install any external software, python library, fully portable executable program, full IDE with any kind of compiler

• in the hard disk, or

• directly in RAM, as if a portion of RAM, decided by OS, behaved the same way as a hard disk (so all files are stored in ram with the same folder hierarchy of hard disk).

As example, let’s suppose I have in my pc (named “Z”) a python environment installed in the hard disk in the path “C:\Python27\” of windows OS (as example, but I ask if it is possible also for linux, mac, other…).

Now suppose that
1. I want to import a python library in my pc (without store it in hard disk with “pip install”). With httpimport library by Torakis it is possible to import temporarily only in ram a pure-python library, but, unfortunately, suppose I want to import in RAM a not-pure python library, like scipy, so I can’t use httpimport lib.
2. Suppose also that the library I want to import only in RAM needs some existing libraries in my OS, that are stored in some folders of hard disk.

Well, can I:

a) compile the not-pure library with a suitable compiler following the common way, that is by installing the compiler in hard disk of a pc and using it to compile the python library,
b) copy the compiled python library, named “Y” and stored in a cloud service accessible via Internet connection, only in a portion of my pc’s ram (as if it was in “C:\Python27\Lib\site-packages\my-compiled-python-lib”) with a python command/function like the one I use to download something , that is the "urllib" python lib,
c) tell to python interpreter of pc “Z” to import the library “Y” in order to use it?

The library “Y”, that needs some external libraries in my OS (see point 2. above), in my idea, would search for the external libraries stored in pc "Z" hard disk even if it is only in RAM.
So, is it possible?

Thank you guys
Regards

@JonB I don't know how ctypes creates callback functions in Pythonista. ctypes internally uses libffi, for which you can find the source at https://github.com/libffi/libffi. However I don't know if @omz had to modify libffi to make it work on iOS, so the official source code may be inaccurate.

What I did find however are the internals of the Objective-C runtime's imp_implementationWithBlock, which effectively lets you create an IMP that calls the given block. See objc-block-trampolines.mm and a1a2-blocktramps-arm64.s in https://opensource.apple.com/source/objc4/objc4-723/runtime/. The way it works is that the assembly code defines a page full of identical trampolines. The trampoline code invokes a block located exactly one page before the entry point of the trampoline. That way all trampolines use the same code, but end up invoking different blocks.

Now to make use of these trampolines, imp_implementationWithBlock allocates two pages of read-write memory. Using the Mach vm_remap function, the page of trampolines from the assembly code is mapped into the second newly allocated page, which also changes the permissions of that page to read-execute. The first page is still read-write and is used to store the blocks, which can then be called by the corresponding trampoline in the next page.

None of this requires an executable stack or any other sort of write-execute memory, since the executable code is loaded unmodified from the library, and then mapped into memory multiple times without modification. I don't think iOS allows write-execute memory, that would make it very easy to circumvent the code signing restrictions.

@Matteo Importing a native Python module from RAM is simply not possible if the OS doesn't allow loading a native library from RAM, and I think Linux doesn't. As far as I know, on Unix systems Python uses the standard dlopen function to load native modules, and that requires the library to be in a file.

I don't know what the SageCellServer Linux environment does and doesn't allow, but if possible you could make a temp directory, download the libraries you want into it, and add it to sys.path. Then any native modules in there will be found by the normal import mechanism. As long as the libraries are compiled correctly for the system used by SageCellServer, they should import without problems.

@dgelessus said:

@JonB I don't know how ctypes creates callback functions in Pythonista. ctypes internally uses libffi, for which you can find the source at https://github.com/libffi/libffi. However I don't know if @omz had to modify libffi to make it work on iOS, so the official source code may be inaccurate.

What I did find however are the internals of the Objective-C runtime's imp_implementationWithBlock, which effectively lets you create an IMP that calls the given block. See objc-block-trampolines.mm and a1a2-blocktramps-arm64.s in https://opensource.apple.com/source/objc4/objc4-723/runtime/. The way it works is that the assembly code defines a page full of identical trampolines. The trampoline code invokes a block located exactly one page before the entry point of the trampoline. That way all trampolines use the same code, but end up invoking different blocks.

Now to make use of these trampolines, imp_implementationWithBlock allocates two pages of read-write memory. Using the Mach vm_remap function, the page of trampolines from the assembly code is mapped into the second newly allocated page, which also changes the permissions of that page to read-execute. The first page is still read-write and is used to store the blocks, which can then be called by the corresponding trampoline in the next page.

None of this requires an executable stack or any other sort of write-execute memory, since the executable code is loaded unmodified from the library, and then mapped into memory multiple times without modification. I don't think iOS allows write-execute memory, that would make it very easy to circumvent the code signing restrictions.

@JonB said:

@dgelessus said:

@JonB I don't know how ctypes creates callback functions in Pythonista. ctypes internally uses libffi, for which you can find the source at https://github.com/libffi/libffi. However I don't know if @omz had to modify libffi to make it work on iOS, so the official source code may be inaccurate.

What I did find however are the internals of the Objective-C runtime's imp_implementationWithBlock, which effectively lets you create an IMP that calls the given block. See objc-block-trampolines.mm and a1a2-blocktramps-arm64.s in https://opensource.apple.com/source/objc4/objc4-723/runtime/. The way it works is that the assembly code defines a page full of identical trampolines. The trampoline code invokes a block located exactly one page before the entry point of the trampoline. That way all trampolines use the same code, but end up invoking different blocks.

Now to make use of these trampolines, imp_implementationWithBlock allocates two pages of read-write memory. Using the Mach vm_remap function, the page of trampolines from the assembly code is mapped into the second newly allocated page, which also changes the permissions of that page to read-execute. The first page is still read-write and is used to store the blocks, which can then be called by the corresponding trampoline in the next page.

None of this requires an executable stack or any other sort of write-execute memory, since the executable code is loaded unmodified from the library, and then mapped into memory multiple times without modification. I don't think iOS allows write-execute memory, that would make it very easy to circumvent the code signing restrictions.

@dgelessus Hi, thank you for suggestion, unfortunately it seems to me that it is not possible to create a folder with remote server by SageMathCell.

I had a simpler idea thanks to your reply: can python (any version) add to sys.path also remote folders (for example any folder stored in a dropbox or github account)?

Example: I want to add to sys.path of my local python environment (for example Pythonista) a folder like this one:

https://github.com/jsbain/objc_hacks

without downloading it neither storing it, with all scripts, in local storage memory of Pythonista.
Can I do it?
Thanks
Regards

Is the issue that you don't have control over your server? I thought you were trying a run a sagecell server on your personal pc. if so, why not just install the packages you need. Or is the issue that sagecell is "locked down" even running on your own pc?

You cannot change sys.path to point to a non-path. on a pc, you could install a drobox client, which automatically syncs a folder, but i think that doesnt satisfy your goals. You could download files using drobox client, import them, then delete them. but if your issue is with sagecell limitations or security, that pribably won't work either.

@Matteo Linux can only load native libraries from the regular filesystem. That doesn't necessarily mean "on a real physical hard disk", it could also be on a network share or RAM disk mounted somewhere in the file system. However if the system doesn't even let you create temp files, it probably won't let you mount remote filesystems either.

On a regular Linux system that you have full access to, that wouldn't be a problem though. You can mount a network share under /mnt/myshare or whatever, add /mnt/myshare/pythonlibs to sys.path, and then you can import any module in that folder. I don't know Linux very well, so I don't know what commands exactly are used to mount remote filesystems. I also don't know if there are ways to mount a Dropbox or a Git repo as a filesystem.

Hi @JonB , I have not my own sagecell provided by my online pc, because of lack of ram and cpu resources of my pc.

I use only the main SageMathCell mainteined by Andrey. But it (seems to me) doesn't allow user to install any external own python library: user can use only built-in libraries like these ones.

So I ask here if we can by-pass this limit by using a set of python scripts or a python library to import in ram (no hard disk) a remote python library compiled in linux stored in any cloud service, in order to use it not with Pythonista but with the remote python interpreter provided by SageCellMath (Pythonista would be a "remote controler" for the SageMathCell).

Hi @dgelessus, ok, I don't know Linux very much, I've tried some times ago to create a virtual machine with linux on my windows pc by using Virtual Box, but it was too slow.

After this long discussion (thanks both dgelessus and JonB!) I'd like if, in a near future, python could allow user to import any working python library (pure and not-pure) stored only in web (cloud or https), without downloading and saving it in the local filesystem. The web would become in this way a read-only big memory resource for own local python env and if user wants to change/modify the imported library, then he/she could do it by downloading the library, changing it and finally reloading the modified library in his/her own cloud account in order to use it with own local python env.

About sys.path python function, now I know that it can be used only for paths in filesystem (hard disk) and not for paths in the cloud (in a read-only mode).

Thanks both again for explanation,
Regards

If you want a full shell, you likely need to pay, such as with Amazon aws, or any number of of full shell accounts. Anders is hosting and paying for the service he provides free, hence the limits on size,etc.

From what I can tell,though, Sagecell is running in it's own VM, complete with access to ’subprocess’ and urllib, you may have be able to do what you want,within the limits of that 64k which might be the limiter.

You might checkout numpy.f2py, which supports interfacing with Fortran libraries.

Actually, check out cocalc, which is just what you need... You get private projects with persistent storage, ability to compile code,etc.

@Matteo
Actually, check out cocalc, which is just what you need... You get private projects with persistent storage, ability to compile code,etc.

@JonB Hi yes, if I will be able to create my own cocalc account and a way to interact with it using Pythonista or any online python environment it would be very useful.
With a cocalc account user could install any library supported by anaconda (I suppose).

For now I will try to create a temporary folder with SageMath, starting from the following piece of code

import os
import tempfile

directory_name = tempfile.mkdtemp()
print(directory_name)

sys.path.append(directory_name)

in order to save inside it, during any math session, some not-built-in libraries I have in my dropbox.
Inside 'directory_name' I will download some external libraries from dropbox and I will test if I will be able to import these libraries with the common 'import ....'. If sagemathcell can edit sys.path and user can add any temp folder in it, maybe the remote python interpreter would import the libs.

Thank you
Regards

This thread seems like it has become the definitive thread for all things pandas/scipy so I’ll just post this here.

Some of you may be interested in the following app:

Python Compiler by Ketan Appa

I’ve played with it a little and you can in fact use scipy and pandas on it. Those are preinstalled. If you upgrade for $5 you can plot as well. Which is nice.

The problem, and for me it’s a pretty big problem, is that your code is sent to a server and compiled remotely (with a max running time of 30s). There are of course many reasons not to do that. But for just learning to work with scipy or pandas or for just playing with generic ideas, I thought perhaps someone could find it interesting. It is buggy but I’ve been successful getting a few basic things to work.

I corresponded a bit with the developer and he is quite responsive. He also said he is thinking of having the code run locally in the future, but there is certainly no reason to expect that to be successful - I would think he will likely hit the same problems as omz.

Anyway for real tasks, I’m still hoping for at least pandas on pythonista one day if there’s no hope for scipy (I would gladly pay $20 more than I have already paid for those - just saying). But I thought someone may find this interesting since it is a relatively new app.

Hi, since cocalc was mentioned, I thought it would be worthwhile to suggest Juno (https://juno.sh), which runs on iPad. It’s basically a Jupyter notebook client running in an iOS app, and you have a choice of different back-ends: Azure Notebooks, Cocalc, or an AWS backend. I’m using the AWS backend for maximum flexibility.

I sure wish I knew pandas won't be supported, before I paid $10.

Sincerely,
Scott

@scotttmiller I would be happy to pay you the $10 back if you create a working version of Pandas in an iOS app and publish it to the AppStore for $10. Please let me know.

In a recent post I tried to see if @omz could share something about what he was working on (no reply). I realize that telling us is a mixed bag for him but perhaps he could say if Scipy and Pandas, which are often requested, are never happening, imminent, or still a possibility.

I can assure those of you interested that Juno is easily worth the $10. Honestly, I’d be happy with my purchase even if I spent $20 or $30. It would be great to be able to use pandas and scipy directly on iOS, but Juno serves most of my purposes just fine. You can use cocalc or azure, or what I usually do is just run both a selenium server and a jupyter server on my laptop on the same network and just run it via Juno. And then I can do anything I want in either Python or R (or many other languages) from my iPad or phone. It’s true you can do that just via safari or other browser too, but I find it unworkable, especially on the phone. The app does make the experience pretty good.

Me too, pandas plz.

Is there a way to volunteer to port pandas to work with pythonista? I don't think it'd take me that long. I could put up a fork on GitHub and then the work to link it into your app and resubmit to the store would be minimal.

@froody It would be a good exercise in any case to see how difficult the port would be. Go for it.

Selenium is the most widely used web automation tool. It has been used to perform an automated test on Windows, Linux, and Unix system. It supports most of the browsers like Internet Explorer, Google Chrome, Safari, Mozilla Firefox, Opera etc.

On the other hand, Live Online Training recording takes place for Selenium with Python Video Tutorials at a scheduled event or time. The Instructor plays an important role throughout the learning process.

Pandas is included on this app. It has also newer versions of NumPy and Matplotlib. Pythonista has old versions of NumPy and Matplotlib.

@dgelessus
Then how come one can use Pandas on Juno?