Forum Archive

I thought I'd start a new thread for this build, the old one is getting a bit unwieldy...

As promised in the release notes, here are a few examples of what you can do with the ctypes module... Be warned: This is pretty advanced stuff – it's very easy to shoot yourself in the foot with this, just like with plain old C... And again: I haven't been able to get ctypes fully working in 64-bit mode, but the examples here should be relatively "future-proof" (I've been able to get them to work in a 64-bit build).

# coding: utf-8
# Some experimental `ctypes` demos for Pythonista on iOS

from ctypes import c_void_p, c_char_p, c_double, c_float, cdll, util
import os

# Load Objective-C runtime:
objc = cdll.LoadLibrary(util.find_library('objc'))
objc.sel_getName.restype = c_char_p
objc.sel_getName.argtypes = [c_void_p]
objc.sel_registerName.restype = c_void_p
objc.sel_registerName.argtypes = [c_char_p]
objc.objc_getClass.argtypes = [c_char_p]
objc.objc_getClass.restype = c_void_p

# Some helper methods:
def obj_to_str(obj):
    objc.objc_msgSend.argtypes = [c_void_p, c_void_p]
    objc.objc_msgSend.restype = c_void_p
    desc = objc.objc_msgSend(obj, objc.sel_registerName('description'))
    objc.objc_msgSend.argtypes = [c_void_p, c_void_p]
    objc.objc_msgSend.restype = c_char_p
    return objc.objc_msgSend(desc, objc.sel_registerName('UTF8String'))

def msg(obj, restype, sel, argtypes=None, *args):
    if argtypes is None:
        argtypes = []
    objc.objc_msgSend.argtypes =  [c_void_p, c_void_p] + argtypes
    objc.objc_msgSend.restype = restype
    res = objc.objc_msgSend(obj, objc.sel_registerName(sel), *args)
    return res

def cls(cls_name):
    return objc.objc_getClass(cls_name)

def nsstr(s):
    return msg(cls('NSString'), c_void_p, 'stringWithUTF8String:', [c_char_p], s)

# Demo:

def print_ipod_title():
    MPMusicPlayerController = cls('MPMusicPlayerController')
    player = msg(MPMusicPlayerController, c_void_p, 'iPodMusicPlayer')
    item = msg(player, c_void_p, 'nowPlayingItem')
    if item:
        artist = msg(item, c_void_p, 'valueForProperty:', [c_void_p], nsstr('artist'))
        title = msg(item, c_void_p, 'valueForProperty:', [c_void_p], nsstr('title'))
        print 'Now Playing: %s -- %s' % (obj_to_str(artist), obj_to_str (title))
    else:
        print 'iPod not playing'

def set_screen_brightness(value):
    UIScreen = cls('UIScreen')
    main_screen = msg(UIScreen, c_void_p, 'mainScreen')
    msg(main_screen, None, 'setBrightness:', [c_float], value)

def save_video(video_file):
    uikit = cdll.LoadLibrary(util.find_library('UIKit'))
    save_func = uikit.UISaveVideoAtPathToSavedPhotosAlbum
    save_func.argtypes = [c_void_p] * 4
    save_func(nsstr(os.path.abspath(video_file)), None, None, None)

def download_and_save_video():
    # Download a short sample file from Apple:
    # http://support.apple.com/en-us/HT201549
    if not os.path.exists('sample_mpeg4.mp4'):
        print 'Downloading test video...'
        import urllib, zipfile
        urllib.urlretrieve('http://a1408.g.akamai.net/5/1408/1388/2005110405/1a1a1ad948be278cff2d96046ad90768d848b41947aa1986/sample_mpeg4.mp4.zip', 'temp.zip')
        zipfile.ZipFile('temp.zip').extractall()
        os.remove('temp.zip')
    print 'Saving video...'
    save_video('sample_mpeg4.mp4')
    print 'Done'

if __name__ == '__main__':
    from time import sleep
    print 'Printing currently playing music (iPod/Music app only)...'
    print_ipod_title()
    sleep(1)
    print 'Downloading a video, and saving it to the camera roll...'
    download_and_save_video()
    print 'Making screen very bright...'
    sleep(1)
    set_screen_brightness(1.0)
    print '...very dark...'
    sleep(1)
    set_screen_brightness(0.0)
    print '...and something in-between.'
    sleep(1)
    set_screen_brightness(0.5)

While probably not terribly useful, these are all things you couldn't do before with Pythonista. :)

As I said – don't get too excited, there are a lot of things you can't do with this... Because of the 64-bit issues, anything that would require a callback is impossible (including things like defining new ObjC classes etc.), and it's also not possible (to my knowledge) to use any block-based APIs, which includes a lot of stuff in the more recent iOS SDKs...

Wow, this is cool! And Apple approves of this? Hopefully... But this is really cool! Does this kind of built a base to also load other kinds of 'modules', like requested a few times?

And Apple approves of this?

No idea, but hopefully, I'll find out soon...

Does this kind of built a base to also load other kinds of 'modules', like requested a few times?

Not really, you can basically just use this to call functions in the frameworks/libraries that are already linked with Pythonista. Anything else wouldn't work because of code signing/sandboxing.

In the unlikely case that Apple doesn't allow this in the Pythonista app would this at least be do-able in the snapshot?

So if I wanted to try and get some basic CoreMIDI access going this might be the way to go if you have included that framework into Pythonista. As one example - would it be possible to write the cb module using this capability? For CoreMIDI I can find plenty of C Python bindings examples so I suspect it may be straightforward to rewrite one of those using this method of interface.

That looks like a powerful thing indeed. Too bad I can barely read and understand, let alone write, C code. And Objective-C even less. Or any of the iOS SDKs for that matter. I know, it's not really code, just a direct interface to (Objective-)C datatypes/objects, but I still have no idea how to correctly use most of this :P

(Let's hope the app review team is sleepy enough when approving this update.)

Here's another fun example:

Audio Recorder.py

(Note: You'll get an iOS permission dialog about using the microphone when you run this; it obviously won't work if you don't give permission, and there isn't really any error handling...)

@omz Really hope this makes it. Could use an audio recorder feature like this (or maybe turn it into its own module?)

Midi?

import qrcode, sys
url = ' '.join(sys.argv[1:]) or 'http://omz-software.com/pythonista/docs'
qrcode.make(url).show()

@omz Very interesting indeed. I'm a bit curious as to what the 64-bit issue has to do with callbacks not being possible. I understand there are hurdles, but I thought they were a little different.

The obvious limitation on iOS is that you are not allowed to execute dynamically generated machine code. If that is how ctypes.CFUNCTYPE works, it cannot be used directly. But there are workarounds. For an arbitrary pure C callback, one may need to resort to tricks like defining lots of trampoline function slots. But in most cases (all?) in Objective-C, that won't be needed.

As you know, often in Cocoa you don't specify a C function pointer at all, but instead an Objective-C object and a selector. That is easily mapped to Python. You just need to supply an Objective-C wrapper class. And blocks, as you mentioned, are just a special case of this. A tiny bit of googling should tell you how. (Or start here.)

@omz you mentioned the possibility of symlink being added. Any update?

@briarfox Should be fixed in this build, forgot to mention.

@omz Long touch on clear button no longer allows for restart of pythonista

Long touch on clear button no longer allows for restart of pythonista

Yes, I intentionally removed that. Kinda hoped no one would notice. ;)

The problem was that on iOS 8, the notification that allows you to restart the app requires explicit user consent, and I didn't really want to have a "do you want to allow Pythonista to send push notifications?" dialog just for this... You could replicate the functionality with an action menu script that uses the notification module though... (and quit the app with os._exit(0)).

@omz - that Audio Recorder sample is great. I would like to hear how you wrote that. Did you start from an Apple developer sample objc sample and convert to this?

Could you explain what parts of the frameworks are (will be) available? Is it only what you have specifically used within the Pythonista App itself or is everything available by loading dynamic libraries that are just out there (doubtful).

I was just looking at another project that has a few examples of ways to browse classes and such using ctypes : https://code.google.com/p/ios-python-objc/ - apparently written because PyObjC was broken??

Thanks omz, Twisted now works :)

@zencuke

In the unlikely case that Apple doesn't allow this in the Pythonista app would this at least be do-able in the snapshot?

You mean the Xcode template? That should actually work already (it's the primary reason why the template doesn't support the 64-bit simulator right now).

@wradcliffe

So if I wanted to try and get some basic CoreMIDI access going this might be the way to go if you have included that framework into Pythonista.

Theoretically yes, though I'm not currently linking against CoreMIDI – and there's another problem: MIDISourceCreate and MIDIDestinationCreate require that the app declares the 'audio' background mode, which is a build-time thing that you can't modify after the app is installed... While declaring this would actually be useful for other things, I'm not sure if I could get the app through review with background audio enabled because it could be abused for all sorts of things (e.g. running in the background forever while looping a silent track...). I don't know, maybe I'll have to integrate a music player to have an official reason. ;)

As one example - would it be possible to write the cb module using this capability?

In principle, maybe, though CoreBluetooth makes heavy use of blocks (which you can roughly think of as the C equivalent of lambdas), and I couldn't get those work with ctypes... Bluetooth support also requires some build-time settings (keys in the Info.plist) to work in the background.

that Audio Recorder sample is great. I would like to hear how you wrote that. Did you start from an Apple developer sample objc sample and convert to this?

I basically went through the Objective-C runtime library documentation, built a few convenience wrappers to make the code a little less verbose (msg, cls etc., already used in the other examples, and a few others to wrap NSNumber and NSURL), and then looked at a basic tutorial of how to use AVAudioRecorder (can't remember which one)... I know Objective-C fairly well, so it was mostly a matter of translating the Objective-C syntax to the lower-level form I needed to work with the runtime library directly...

@mteep

Very interesting indeed. I'm a bit curious as to what the 64-bit issue has to do with callbacks not being possible. I understand there are hurdles, but I thought they were a little different.

The obvious limitation on iOS is that you are not allowed to execute dynamically generated machine code. If that is how ctypes.CFUNCTYPE works, it cannot be used directly. But there are workarounds. For an arbitrary pure C callback, one may need to resort to tricks like defining lots of trampoline function slots.

ctypes is based on libffi, and from my (limited) understanding, this is how the ARM versions of the library work. ARM64 support has only been added to libffi quite recently, and in my testing, it always crashed when trying to use CFUNCTYPE and friends... I haven't found a lot of information about this (there was a bug report on a mailing list somewhere that sounded related, but I can't find it anymore). It could very well be that I'm doing something wrong, but it also doesn't seem unlikely that the ARM64 support in libffi just isn't very robust yet.

But in most cases (all?) in Objective-C, that won't be needed.

As you know, often in Cocoa you don't specify a C function pointer at all, but instead an Objective-C object and a selector. That is easily mapped to Python. You just need to supply an Objective-C wrapper class. And blocks, as you mentioned, are just a special case of this. A tiny bit of googling should tell you how. (Or start here.)

Well, yes, when you write actual Objective-C, you'll see very few function pointers, but if you want to do things like creating a new Objective-C class dynamically at runtime, you actually do need them (for things like class_addMethod()...).

It might very well be possible to work around these issues by generating wrapper classes/functions etc., but I haven't looked into that very much yet.

@omz - I have been scrounging around for more educational info on how to use ctypes to interface to objc and ran into this project: http://pybee.org

They have a module called rubicon for Java and Objective C. Rubicon-ObjC is a bridge between the Objective C libraries and Python. It has been tested on both desktop/laptops, and on iOS devices.

It is basically a very well fleshed out set of wrappers that looks very much like the code you are writing in your tests and demos but completely generalized. The amount of code is huge! It is handling all kinds of hairy stuff like object reference counting and being able to dynamically add methods to classes.

It enables you to use Python to instantiate objects defined in Objective C, use Python to invoke methods on objects defined in Objective C, and subclass and extend Objective C classes in Python. It also includes wrappers for some of the some key data types from the Core Foundation framework (e.g., NSString and NSObject).

There seems to be support both 32 and 64 bit usage that you can see in the "CoreFoundation" module.

Other goodies include templates for building app that include Python and bunch of visual tools like a debugger that look very interesting.

Seems to be very well supported and actively being worked on. I downloaded it and was able to use it to list classes methods and ivars so it looks like it works. I have not found an example of it being used for block code or even just delegates or callbacks.

This group looks to be doing a lot of interesting things that are very synergistic with Pythonista.

@omz - this is just a message of thanks for continuing to keep this project up to date.

I ran into a recent thread of discussion started by the pybee author Russell Keith-Magee at https://mail.python.org/pipermail/python-ideas/2014-October/029856.html where he proposes that Python be modified to formally support building it for IOS and Android. This thread goes into many of the tough decisions that currently need to be made to get a build of any kind done on these platforms, but IOS is the obvious standout for having the worst. Russell is just discussing how this could be made a standard and supported build for Python in general makes your head spin. Russell is coming at it with a "proof of concept" implementation and just wants to give it back to the community and it looks like a difficult and tedious effort to do that.

The main point we should all keep in mind is that base level of work on this is substantial and it just keeps getting harder - not easier. With 64-bit and Python 3 bearing down - it makes me wonder if there is something more the users of this product could be doing to lighten that load. I get the sense that there are a lot of old timers in this community that have spare time on our hands that might be put to use doing some of the more tedious aspects of building and testing.

UPDATE: Here is a good Landon Fuller rant on what is behind a lot of this: http://landonf.bikemonkey.org/code/ios/Radar_15800975_iOS_Frameworks.20140112.html

@omz
I see. I looked at the libffi source, and it seems they fixed something very close to this in the AArch64 port in a commit three weeks ago. They use a trampoline table as I suggested, but with PC-relative addressing and fancy virtual memory mirroring so there's no limit to how many they can have. (Although there's a bug in that they forgot to unlock the ffi_trampoline_lock in case of failure, line 871 in ffi.c.) So maybe this haven't been included in ctypes yet, causing your crashes.

However, as when using Objective-C APIs, the Python programmer indeed doesn't have to provide pure C functions in order to subclass Objective-C classes. The function you pass to class_addMethod() receives the selector _cmd as well as the self pointer. So you can always add the same single function (at least one per value of the types parameter) that further dispatch to Python code based on [self class] and _cmd. Not super efficient, but it works in most cases.

There are many projects in many languages that do this. Others have mentioned some in Python that might do it this way. I myself am familiar with two in Java: the SWT Platform Interface for Cocoa from eclipse.org, and a vaguely similar thing for AWT in OpenJDK (since JDK 7).

And for blocks, what you actually pass is a pointer to a data structure (an Objective-C object) which (after the header including the function pointer) can contain anything you like. So it is even simpler to dispatch to the correct Python code.

So while this will require some work, it is definitely doable.

I've playing with the ctypes module for a while now and have for instance been able to integrate a live feed from the camera in a ui.View. I have also defined a custom Objective-C class that appears to work from the main interpreter thread. I can add an instance of that class as a delegate to a AVCaptureMetadataOutput, to be called on a newly created serial dispatch queue when "metadata" is detected in the stream. However, when metadata, such as a bar code or a face, is actually detected, Pythonista crashes.

I've looked over the code many times and tried many variations, but I can't get it to work. I hoped there would be crash reports that I could look at and get hints from, but since iOS 8 they seem to no longer be accessible on the device, only from Xcode. But even from Xcode, there are no recent crash reports from Pythonista. Why is that? (I have crash reports from older Pythonista versions and recent from other apps.) Is there any way to enable them?

The only hint I've got is a few lines in the system log that I got when running my code while the iPad was tethered to Xcode. But it basically just said "segmentation fault", no further details. I've tried to send retain to my delegate, but fear that maybe it is some Python object that gets garbage collected. (Comments in the PyBee Rubicon code suggest that CFUNCTYPE instances could be GC'd, but I don't quite see how that would happen in my code. That's about the only path I have left to explore.)

Has anyone else been able to get something similar to work? Or get crash reports? omz?

As a side note, I looked at the SWT PI Java code that I mentioned earlier, and it had comments suggesting that the Objective-C classes it defined only would work in the thread in which it was created (in this case the AppKit UI thread). But there was no explanation and can't really see why that would be the case.

@mteep As I said earlier, CFUNCTYPE will NOT work AT ALL in 64-bit builds, and I will probably move to 64-bit sooner than I had initially planned. I will continue looking into this because it's quite interesting, but honestly, it's also pretty much an 'edge use case'. It's nice that it works at all (to some degree), but it's not a feature I plan to be supporting "officially"...

As for crash reports, they've moved from the General section to Privacy/Usage & Diagnostics in iOS 8, took me a while to figure this out as well... You won't be able to get symbolicated crash reports anyway though, so it's probably not very useful to you.

@mteep wrote: "[I have] been able to integrate a live feed from the camera in a ui.View."

That sounds like code worth sharing. ;-)

@omz
Thanks, I now found the crash reports. But I find it slightly weird that they didn't show up in Xcode. Maybe it's because they end in .ips.beta instead of just .ips. Anyway, at least I now know which thread that crashed and why, in some sense. Then I can hopefully detect differences when I change stuff. (Interesting to note that it is a 64-bit address on an A8X, everything else is 32-bit, but maybe that's how it works.)

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000040000004
Triggered by Thread:  5

Thread 5 name:  Dispatch queue: com.apple.avfoundation.metadataoutput.objectqueue
Thread 5 Crashed:
0   Pythonista                      0x006915e7 0xea000 + 5928423
1   Pythonista                      0x0069406c 0xea000 + 5939308
2   AVFoundation                    0x218bb6f5 0x217b0000 + 1095413
3   CoreMedia                       0x234822c1 0x23455000 + 185025
4   CoreMedia                       0x23494713 0x23455000 + 259859
5   libdispatch.dylib               0x30dc224f 0x30db0000 + 74319

And yes, I am fully aware that you said CFUNCTYPE didn't work in the 64-bit builds. That's why I had to try it now while it is still 32-bit. (Otherwise, I couldn't compare face/barcode detections between newer/64-bit and older/32-bit devices.) My motivation is twofold.

First, this is functionality I've long wished for in Pythonista, but always assumed it would be out of scope. (Over a year ago, I prototyped something similar in Codea using a custom GLSL shader for bar code decoding, but due to the lack of a native UI I never polished it enough.)

Second, being able to use almost any iOS framework is of course very powerful, and before you added the ctypes module the thought hadn't really crossed my mind that such capabilities could come to Pythonista. But now, I know it is possible, although I understand that it likely wouldn't be a high priority for you but rather an 'edge use case', as you wrote. If Pythonista has this capability, a lot of user requests for features could be solved in Python, and you wouldn't need to spend time on the edge cases (like perhaps, the cb module, which I love too). I wanted to help make this a reality by both providing a usage example and a way in which it could be implemented.

As I wrote earlier, I think it is possible to enable access to a large part of the iOS frameworks without requiring a working CFUNCTYPE. I had hoped to be able to prototype such a solution that ultimately would be using a fixed set of predefined native dispatch functions. In the prototype however, the dispatch functions would need to be defined using CFUNCTYPE, while they work. (Alternatively, after seeing the libffi source code and the recent ARM64 patch, it doesn't seem implausible that CFUNCTYPE soon could work in 64-bit.)

@ccc Yes, but I had hoped I could make it do a little more. It is rather similar to omz' AudioRecorder. I just added a AVCaptureVideoPreviewLayer as a sub layer to the CALayer of a ui.View, whose pointer I got from the undocumented ui.View._objc_ptr. That part should work also in the 64-bit builds, as long as the pointer remains accessible.

Wow this is amazing. I am actually an iOS dev and would really love to actually write iOS specific code on iOS. I just hope Apple lets this through.

BTW is this a public beta? And if so, do you have any invites left?

@hvd, See https://omz-forums.appspot.com/pythonista/post/5792132698734592 for info on the beta.

I finally got the bar code detection to work. It was the Python method corresponding to the Objective-C method implementation (IMP) that got GC'd. Now I just have to clean the code up, a lot. In particular, I need to find a good place to dispose the Objective-C classes I registered. Otherwise, crashes are likely.

@mteep - very impressive. How did you manage to figure the crash issue out? Did you force GC manually or something like that? Also - did you make use of the PyBee projects helper classes?

@wradcliffe Thanks, but I can't really say it was a very structured approach. I started on two different approaches: rewriting using PyBee, and logging the address and memory contents of the ctypes objects to be able to compare with the register contents in the crash reports. But while doing that I came to realize that the way I had written the code, nothing in Python (nor in Objective-C) could actually hold on to the function trampoline slot. So I added a direct reference to the Python object which allocated it. Luckily it was that simple.

So I never completed the rewrite using PyBee, but that would most likely have eliminated the crashes too . Except that I could only have run the code once without restarting Pythonista, due to the Objective-C classes remaining registered. I currently deal with this by disposing the old class pair if I cannot allocate a new class pair, but it would be better to clean up before exiting.

In parallel, I also did a stripped down version without the metadata detection, in order to post it. However, when I added the ability to switch between the front and back cameras, it crashed again. But this does not seem to be a GC thing, rather that an AVCaptureSession only should be manipulated from a single thread (or serial queue). I'm looking at Apple example code, but haven't yet had the time to re-create the exact scheduling of the code across threads/queues. A ui.in_foreground would have made that easier.

@mteep - I think you are onto something that is going to be very valuable information going forward. If this has to do with a mechanism that is needed to make sure that an object is only manipulated from the same thread it was created on then a fix may be quite difficult to come up with.

The last time I tried to debug this kind of problem I gave up because adding the logging seemed to make problems go away or change to something else. I also found that I had to restart Pythonista for every single change I would make or repro was impossible. I did not have the crash report to look at so that could have yeilded a clue as well.

My only thought on this is that there should be some way to force GC related issues to occur by directly manipulating the gc system. Perhaps forcing gc to happen continuously could flush out any issues. Multithreading issues are much more difficult, in general, to debug - but there must be ways of forcing objects to be accessed from various threads in order to force these types of bugs to occur.

I am totally impressed with your tenacity and perseverance on this.

@omz - could you discuss what the background is with cffi and how it relates to the ctypes - ffi combination? I keep seeing references to it as an active project that makes writing the python code simpler somehow but does basically the same thing as ctypes and ffi in combination. Is this old stuff - or new? If new - maybe it solves the 64-bit issues already. Just curious.

UPDATE: I just noticed that cffi is included in Pythonista under pylib/site_packages along with a copy of pycparser. Interesting.

./pylib/site-packages/cffi/__init__.py
./pylib/site-packages/cffi/api.py
./pylib/site-packages/cffi/backend_ctypes.py
./pylib/site-packages/cffi/commontypes.py
./pylib/site-packages/cffi/cparser.py
./pylib/site-packages/cffi/ffiplatform.py
./pylib/site-packages/cffi/gc_weakref.py
./pylib/site-packages/cffi/lock.py
./pylib/site-packages/cffi/model.py
./pylib/site-packages/cffi/vengine_cpy.py
./pylib/site-packages/cffi/vengine_gen.py
./pylib/site-packages/cffi/verifier.py