Forum Archive

After playing around with Vision and coreML I rich "new file/project couldn't be created" situation.

I even can't edit/save existing scripts. If script need to read/write from Pythonista folder, Pythonista crashes.

I'm using iPhone8 not jail broken and lates Pythonista from appstore.

Do you have solution?

Regards.

@pavlinb did you try by removing app from memory?

In the past, did you jailbreak?

@cvp said:

@pavlinb did you try by removing app from memory?

I restarted the phone several times. I hope it has gone from memory.

In the past, did you jailbreak?

No, I didn't.

@pavlinb Remove it from active apps and then, try, in Safari pythonista3://

Tried already w/o success.

Now I'm trying with remove app and data and reinstalling.

After delete and reinstall Pythonista works now.

Sorry for your time.

@pavlinb no sorry, good to know, could happen to (almost 😀) everybody

Just for information - I played alot today with Vision coreML and Pythonista crashes again. ‘Documents and data’ in storage has increased to 5GB.

@pavlinb Oups. I suppose you will say on Settings/Général/iDevice storage/Pythonista

I also play, for you, with mlmodels and my storage was up to 5,44GB (normal is 3).
Remove app from memory, not reinstall, wait 10min and storage 4,6G

10min later: 3,7 GB

No explanation

Does waiting solve the crashing?

May be we have a way to delete junk programmatically from Pythonista?

@pavlinb no idea at all

Something I had never remarked before, in Settings/Général/IPad Storage,
Pythonista: 4,4 GB
If I tap, documents and data: 3,7GB
At return, it displays 1,43GB during a second
But I really have 2,13GB of personal files.

And 2,13 + 1,43 = 3,7 (almost)
Perhaps are these 1,43GB junk, or perhaps due to beta version?

Could people using or not the beta version check if the setting also shows a different storage during one second when returning from documents and data?

I uninstalled beta, but the problem remains.

I'm using iPhone8.

Actually my problem is that Pythonista crashes, not the amount of data itself.

I just thought they are connected.

@pavlinb crash with any script?

Only with those, which require reading/writing to file system.

At the same time I can't edit and save existing projects.

And can't create new projects.

@pavlinb and you have still free space?

Yes, I have about 30GB free in iPhone storage.

@pavlinb and between reinstall and crash, what did you do?

You mean what I do with Pythonista?

I played with coreML OCR from your example.

Before I have used Pythonista every day for small things and I haven't got any isues.

Probably Vision and coreML frameworks generates a lot of junk
, who knows...

I'll try to find a way to clean data programmatically.

@pavlinb if you reinstall, could you install the pythonista startup of @dgelessus , that could help to find the crash reason

And post here the exact script you use

Sure, I'll post details.

Settings/Général/storage today 3,77GB instead of 4,14
without removing anything...

The content of fault log:

Fatal Python error: Aborted

Current thread 0x000000016df9b000 (most recent call first):
File "/var/containers/Bundle/Application/0D357A4D-0366-4446-8E8F-34E0CF0C4E41/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/objc_util.py", line 772 in call
File "/private/var/mobile/Containers/Shared/AppGroup/788B3F26-6E63-4760-B7CD-1B8879F98443/Pythonista3/Documents/MyPython/visionOCR_4.py", line 49 in load_model
File "/private/var/mobile/Containers/Shared/AppGroup/788B3F26-6E63-4760-B7CD-1B8879F98443/Pythonista3/Documents/MyPython/visionOCR_4.py", line 57 in _classify_img_data
File "/private/var/mobile/Containers/Shared/AppGroup/788B3F26-6E63-4760-B7CD-1B8879F98443/Pythonista3/Documents/MyPython/visionOCR_4.py", line 75 in classify_image
File "/private/var/mobile/Containers/Shared/AppGroup/788B3F26-6E63-4760-B7CD-1B8879F98443/Pythonista3/Documents/MyPython/visionOCR_4.py", line 139 in classify_asset
File "/private/var/mobile/Containers/Shared/AppGroup/788B3F26-6E63-4760-B7CD-1B8879F98443/Pythonista3/Documents/MyPython/visionOCR_4.py", line 155 in main
File "/private/var/mobile/Containers/Shared/AppGroup/788B3F26-6E63-4760-B7CD-1B8879F98443/Pythonista3/Documents/MyPython/visionOCR_4.py", line 158 in

Code at line 49:

ml_model = MLModel.modelWithContentsOfURL_error_(c_model_url, None)

Before crash Documents & Data iPhone storage was about 150MB.
After crashing - 3GB.

Did you test with a very long text?

Do you use the beta? Because in my (beta) version, line 772 of objc_util is relative to restype and I don't know if it is the same in your version

Does the used space increase of some GB at each test?

@cvp said:

Did you test with a very long text?
What do you mean?

Do you use the beta? Because in my (beta) version, line 772 of objc_util is relative to restype and I don't know if it is the same in your version
No, I use official version from Appstore. I was with beta when this happens for a first time.

Does the used space increase of some GB at each test?
No, space was around 150MB after each teast.

@pavlinb Could you tell me which is the line 772 in objc_util in the App Store version.

Thus, you don't have a crash each time.

Do you use another mlmodel than OCR.mlmodel?

Is the image containing a text with a lot of characters?

@cvp said:

@pavlinb Could you tell me which is the line 772 in objc_util in the App Store version.
res = objc_msgSend(cls, sel(self.sel_name), *args)

Thus, you don't have a crash each time.
Only with script that reads/writes to memory

Do you use another mlmodel than OCR.mlmodel?
Yes, used other models, it doesn’t depend from model.

Is the image containing a text with a lot of characters?
It looks that script tough the image has lot of characters.

@pavlinb you said that memory was 150MB after a test, and this test was not trying to write to memory?
I don't understand the difference between two tests of mlmodel, one writing and the other one no

Memory was about 150MB when test finish without crash - I checked the memory regularly.

When Pythonista crashed, I checked the memory again - it was 3GB.

I tried to OCR an image, in wich algo recognizes lot of characters. May this is the main reason.

Do you want a copy of that image to try?

Now I can't edit scripts and can't create new ones in Pythonista.

Some scripts still work.

But those that uses mlmodels crashes when script tries to load the model.

@pavlinb I don't want to take the risk to need to reinstall, sorry.
If you remove Pythonista app from active apps list, and wait some time, does the memory vary?

@cvp said:

@pavlinb I don't want to take the risk to need to reinstall, sorry.

Sure, no problem.

If you remove Pythonista app from active apps list, and wait some time, does the memory vary?

After 1 hour memory is the same - 3GB.

@pavlinb I think it is now time to ask help from our big guru's...

I wonder can you check the pythonista temp directory?

Are you using a local file when you load your model? Or an internet url?

Also... What is your model_url? We need to check that is actually valid.

@JonB said:

I wonder can you check the pythonista temp directory?

Sure, what do you need?

Are you using a local file when you load your model? Or an internet url?

I use local files.

I mean check it for excessively large temp files.

What is your exact model_url?

Can you get a small model to work?

Have you already pre-compiled the model?

https://alexsosn.github.io/ml/2017/06/09/Core-ML-will-not-Work-for-Your-App.html

This mentions that complex models can turn into several GB on the device. You might also try backing up after, and looking at what files get created and where. Possibly, I'm thinking if you put your file into a folder starting with a period, that keeps pythonista from showing /indexing it.

If he uses the mlmodel I found, it is here

but I also used it without any problems, with images of short texts

Yes, I used mentioned model.

And with clean images with few symbols all is ok.

Probably the problem occurs with complex images.

@JonB said:

Have you already pre-compiled the model?

The omz code compiles the model before using it.

For each character, the compiled model is generated as
file:///private/var/mobile/Containers/Data/Application/C285FD04-6489-45E5-A6C5-D4A44D300BBC/tmp/(A%20Document%20Being%20Saved%20By%20Pythonista3%20643)/OCR.mlmodelc/

Omz code needs to be improved so this compilation is done only once, not for each character...

It might be a good idea to compile it once, copy it to a regular location, then delete the tmp file. That's what Apple recommends

If you run the code that does a fresh compile every time, you will quickly fill up the tmp storage. I'm not how reliable the cleaning of tmp files is.

Agree. .

Set vn_model global,
call load_model at begin of main and
remove load_model in _classify_image_data

Script is quicker and does not generate a compiled model at each character

After run, tmp folder is empty

def load_model():
    global vn_model

def _classify_img_data(img_data):
    global vn_model
    '''The main image classification method, used by `classify_image` (for camera images) and `classify_asset` (for photo library assets).'''
    #vn_model = load_model()

def main():
    global vn_model
    vn_model = load_model()

After some tests with small texts, Pythonista storage has increased of 1GB but I hope this will decrease in some hours...

import os
d = '/private/var/mobile/Containers/Data/Application/C285FD04-6489-45E5-A6C5-D4A44D300BBC/tmp/'
os.chdir(d)
l = os.listdir()
for e in l:
    print(e)

Shows ....a lot of

(A Document Being Saved By Pythonista3 92)
(A Document Being Saved By Pythonista3 315)
(A Document Being Saved By Pythonista3 262)
(A Document Being Saved By Pythonista3 628)
(A Document Being Saved By Pythonista3 563)
(A Document Being Saved By Pythonista3 54)

Trying to remove it gives

PermissionError: [Errno 1] Operation not permitted: '(A Document Being Saved By Pythonista3 92)'

Each file has a size of 96 bytes??????????

Works.

No more crashes with complex images.

Shame on me, each line is a folder, that's why it is 96 bytes and unremovable 😢 because not empty

The .mlmodelc are not permitted for removal...

.mlmodelc is still a folder, one more time, shame on me...
1GB of temp files...

In the text detection script, add

    print(c_model_url)

Note the code d = '/private/var/mobile/Containers/Data/Application/C285FD04-6489-45E5-A6C5-D4A44D300BBC/tmp/'
Replace it in the little script here-under to **clean /tmp folder****

import os

n = 0
s = 0
def loc_dirs(path):
        global n,s
        try:
            file_path = path
            os.chdir(file_path)
            filelist = os.listdir(file_path)
            for file in filelist:
                print(file)             
                f_full = file_path + file
                if os.path.isdir(f_full):
                    loc_dirs(path+file+'/') # explore it
                else:   

                    loc_size = os.path.getsize(f_full)
                    n = n + 1
                    s = s + loc_size
                    os.remove(f_full)
            os.rmdir(file_path)
        except Exception as e:
            print('exception: ',e)
        return


d = '/private/var/mobile/Containers/Data/Application/C285FD04-6489-45E5-A6C5-D4A44D300BBC/tmp/'
#loc_dirs(d)
print(n,s)

print(n,s)

3637 2027261814

@pavlinb 20GB?

May be 2GB.

@pavlinb 2.027.261.814 = 2GB, Sorry for my old eyes 😢

The most important is that we
- find the reason (too much .mlmodelc)
- change the code
- can clean the tmp

@cvp Thanks to you.