Forum Archive

I don't know if anyone's interested, but I took a combination of the Sketch example, along with some other examples I've seen online, and come up with the following.

It's a simple Neural Network class, prepare data, train it and it can hopefully guess what you draw to test it with.

Simply:
- Draw three positive image, all the same. I.e. Draw three smiley faces.
- Then draw three negative images, all the same. I.e. Draw three sad faces.
- Then press the Train button.
- OK so now it's ready. Draw either a copy of the positive or negative image, and see if it gets it right 😀

I don't claim this is the best code ever, or even efficient algorithm and could be tuned much better, but thought I'd share it anyway 😀

https://gist.github.com/d77486f38d5d23d6ffd696e7bf0545fc

Updated gist link to the right version. Good start, eh 😀

@mkeywood Nice code, I've tried, not always successfully but I like it....

@mkeywood thank you! Incredible that it is so simple to make some NN. You demystified it.
How can you add layers?
Can you make CNN too?

@jmv38, thanks.

Currently it’s quite simple and so not massively extensible for things like CNN. Although that is the area I’m looking at now 😄

As for additional layers, we could add them relatively easy. Basically it’s all geared around variables W1 and W2 that are the weights between the input and hidden layer, and hidden and output layers respectively.
Extending the init, forward, backward etc should be easy enough to extend.
Something else I can look at 😄

@cvp, thanks 😄 My daughter played with lots of different sketches and found some more reliable than others.

@mkeywood i have tried to add some robustness: i shift the drawings in 9 position for the training. I had to decrease the learning rate by x0.1. Is it ok? what do you think?
Not sure the performance is better.

note: I also changed the layout because i work in landscape.

https://gist.github.com/d87a0833a64f0128a12c59547984ad2f

I have changed the learning rate to 0.02 and increased the number of training epochs to 200: i am getting good results now.

@mkeywood really having fun with your code, thanks so much for sharing!
Now i made some more changes: using 2 neurons in the output, to get independant estimates and reliability (and also to see the result with a 3rd template that is not 1 or 2).
I have also added a small rotation in the training, and tweeked a little bit the training parameters.
https://gist.github.com/e373904d3ccba03803d80173f44b5eee

@mkeywood
update: i have made a Layer class to more easily add layers. I have now 4 layers. It seems to work, but i have not checked if the undeground maths are correct. I’ve assumed your formula to backpropagate the error is recursive.
With this implementation you can use the number of layers you want, with the number of neurons you want inside.

https://gist.github.com/aea7738590793eefcd786be8657fa88b

@jmv38 this is awesome!!

Glad it was helpful, but you've taken it to another level completely 😀
That's some really great additions. Really amazing. I look forward to see what you do next 😀

Thanks 😀

@mkeywood here is an update:
https://gist.github.com/3c9f5917224d8a70ea319af1df973c73

i have made 3 inputs, many duplicates, and some live feedback during learning. Works not bad sometimes.

another one: now i display the learning set while creating it
https://gist.github.com/549d071893cac00e84fcd1875d422d1a

hello
v08: added a white image and a random image should return 0: to improve robustness
https://gist.github.com/d21c832208f33fe083b9200b29e1f073

@jmv38 Hi interesting project! I'd like to ask you two things about it:

1. could you easly modify your script in order to allow execution also on little screens (4 in)? Does anyone here (Pythonista forum) know a general way to modify easly a script with UI in order to adapt it automatically according to screen size (through the automatic recognition of the screen size of the device where the script is executed)?
2. only for fun, if you are interested: how about a script (by following your original script) that tries to learn to play tic-tac-toe game? For example with random choice of moves at beginning and positive weight to set of moves for winner player in several matches, in order to create a set of moves (getting closer to being the best ones) for each situation? What would you suggest?

Thank you and feel free to share some reasoning about it.
Regards

@Matteo hello.
Note that mkeywood made the original programm and ui layout. I just made a set of small changes each time that lead me here.
For you questions:
1/ the ui part is at the bottom of the script. You can change the numbers and the layout to match your screen definition. That is some work though (1 hour or less).
2/ that would be quite some thinking to do that. For the moment i am just doing simple things to learn python, by tweaking mkeywood programm, so it is beyond me.
Thanks.

@jmv38 Oops, you are right , my mistake, sorry @mkeywood :-), I didn't read the whole thread...

Anyway thank you for the answer. Some times ago I started for fun to study something about ML, and the first test example in my mind was an algorithm able to learn how to play a simple game like tictactoe, without studying any python specific library for ML.

The interesting thing in my opinion is how to create a general algorithm able to learn something without any big python libraries, only as a concept proof and with some little constraints defined by user for the research of the Ml goal/goals. The constraints could change in the algorithm when some situations occur during calculation. So thank you again both for your work, maybe it could give me some technical info for the ML game solver I've in mind.

Regards

v09:
1/ cleaned up some code
2/ live color feedback during training on samples
https://gist.github.com/89684d9166746504bba88348240e26ff

v10: 1/ added a [learn more] button to ... learn more.
https://gist.github.com/f7fc75b727c953e4dbb59c04f88acf74

@jmv38, looks like some great updates. I look forward to looking over them this weekend :)

@Matteo, no problem at all. @jmv38 has built on it massively from what I did. Really this is here for anyone to enjoy.

Thanks

Please help wanted!
I am stuck on a bug and i cant find what i am doing wrong.
here is the gist https://gist.github.com/3af5cf10e59944648ee38d3628282324
it runs fine. But i have tried to move a small piece of code to a class and then i crash all the time. no idea what is wrong.
To see the bug happen, replace False by True in this line 294

    testBug = False # set True to show the bug

with False i directly execute the code, with True i use the piece embedded in the SketchView class.
the problem seems to appear when i ask the SketchView instance to remember an image: line 230

self.sImage = pil_image

can anyone help me?
thanks

@jmv38, if Pythonista crashes, google for ”dgelessus faulthandler” to get the ObjC exception. If it is not a Pythonista crash, what is the trace?

it is a pythonista crash

note that the images stored are very small (25x25) and only 3 SketchView objects are using them. So it cannot be memory overflow.
I am probably doing something very wrong somewhere, but what?
What is puzzling is that the very same code, executed 200 times works fine. I try to execute it only 3 times, and re-use the result, but then it does not work...
Note that it works the 1rst time. This code never crashes

  def run(self):
    global X, y, pts, NN
    n = len(self.vars)
    count = self.count
    if count<n:
      if count == 3: exit()

the crash occurs ramdomly during one of the next calls. Not always the same. I must be writing in the memory at a wrong place.

Crash, with faulthandler, gives an empty faultlog-temp.txt...strange

@jmv38 Not sure if that helps but no crash with

  def getImg(self):
    if self.sImage == None:
      pil_image = ui2pil(snapshot(self.subviews[0]))
      _,_,_,pil_image = pil_image.split()
      pil_image = pil_image.resize((100, 100),PILImage.BILINEAR)
      pil_image = pil_image.resize((50, 50),PILImage.BILINEAR)
      pil_image = pil_image.resize((25, 25),PILImage.BILINEAR)
      return pil_image.copy() <-------------------
      self.sImage = pil_image
    return self.sImage.copy()

@cvp thank you, i feel less alone...
your last proposal is not a solution to the problem: the sImage is never updated, so it is recomputed at each cycle, that is what want to avoid.
I just tried some more changes (slow down, predefine self.sImage), but nothing works.
Must be something stupid (a bad local name, messing with a global?). Or are the some memory bugs in pythonista?
I think i must be degrading self.sImage, but how? i return a copy, not the image itself, and i dont modify sketch[] during learning...

@jmv38 this shows, on my iPad mini 4, that crash arrives after preparing 71/243

    if testBug:
      pil_image = v.getImg()
      time.sleep(0.05)

Preparing 74/243... .....always 74 even with time.sleep(0.5)

@jmv38 Not sure if this modification does destroy the algorithm

1) in updateLearninImage

  BWlearningSetImage = temp#.convert('RGB')

2) in showTraining

    global BWlearningSetImage
    BWlearningSetImage = BWlearningSetImage.convert('RGB')

@cvp does it solve the bug?

@cvp no crash, you are right!
Incredible that you found that.
Any insight of what is going on there?

@cvp thank you so much for solving my problem!
I wish i understood what was wrong in my code, though...

@jmv38 I'm just happy to have been able to help.
Sincerely, I don't understand all your code but I've tried to follow it, step by step by skipping some process until I found this "solution". I agree that it does not explain the problem

Doing the conversion at end is less work, I think, because not converted at each iteration.
Perhaps, a problem of cpu consumption

v12 https://gist.github.com/8fa3ac1516ef159284f3090ba9494390
big pef improvement with prior normalization of image size and position

Huge update! Now you can inspect the states and weights of the internal layers and get a feeling of how the network decides!
https://gist.github.com/ef4439a8ca76d54a724a297680f98ede

Also I added a copyright because this is a lot of work. @mkeywood if you are uncomfortable with this let me know.

Here is a video showing the code in action
https://youtu.be/yBR80KwYtcE

screenshot

300 views and not a word...???
Hey guys (and gals), if you like the code i share, some encouragement is always welcome!
Thanks.

v14 colors modified to better understand the network computation.
now it is much easier.
https://gist.github.com/94a8d1474a6ef6e49972518baa730f1b

Can you explain what is happening in the bottom set of plots?

@JonB hello.
the bottom plot shows:

during training: the training set and the color of each element corresponds to the recongnition result of this element.

during guessing (what you can se above):
the states of input layer0, then weights 0>1, then states of layer1, then weights 1>2, then states of layer2, then weights 2>3, then states of layer2, then output layer3 = 3 neurons.

First i display the weights in black and white (white=positive, black=negative), then i display in color and in sequence the signal propagation through the network:
- the neuron states: green means active (1) and red means inactive (0).
- neurons x weights values: Green means positive influence on the neuron (excitation) and red negative influence (damping). The brighter the stronger. I.ll call this wxn for weights x neuron.

The colors of the 3 neurons of layer 3 (red,red,green) are the same under the sketches: they correspond to the same data.
the group of 3 blocs on the left of these neurons (weights2>3) are the input weights of each neuron x the input of previous layer.

Let’s analyse this picture

Here you can see that a single wxn is mainly responsible for the 3 neuron states: its the weight (1/2, 0/2) ((0,0) is top left of each bloc), excited by neuron (1/2, 0/2) (green). Other neurons are desactivated (red) so the wxn are insignificant (black). This neuron is damping the 2 first neurons (red wxn) and exciting the last one (green wxn).

Now let’s see why neuron (1/2, 0/2) of layer2 is excited. Look in wxn1>2 at the bloc (1/2, 0/2). Several wxn are green, they are responsible for the response. There is no opposite response (red).

Let’s look at the stonger response wxn (2/4,3/4). The previous layer neuron corresponding is green too (active). look at the corresponding wxn0>1: you can see the the top left part of the ‘o’ drawing is green = detected.

so we can say the ‘o’ has been detected because of its top left part, which is not present in the 2 other drawings. That makes sense.
And the 2 other choices have been rejected for the same reason (it might not be the case).

I hope this explaination is what you were expecting.

here is a video that is the same explanation
https://youtu.be/L-6ZwbXjG48

On the global thing... the reason that works is because it forces del to be called on older images that are no longer being used. Elsewhere in this forum you can find discussion that the garbage collector is not fast enough at deleting images when cycling thru lots of images. You can also fix this issue by calling del yourself but I like the elegance of the global solution.

@jmv38 Hi, very interesting, congratulations and thank you for your explanations!

Unfortunately I can't use your script due to little screen of my 5s (no, I haven't an iPad).

I'd like to ask you some things, if you are interested and have time:

1. would it be too difficult for you to modify your script in order to allow user to swipe/move with fingers the full graphical panel of your script in little idevices to be able to see and access the full sub graphical views ("Prepare the data", "Train the model", etc...)? I think that by using the powerful scripts written by @mikael it could be possible, but really I don't know how.
2. could you think a good proposal to add a "delete" button for each single square in order to delete draw inside only one square, instead of to delete all by touching the Reset!! key?
3. in my opinion it would be nice to implement (but I repeat only if you are interested and have time) a way to have a grid (with adjustable dimensions) for when user draws very simple things on squares, only to test the algorithm also with simple draws like for example 2x3 big pixels images.

Thank you
Regards

@ccc not quite sure what you reffer to.

@Matteo thanks
1/ and 3/ are quite some work and are not in my top priorities now.
For 1/ maybe you could modify the code to fit your needs?
2/ is easy, i could add it, i’ll put that in a future version.

@Matteo A quick and dirty solution that should work for any future version of @jmv38 code:
- replace mv = ... by

#mv = ui.View(canvas_size, canvas_size)
mv = ui.ScrollView(canvas_size, canvas_size)

• add this at the end

mv.present('full_screen', orientations='landscape')
#=== added
wm = mv.width
hm = mv.height
for sv in mv.subviews:
    wm = max(wm,sv.x+sv.width)
    hm = max(hm,sv.y+sv.height) 
#print(w,h,wm,hm)
mv.content_size = (wm,hm)
mv.scroll_enabled = False
scroll_right = ui.ButtonItem()
scroll_right.title = '➡️'
def scroll_right_action(sender):
    ws,hs = mv.content_offset
    ws = min(ws+w/2,wm-w)
    mv.content_offset = (ws,hs)
scroll_right.action = scroll_right_action
scroll_left = ui.ButtonItem()
scroll_left.title = '⬅️'
def scroll_left_action(sender):
    ws,hs = mv.content_offset
    ws = max(ws-w/2,0)
    mv.content_offset = (ws,hs)
scroll_left.action = scroll_left_action
scroll_bottom = ui.ButtonItem()
scroll_bottom.title = '⬇️'
def scroll_bottom_action(sender):
    ws,hs = mv.content_offset
    hs = min(hs+h/2,hm-h)
    mv.content_offset = (ws,hs)
scroll_bottom.action = scroll_bottom_action
scroll_top = ui.ButtonItem()
scroll_top.title = '⬆️'
def scroll_top_action(sender):
    ws,hs = mv.content_offset
    hs = max(hs-h/2,0)
    mv.content_offset = (ws,hs)
scroll_top.action = scroll_top_action
mv.right_button_items = [clearAll_button, scroll_right, scroll_left, scroll_bottom, scroll_top]

You will have 4 menu buttons to scroll in the 4 directions
I hope it works because I was not able to test on an iPhone 5s but it is ok on my iPad mini 4 in portrait mode where I don't see all like you.

Some labels of code do not have a width, thus set as 1024, that's the reason why you can scroll too much at right...

Put this alignment line to check

lb.text='OK now lets see if it can Guess right'
lb.alignment = ui.ALIGN_RIGHT

@Matteo Better solution, use this code at end, to swipe with two fingers...

#=== added
wm = mv.width
hm = mv.height
for sv in mv.subviews:
    wm = max(wm,sv.x+sv.width)
    hm = max(hm,sv.y+sv.height) 
#print(w,h,wm,hm)
mv.content_size = (wm,hm)
mv.scroll_enabled = True
mvo = objc_util.ObjCInstance(mv)
mvo.panGestureRecognizer().setMinimumNumberOfTouches_(2)
mvo.panGestureRecognizer().setMaximumNumberOfTouches_(2)

Tested on iPhone 5s

Don't forget to replace mv = ui.View by ui.ScrollView

Hi @cvp, wondeful, it works perfectly! Very exciting! Now I can use the jmv38 code also in my little phone :-)

Unfortunately when I touch button 1/ Train after adding the draws the code tells me "Type Error: integer argument expected, got float".

I will perform some tests and if needed I will post here the full traceback.

Thank you again
Regards

@cvp hello
i tried your code: it works fine on my ipad. No error.
But i can scroll only horizontally. How could i scroll vertically too?
thanks.

@jmv38 I think your view does not need to scroll vertically.
If I add a label at y=1000, vertical scroll with two fingers works

@Matteo I suppose it is a problem for @jmv38 , he will need at least the line number

@jmv38 uncomment my line

print(w,h,wm,hm)

To check vertical dimension and usage

@Matteo for your request 2/: it already works. Dont press reset, just start another drawing in one of the boxes and it will replace the previous one. Then tap train.

Hi @jmv38 , I'm sorry for the delay, I've been busy and never used Pythonista for one week..Now I tested something and solved the problem related to integer argument expected of version 14 by adding int(argument) where needed in your code and since I use python 2.7 by default, I forgot to put the command #!python3 on the first line of the script (without it I had another problem with python 2.7). Now all work well!

For request 2 about erasing only one draw, I can't understand, sorry: if I try to draw something else on the square the old draw remains. Am I wrong?

Anyway thank you again (also @cvp) for support.
Regards

@Matteo For your request 2, when you draw on an existing drawing, this one stays until you terminate your move, when your finger leaves the screen.

@cvp Hi, sorry but it doesn't work with me , the reason could be I use Pythonista 3.1 (301016)?
When I draw something on a square, leave my finger from the screen, try to draw something else on the same square by touching again the screen on the square, and leave my finger from the screen , the old draw remains, it doesn't disappear to show anly the new one.

But don't worry, it is not so important, the script works well now.
Thank you
Bye

@Matteo You're right, but try after a training run. If I redraw before the training, both drawings stay, if I redraw after a training, the first drawing disappears

@jmv38 No problem at all 😀

Hi @jmv38, great job!

Not sure if you have already known that coreML can be used in Pythonista. Snippet in OMZ’s gist.

Cheers.

@cvp Yes, in this way it works! Thank you @cvp.

@jmv38 and @mkeywood : can the script recognize also the same picture drew in different positions inside the three squares? I tried it but maybe I'm wrong with something because it doesn't work, I mean if I draw the same picture in different positions, I can't obtain the good choice of the picture inside the guess square.

Thank you
Regards

@st84 hi. I dont see which omz post you are referring to. Can you put a link? thanks

@Matteo my version ofthe script tries to recognize 3 different objects. So draw the same object in 2 boxes will not work: the software assumes they are different.
The initial version of mkeywood is different.

@jmv38 here

@cvp and @st84 thank you.
it is very powerfull. But somewhat opaque.
Do you know
- how much memory is used? where is the model stored?
- what is the NN structure?
- is it fully local?
Thanks

@jmv38 the models are downloaded from https://docs-assets.developer.apple.com/coreml/models/MobileNet.mlmodel and copied in a local file, then next time, it is entirely local.

Big file of 170 MB

@cvp thanks
more in https://arxiv.org/pdf/1704.04861.pdf

@jmv38 this is interesting

@cvp thanks.