Forum Archive

@omz, just would to request to have TinyDb https://github.com/msiemens/tinydb included in Pythonista distribution or equivalent.
To me at least, this would be a great addition.
TinyDB seems to sit somewhere between shelve and SQLite. where shelve is too simple and SQLite is to complicated for simple things.

What I like about it is:
1. super simple to use
2. only has one file, unlike say shelve creating 3 files
3. as simple as it, it supports tables
4. 100% pure python and no external dependencies
5. is pip installable, good for updates between Pythonista updates
6. if you want to dig in a little, you can extend it
7. the queries are straight forward

Also it has 100% test coverage, good docs and over 1,700 stars on github.

Look, I am sure you have to be very selective of what you include in Pythonista as batteries Inc. I don't know your criteria, but at least on the surface this seems like it would be a good candidate.

I posted here instead of the issues repo, in case people here had there own ideas or comments.

Edit should have mentioned as another plus, support for py2.7 & py3.6 among others.

I was just playing around using tinydb. The below code is a little stupid and is not suppose to be great. But I wanted to try somethings. Subclassing the list for one and using tinydb to save the lists contents for another. This is a very simple example. But I have to walk before I can run :)

# this example needs tinydb installed to run.  
# this could be done with StaSh pip cmd, pip install tinydb
'''
    not so important to run this code. i was just playing with tinydb and subclassing list.
    i didnt try to use any tricks or optimise it. just playing to see the potential.
    I am sure many have done something like this before. Subclassing lust that is and adding
    persistance of some form or another. 
    Dont need a Database to do this of course.  but it may come in handy in more complicated
    requirements.  
    At the end, i just dispay dialogs.list_dialog with a slice of a loaded list.
    But for something like the dialogs.form_dialog, something like this class might
    be helpful...
    maybe i have approached this totally the wrong way. i just wanted to try it.
'''
try:
    from tinydb import TinyDB, Query
except :
    print('tinydb is not installed, exiting.')
    exit(0)

import dialogs
import warnings

class ListPersitant(list):
    def __init__(self, filename=None):
        self.filename = filename
        self.query = Query()

    def save(self, filename=None):
        '''
            save the list into the database. doing like a blob save. eg, saving the list
            contents under a single key, rather than saving the lines of the list
        '''

        fn = self.resolve_filename(filename)
        if not fn:
            warnings.warn('The list not saved as a filename has never been set')
            return      # no filename has been set

        db = TinyDB(fn)
        el = db.get(self.query.key == (fn))
        if el:
            db.update({'data':self[:]}, eids=[el.eid])
        else:
            e = db.insert({'key':self.filename, 'data': self[:]})
        db.close()

    def load(self, filename=None):
        fn = self.resolve_filename(filename)
        if not fn : return

        db = TinyDB(fn)
        el = db.get(self.query.key == (fn))
        if el.eid:
            self[:]= el['data']
        db.close()

    def resolve_filename(self, filename=None):
        if not filename:
            if not self.filename:
                return
            else:
                return self.filename
        else:
            return filename


if __name__ == '__main__':
    fn = 'astro_turf.json'
    l2 = ListPersitant(fn)
    l2.append('a')
    l2.append('b')
    l2.append('c')
    l2.append('d')
    l2.append('hhh')
    l2.save()

    # print out the db
    db = TinyDB(fn)
    print(db.all())
    db.close()

    # create a new list, and load in the data
    l1 = ListPersitant()
    print('l1', l1)
    l1.load(fn)
    print('l1', l1)
    dialogs.list_dialog('List Dialog', l1[:3])
    l1.append('Pythonista')
    print(l1)
    l1.save(fn)

    l3 = ListPersitant(fn)
    l3.load(fn)
    dialogs.list_dialog('List Dialog', l3)

    def resolve_filename(self, filename=None):
        return filename or self.filename or None

@ccc , hmmmm. I knew it :). For sure you would say something. I should have used a context manager for opening tinydb, also could have cached an id on the first insert/load/save etc. for updating. Also more error checking/warnings. I have never subclassed a list before, at least that can remember. But it's such a nice idea. Get so much for free.
But honestly after this small experiment, I am left wondering why python does not have a built in persistence methods for objects or at least the data for objects. Also, I don't see why py3.6 did not attempt handle the pickling datetime objects natively. I understand dates and times are complex, but there seems to be standard ways and views how not to screw this up, well to a minimum. Ok, they put in the fold to help with DST. But one of the biggest reoccurring problems I see is with serialisation of datetime objects when it comes to storing them. Seems crazy to me. But ok, I am ranting now :)

Really nice discussion on DST and the Maya project at https://talkpython.fm/episodes/show/115/python-for-humans-projects

Highly recommended.

@ccc , thanks. I listened to that one. Also watched the vid from PyCon2017 It's time for datetime. Which is good. There also was a Talk about tinydb on Talk Python to me also link

Just wanted to say I tried this TinyDB middleware serialisation extension written by one of the users. Worked as expected. I know JSON also has hooks to intercept and serialise objects, I just have never done it.
I just used the datetime example listed in the site, just changed it to use isoformat. Of course you can serialise anything you want.

I was interested about using TinyDB to act as virtual list for ui.TableView (just store an id number until the row requests to be drawn) I asked the developer if there was an optimised way to get all the ids in a table as a list.
He mentioned i could use an internal method eids = list(db._read().keys())
I just did a test below. Nothing scientific about it because you have creation, faker, printing involved. Regardless it is still fast.
Maybe not important, but still interesting in my view. The thing I don't understand is how much memory is being consumed.
Anyway, this was just meant to see if it's in the realm of possibilities

from random import choice
try:
    from tinydb import TinyDB
except :
    print('tinydb is not installed, exiting.')
    raise ImportError

from faker import Faker
fake = Faker()

def create_fake_db(filename, num_records):
    lst = []
    for i in range(num_records):
        d = dict(first=fake.first_name(),
                last=fake.last_name(),
                age=fake.random_int(min=18, max=89) 
                )
        lst.append(d)

    with TinyDB(filename) as db:
        db.insert_multiple(lst)
        eids = list(db._read().keys()) # The developer suggested this!!!
    return eids

if __name__ == '__main__':
    filename='eid_test.json'
    eid_list = create_fake_db(filename,500)
    print(eid_list)

    # pick a random record to display and get a record count
    with TinyDB(filename) as db:
        item = db.get(eid=choice(eid_list))
        num_records = len(db)

    print('Records in database={}'.format(num_records))
    print(item)

LOL, still here beating this drum :) but below is a simple example of saving views into a TinyDB and recovering them.

I know it's simple, and i guess that is the point. It should be simple. Look I know its far from being an exhaustive test. Just a way that muliple UIFiles could be packaged and easily addressable. The real break through is the ui.dumps method, make this easy

import ui

try:
    from tinydb import TinyDB, Query, __version__ as tinydb_version
except ImportError:
    print('tinydb v3.4.1 or greater needs to be insalled to run this app')
    exit(1)


def save_view(filename, key, jstr):
    '''
    save or update the json str of a view into the database with a key.
    The key, is how we recover the view later
    '''
    db = TinyDB(filename)

    el = db.get(Query()['key'] == key)
    pay_load = {'key': key, 'data': jstr}

    if el:
        db.update(pay_load, eids=[el.eid])
        print('Updated View-{}'.format(key))
    else:
        db.insert(pay_load)
        print('Created View-{}'.format(key))


def get_saved_view(filename, key):
    '''
    return a ui.View given a key that has been previously saved to
    the database.
    '''
    db = TinyDB(filename)
    el = db.get(Query()['key'] == key)
    if el:
        return ui.load_view_str(el['data'])


class MyClass(ui.View):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.make_view()

    def make_view(self):
        pass


class MyClass2(ui.View):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.make_view()

    def make_view(self):
        pass

if __name__ == '__main__':
    db_filename = 'myviews.json'
    f = (0, 0, 300, 400)
    v = MyClass(frame=f)
    v.present(style='sheet', animated=False)

    v2 = MyClass2(name='MyClass2', bg_color='purple', frame=f)

    save_view(db_filename, 'myClass', ui.dump_view(MyClass()))
    save_view(db_filename, 'myClass2', ui.dump_view(v2))

    v3 = get_saved_view(db_filename, key='myClass2')
    v3.present('sheet')
    v3.wait_modal()
    v3.bg_color = 'orange'
    v3.title = 'Orange View'
    v3.add_subview(ui.DatePicker())
    save_view(db_filename, 'myClass2', ui.dump_view(v3))
    v4 = get_saved_view(db_filename, key='myClass2')
    v4.present('sheet')

@omz, I hope its not inappropriate to ask this, but have you given any thought to adding TinyDB to the undocumented modules supplied with Pythonista? If so did you come to conclusion either way? I am convinced I am never going to write any compelling app. I get distracted to easily these days. But i do enjoy trying to make small helpers for Pythonista. I personally really like the simplicity of TinyTB. But it doesn't make sense for me to use it if people are going to have to install/copy it. Especially as the things I do are pretty small.
Anyway, not to try and pressure you. But if you have made up your mind one way or another it would be good know. If not, I will wait to see what happens.

@omz, sorry back to this subject. But if you decide to include TinyDB you might also consider including ujson also. Apparently a ultrafast json encoder/decoder written in pure C. However does not have the extensibility of python json. But if the extensibility is not required apparently is a big speed up for TinyDB. I am sue it could be used else where also. However I do see a problem. That is the package name. If a package called ujson is present it will just use it, it imports ujson as json. This maybe not what you want to do. On a desktop this would be more manageable. So maybe the package name could be ultrajson and if you wanted to use it in TinyDB you could import ultrajson as ujson before starting to work with TinyDB. I think that would work. I am not sure. Another way would be to ask the dev of TinyDB to have a switch to use ujson if its present. At the moment if ujson is seen, its used. Anyway, just more food for thought.