Forum Archive

The update comes with many new features mostly thanks to the awesome objc_util module. It is available on the GitHub repo as branch beta16.

To install it, copy and run following line in the Pythonista prompt.

import urllib2; exec urllib2.urlopen('http://bit.ly/gsb16').read()

It installs StaSh under site-packages and copy the launch_stash.py script under ~/Documents for easier access. If it is not obvious enough, you'll need Pythonista 1.6 beta for it to work. Try a few commands like ll and grep and see the colors and styles. (please note not all commands are updated to use attributed texts).

Features and Changes:
* Attributed texts
- Colors, bold, italic, underline, strikethrough. Default font size is now 14 on iPad (was 12).
* Real external keyboard support (no more approximation)
- Ctrl-C interrupts running command. Ctrl-D ends user input. Ctrl-P/N navigates through command history. Ctrl-A/E jump to start/end of the input line. Ctrl-U erases the input line. Ctrl-W deletes one word from the input line.
* Killable threads
- Running scripts can now be properly terminated by the C-C button (Ctrl-C) which raises KeyboardInterrupt.
* Better modularity
- Code is now splitted into 7 files. The total length is over 4000 lines.A single file really gets difficult to handle.
* It is now a module
- The application is now made as a module to be installed under site-packages to survive through the global variable clearing.
* Better scrolling
* Largely redesigned the logic of user/script IO handling.
- The on screen and in memory buffers should now be more consistent.

There are also drawbacks. The most noticeable one is probably the speed. On iPad Air 2, it runs OK but cat through a large file still takes quite a few seconds. I am not sure how it would work on older devices.

This version is still at its early stage and you may expericence crash and freeze. There is still much room for improvements. I have a few already in my mind and will implement them when time permits.

All feedbacks are welcome!

I just hope the objc module survives app store review!

@JonB said:

I just hope the objc module survives app store review!

Yep fingers crossed!

I'd add the built in extended keyboard

i don't think we have an easy way of forcing the built in kb to show, or target the textview. well, we might be able to find it, traversing through the UIApplication.... hmmmm

I found OMKeyboardAccessoryView ...

Tried to implement it but the result for me is the view without buttons

import ui
from objc_util import *
v = ui.load_view()
editor_view=ObjCInstance( v['textview1'])
OMKeyboardAccessoryView = ObjCClass('OMKeyboardAccessoryView')
accessory_view = OMKeyboardAccessoryView.alloc().initWithType_dark_('python', False).autorelease()
editor_view.setInputAccessoryView_(accessory_view)
v.present('sheet')

The pyui file is just a view with a textview inside it

@filippocld I did think about using the builtin accessory keyboard and studied the example that @omz provided. However it turned out to be rather inadequate for following reasons:

• Unlike a text editor, a Terminal does not allow users to freely type any texts in any locations. It has strict control on where you can type texts. For the accessory keyboard, this means I need to register my own callbacks for the button actions and swipes. This may be possible but may not be trivial, especially when there is no documentation about its internals.

• The builtin accessory keyboard does not provide Control keys (e.g. Ctrl-C, Ctrl-D) or any utility buttons like history navigation.

• Some of the utility buttons should be context aware. For an example, behaviour of the Tab button depends on whether a script is running. This again requires a custom callback for the buttons.

With above being said, the extra keyboard really could use some improvements. A possible way is to create my own inputAccessoryView for better look and encapsulate the functions.

Awesome! The site-packages installation will really make integration a lot easier for scripts like the Pythonista Tools client that rely on StaSh. The previous integration method was pretty hack-ish. Nice work!

I am sorry to report that external bluetooth keyboard support is no longer working as it did in the prior version. History via Cmd up and down arrows is no longer functioning for me now. Output scrolling is kinda jerky and looks like some kind of buffering is going on that causes it to stop and start.

I have not seen any other issues and it looks to be functioning well running from site_packages. I made sure to delete my old stash folder in Documents and remove and re-add stash_launcher to my tools menu.

@wradcliffe Thanks a lot for the feedbacks!
* The cmd + up/down history navigation is removed from the new version. I replaced them with Ctrl + p/n. But it does make sense to have arrow keys involved for browsing history. So I have just added support for using only the up/down arrow keys (no need to press Cmd, just the arrow key itself). I hope this would work better for you.

• As for the jerky scrolling, what device do you have? How frequent did it happen? Did it happen when you type ll or cat a large file or all the time? I had a few optimisations pushed since the original post. So it may also be worthwhile to try the latest version by type and run selfupdate.py beta16 from within the shell (you may need restart the whole Pythonista app afterwards).

Please let me know how it works for you. Thanks!

@Gerzer Glad to hear you like the changes! I wasn't sure about the site-packages change but now it seems to make more sense.

@ywangd - Thanks for adding the simpler up arrow for history. Makes a lot of sense and has the nice benefit of preventing the user from moving the cursor off the input line.

With further testing, I don't see any real problem with scroll speed and smoothness. Testing using cat on a large file works fine and the speed is dictated by screen rendering time as you would expect. The jerkiness I noticed was for file system traversal. Seems slower then what I remember but it is hard to imagine how stash could be effecting that.

As usual, your releases are superb. Thanks for all the hard work on this.

One more question. How can users contribute scripts for potential inclusion into stash? I have one that I use often that can "sanatize" one or more text files that may contain odd character encodings. It is used for cases where a text file won't open properly in the editor. Here is the script ...

# coding: utf-8
"""Sanitize one or more files to pure ascii and MAC EOL chars"""

from __future__ import print_function

import argparse
import fileinput
import os
import sys
import codecs

def sanitize_file(fn_in, fn_out):
    with codecs.open(fn_in, 'rt', encoding='ascii', errors='ignore') as in_file:
        text = '\n'.join(line.rstrip() for line in in_file.readlines()) + '\n'
    with open(fn_out, 'wt') as out_file:
        out_file.writelines(text)

def main(args):
    ap = argparse.ArgumentParser()
    ap.add_argument('files', nargs='*', help='files to be searched')
    ns = ap.parse_args(args)

    # Do not try to sanitize directories
    files = [f for f in ns.files if not os.path.isdir(f)]

    fileinput.close()  # in case it is not closed
    try:
        for line in fileinput.input(files):
            line=line.rstrip()
            if os.path.exists(line):
                if fileinput.isstdin():
                    fmt = '{lineno}: {line}'
                else:
                    fmt = '{filename}: {lineno}: {line}'
                print(fmt.format(filename=fileinput.filename(),
                                 lineno=fileinput.filelineno(),
                                 line=line))
                sanitize_file(line, line)
    except Exception as err:
        print("sanitize: {}: {!s}".format(type(err).__name__, err), file=sys.stderr)
    finally:
        fileinput.close()

if __name__ == "__main__":
   main(sys.argv[1:])

@wradcliffe You can make a pull request to the repo for your changes:

• Fork the repo first
  • navigate your browser to the repo and click Fork at the top-right corner.
• Make changes against the forked repo
  • You can make any changes since forked repo is owned by you
• Create a pull request back to the original repo once you are done with the changes
  • The pull request is created by clicking the green button on the forked repo's main page.
  • Please create the pull against the dev branch of the original repo. The dev branch will be merged into master when things are stable.

Thanks again for the feedback. This is what I could really use to make StaSh progress.

Ran launch_stash.py and it crashes (returns to home screen) after 7 seconds.

@ihf
I cannot reproduce it. What device do you have? Are you on the latest beta (160033)? Could you please try a fresh install:
* First delete any existing stash folders and launch scripts, including the new one and any old ones (v0.4.x).
* Run import urllib2; exec urllib2.urlopen('http://bit.ly/gsb16').read() to install
* Unload Pythonista app completely and relaunch to ensure a fresh state
* Make sure you are not loading any custom resource files, e.g. .stashrc.
* Now run ~/Documents/launch_stash.py

Please let me know if above steps work for you.

I will just report I have the latest version of stash, and the latest beta, and launch_stash.py works.
IPad Air 2 , 128gb , wifi + cell, iOS 8.4.1

@Phuket2 Good to know it works for you. Thanks a lot for the feedback.

@ywangd , no problems. However there is always a but :) on the previous version I tried to find where to change the font and size used. It was described clearly, but I could not find the file that was referred to. When you have time, could you please point me in the right direction to change the font and size. My eyes are bad, so I need a bigger size

@Phuket2
You can configure the text size, but the font can no longer be changed in the new version (at least not recommended).

• If you prefer to modify the source file, you can do it by change the value on Line 103 of stash.py. Default size is 14 for iPad and 12 for iPhone.
• Alternatively, I'd recommend creating a .stash_config file under the stash installation root (i.e. ~/Documents/site-packages/stash/). Any settings in this file override the same settings in the source file. For an example, to increase the text size:
  [display] TEXT_FONT_SIZE=20

The text font can no longer be configured because not every font has all of the bold, italic and bold-italic variants. If stash tries to display a non-existing font variant, it crashes the entire Pythonista app.

@ywangd, thanks I did it in your code. I know it seems simple, but I tried the config file and failed big time. But did you mean literally '.stash_config' or 'stash_config.txt' or ...... Sorry, I was not sure

@ywangd I followed your instructions and the problem persists. I am running v1.6 (160033) on an iPad 3.

@Phuket2 I mean literally .stash_config. It can be tricky to create this file just using Pythonista editor because it does not like files without extension. You can create as a regular python file, say stash_config.py in ~/Documents. Then from inside stash, rename it by running mv stash_config.py site-packages/stash/.stash_config.

@ihf I am not sure if it has anything to do with 32bit vs 64 bit devices. As I remember, iPad3 is 32 bit. Unfortunately I don't have a 32bit device around to do any tests. I'll try add some debug loggings so the launching process can be recorded into a file. I'll let you know once it is ready. Sorry for the inconvenience.

i have ipad 3, and have launched it no problem. I might be a few commits behind, will try updating later tonight.

I did have a crash once recently, while i had another view opened. also cancelling during the launch, then running again is likely to cause a crash.

@ywangd , I really can't work it out. I tried many things and get weird results. I have a feeling that the preceding dot is the problem.
Even i do this in the site-packages/stash/ dir

_settings =  '''[display] \n
TEXT_FONT_SIZE=10'''

with open('.stash_config', 'w') as f:
    f.write(_settings)

As far as I can see the file does not show up in the browser. If I remove the preceeding dot, the above works as expected. But I suspect it's there, but not visible. Look maybe not my place to say, but would it make more sense to create a file like stash.cfg on start up if it didn't exist and write your defaults to it.

@Phuket2 Yes files with names start with a dot are hidden files. It is an Unix convention. I agree it could be confusing for people not familiar with Linux/Unix. Also the Pythonista builtin browser and editor are not designed to take care of this type of files.

So I modified the code and now it also reads a stash.cfg file under the installation root (thanks for the idea). Could you please try again with this file? You can get the changes by running selfupdate.py beta16 from within the shell.

Apparently there are multiple ways you can create this file. As a demo, you can do this from within StaSh by running following command:

echo "[display]\nTEXT_FONT_SIZE=20" > $STASH_ROOT/stash.cfg

You need to restart stash afterwards for changes to take effect.

@JonB Thanks for the test. Please do let me know the results with the latest push. I just did one about half an hour ago.
If it works for your iPad3, I really have no idea what exactly could cause the crash for @ihf . Though it is most likely due to usage of the objc_util module. A single error on text range etc can crash the entire app. But exactly where I don't know ...

beta16 (installed via git clone) worked fine on IPad 3. I also tried via getstash.py.

@ywangd, thanks your mod works perfectly, as well as your selfupdate

@ihf
I have created a new branch beta16-log with detailed logging statements for debugging the launch process. Could you please give it a go?

• First delete any existing stash folders and launch scripts
• Run import urllib2; exec urllib2.urlopen('http://bit.ly/gsb16').read() in {'_br': 'beta16-log'} to get the new branch. please note the trailing dict for selecting the correct branch
• Unload Pythonista completely and restart
• Tap to run launch_stash.py found inside ~/Documents folder.
• If Pythonista crashes during the process, restart it and you should be able to find a log file as ~/Documents/stash.log.
• Please copy contents of the log file and create a new ticket at the GitHub repo (better not flood this thread with logging outputs)

Thanks!

@JonB @Phuket2
Great to know the latest changes work for you. Thanks!

@ywangd Done. Ticket #136. Thanks very much.

@ywangd , i know the git distrib is not yours , but would also be nice if they didn't use '.git' for a dir name.

@ihf I updated the ticket with steps for further debug. Thanks for taking time to work on this!

@Phuket2 said:

@ywangd , i know the git distrib is not yours , but would also be nice if they didn't use '.git' for a dir name.

@JonB is the author of the git command. But here are my takes on the .git folder:
* It is consistent with the original (PC) version git.

• Often a hidden file/folder indicates that it is not supposed to be directly edited by users. This is the case of .git. This folder stores the version control information and is the internal working directory of the command. If you look inside the folder, there is little information that is human readable.

• Many Unix/Linux tools use .hidden files/folders as resource files and working directories. It is a convention. For an example, both PC version pip and ipython create hidden folders (.pip and .ipython) under your home directory to store their working states.

• With StaSh, you can list hidden files by ls -a and you use cd to get inside a hidden folder, i.e. cd .git.

@ywangdk thanks for nice complete definition. Personally, I think the days of hidden dirs and files should be over. Maybe I am wrong, but it just seems to me in this day and age , not required. I would think if you have files that should not be edited, better just to set the read only flag. If people go out of their way to change that flag, well...better still at the OS level, files could have a imbedded password to change the flags. That's why I still love the old idea of the old mac files. Resource fork and data fork. I know this was a trick by the OS. But in my idea, every file should have a header. .txt or whatever. If every OS supported it, would not be a problem.
Yes, I am dreaming :)

@Phuket2 Keep in mind that "hidden" files are much less hidden in shell environments than on a desktop system like Windows or OSX. In the shell there are no hidden files as such, it's just that ls by default doesn't show files starting with a dot, which are by convention "hidden". Desktop systems like Windows or OSX have a different concept of "hidden", namely an extra file flag that has nothing to do with Unix dot files. To see that kind of hidden files on a desktop system, you often need to change some hard to find setting. Which is probably a good thing, considering how many users (want to) know nothing about how their OS works internally and would be worried about all those desktop.ini files that they never created.

ls .* will show you hidden files and directories on the commandline.

@dgelessus and @ccc, I really do get your points. But we are in 2015 now. I know in some way it's very easy to have hidden files. I just think the days of hiding things should be over. But I also think every file known to man should have a Json style header which every OS adheres to. I know we need another committee to make that happen. But pure text files etc... Are just an idea in the past based on transmission speeds , storage space etc. is a old idea.
Every single file could have a header (binary or text) with a set of attributes. So whatever the committee come up with , they will be there regardless. Then of course you should be able to extend it for app specific use. I am not really smart, but I could see how this could help revolutionise mobile/desktop computing.

Think of the period like a leading _ in python, which is a way of signalling something is intended to be private. like in python, if you dare you can still use or modify it, but by default it won't be shown by some dir type commands. But it is very easy to see (i believe in stash we have the alias la defined which lists all files, basically ls -la), and nothing prevents you from modifying anything in the folder, though you are likely to completely break your repo unless you know what you are doing. Like python, these are often poorly documented or likely to change... you can use the files, but dont't depend on them.

i don't quite get what you mean about a json file for every file in the system. what would go in such a header? for instance, for git, a file in /refs/heads/branches is the name of the branch, and the contents is a 20 character sha1 pointing to a commit in the object store, which makes branch creation basicslly free. what would the header say for such a file, and how would that help you muck about in the .git folder? How many bytes are you going to spend documenting the format of a 20 byte file? While HD space is cheap, device space is not....

@JonB , maybe I am out of step. But I don't think I am. Unix, and all modern operating systems are living in the past (well, just what they know). No new ideas are coming as far as file wide Standardisation.
You talk about a 20 byte header. The header from what ever crazy ISO committee could be implied by the OS. Then extendend , for example if the header decided is a JSON header. Not saying it should be JSON, just one idea to illustrate a point. Then the OS could imply the header, you could then extended it. In compatibility mode, the OS could strip off the header, and pass the text file, PIL or whatever format im
I know it seems out there, but it's not really.

@Phuket2 You mean something like SGML DTDs or XML schemas, but for any kind of file? That might be possible, but such a StandardBinarySchema should not go at the beginning of every file. If anything, the file header should contain some kind of unique reference that points to a StandardBinarySchema. That would mean much less duplication, but you'd still have a lot of extra data when you have e. g. many small files.

Then there's also the problem of how such a StandardBinarySchema would look. Should it be a BNF variant for binary data? (That would make certain file structures difficult to describe.) Should it be regex? (Now you have two problems.) Should it be a Turing-complete language? (Now Windows has another file type to "protect" the user from.) How would you handle nested structures (a PDF file containing a PNG image)? Where would the OS get schemas that it doesn't have locally? How would updates or different versions of a schema be managed without conflicts and breaking old applications?

I don't think the current situation is so bad that you'd need this kind of "file content description". Many formats already have some kind of relatively unique header, for example UTF-16 text files start with the byte order mark "\xFE\xFF" (which may be reversed if the file is stored in this crazy little-endian byte order), ZIP archives start with "PK\x03\x04", MS-DOS/Windows exe files start with "MZ". If you want to know the format of a file, use the awesome file utility that is available on almost any Unix-like system. If you want to know how a certain file format is structured, ask your favorite search engine.

@dgelessus , thanks for the complete reply. Yes, depending on the block size a one byte file will still consume the size of the formatted block size as far as I know it.
But I don't think a OS/device wide std file header (whatever format) would be prohibitive, size wise. But let's say a JSON style header. Also variable length, so after the standard stuff the developer could insert his/hers own nodes. Maybe there are some issues such as byte order etc, but this would be handled by the OS.
Look, I understand this would not be easy, but these committees are setup to handle the difficult questions as well as our futures.
Look maybe I am dreaming. But the concept at least sounds easy and clear enough to me.

I personally think if you ask any developer, if you could attach a JSON style header to any file you create, eg PNG, txt, TIFF, CSV etc... And not have that header interfere with another program that expects a file type of a certain kind, I think it would be a resounding HELL YES! Yes, also security issues. Would also be easy to rewrite the header. But again, this would be for the big brains in the ISO committee to work out.

I still think there was a big improvement with the release of xml. But it solved only a few problems.

Sorry, I know this is not the place to discuss these issues. But my frustration takes over sometimes :) I have always been the can do type of person. My idea is that if we can put people on the moon, then there is not much we can't achieve here only lonely planet Earth.

I am still not sure you have explained what you think should go IN the header?

Keep in mind, as well, that as a developer you would need to be responsible for creating such a header (how is the os supposed to know what the file you created is for?). For instance, maybe you want to dump out a log file from your virtual cell class .... NOT SO FAST ! first you need to spend time documenting the format of the log file. What if you change your mind and realize you need to add a time field.... not so fast, better update that json first. If such a header is somehow mandatory, it means you cannot touch a file unless you modify the header. If it is not somehow mandatory, or if "unstructured data" is an option, I think you will find most developers opt for that one.

If you are not talking about some sort of full schema definition, but just some kind of "comment" left by the defeloper, many file systems already provide such a mechanism. NTFS offers an "Alternative Data Stream", which a developer can append whatever info he/she wants. The file system stores a reference to the alternate stream. Funny thing about it, though, is that Windows does not make it easy to find or access, partially because the intent is to store info intended to be inaccessible (for instance, on a windows pc it knows that you have downloaded a file via the internet... that info is stored in an ADS record for each file). Of course the problem is that this exists only on windows, and so emailing a file, zipping it up, copying it to a FAT32 thumbdrive, etc loses that record.

Also... this discussion started with frustration about hidden files. I guess I am still unclear how a json header would help protect the user from files that the developer has suggested they neednt mess with.

@JonB, your right. I didn't describe what should go in the header. That would be left to a committee to work out.
All I am trying to say is while all files need their varing formats internally, I would think to have a enclosing format around the file that could be read in a consistent way by any application on any platform without knowing the internals of the file would be a great thing. Yes, I might be as simple as a web address, authors name, creation location (gps data). Yeah, I can see why someone would say, why would you want gps data in a text file. It's the old saying you build it they will come. Meaning people will invent ways to use this extra information in new and exciting ways.
But again, I am not thinking so much about the data that would be stored (I am confident that a committee could come up with some smart relevant information).
In my simplistic idea, this type of idea could help harmonise/update OS file deficiencies/differences.
@JonB, look I know I am really simplifying it and there are would be Technical difficulties to solve. But it has to getting than it is today. Regardless if it is good or bad. File systems have to evolve. Also I know they have. Everyone's off in there own world doing there own thing. That's my impression anyway.
I am guessing the poor lonely text file's upgrade has only been double byte/Unicode. I remember .rtf files, PICT files etc. This is good, but file specific, I am looking for a global upgrade.
I really wish I could express myself better on this topic in writing. I am sure I would have no problems articulating my views verbally.
So, I think I end my message with, We can wait for operating systems to get with the program and supply a wealth of information to files globally, or a new format describing a heading for the want of a better word that wraps around all files.
Lol, thinking if I should post or not.... Yeah I will. I hate to appear stupid....but it's clear in my head. Maybe I should make a video instead. But debating verbally is not my weakest point :)

@JonB, not exactly what we were talking about, but same idea would apply.
JPeg lockdown: Restriction options sought by committee
http://www.bbc.co.uk/news/technology-34538705
With a well thought out universal file header, all sorts of benefits could be gained and implemented reasonably easily. But now a the JPEG committee will spend a lot of time trying to solve a problem for only one file format. I think the issue they are trying to take here and other issues should be addressed across the whole spectrum of files.
I know is an old discussion we had, but seen this article today and instantly reminded me of our discussion here