Forum Archive

I haven't seen this error before, until I loaded up my script today:

you seem to be missing the utf-8 encoding line. likely you have some unicode chars in the script?

@JonB Weird.. never happened before. Even when I added that line though - I got a invalid syntax, even when commenting out.

You sure you don't have a LATIN CAPITAL LETTER A WITH CIRCUMFLEX somewhere in your file? :P

Could you upload the file as a Gist?

@techteej - I have had a long history of trouble with character encoding problems in text files because I download lots of stuff or copy paste stuff from various sources. I wrote a script to remove bad characters that you can use to see if this is your problem. It is designed to run in stash.

WARNING: This cleans the file down to pure ascii encoding - not UTF-8

# coding: utf-8
"""Sanitize one or more files to pure ascii and MAC EOL chars"""

from __future__ import print_function

import argparse
import fileinput
import os
import sys
import codecs

def sanitize_file(fn_in, fn_out):
    with codecs.open(fn_in, 'rt', encoding='ascii', errors='ignore') as in_file:
        text = '\n'.join(line.rstrip() for line in in_file.readlines()) + '\n'
    with open(fn_out, 'wt') as out_file:
        out_file.writelines(text)

def main(args):
    ap = argparse.ArgumentParser()
    ap.add_argument('files', nargs='*', help='files to be searched')
    ns = ap.parse_args(args)

    # Do not try to sanitize directories
    files = [f for f in ns.files if not os.path.isdir(f)]

    fileinput.close()  # in case it is not closed
    try:
        for line in fileinput.input(files):
            line=line.rstrip()
            if os.path.exists(line):
                if fileinput.isstdin():
                    fmt = '{lineno}: {line}'
                else:
                    fmt = '{filename}: {lineno}: {line}'
                print(fmt.format(filename=fileinput.filename(),
                                 lineno=fileinput.filelineno(),
                                 line=line))
                sanitize_file(line, line)
    except Exception as err:
        print("sanitize: {}: {!s}".format(type(err).__name__, err), file=sys.stderr)
    finally:
        fileinput.close()

if __name__ == "__main__":
   main(sys.argv[1:])

I get a similar error trying to parse mobypos.txt in pythonista, even if it works perfectly on my MacBook with python2.7, and I'm not using any libraries. I just gave up on that one eventually.

@dgelessus Here's a link to the script.

Your code has a bad case of NO-BREAK SPACE.

For some reason, every space character in the file is a non-breaking space. Python 2 doesn't recognize those as spaces (and neither might Python 3). The easiest fix is to view the file on GitHub, click the "Raw" link and manually copy-paste the entire text into Pythonista. When I get the script that way I don't have any issues running it.

@dgelessus Thanks!