Forum Archive

Is there a way to construct a webview in reader mode when it is available?

@ihf, according to this stackoverflow answer, neither UIWebView or WKWebView have such a mode built in. SKSafariViewController has it, naturally, but I do not know if it is usable in Pythonista.

As a quick fix, they suggest using a web service to scrape the readable content for you. I.e. load this url to the webview: http://www.readability.com/m?url=YOUR_URL_HERE

I guess you could also load the page first with e.g. requests, then get just the text from the page with BeautifulSoup, and present that.

@mikael Thanks! Unfortunately, Readability is no longer available and the browser's Reader view/mode produces much prettier output than what I am likely to get by merely parsing text (no images, etc.)

@ihf, there’s a Python port of the Ruby port of Readability’s original JS version here. ... But it also seems to have a problem with images, based on this issue.

Apparently there are some ways to get Apple's version of readability on the desktop:

https://github.com/amumu/safari-reader-js

Should be possible, I think, to execute this in a webview, or @mikael's JavaScript framework, though I have not tried.

My goal is to permit the RSS Reader script to load articles in Reader mode when possible.

Have you tried executing the JavaScript in a webview?

@jonb I tried this:

(reader.js is a string with the js code)

wv = ui.Webview()
wv.load_url(‘https://www.nytimes.com/2019/01/29/climate/global-warming-extreme-weather.html’)
wv.eval_js(reader.js)
wv.present()

Obviously, missing something...

@ihf I think that between load_url and eval.... you need to wait the page is loaded

web.load_url(....)
    # wait for documentState to start loading, 
    for i in range(10):
        if web.eval_js('document.readyState')!='complete':
            break
        time.sleep(1)

    # ...then wait for it to complete
    while   web.eval_js('document.readyState')!='complete':
        time.sleep(1)

My web view is a subview of an ui.View already presented

@ihf or use webview.delegate (see help) to check if url did finish loading

@ihf Try

import ui

class MyWebView(ui.View):
    def __init__(self,url):
        self.width, self.height = ui.get_screen_size()
        self.web = ui.WebView(frame=self.bounds)
        self.web.scales_page_to_fit = True
        self.web.delegate = self
        self.web.load_url(url)
        self.add_subview(self.web)

    def webview_should_start_load(self,webview, url, nav_type):
        print('should start:{}'.format(url))
        return True

    def webview_did_start_load(self,webview):
        print('did start')
        return True

    def webview_did_finish_load(self, webview):
        print('did finish')
        pass

    def webview_did_fail_load(self, webview, error_code, error_msg):
        print('did fail {} {}'.format(error_code,error_msg))
        pass


MyWeb = MyWebView('https://www.nytimes.com/2019/01/29/climate/global-warming-extreme-weather.html')
MyWeb.present('full_screen')

@cvp I understand that I must wait for the page to load but my testing has been in the console so I would have thought that by the time I type the eval_js, the page has loaded. No?

@ihf obviously.
Did you see that your site loads several pages

Yes, indeed. I just want to see if this js code will, in fact, implement a reading mode. What I type in the console is:

import ui
with open ("safari-reader.js", "r") as myfile:
data=myfile.readlines()
data=str(data)
wv = ui.WebView()
wv.load_url('https://www.nytimes.com/2019/01/29/climate/global-warming-extreme-weather.html')
wv.eval_js(data)
wv.present()

I wait a bit after the load_urI and end up with the unmodified webpage.

@ihf Not sure you can eval_js before webview is presented

Could you paste here your .js file if it is text

It is too long to post but I got it from @JonB post above: https://github.com/amumu/safari-reader-js

@ihf ok, sorry, too complex for me. I don't know almost anything about js

@ihf Try this, it will be ok for you, I hope

import ui
from objc_util import *

def safariViewControllerDidFinish_(_self, _cmd, _controller):
    #print('SafariViewControllerDidFinish_')
    SFSafariViewController = ObjCInstance(_controller)  
    SFSafariViewController.uiview.close()

methods = [safariViewControllerDidFinish_,]
protocols = ['SFSafariViewControllerDelegate']
try:
        MySFSafariViewControllerDelegate = ObjCClass('MySFSafariViewControllerDelegate')
except:
    MySFSafariViewControllerDelegate = create_objc_class('MySFSafariViewControllerDelegate', methods=methods, protocols=protocols)

#@on_main_thread    
def MySFSafariViewController(url, w, h, mode='sheet', popover_location=None):
    uiview = ui.View()
    uiview.frame = (0,0,w,h)
    uiview.background_color = 'white'
    if mode == 'sheet':
        uiview.present('sheet',hide_title_bar=True)
    elif mode == 'popover':
        if popover_location:
            uiview.present('popover', hide_title_bar=True, popover_location=popover_location)
        else:
            return
    else:
        return

    SFSafariViewController = ObjCClass('SFSafariViewController').alloc().initWithURL_entersReaderIfAvailable_(url,True)


    # Use new delegate class:
    delegate = MySFSafariViewControllerDelegate.alloc().init()
    SFSafariViewController.delegate = delegate      
    SFSafariViewController.setModalPresentationStyle_(3)
    SFSafariViewController.uiview   = uiview        # used by delegate

    objc_uiview = ObjCInstance(uiview)
    SUIViewController = ObjCClass('SUIViewController')
    vc = SUIViewController.viewControllerForView_(objc_uiview)  

    vc.presentViewController_animated_completion_(SFSafariViewController, True, None)


def main():
    # demo code
    mv = ui.View()
    mv.background_color = 'white'
    mv.name = 'Test SFSafariViewController'
    mv.present()
    url = nsurl('https://www.nytimes.com/2019/01/29/climate/global-warming-extreme-weather.html')
    #MySFSafariViewController(url,600,500)
    MySFSafariViewController(url,600,500, mode ='popover', popover_location=(mv.width-40,60))

if __name__ == '__main__':
    main()

My code seems to have a problem with sheet mode, test it with popover mode.
I'm searching why

Edit: remove the @on_main_thread line.... ==> ok

I took a stab at the js code. With the code below I get the article part of the original page, but all attempts to get the cleaned version end in a non-helpful js error, which I do not have time to track right now.

import wkwebview
import ui

with open('safari-reader.js','r',encoding='utf-8') as f:
  safari_script = f.read()

class Delegate:

  @ui.in_background
  def webview_did_finish_load(self, webview):
    w.eval_js('r = new ReaderArticleFinder(document); document.body.innerHTML = r.articleNode().innerHTML;')

w = wkwebview.WKWebView(delegate=Delegate())

w.present()
w.add_script(safari_script)
w.load_url('http://blog.manbolo.com/2012/11/20/using-xcode-opengl-es-frame-capture')

@mikael I tried your script with the NYTimes url above and got these errors:

WARN: et2 snippet should only load once per page
ERROR: Pinterest Tag Error: 'load' command was called multiple times. Previously for tag id '%s', now for tag id '%s'.
ERROR: TypeError: null is not an object (evaluating 'r.articleNode().innerHTML') (https://www.nytimes.com/2019/01/29/climate/global-warming-extreme-weather.html, line: 1, column: 81)
ERROR: Script error. (, line: 0, column: 0)

Is this what you are talking about?

@ihf Just to know what you think, did you try mine?

@cvp Yes, just now and it also fails on the Nytimes articles. Is it working for you? If so, I will send the error messages.

@ihf

Yes, but look at where the article ends.

@ihf I don't have any error.

I can’t reproduce the error (I may have run it under python 2.7). I can get what you are seeing but the article is truncated.

@ihf When I try in Safari, I get the same result, except some lines at end saying there are 4 articles more

@ihf As I use the standard Apple Safari view, I suppose we can't get better, no?

@cvp I tried on some Macstories and it worked perfectly. What I was seeing may be peculiar to the NYtimes as I cannot login so there are limits to what it will show.

Thanks so much for working on this. Now I will try to incorporate this into rss_reader.py

@ihf 👍 good luck

@ihf If you go back in normal mode, at bottom, you get

@ihf Last post for today: perhaps you will need to use other delegates, see here

Safari view is new to me and very much fun, but if the original intent was to get some RSS items cleaned up, I do not see anything in the Safari view API to get the cleaned HTML out of the view. Can it take a file: URL so that we can use it just as a viewer? Or would it be able to handle just the online RSS feed address?

From what I read, it ONLY supports http and https urls -- no file://, data://, or other loading from stored HTML. Furthermore, you basically have no access to anything happening inside it, for security. There seem to be delegate methods for things like redirect, and Done/Action buttons.

I still think that readability.J's combined with wkwebview might be the way to handle custom rss reader

You are right, Apple doc says

Choosing the Best Web Viewing Class
If your app lets users view websites from anywhere on the Internet, 
use the SFSafariViewController class. 
If your app customizes, interacts with, or controls the display of web content, 
use the WKWebView class.

Using Flask, we could access local html Files...

import ui
from objc_util import *

from flask import Flask
import threading
import os

app = Flask(__name__)

@app.route('/')
def entry_point():
    global local_html
    fil = os.path.expanduser('~/Documents/'+local_html)
    data = open(fil).read()
    return data 

def safariViewControllerDidFinish_(_self, _cmd, _controller):
    #print('SafariViewControllerDidFinish_')
    SFSafariViewController = ObjCInstance(_controller)
    #print(SFSafariViewController.uiview)
    try:
        SFSafariViewController.uiview.close()
    except Exception as e:
        SFSafariViewController.uiview().close()

methods = [safariViewControllerDidFinish_,]
protocols = ['SFSafariViewControllerDelegate']
try:
        MySFSafariViewControllerDelegate = ObjCClass('MySFSafariViewControllerDelegate')
except:
    MySFSafariViewControllerDelegate = create_objc_class('MySFSafariViewControllerDelegate', methods=methods, protocols=protocols)

#@on_main_thread    
def MySFSafariViewController(url, w, h, mode='sheet', popover_location=None):
    uiview = ui.View()
    uiview.frame = (0,0,w,h)
    uiview.background_color = 'white'
    if mode == 'sheet':
        uiview.present('sheet',hide_title_bar=True)
    elif mode == 'popover':
        if popover_location:
            uiview.present('popover', hide_title_bar=True, popover_location=popover_location)
        else:
            return
    else:
        return

    SFSafariViewController = ObjCClass('SFSafariViewController').alloc().initWithURL_entersReaderIfAvailable_(url,True)

    # Use new delegate class:
    delegate = MySFSafariViewControllerDelegate.alloc().init()
    SFSafariViewController.delegate = delegate      
    SFSafariViewController.setModalPresentationStyle_(3)
    SFSafariViewController.uiview   = uiview        # used by delegate

    objc_uiview = ObjCInstance(uiview)
    SUIViewController = ObjCClass('SUIViewController')
    vc = SUIViewController.viewControllerForView_(objc_uiview)  

    vc.presentViewController_animated_completion_(SFSafariViewController, True, None)

def main(url,local=None):
    global local_html
    # demo code
    mv = ui.View()
    mv.background_color = 'white'
    mv.name = 'Test SFSafariViewController'
    mv.present()
    ns_url = nsurl(url)
    if 'http://127.0.0.1:5000/' in url:
        # local html
        local_html = local
        threads = threading.enumerate()
        already = False
        for thread in threads:
            if thread.name  == 'flask':
                already = True
                break
        if not already:
            # run Flask in separate thread
            threading.Thread(target=app.run,name='flask').start()
    MySFSafariViewController(ns_url,600,500)
    #MySFSafariViewController(url,600,500, mode ='popover', popover_location=(mv.width-40,60))

if __name__ == '__main__':  
    #main('https://www.macstories.net/stories/my-must-have-ios-apps-2018-edition/2/')
    main('http://127.0.0.1:5000/'   ,local='MesTests/test.html')

Maybe I best go back and explain what exactly I was trying to achieve. I have been using this RSS reader script for years ([https://github.com/dlo/PythonistaRSSReader]. It takes RSS feed urls and displays the latest titles for each feed. The titles can be touched to take you to the actual posting. I would like it to utilize the Reader mode where it is available. If there is a better RSS reader script available (or an enhanced version of this one, that would be nice to know as well).

@ihf, if I understand correctly, you are not downloading the articles themselves, thus just opening them in the Safari view would be an working solution. Looking at the docs, the method for opening directly into Reader mode has been deprecated, and the option is now a bit hidden in the configuration object, but still there.

you could modify rss.py, using cvp's solution

webview = ui.WebView()
        webview.name = entry['title']
        webview.load_url(entry['link'])

        tableview.navigation_view.push_view(webview)

instead, you would use cvp's conttroller, but need to modify it to return uiview instead of presenting it, then you would replace webview with that uiview.

@mikael I had seen this depreciation and I had searched the new way but not found easily then, just to let it working, I used the depreciated way. But, you're right, I should modernize it.

@ihf as advised by @mikael , replace depreciated

    #SFSafariViewController = ObjCClass('SFSafariViewController').alloc().initWithURL_entersReaderIfAvailable_(url,True)

by

    SFSafariViewControllerConfiguration = ObjCClass('SFSafariViewControllerConfiguration').alloc().init()
    SFSafariViewControllerConfiguration.setEntersReaderIfAvailable_(True)
    SFSafariViewController = ObjCClass('SFSafariViewController').alloc().initWithURL_configuration_(url,SFSafariViewControllerConfiguration)

I have packaged cvp's solution into a module, which works like a charm, thanks a lot cvp ! (I use it for a newsreader).

I have come across two limitations, though:
- reader_view() cannot be used if 'uiview' has a NavigationView, or if any view presented below it has a NavigationView (even if the view has no relationship with 'uiview' - very strange)
- No resizing is performed when the reader view changes orientation

I am not very proficient with objc, would anyone know how to improve this ?

""" Modal view which displays a web page in reader mode.

Based on cvp's code:
https://forum.omz-software.com/topic/5392/webview-and-reader-mode/18

When run as a script, the module shows a small demo application.

Implementation note:

    Transmitting the callback function from reader_view() to safariView-
    ControllerDidFinish_() turns out to be trickier than expected, especially
    when combining reader_view.py with app_single_launch.py.

    In this case, when the user launches the same app a second time, a new
    'safari_view_controler' instance seems to be created automagically. We
    therefore cannot store the callback function as one of the 'safari_view_
    controler's properties, since the instance created by reader_view() is
    silently replaced by a new instance when the app is relaunched, and
    ControllerDidFinish_() will receive the new instance, not the old one.

    In the end, the simplest solution works, i.e. storing the callback in a
    global. This does of course come with theoretical limitations: an applica-
    tion should not make a second call to reader_view() until the reader view
    displayed by the first call has been closed. This should not be too much of
    an issue, as I really cannot imagine a practical use case for doing this. I
    have included a simple protection against this edge case anyway.

Revision history:
 6-Fev-2019 TPO - Initial release
28-Fev-2019 TPO - Added the 'enter_reader_mode' argument to reader_view()
 7-Mar-2019 TPO - reader_view() now works when used with app_single_launch.py """


from typing import Callable, Optional

from objc_util import create_objc_class, nsurl, ObjCClass, ObjCInstance
import ui


__all__ = [
    'reader_view',
]


Callback = Callable[[], None]
__CALLBACK: Optional[Callback]
__READER_VIEW_DISPLAYED = False


def safariViewControllerDidFinish_(_self, _cmd, _controller):
    global __CALLBACK, __READER_VIEW_DISPLAYED
    if __CALLBACK:
        __CALLBACK()
    __READER_VIEW_DISPLAYED = False


def reader_view(url: str,
                uiview: ui.View,
                callback: Optional[Callback] = None,
                enter_reader_mode: bool = True) -> None:
    """ Modal view which displays a web page in reader mode.

    Arguments:
    - url: of the web page to be displayed. The only URL schemes allowed are
      http and https.  Any other URL scheme will raise an Objective-C exception
      and cause Pythonista to be terminated.
    - uiview: ui.View instance on which the web view is to be displayed. Note
      that reader_view() does not work if this is an instance of ui.Navigation-
      View.
    - callback: function of no argument, which is called when the user closes
      the view.
    - enter_reader_mode: if true and if reader mode is available, it will be
      entered automatically; if false, the user will have to tap the reader
      mode icon to enter reader mode (when available). """
    global __CALLBACK, __READER_VIEW_DISPLAYED
    if __READER_VIEW_DISPLAYED:
        raise ValueError("reader_view() already in use")
    __READER_VIEW_DISPLAYED = True
    __CALLBACK = callback

    MySFSafariViewControllerDelegate = create_objc_class(
        'MySFSafariViewControllerDelegate',
        methods=[safariViewControllerDidFinish_],
        protocols=['SFSafariViewControllerDelegate'])
    delegate = MySFSafariViewControllerDelegate.alloc().init()
    safari_view_controler_conf = (
        ObjCClass('SFSafariViewControllerConfiguration').alloc().init())
    safari_view_controler_conf.setEntersReaderIfAvailable_(enter_reader_mode)
    safari_view_controler = ObjCClass('SFSafariViewController').alloc() \
        .initWithURL_configuration_(nsurl(url), safari_view_controler_conf)
    safari_view_controler.delegate = delegate
    safari_view_controler.setModalPresentationStyle_(3)  # 3 = currentContext
    vc = ObjCClass('SUIViewController').viewControllerForView_(ObjCInstance(uiview))
    vc.presentViewController_animated_completion_(safari_view_controler, True, None)


if __name__ == '__main__':
    from datetime import datetime

    def done():
        print("After user has closed reader view", datetime.now())

    def test(sender):
        print("Before call to reader_view()", datetime.now())
        reader_view(url=url, uiview=view, callback=done)

    url = 'http://www.nytimes.com/2019/01/29/climate/global-warming-extreme-weather.html'
    view = ui.View(
        flex='WH',
        background_color='white',
        name='reader_view demo',
        right_button_items=[ui.ButtonItem(title='test', action=test)])
    view.present()

@TPO, this I can help you with:

safari_view_controler.view().autoresizingMask = 2 + 16 # WH

In addition to adopting cvp's SFSafariView solution, I have been experimenting with JonB's initial suggestion of using Safari's JS code for the reader mode.

The reason for this is that SFSafariView is a black box which does not provide a way of getting / setting the position in the page. Being able to do this would allow a newsreader app to remember where the user stopped reading an article, and later re-open the article at this very position. This would be convenient for long articles which cannot be read in one sitting.

If we had a way a generating a Safari-reader-mode-like filtered version of the web page, we could display it in a WebView and then get / set the position (as demonstrated by MarkdownView, https://github.com/mikaelho/pythonista-markdownview) to achieve this result.

I have digged a bit into this, have found a more recent and more complete version of Safari's reader mode source code (both JS and CSS), and have had some success with it:
- The web page is cleaned of most useless content
- It is displayed in the mode selected (for example : sepia), with the font selected

However:
- Not all useless content is removed, and the result is not as good as what Safari produces
- Margins and spacing between paragraphs are not handled properly

You will find below my test code and pointers to the Safari sources, in case anyone is interested in investigating this further.

Download the following files :
* safari.js : https://github.com/liruqi/Safari/blob/master/safari.js
* article-content.css : https://github.com/andjosh/safari-reader-css/blob/master/article-content.css
* reader-ui.css : https://github.com/andjosh/safari-reader-css/blob/master/reader-ui.css
* theming.css : https://github.com/andjosh/safari-reader-css/blob/master/theming.css

Then run the following in the same directory :

import wkwebview
import ui

with open('safari.js', 'r', encoding='utf-8') as f:
    safari = f.read()

with open('reader-ui.css', 'r', encoding='utf-8') as f:
    reader_ui_css = f.read()

with open('article-content.css', 'r', encoding='utf-8') as f:
    article_content_css = f.read()

with open('theming.css', 'r', encoding='utf-8') as f:
    theming_css = f.read()

js_script = f'''
    r = new ReaderArticleFinder(document);
    document.body.innerHTML = r.articleNode().innerHTML;
    document.body.classList="system sepia";
    var style = document.createElement('style');
    style.innerHTML = `{reader_ui_css}\n{article_content_css}\n{theming_css}`;
    var ref = document.querySelector('script');
    ref.parentNode.insertBefore(style, ref);
    document.getElementById("dynamic-article-content").sheet.insertRule("#article {{ font-size:15 px; line-height:25px; }}"); '''


class Delegate:

    @ui.in_background
    def webview_did_finish_load(self, webview):
        w.eval_js(js_script)


w = wkwebview.WKWebView(delegate=Delegate())
w.present()
w.add_script(safari)
w.load_url('https://www.technologyreview.com/s/612929/wristwatch-heart-monitors-might-save-your-lifeand-change-medicine-too/')

@mikael : works like a charm, thanks a lot !

@TPO, looks like there is at least a contentScrollView attribute. Maybe you could dig deeper and get hold of the scroll position.

@mikael doc says

The user's activity and interaction with SFSafariViewController are not visible to your app, which cannot access AutoFill data, browsing history, or website data

Thus, I'm not sure we could...

@mikael you are quite right, dir() shows that safari_view_controler has a 'contentScrollView ' attribute.

However, safari_view_controler.contentScrollView() returns None, both in reader_view() and in safariViewControllerDidFinish_() :-(

I have gone through the official Apple doc on SFSafariViewController, 'contentScrollView ' is not mentionned anywhere.

@TPO said:

presentViewController

I wonder if addChildViewController (along with uiview.objc_instance.addSubview_(vc.view())) might be more appropriate than presentViewController here, if you are trying to essentially add a subview instead of creating a new fullscreen view.