Forum Archive

Copy code from forum

Phuket2

This code was for the old forum. It still appears to work. One difference being that it appears to be a paged system here, so for long posts it will not get all the code. I will think about that later. Still wanted to post it here for any feedback/suggestions anyone may have.

A few warnings up the top of the file worth reading....

# coding: utf-8

# WARNING
# If you click write to file, a directory called
# _temp_dir_name will be created in your root dir
# i think the normal temp/tmp dirs are invisible
# in pythonista. i refer to as a temp dir as in 
# my case, i only download code from the forum
# to test it. i want it all to go to one dir. 
# easier to clean up 

# WARNING
# Something wrong with the clipboard code. I have
# hidden the button with a constant below until i
# can workout why i am corrupting the clipboard.
# at least least, i think thats what is happening 
_SHOW_CLIPBOARD_BTN = False 

# if this is set to true, the file created will
# be opened in the editor. According to the docs
# the current file is automatically saved
_OPEN_FILE_AFTER_WRITE = True  

import ui
import clipboard, console, editor
import os, time
import tempfile
import requests
import bs4


# name of the directory that will be used to save
# the code files if you write to disk. if the dir
# does not exist it will be created.
_temp_dir_name = 'forum_temp/'

# the full path of the temp dir, saving here!
__TEMP_DIR__ = os.getenv('HOME')+'/Documents/' +  _temp_dir_name

class OmzFourmCopyUI(ui.View):
    def __init__(self, data_list, post_title):
        #the list of code blocks, scraped from the forum
        self.data = data_list
        self.name = post_title

        # create acopy title btn
        # using the var below to hide it. i think
        # i could be corrupting the clipboard somehow
        if _SHOW_CLIPBOARD_BTN:
            btn = ui.ButtonItem('Clipboard')
            btn.action = self.copy_code
            self.left_button_items = [btn]

        # create copy to file btn
        btn = ui.ButtonItem('Write to File')
        btn.action = self.write_to_File
        self.right_button_items = [ btn]

        # create the tableview, set the data_source
        # and delegate to self.
        tv = ui.TableView(name = 'tv1')
        tv.name = 'tv1'
        tv.data_source = self
        tv.border_width = .5
        tv.delegate = self
        self.add_subview(tv)

        #create textview
        txtv = ui.TextView(name = 'txtv')
        txtv.font = ('Courier', 14)
        self.add_subview(txtv)

        # select the first code block
        tv.selected_row = 0
        txtv.text = self.data[0]

    # copy the selected code block to the clipboard 
    def copy_code(self, sender):
        if self['txtv'].text:
            code = self.data[self['tv1'].selected_row[1]]
            clipboard.set(code)
            console.hud_alert('Copied', duration = .5)
            self.close()

    # write the selected code block to a file
    # tempfile is used, delete = False
    # a var at the top of this file determines
    # if the file is opened after its written
    def write_to_File(self, sender):
        if not self['txtv'].text:
            console.hud_alert('No text to write to file', duration = 2)
            return

        # create a prefix for the file, using local
        # time of the device
        the_prefix = time.strftime('%X', time.localtime()).replace(':', '_') + '_'

        # if no directory exits, create one
        if not os.path.exists(__TEMP_DIR__):
            os.mkdir(__TEMP_DIR__)

        # create and return a ref to the new file
        fh = tempfile.NamedTemporaryFile(dir = __TEMP_DIR__, prefix =  the_prefix, suffix = '.py', delete = False)

        code = self.data[self['tv1'].selected_row[1]]
        fh.write(code)  # write the code to the file
        fspec = fh.name
        fname = fh.name.split('/')[-1:][0]
        fh.close()

        #close the view
        self.close()

        console.hide_output()

        if _OPEN_FILE_AFTER_WRITE:
            editor.open_file(fspec)
        else:
            console.hud_alert('written to file {}'.format(fname, _temp_dir_name), duration = 2)

    # handle the resizing more correctly.       
    def layout(self):
        self.frame = self.bounds
        tv = self['tv1']

        tv.frame = (0,0,self.width * .20, self.height)

        self['txtv'].frame = (tv.width, 0, self.width - tv.width, self.height)

    # tableview data_source callback methods    
    def tableview_cell_for_row(self, tableview, section, row):

        cell = ui.TableViewCell()
        cell.text_label.text = 'Code {}'.format(row + 1)
        return cell

    def tableview_number_of_rows(self, tableview, section):
        # Return the number of rows in the section
        return len(self.data)

    # tableview delegate callback method
    def tableview_did_select(self, tableview, section, row):
        self['txtv'].text = self.data[row]


def extract_code_blocks(url): 
    # code from omz as a quickie, i made some mods
    lst = []

    html = requests.get(url).text
    soup = bs4.BeautifulSoup(html)
    pre_tags = soup.find_all('pre')

    if pre_tags:
        for p in pre_tags:                               
            lst.append(p.get_text())

    title = ''
    if len(lst):
        # extract the title of the post
        title = soup.title.get_text()
        title = title.split('—')[-1:][0].lstrip(' ')

    # if no code block found, lst will be an empty
    # list and title will be ''
    return (lst, title)


if __name__ == '__main__':
    url_fail = True 
    console.show_activity()
    # copy a url from the clipboard
    # url should contain a link to a omz forum
    # that you want to extract code from.
    url = clipboard.get()

    #check the url
    if 'forum.omz-software.com' not in url:
        console.alert('omz-forums, not in url')
    elif len(url.splitlines()) > 1:
        console.alert('url, has multiple lines')
    else:
        # if the url looks ok, try and get
        # a list of code blocks from the
        # omz fourm
        url_fail = False 
        result =  extract_code_blocks(url)
        lst , post_title = result
    console.hide_activity()


    if not url_fail:
        if len(lst) == 0:
            # url appeared ok, but no code blocks found
            console.alert('No code found on page')
        else:
            x = OmzFourmCopyUI(lst, post_title).present()
    else:
        # if the url does not pass the checks
        # print the url to the console
        print 'Clipboard Contents - {}'.format(url)

Phuket2

Or code in gist I guess.

link text

Phuket2

Code on GistHub

Sorry, getting used to things here

cook

Maybe try getting your info from the RSS feed of the thread rather than using the URL.

Compare:

(A) https://forum.omz-software.com/topic/2037/copy-code-from-forum
(B) https://forum.omz-software.com/topic/2037.rss

It's easy enough to convert A to B. Then you can load the rss information using feedparser module.

usually something like this...

import feedparser
a = feedparser.parse('url')
for i in a['entries']:
    print i

I don't have time to look at this specific feed - but just wanted to say you may be able to get ALL of the information from a thread using the rss method. Maybe you need to test it on a super long thread.

Easy to load into a dictionary :)

Phuket2

@cook. Actually the code itself is ok besides the strange bug. The bug only occurs when you are doing something that is not normal. But of course we have to test for all conditions.
I am a bit quiet now, some friends from overseas arrived. Party party!
Also when I get a chance to do someThing here, I am focusing on github. But it's driving me crazy. It's ok, I will eventually work it out, I hope. Not for the faint hearted!