Forum Archive

I would like to allow the user of a progam I'm developing to create a pipeline of stages, probably separated by the | character (or a replacement). They would pass this text string to my program.

For example:

fred.txt | takeLines 5 | takeColumns 3 | makeHTML | fred.html

These are not the real stages and I probably haven't got the Unix file handling syntax quite right but bear with me, please.

Has anyone done this? And do you know of any modules that make it easy?

The pipeline is probably going to be invoked by:

1) Command line parameters when running (from the console?)

2) As part of an iOS extension.

I'm not sure if there are any gotchas or if I can get any help with parsing the stages.

I also can tweak the syntax. If I can't use | within the input string I could choose something else; I don't mind much.

One thing I'm sure of is I don't want to have the pipeline invoke multiple Pythonista programs. The data is too cumbersome to keep parsing and externalising at each stage. Or, put another way, it would introduce unnecessary inefficiency.

Thoughts?

There is nothing magic about putting all your stages on a single line with a pipe in between and a return being the end of the pipe
vs. putting each stage on a line by itself with a blank line or end of file being the signal of the end of a pipe.

The first starts with commands = user_text.split('|') and the second starts with commands = user_text.splitlines().

It looks like a nice pattern would be for the first word of each stage/command to be the action verb and the later words are the modifiers.

I might rewrite it adding the read and write verbs as:

read fred.txt | takeLines 5 | takeColumns 3 | makeHTML | write fred.html

[ or ]

read fred.txt
takeLines 5
takeColumns 3
makeHTML
write fred.html

There is an implication of a single, unnamed variable (I will call it state) that carries the state of the workflow from stage to stage. Thus read puts data into that variable and and write gets data out of that variable. The two take verbs reduce the size of the data in the variable via selection. makeHTML transforms the data in the variable from one format to another.

I would start by writing 5 functions: read, takeLines, takeColumns, makeHTML, and write. Once you have those five done, the rest will become much simpler.

def execute_command(command, state):
    # [ ... put in logic to call the appropriate command and put the execution result into `state` ... ]
    return state

def main(commands):
    state = None
    for command in commands:
        state = execute_command(command, state)
    if state:
        print(state)
    return state

I would use a dictionary to call your functions. Do what @ccc said and under

```
def execute_command(command, state):
d = {'someCommand':someCommand,
'command2':command2}
for key in d.keys():
if command in key:
func = d[key]
tokens = key.split()
tokens.remove(tokens[0]) #get rid of function name
func(state, *tokens)

stash supports piping of commands. in that case the stdout from one command is fed into the stdin of the next.

May be you can look at the following python recipe.
http://code.activestate.com/recipes/276960-shell-like-data-processing/

It needs to be changed slightly to run it on python 3.

Thanks to everybody for your responses so far.

I like the dictionary idea and @ccc had a particularly good thought in terms of stages being in a file, perhaps new line separated.

@ccc I'll be in touch with you during the week as you emailed me. For those who don't know, we both work for IBM - which has a positive stance on Open Source but some governance around it.

@MartinPacker I have been working on a script that does something similar with a UI where you have functions and data is passed from one to the other, I had to keep track of input and output types to make sure the function the data is passed to can handle it. Anyway if you would like to take a look it is available here this might be totally overkill to what you want but you can have a look how I pass data etc