Hi.
I have this repetitive task at work, download data from SAP and upload it to a ftp server.
I have managed to script the steps making semiautomatic downloads trough a VBA script, the files are stored in a OneDrive folder, after that, I access that folder and run a Pythonista script to upload the files to the ftp server.
Could somebody give me a hand to do it on a single step?
Thanks.
@Tito, do I understand correctly that you can access the OneDrive folder and file, but do not know what path to use in the Pythonista script that uploads it to the ftp server?
If that is the case, probably simplest is to copy your script to the same folder, open it from there and run it. Then it has access to your file without any path.
@mikael When I open external folder in Pythonista, OneDrive stays gray while it isn't in the Files app.
@cvp, I think it is the same for me. But for this, it is enough to open the .py file, not a folder. And the file can be moved to OneDrive by sharing.
@mikael I did understand but in this topic, I thought you could access OneDrive in Pythonista
@mikael so, would it be possible to make a script that select two files and upload them to the ftp?
I have would take the data to the folder and then exec the script and upload the files in the same execution?
Following the logic I only had to say the file name because the script would be in the same folder.
I think this is it...
fil = appex.get_file_path()
if fil == None:
print('no file passed')
return
I have to find the code that says the file names, now it shows the document picker but I want two fixed files.
Thanks.
@Tito I think that even a script located in your OneDrive folder would not access to other files while OneDrive is not open first as external folder IN Pythonista
I've tried with
import os
print(os.listdir())
But that gives a permission error.
And giving the link obtained trough files?
shareddocuments:///private/var/mobile/Containers/Shared/AppGroup/...
@Tito I guess you could use MS API with token authorization etc....but I don't know how..
@Tito said:
And giving the link obtained trough files?
Keep from /private.. and try
os.listdir('/private...')
Now I’m open OneDrive app, go to the folder, I select a file, run Pythonista Script and repeat with the other file.
I can use iOS to access the route give it to Pythonista and upload a file. It has to be a way for doing this with a fixed route.
@Tito when you share an OneDrive file to send it to Pythonista, iOS creates a temporary copy and its path is not always the same
@cvp what a pity... in that case I’m have to do it manually forever.
Thanks for your help. :)
@Tito said:
in that case I’m have to do it manually forever.
Or try via theMS API
@cvp I’ve found a post about the compatibility between OneDrive and Shortcuts but seems to be only for scan documents and related stuff nothing related to file routes.
@Tito I had tried also a shortcut opening the url, without success.
Another solution could be perhaps web scraping on the OneDrive site...but surely not easy.
@Tito I think I have found a solution.
You have to share (in OneDrive app or web page) your folder once and copy the generated link.
Then, try this little script, where you have to replace my url with yours
import ui
url = 'https://1drv.ms/u/s!AnAqP7kYeMWrgXwwda6qlAmt-bsl?e=wNu7Tb'
w = ui.WebView()
w.load_url(url)
w.present()
You will get a web page where we will still have to find a way to press the download button to get a zip with all files in the folder.
@Tito I'll (try to 🙄)do it this evening
@Tito Finally, easier that I thought. For each OneDrive file that you need to upload, in the OneDrive app, once, you share it, and 'copy the link'. In the little script here under, you store in the files dict, the file name name and it's pasted link.
Even if the OneDrive file is modified, its link stays the same, at least during my tests.
The script downloads the OneDrive file and copies it locally.
The FTP unload part is another process, if you want help for it, ask me.
Based on Generate OneDrive Direct-Download Link with C# or Python
import requests
import base64
def create_onedrive_directdownload (onedrive_link):
data_bytes64 = base64.b64encode(bytes(onedrive_link, 'utf-8'))
data_bytes64_String = data_bytes64.decode('utf-8').replace('/','_').replace('+','-').rstrip("=")
resultUrl = f"https://api.onedrive.com/v1.0/shares/u!{data_bytes64_String}/root/content"
return resultUrl
def main():
files = {'a.py':'https://1drv.ms/u/s!AnAqP7kYeMWrgX9L-9sMT08Fkgkf', 'aa.txt':'https://1drv.ms/t/s!AnAqP7kYeMWrggCb2Umu3Wn_Hr4G'}
for nam_file in files:
url_file = create_onedrive_directdownload(files[nam_file])
r = requests.get(url_file)
with open(nam_file, mode='wb') as fil:
fil.write(r.content)
if __name__ == '__main__':
main()
@Tito including SFTP
import base64
import os
import paramiko
import requests
import sys
# https://towardsdatascience.com/how-to-get-onedrive-direct-download-link-ecb52a62fee4
def create_onedrive_directdownload (onedrive_link):
data_bytes64 = base64.b64encode(bytes(onedrive_link, 'utf-8'))
data_bytes64_String = data_bytes64.decode('utf-8').replace('/','_').replace('+','-').rstrip("=")
resultUrl = f"https://api.onedrive.com/v1.0/shares/u!{data_bytes64_String}/root/content"
return resultUrl
def main():
path = sys.argv[0]
i = path.rfind('/')
path = path[:i+1]
ip = 'my ip'
user = 'user'
pwd = 'password'
try:
sftp_port = 22
transport = paramiko.Transport((ip, sftp_port))
transport.connect(username=user, password=pwd)
sftp = paramiko.SFTPClient.from_transport(transport)
files = {'a.py':'https://1drv.ms/u/s!AnAqP7kYeMWrgX9L-9sMT08F', 'aa.txt':'https://1drv.ms/t/s!AnAqP7kYeMWrggCb2Umu3Wn_'}
for nam_file in files:
url_file = create_onedrive_directdownload(files[nam_file])
r = requests.get(url_file)
with open(nam_file, mode='wb') as fil:
fil.write(r.content)
sftp.put(path + nam_file, nam_file)
os.remove(nam_file)
sftp.close()
transport.close()
except Exception as e:
print(str(e))
if __name__ == '__main__':
main()
You could even find automatically the file name from its link
.
.
.
for nam_file in files:
r = requests.get(files[nam_file])
c = r.content.decode('utf-8')
t = 'property="og:title" content="'
i = c.find(t)
if i >= 0:
# <meta property="og:title" content="a.py"/>
j = c.find('"/>',i)
print(c[i+len(t):j])
url_file = create_onedrive_directdownload(files[nam_file])
.
.
.
@cvp wow this is really awesome!!
Thanks so much, I will study your code and try to use it on my workflow.
Thanks again.
for url in files.values():
text = requests.get(url).text
tag = 'property="og:title" content="'
_, _, content = text.partition(tag)
if content:
content, _, _ = content.partition('"/>')
print(content)
url_file = create_onedrive_directdownload(url)
... or maybe ...
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
content = soup.find("content").text
@ccc I had tried with Eval_js on ui.WebView but there was a lot of "content" and other tries.
I'm sure there is a better analysis but my initial challenge was to get the links of the shared files.
@ccc said:
for url in files.values():
I always forget this function. I'm sure that I become too old and that I'll have to find soon a new hobby 😢
Anyway, as usual, thanks for your advices. Even if I can't use them, other Python beginners will.
Dude... I am 61 years old... No excuses... Keep pushing ;-)
@ccc too late I just bought one
@ccc said:
for url in files.values():
But later in the loop, I also use the files key (nam_file), thus...
@cvp
for filename, url in files.items():
😁😉
Hi, I’ve been trying but I think this is too complex for my skills.
You’ve been saying you were old guys, I’m felling useless at my almos brand new forties.
I don’t really understand how the function resolves the file and in my procedure I have to deal with an FTP server instead of an SFTP, so I don’t know where to touch the code and switching functions.
Many thanks for your help, and sorry for bothering, I’ll keep trying.
What error messages are you seeing?
https://docs.python.org/3/library/ftplib.html
@ccc thanks for answer and the link
I’m starting from here:
# coding: utf-8
import appex
import console
import os
import ui
from ftplib import FTP
def main():
# Sharing: receive file
fil = appex.get_file_path()
if fil == None:
print('no file passed')
return
server = 'address'
user = 'account'
pwd = 'pass'
server_file = os.path.basename(fil)
try:
ftp = FTP(server) #connect
ftp.encoding = 'utf-8'
ftp.login(user,pwd)
ipad_file = open(fil,'rb')
ftp.storbinary('STOR '+server_file,ipad_file,blocksize=8192)
ipad_file.close()
ftp.close()
except Exception as e:
print(str(e))
appex.finish()
if __name__ == '__main__':
main()
With this code I take a file on OneDrive and upload it to a FTP server, but the code is made for taking files from iOS menu (I think that’s the appex part for the code structure).
Now here it comes the code that @cvp kindly searched and found it points to a web link route, it points to a file through an extracted link (is really awesome, black magic for me...).
As you can see, I’m not a coder, I tried to learn python, bought a book (Learn Python the Hard Way) but got jammed soon. And I have this problem with Pythonista when I search for info about python, the code is tailored for a computer environment, and making it work on Pythonista changes the game a lot.
Sorry I lost focus on this and I’m telling you my sorrows.
I tried to do it in parts first try to understand the code and make it run alone.
import base64
def create_onedrive_directdownload (onedrive_link):
data_bytes64 = base64.b64encode(bytes(onedrive_link, 'utf-8'))
data_bytes64_String = data_bytes64.decode('utf-8').replace('/','_').replace('+','-').rstrip("=")
resultUrl = f"https://api.onedrive.com/v1.0/shares/u!{data_bytes64_String}/root/content"
return resultUrl
I tried to put the OneDrive link on (onedrive_link) I thought it was the first position working as a shortcut but I got an “invalid syntax” error... I’ve noticed that my OneDrive links are not the same as the ones from the article (1drv vs my works personalised direction https:://xxxxxx-my.share point.com/:x:/g/personal/... maybe could be there the issue.
My next step would be to get the file and upload to the FTP address and after that try to make it work with two files and upload the together.
Thanks for your valuable help, I’m a mess with this and I’m trying, for me was a big win to understand the previous code for uploading files from iOS with the script. In the beginning I had to open an ftp app and navigate to the route every time I wanted to upload a file with this script I only have to check that everything worked smoothly, my goal was doing both things on a tap.
I’m asking myself, if I put a Pythonista script pointing to the files I have to upload in the OneDrive folder, would it found them? I suppose this shouldn’t need any route, only file names, am I right?
@Tito if you want to try my script, you have to
1) choose in your OneDrive one file to test, assume named myfile.xxx
2) set it as shared in OneDrive
3) you will get a link, like https://1drv.ms/u/sxxxxxxxxx
4) modify in my script like
files = {'myfile.xxx':'https://1drv.ms/u/sxxxxxxxxx'}
5) modify my script to set your iP, user,password for FTP
Don't change something else, and sure not in the create_onedrive_directdownload function.
Good luck
@cvp done
I think is the function sftp, the server is a plain FTP...
Exception: Error reading SSH protocol banner
Traceback (most recent call last):
File "/var/containers/Bundle/Application/E5EADB2F-39AA-4401-9C74-050227C872BA/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/paramiko/transport.py", line 1855, in _check_banner
buf = self.packetizer.readline(timeout)
File "/var/containers/Bundle/Application/E5EADB2F-39AA-4401-9C74-050227C872BA/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/paramiko/packet.py", line 327, in readline
buf += self._read_timeout(timeout)
File "/var/containers/Bundle/Application/E5EADB2F-39AA-4401-9C74-050227C872BA/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/paramiko/packet.py", line 497, in _read_timeout
raise socket.timeout()
socket.timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/containers/Bundle/Application/E5EADB2F-39AA-4401-9C74-050227C872BA/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/paramiko/transport.py", line 1711, in run
self._check_banner()
File "/var/containers/Bundle/Application/E5EADB2F-39AA-4401-9C74-050227C872BA/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/paramiko/transport.py", line 1859, in _check_banner
raise SSHException('Error reading SSH protocol banner' + str(e))
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
Error reading SSH protocol banner
Thanks.
@Tito said:
server is a plain FTP
Sorry, of course you have to replace sftp process by ftp process, like in your script, sorry one more time