Forum Archive

Finished a new sync program called Synchronator to sync from Pythonista to Dropbox and visa versa. It uses the new Dropbox API (the old one is deprecated). It can be found in Github at https://github.com/markhamilton1/Synchronator. I have now been using this to sync between iPad-Dropbox-iPhone. Take a look at the project wiki at https://github.com/markhamilton1/Synchronator/wiki.

Any feedback would be appreciated. This should work in either Python 2 or 3.

I tried it, and I get this stack trace:

****************************************
*     Dropbox File Syncronization      *
****************************************

Loading Local State

Updating From Dropbox

Saving Local State

Checking For New Or Updated Local Files
    Uploading:  DropboxSetup.py -- Not Found Remotely
Traceback (most recent call last):
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/Synchronator/Synchronator.py", line 300, in <module>
    check_local(dbx, state)
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/Synchronator/Synchronator.py", line 207, in check_local
    state.check_state(dbx, path)
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/Synchronator/Synchronator.py", line 66, in check_state
    self.upload(dbx, path, '-- Not Found Remotely')
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/Synchronator/Synchronator.py", line 171, in upload
    mute=True)
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/site-packages/dropbox/base.py", line 1252, in files_upload
    f,
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 249, in request
    timeout=timeout)
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 341, in request_json_string_with_retry
    timeout=timeout)
  File "/private/var/mobile/Containers/Shared/AppGroup/256C51EF-C1A2-4394-964A-C26B18D4F04B/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 385, in request_json_string
    type(request_binary))
TypeError: expected request_binary as binary type, got <class 'str'>

Sorry for the delay. I have been looking for a solution but haven't found it yet.

Someone pointed out that they think this is because something in my code is not Python 3 compatible. I have been looking at what it could be, but I realized there is a simple work-around in the interim, just put #!/usr/bin/python2 as the first line in Synchronator.py. Because Pythonista is capable of running both it should work and Synchronator will sync all files. This is why I love Pythonista!

I have updated the code on GitHub with this change.

I still want to understand what is not working, but at the moment it seems that it is failing in the dropbox package. Still more than likely something I am doing wrong but haven't found it yet.

https://github.com/dropbox/dropbox-sdk-python/issues/76

this might be your problem? See link in the link..

import dropbox
print(dropbox.__version__)  # '6.4.0'

change
with open(path, 'r') as local_fr:
to
with open(path, 'rb') as local_fr:

in two places. files_upload takes bytes, not str, which you get by reading in binary mode, or else encoding.

Thanks Jon, I have made the change and uploaded it to the GitHub repo.

Works perfectly now, thanks

Hi Mark, everyone.

I am new to pythonista and came across this, thank you Mark for developing and sharing it!

I followed your guide / wiki and have installed the latest Dropbox package (8.0.0) via slash.

Both your scripts setup in the app and when I run it it gives a stack error of ValueError:

Thanks in advance for any help!

Joao

• Dropbox File Syncronization *

Loading Local State

Cannot Find State File -- Creating New Local State

Updating From Dropbox
Traceback (most recent call last):
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/SyncDropbox/Synchronator.py", line 297, in
check_remote(dbx, state)
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/SyncDropbox/Synchronator.py", line 218, in check_remote
state.execute_delta(dbx)
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/SyncDropbox/Synchronator.py", line 104, in execute_delta
results = dbx.files_list_folder('', True)
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/site-packages/dropbox/base.py", line 715, in files_list_folder
None,
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 234, in request
timeout=timeout)
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 326, in request_json_string_with_retry
timeout=timeout)
File "/private/var/mobile/Containers/Shared/AppGroup/7C871155-8C31-4773-B218-2D35DDD5C4CC/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 411, in request_json_string
timeout=timeout)
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/requests/sessions.py", line 505, in post
return self.request('POST', url, data=data, json=json, kwargs)
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/requests/sessions.py", line 462, in request
resp = self.send(prep, send_kwargs)
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/requests/sessions.py", line 574, in send
r = adapter.send(request, **kwargs)
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/requests/adapters.py", line 371, in send
timeout=timeout
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/urllib3/connectionpool.py", line 566, in urlopen
timeout_obj = self._get_timeout(timeout)

File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/urllib3/connectionpool.py", line 309, in _get_timeout
return Timeout.from_float(timeout)
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 155, in from_float
return Timeout(read=timeout, connect=timeout)
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 98, in init
self._connect = self._validate_timeout(connect, 'connect')
File "/var/containers/Bundle/Application/FC4FD503-11FB-4FF7-AFBC-6134FAAE69C8/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 128, in _validate_timeout
"int or float." % (name, value))
ValueError: Timeout value connect was Timeout(connect=30, read=30, total=None), but it must be an int or float.

Thanks for bringing this to my attention!

So everything I can find online seems to indicate that this was a bug that was exposed in the Requests library when Dropbox updated their library to 8.0. I hate to ever point fingers since I have caused my own share of bugs.

If you run 'pip install requests' you should get version 2.18.1 or better. They fixed the issue in their 2.17.1 release from what I can find.

After installing this later version of Requests you will need to quit Pythonista and restart it to have it load the newest versions of the libraries.

After that Synchronator should run just fine.

Mark
Keep Calm and Code Python

@markhamilton1, just want to say thanks. I just used Synchronator to do a backup. Worked very well. On my first run it took a long time to run because of many files, i was worried about it timing out. Maybe using console.set_idle_timer_disabled True/False at the start and end of the sync might be a worthwhile thing.
But again thanks, it did work perfectly

@markhamilton1
Works perfectly now!

Thank you! This opens a new whole new world

Cheers

Hi Mark,

Synchronator seems great, unfortunately I keep getting this error when I run it on Pythonista 3 using Python 3.5.
I used StaSh pip install to install the latest dropbox and requests to no avail, here is the stack trace, it seems to be a timeout problem :(

Updating From Dropbox

• Dropbox File Syncronization *

Loading Local State

Cannot Find State File -- Creating New Local State

Updating From Dropbox
Traceback (most recent call last):
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 125, in _validate_timeout
float(value)
TypeError: float() argument must be a string or a number, not 'Timeout'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/bin/Utilities/Synchronator/Synchronator.py", line 425, in
check_remote(dbx, state)
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/bin/Utilities/Synchronator/Synchronator.py", line 333, in check_remote
state.execute_delta(dbx)
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/bin/Utilities/Synchronator/Synchronator.py", line 174, in execute_delta
results = dbx.files_list_folder('', True)
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/site-packages/dropbox/base.py", line 1336, in files_list_folder
None,
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 234, in request
timeout=timeout)
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 325, in request_json_string_with_retry
timeout=timeout)
File "/private/var/mobile/Containers/Shared/AppGroup/895491B4-6F48-4CA2-AF6A-1A663749BD60/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 409, in request_json_string
timeout=timeout,
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/requests/sessions.py", line 512, in post
return self.request('POST', url, data=data, json=json, kwargs)
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/requests/sessions.py", line 469, in request
resp = self.send(prep, send_kwargs)
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/requests/sessions.py", line 577, in send
r = adapter.send(request, **kwargs)
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/requests/adapters.py", line 377, in send
timeout=timeout
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/urllib3/connectionpool.py", line 547, in urlopen
timeout_obj = self._get_timeout(timeout)
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/urllib3/connectionpool.py", line 303, in _get_timeout
return Timeout.from_float(timeout)
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 155, in from_float
return Timeout(read=timeout, connect=timeout)
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 98, in init
self._connect = self._validate_timeout(connect, 'connect')
File "/var/containers/Bundle/Application/44E531D9-C860-4F8B-9A15-16BE6F3BE756/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/site-packages/urllib3/util/timeout.py", line 128, in _validate_timeout
"int or float." % (name, value))
ValueError: Timeout value connect was Timeout(connect=30, read=30, total=None), but it must be an int or float.

Thank in advance
Cheers
Andre

I've just installed Pythonista, the dropbox and requests packages and Synchronator. When I run DropboxSetup.py I get the following:

/private/var/mobile/Containers/Shared/AppGroup/EAEDE395-145A-4A8B-A282-E6D1409E3D51/Pythonista3/Documents/site-packages/requests/init.py:80: RequestsDependencyWarning: urllib3 (1.4) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "/private/var/mobile/Containers/Shared/AppGroup/EAEDE395-145A-4A8B-A282-E6D1409E3D51/Pythonista3/Documents/Dropbox/DropboxSetup.py", line 54, in
import dropbox
File "/private/var/mobile/Containers/Shared/AppGroup/EAEDE395-145A-4A8B-A282-E6D1409E3D51/Pythonista3/Documents/site-packages/dropbox/init.py", line 3, in
fronds in my UJ UK m .dropbox import version, Dropbox, DropboxTeam, create_session # noqa: F401
File "/private/var/mobile/Containers/Shared/AppGroup/EAEDE395-145A-4A8B-A282-E6D1409E3D51/Pythonista3/Documents/site-packages/dropbox/dropbox.py", line 17, in
import requests
File "/private/var/mobile/Containers/Shared/AppGroup/EAEDE395-145A-4A8B-A282-E6D1409E3D51/Pythonista3/Documents/site-packages/requests/init.py", line 90, in
from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'

1) make sure you are using the latest stash.

selfupdate

2) Install requests

pip install requests

3) For me, I had to update chardet and idna. Update failed, but install then fixed it.

pip update chardet
pip install chardet
pip update idna
pip install idna
pip update dropbox
pip install dropbox

4) in console:

M=[m for m in sys.modules if 'requests' in m]
for m in M:
   del sys.modules[m]
import requests

@JonB can you explain the purpose of step 4? And with all the packages that have been installed, why am I only interested in requests? Thanks

you could just force quit pythonista. The issue is that requests is imported by stash, so to force the new version to import, you have to delete everything related from sys.modules.

@JonB why don't I have to do the same for the other packages that have been installed? Do you mean StaSh doesn't use the other packages?

@JonB I tried the first line at the console and got:

>>> M=[m for m in sys.modules if 'requests' in m]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'sys' is not defined

import sys

Again, the only reason I did this was to check that the import was successful, without a full quit -- it took a few tries, to figure out that I needed to update chardet, then idna.

Since you know what you need to update already, just double tap the home button and swipe to kill pythonista. When you come back, import requests and if it works, you are good to go

@JonB I killed and restarted Pythonista and got:

>>> import requests
/private/var/mobile/Containers/Shared/AppGroup/77E4F4DB-EDA3-49C3-8AAD-D4452754B812/Pythonista3/Documents/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.4) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/mobile/Containers/Shared/AppGroup/77E4F4DB-EDA3-49C3-8AAD-D4452754B812/Pythonista3/Documents/site-packages/requests/__init__.py", line 90, in <module>
    from urllib3.exceptions import DependencyWarning
ImportError: cannot import name 'DependencyWarning'

Did some more digging and found that StaSh had installed urllib3 version 1.4 whereas the latest version is 1.22. Fortunately I am still at the beginning of my journey with Pythonista so I have deleted and reinstalled it. I will try again.

There is a problem with the way StaSh pip handles installing minimum versions. I have raised an issue on GitHub (see Pip installed wrong version of urllib3). To get round the problem, check the latest version of the package you are trying to install using PyPI - the Python Package Index and install that version using:

pip install [-d <directory>] <package>==<version>

And if a package gets installed as a dependency for another, check its version and if it is wrong perform a pip remove and install the correct version.

I am documenting exactly how I am getting this to work and, once it does, I will post it on this forum.

@Dangthrimble

I have the same issue as you, but when I tried to install/update/ remove requests, I get the same issue:

StaSh v0.6.20
Tip: You can redirect output to Pythonista console, e.g. ls > &3
[~/Documents]$ pip install requests
/private/var/mobile/Containers/Shared/AppGroup/8D798770-A6EB-4040-9568-3BCBFB1CF8E8/Documents/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.4) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
stash: <type 'exceptions.ImportError'>: cannot import name DependencyWarning

Is there a way to fix the requests module without deleting Pythonista altogether?

@HerniaChair I'm not an expert and so I don't know if it will work, but you could try deleting the requests folder in site-packages and close down and restart Pythonista

Yes, remove requests from site packages, then follow the instructions on the other thread.

I’m having exactly this problem and cannot even uninstall the package with pip (I keep getting the same error).

Can someone point me to the “other thread” that is mentioned in this post? Nothing in this post seems to hint at how to get rid of the offending package. Can I do it manually?

Thanks for any help.

Heather

@heatherdrury
I have to say that Pythonista is an awesome program that has been a huge help to me in working on planes, etc.

Probably bad form to answer my own post, but I ended up deleting the package directories (/site/packages/urllib3 and /site/packages/requests). This let me run pip and “uninstall” the packages correctly.

Now I’m trying to get back to the original task to figure out what I need to download latest Dropbox API, resolve this dependency issue with urllib3, and get Synronator.py to run so I can backup the many python programs I’ve written in Pythonista.

If anyone has more details on these steps, I’d appreciate it.

Heather

This thread was very helpful to me with resolving the issue with the outdated version of the Requests module. Thanks.

Now that the Synchronator script is running, I am wondering: Is there an ignore list? I would prefer not be syncing site-packages.

Hmm I should also comment that I'm having to run synchronator repeatedly. I keep getting a LookupError('malformed_path', None). It appears to be related to the fact that the dropbox contains filenames starting with '.' (uploaded by DropboxSync) that Synchronator doesn't consider valid to sync (based on the valid_filename_for_upload function).

I think that my sync may be completely fubar'd due to the remnants left over from DropboxSync, but I'm afraid if I empty out the app folder and resync, it's probably going to try to empty and delete the site-packages folder.

EDIT: I ended up deleting the old app and creating a new one, and clearing out all the state files and token. Hopefully that'll resolve the issues with existing files from DropboxSync and previously-crashed Synchronator instances.

Alternate solution without using API's
I was also looking for something similar and thought it shouldn't be that difficult or time taking. That's when I came across "Shortcuts" app which I use for lot of other workflows.

I managed to get it done with 2 simple steps which is already available in iPad or iOs :

1. Create a shortcut app
2. From Pythonista on the right corner top click on share and select "Shortcuts" and your created shortcut.

( File is now received in your dropbox )

To create workflow in Shortcuts
a. Search for "Get Text from Input" and drag the same to your workflow.
b. Search for "Save File" and select "Service" as "Dropbox" (Note : You'll be asked to login to your dropbox once)

That's it. Worked for me like charm.
iPad version : 12.4

P.S. This is only for those lookng for just syncing the files to dropbox from Phythonista not caring about using dropbox apis.

@timmymathew If you want to send your file to Shortcuts app, there is a quicker way.
Actually, you tap on Tools, Share, Shortcuts, your shortcut.
With this way, you tap Tools, "Share to shortcut" (or the text you want, of course)

You have to do, once,
1) your shortcut, named here, My_shortcut, containing only:
Get Clipboard
Save file.....as yours shortcut
2) create the little script here-under and add it to Tools
That's all, folks

import clipboard
import editor
import webbrowser
t = editor.get_text()
clipboard.set(t)
url = 'shortcuts://run-shortcut?name=My_shortcut&input=clipboard'
webbrowser.open(url)