Django tasks, reminders, notifications - python

I have a fairly simple web app in Django (Apache, Ubuntu) for keeping some meetings documentation. Meetings have their appointment time stored in database (postgres) in datetime format. Now, I'd like to have a custom reminder module that would enable to user to setup their preferred reminders.
For example:
Meeting will be held at a certain date (let's say 25th of April 2018, 8:00 PM) and user should be able to setup a custom reminder (via SMS for example but the question is not about the sending texts or emails - I got this covered) to be fired up 24h before the time of meeting.
It got me thinking that this requires some kind of a permanent process browsing through the meetings table and checking if now() is the appointment time -24h and if yes then perform the reminder.
I started with rabbitmq and celery but these look a bit complicated at first glance (here's one of the tutorials I found) and looks like it's not designed for what I need.
So, question is - how to setup a permanent process that would check if a set reminder time is now and if yes - perform the reminder task?
EDIT: some errors after trying to follow the tutorial in answer. Got stuck at step 3:
(dj2_env) adrian#dev:~$ celery -A dj2 worker -l info
Traceback (most recent call last):
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/app/utils.py", line 361, in find_app
found = sym.app
AttributeError: module 'dj2' has no attribute 'app'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/adrian/dj2_env/bin/celery", line 11, in <module>
sys.exit(main())
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/__main__.py", line 14, in main
_main()
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/bin/celery.py", line 326, in main
cmd.execute_from_commandline(argv)
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
self.app = self.find_app(app)
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/bin/base.py", line 503, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/home/adrian/dj2_env/lib/python3.5/site-packages/celery/app/utils.py", line 366, in find_app
found = sym.celery
AttributeError: module 'dj2' has no attribute 'celery'
Ok, got the above error figured out - it's weird however, as when I ran the worker command from dj2 (project) directory it worked 8|

You can use celery and rabbitmq for this. There is something called periodic tasks in celery which lets you runs tasks given a specific time just like a cron job.
Here is a simple tutorial for Asynchronous tasks in Celery
It gives a good understanding of how celery works. It also includes the configuration, etc.
What you probably need to do is to fire up a task (every 15 mins) which queries the database and checks for meetings within next 24 hours and then notify the user about it. The 4th part of the tutorial shows how you can use periodic tasks.

A Celery periodic task may be the best solution for this, however a simpler alternative may be a Django custom management command running from a crontab.
The management command would automatically have access to the Django ORM and could thus perform the database check and send the reminders. You could run it from the crontab such as:
# Run the command every 15 minutes
*/15 * * * * python /path/to/manage.py your_command_name
Using a management command would also give you the ability to execute the reminder process manually from the command line should you ever have to.

Related

using django celery beat locally I get error 'PeriodicTask' object has no attribute '_default_manager'

using django celery beat locally I get error 'PeriodicTask' object has no attribute '_default_manager'. I am using Django 1.10. When i schedule a task it works. But then a few moments later a red error traceback like the following occurs
[2016-09-23 11:08:34,962: INFO/Beat] Writing entries...
[2016-09-23 11:08:34,965: INFO/Beat] Writing entries...
[2016-09-23 11:08:34,965: INFO/Beat] Writing entries...
[2016-09-23 11:08:34,966: ERROR/Beat] Process Beat
Traceback (most recent call last):
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/billiard/process.py", line 292, in _bootstrap
self.run()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/beat.py", line 553, in run
self.service.start(embedded_process=True)
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/beat.py", line 486, in start
self.scheduler._do_sync()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/celery/beat.py", line 276, in _do_sync
self.sync()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/djcelery/schedulers.py", line 209, in sync
self.schedule[name].save()
File "/Users/ray/Desktop/myheroku/practice/lib/python3.5/site-packages/djcelery/schedulers.py", line 98, in save
obj = self.model._default_manager.get(pk=self.model.pk)
AttributeError: 'PeriodicTask' object has no attribute '_default_manager'
after this happens the next schedule wont run unless I "control+c" out of the terminal and start it again. I saw on git hub that this may be because I am using django 1.10. I have already git pushed this to my heroku server. How can I fix this issue? The git hub post said he fixed it by doing this
Model = type(self.model)
obj = Model._default_manager.get(pk=self.model.pk)
I was willing to try this but I don't know where to put this and I don't want to cause a bigger unforeseen issue that this could cause. What are my options? am I supposed to manually go inside my remote app and reset it after every time it runs? thats unfeasible and defeats the purpose of task automation.
I figured it out. At line 98 in schedulers.py it was
obj = self.model._default_manager.get(pk=self.model.pk)
so a line above it I added
Model = type(self.model)
and changed
obj = self.model._default_manager.get(pk=self.model.pk)
to
obj = Model._default_manager.get(pk=self.model.pk)
so completed it looks like this
98 Model = type(self.model)
99 obj = Model._default_manager.get(pk=self.model.pk)

gdata spreadsheet library for python not working anymore?

I was trying to run a query for data in one of my google docs, and it's worked for several months. Starting yesterday or the day before, I noticed that my script no longer works. Has Google updated their api for spreadsheets? Has anybody found a workaround?
My error looks like this:
Traceback (most recent call last):
File "build_packer_image.py", line 311, in <module>
for index, entry in enumerate(client.GetWorksheetsFeed(doc_key).entry):
File "/build/toolchain/mac-10.5-32/lib/python2.7/site-packages/gdata/spreadsheet/service.py", line 129, in GetWorksheetsFeed
converter=gdata.spreadsheet.SpreadsheetsWorksheetsFeedFromString)
File "/build/toolchain/mac-10.5-32/lib/python2.7/site-packages/gdata/service.py", line 1074, in Get
return converter(result_body)
File "/build/toolchain/mac-10.5-32/lib/python2.7/site-packages/gdata/spreadsheet/__init__.py", line 411, in SpreadsheetsWorksheetsFeedFromString
xml_string)
File "/build/toolchain/mac-10.5-32/lib/python2.7/site-packages/atom/__init__.py", line 93, in optional_warn_function
return f(*args, **kwargs)
File "/build/toolchain/mac-10.5-32/lib/python2.7/site-packages/atom/__init__.py", line 127, in CreateClassFromXMLString
tree = ElementTree.fromstring(xml_string.replace('doctype','DOCTYPE'))
File "<string>", line 125, in XML
cElementTree.ParseError: no element found: line 1, column 0
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I am using:
Python 2.7.5
gdata 2.0.18
I am just using an document key and no oauth in my code, if that makes a difference (I am passing in the username and password to the ClientLogin method)
Actually here is the answer to the problem:
The use of client login (using username/password instead of oauth2) is
likely the cause of the error. That protocol was deprecated 3+ years
ago and was just shutdown. If you capture the HTTP response (which
appears to have some HTML content), that might confirm if it is
related to the shutdown. Migrating to OAuth 2 would get your apps
working again.
After sending xml for update in spreadsheet google respond with a login page.
It means the authentication is not working for gdata now
https://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3851#c2

AppRegistryNotReady in Django 1.7 and Python 3.4 when using Multiprocessing

In a Django view I spawn a thread (class of threading.Thread) which in turn creates a multiprocessing pool of 5 workers.
Yes, I know using a task queue like Celery is usually the accepted way of doing things, but in this case we needed threads/multiprocessing.
Both the Thread and each of the Multiprocess Workers access items in the database. However, doing any call to a Django Model in the Thread or Worker causes a "django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet" exception.
Here is the full stack trace:
Process SpawnPoolWorker-2:
Traceback (most recent call last):
File "C:\Python34\lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "C:\Python34\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Python34\lib\multiprocessing\pool.py", line 108, in worker
task = get()
File "C:\Python34\lib\multiprocessing\queues.py", line 357, in get
return ForkingPickler.loads(res)
File "d:\bryan\Documents\Projects\spreadmodel_3.4_venv\lib\site-packages\djang
o\db\models\fields\__init__.py", line 59, in _load_field
return apps.get_model(app_label, model_name)._meta.get_field_by_name(field_n
ame)[0]
File "d:\bryan\Documents\Projects\spreadmodel_3.4_venv\lib\site-packages\djang
o\apps\registry.py", line 199, in get_model
self.check_models_ready()
File "d:\bryan\Documents\Projects\spreadmodel_3.4_venv\lib\site-packages\djang
o\apps\registry.py", line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
It is odd to me that I don't see any part of my code in the stack trace.
I've tried doing django.setup() in the Thread init and at the beginning of the method the Workers start, still with no success.
At no point in my models do I try to read anything from the database like the common issue of doing a foreign key to the user model.
EDIT:
I can get the database queries to work in the Thread by putting django.setup just under the Simulation class instead of having it in the init method.
But I'm still having issues with the queries in the Workers.
Edit2:
If I modify Python's queue.Queue file and put the django.setup() call in the get function, everything works great.
However, this is not a valid solution. Any ideas?
Edit3:
If I run the tests inside PyCharm, then the test associated with this problem works. Running the test in the normal command line outside of PyCharm (or running the view server from a server [django test server or CherryPy]) results in the above error.
If it helps, here is a link to the views.py on GitHub.
https://github.com/NAVADMC/SpreadModel/blob/b4bbbcf7020a3e4df0d021942ddcc5039234bd88/Results/views.py
For future reference (after we fix the bug), you can see the odd behaviour on commit b4bbbcf7 (linked above).

ImageKit async error - can't decode message body

I'm using Django 1.6 and Django-ImageKit 3.2.1.
I'm trying to generate images asynchronously with ImageKit. Async image generation works locally but not on the production server.
I'm using Celery and I've tried both:
IMAGEKIT_DEFAULT_CACHEFILE_BACKEND = 'imagekit.cachefiles.backends.Async'
IMAGEKIT_DEFAULT_CACHEFILE_BACKEND = 'imagekit.cachefiles.backends.Celery'
Using the Simple backend (synchronous) instead of Async or Celery works fine on the production server. So I don't understand why the asynchronous backend gives me the following ImportError (pulled from the Celery log):
[2014-04-05 21:51:26,325: CRITICAL/MainProcess] Can't decode message body: DecodeError(ImportError('No module named s3utils',),) [type:u'application/x-python-serialize' encoding:u'binary' headers:{}]
body: '\x80\x02}q\x01(U\x07expiresq\x02NU\x03utcq\x03\x88U\x04argsq\x04cimagekit.cachefiles.backends\nCelery\nq\x05)\x81q\x06}bcimagekit.cachefiles\nImageCacheFile\nq\x07)\x81q\x08}q\t(U\x11cachefile_backendq\nh\x06U\x12ca$
Traceback (most recent call last):
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/messaging.py", line 585, in _receive_callback
decoded = None if on_m else message.decode()
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/message.py", line 142, in decode
self.content_encoding, accept=self.accept)
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 184, in loads
return decode(data)
File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
self.gen.throw(type, value, traceback)
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 59, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 55, in _reraise_errors
yield
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 184, in loads
return decode(data)
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 64, in pickle_loads
return load(BytesIO(s))
DecodeError: No module named s3utils
s3utils is what defines my AWS S3 bucket paths. I'll post it if need be, but the strange thing I think is that the synchronous backend has no problem importing s3utils while the asynchronous does... and asynchronous does ONLY on the production server, not locally.
I'd be SO greatful for any help debugging this. I've been wrestling this for days. I'm still learning Django and python so I'm hoping this is a stupid mistake on my part. My Google-fu has failed me.
As I hinted at in my comment above, this kind of thing is usually caused by forgetting to restart the worker.
It's a common gotcha with Celery. The workers are a separate process from your web server so they have their own versions of your code loaded. And just like with your web server, if you make a change to your code, you need to reload so it sees the change. The web server talks to your worker not by directly running code, but by passing serialized messages via the broker, which will say something like "call the function do_something()". Then the worker will read that message and—and here's the tricky part—call its version of do_something(). So even if you restart your webserver (so that it has a new version of your code), if you forget to reload the worker (which is what actually calls the function), the old version of the function will be called. In other words, you need to restart the worker any time you make a change to your tasks.
You might want to check out the autoreload option for development. It could save you some headaches.

Django exception bugging me, don't know how to debug it

I recently upgraded to python2.7 and django1.3 and since then
Unhandled exception in thread started by <bound method Command.inner_run of <django.core.management.commands.runserver.Command object at 0x109c57490>>
Traceback (most recent call last):
File "/Users/ApPeL/.virtualenvs/myhunt/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 88, in inner_run
self.validate(display_num_errors=True)
File "/Users/ApPeL/.virtualenvs/myhunt/lib/python2.7/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/Users/ApPeL/.virtualenvs/myhunt/lib/python2.7/site-packages/django/core/management/validation.py", line 36, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/Users/ApPeL/.virtualenvs/myhunt/lib/python2.7/site-packages/django/db/models/loading.py", line 146, in get_app_errors
self._populate()
File "/Users/ApPeL/.virtualenvs/myhunt/lib/python2.7/site-packages/django/db/models/loading.py", line 67, in _populate
self.write_lock.release()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 137, in release
raise RuntimeError("cannot release un-acquired lock")
RuntimeError: cannot release un-acquired lock
Your help would be greatly appreciated.
A usual first recommendation is to apply the latest updates to gevent or greenlet or what you use related to threads. Implementation of threading.Thread.start has been changed between Python 2.6 and 2.7. There are many recipes how to start green... or green... with Django. Try to read any recent for Python 2.7. and send a link which one makes the problem.
Debugging:
Add following lines to your manage.py to enable logging of thread start etc. to stderr:
import threading
setattr(threading, '__debug__', True)
Add the argument verbose to django/db/loading.py line 39 in order to see also what threads acquire and release the lock.
- write_lock = threading.RLock(),
+ write_lock = threading.RLock(verbose=True),
Run development server. For only one thread without autoreload you should see something like:
$ python manage.py runserver --noreload
Validating models...
MainThread: <_RLock owner='MainThread' count=1>.acquire(1): initial success
MainThread: <_RLock owner=None count=0>.release(): final release
Notes:
count=1 acquire(1) -- the first acquire by a blocking lock
owner=None count=0>.release() -- the the lock is currently being unlocked
$ python manage.py runserver
Validating models...
Dummy-1: <_RLock owner=-1222960272 count=1>.acquire(1): initial success
Dummy-1: <_RLock owner=None count=0>.release(): final release
This is the same with autoreload. Models are validated by the child process.
"Dummy-1" is a symbolic name of the thread. This can be repeated for more threads, but no threads should/can acquire the lock until it is released by the previous thread. We can continue according the results.

Resources