From: Joar Wandborg Date: Fri, 24 May 2013 21:07:09 +0000 (+0200) Subject: Improved docs X-Git-Url: https://vcs.fsf.org/?a=commitdiff_plain;h=5471e08e7eb27872d7a656ed87911be307dd8290;p=mediagoblin.git Improved docs - Fixed an outdated URL - Rewrote "Separate Celery" section - Changed literal blocks to bash code-blocks - Changed wording when referring to the MediaGoblin WSGI application --- diff --git a/docs/source/siteadmin/production-deployments.rst b/docs/source/siteadmin/production-deployments.rst index 1a32d95e..839d3ce5 100644 --- a/docs/source/siteadmin/production-deployments.rst +++ b/docs/source/siteadmin/production-deployments.rst @@ -22,12 +22,14 @@ MediaGoblin in actual production environments. Consider Deploy with Paste ----------------- -The instance configured with ``./lazyserver.sh`` is not ideal for a -production MediaGoblin deployment. Ideally, you should be able to use -an "init" or "control" script to launch and restart the MediaGoblin +The MediaGoblin WSGI application instance you get with ``./lazyserver.sh`` is +not ideal for a production MediaGoblin deployment. Ideally, you should be able +to use an "init" or "control" script to launch and restart the MediaGoblin process. -Use the following command as the basis for such a script: :: +Use the following command as the basis for such a script: + +.. code-block:: bash CELERY_ALWAYS_EAGER=true \ /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ @@ -41,7 +43,9 @@ synchronously, and the user will advance to the next page only after processing is complete. If we take Celery out of "always eager mode," the user will be able to immediately return to the MediaGoblin site while processing is ongoing. In these cases, use the following command -as the basis for your script: :: +as the basis for your script: + +.. code-block:: bash CELERY_ALWAYS_EAGER=false \ /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ @@ -52,30 +56,40 @@ as the basis for your script: :: Separate Celery --------------- -While the ``./lazyserver.sh`` configuration provides an efficient way to -start using a MediaGoblin instance, it is not suitable for production -deployments for several reasons: +MediaGoblin uses `Celery`_ to handle heavy and long-running tasks. Celery can +be launched in two ways: + +1. Embedded in the MediaGoblin WSGI application [#f-mediagoblin-wsgi-app]_. + This is the way ``./lazyserver.sh`` does it for you. It's simple as you + only have to run one process. The only bad thing with this is that the + heavy and long-running tasks will run *in* the webserver, keeping the user + waiting each time some heavy lifting is needed as in for example processing + a video. This could lead to problems as an aborted connection will halt any + processing and since most front-end web servers *will* terminate your + connection if it doesn't get any response from the MediaGoblin WSGI + application in a while. + +2. As a separate process communicating with the MediaGoblin WSGI application + via a `broker`_. This offloads the heavy lifting from the MediaGoblin WSGI + application and users will be able to continue to browse the site while the + media is being processed in the background. + +.. _`broker`: http://docs.celeryproject.org/en/latest/getting-started/brokers/ +.. _`celery`: http://www.celeryproject.org/ + -In nearly every scenario, work on the Celery queue will need to -balance with the demands of other processes, and cannot proceed -synchronously. This is a particularly relevant problem if you use -MediaGoblin to host video content. Processing with Celery ought to be -operationally separate from the MediaGoblin application itself, this -simplifies management and support better workload distribution. +.. [#f-mediagoblin-wsgi-app] The MediaGoblin WSGI application is the part that + of MediaGoblin that processes HTTP requests. -Basically, if you're doing anything beyond a trivial workload, such as -image hosting for a small set of users, or have limited media types -such as "ASCII art" or icon sharing, you will need to run ``celeryd`` -as a separate process. +To launch Celery separately from the MediaGoblin WSGI application: -Build an :ref:`init script ` around the following -command:: +1. Make sure that the ``CELERY_ALWAYS_EAGER`` environment variable is unset or + set to ``false`` when launching the MediaGoblin WSGI application. +2. Start the ``celeryd`` main process with - CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd + .. code-block:: bash -Modify your existing MediaGoblin and application init scripts, if -necessary, to prevent them from starting their own ``celeryd`` -processes. + CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd .. _sentry: @@ -102,7 +116,7 @@ These are scripts provided by the MediaGoblin community: Debian * `GNU MediaGoblin init scripts - `_ + `_ by `Joar Wandborg `_ Arch Linux