docs/source/siteadmin/production-deployments.rst

   1 .. MediaGoblin Documentation
   2
   3    Written in 2011, 2012 by MediaGoblin contributors
   4
   5    To the extent possible under law, the author(s) have dedicated all
   6    copyright and related and neighboring rights to this software to
   7    the public domain worldwide. This software is distributed without
   8    any warranty.
   9
  10    You should have received a copy of the CC0 Public Domain
  11    Dedication along with this software. If not, see
  12    <http://creativecommons.org/publicdomain/zero/1.0/>.
  13
  14 =========================================
  15 Considerations for Production Deployments
  16 =========================================
  17
  18 This document contains a number of suggestions for deploying
  19 MediaGoblin in actual production environments. Consider
  20 ":doc:`deploying`" for a basic overview of how to deploy MediaGoblin.
  21
  22 Deploy with Paste
  23 -----------------
  24
  25 The instance configured with ``./lazyserver.sh`` is not ideal for a
  26 production MediaGoblin deployment. Ideally, you should be able to use
  27 an "init" or "control" script to launch and restart the MediaGoblin
  28 process.
  29
  30 Use the following command as the basis for such a script: ::
  31
  32     CELERY_ALWAYS_EAGER=true \
  33      /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
  34      /srv/mediagoblin.example.org/mediagoblin/paste.ini \
  35      --pid-file=/var/run/mediagoblin.pid \
  36      --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543
  37
  38 The above configuration places MediaGoblin in "always eager" mode
  39 with Celery, this means that submissions of content will be processed
  40 synchronously, and the user will advance to the next page only after
  41 processing is complete. If we take Celery out of "always eager mode,"
  42 the user will be able to immediately return to the MediaGoblin site
  43 while processing is ongoing. In these cases, use the following command
  44 as the basis for your script: ::
  45
  46     CELERY_ALWAYS_EAGER=false \
  47      /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
  48      /srv/mediagoblin.example.org/mediagoblin/paste.ini \
  49      --pid-file=/var/run/mediagoblin.pid \
  50      --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543
  51
  52 Separate Celery
  53 ---------------
  54
  55 While the ``./lazyserer.sh`` configuration provides an efficient way to
  56 start using a MediaGoblin instance, it is not suitable for production
  57 deployments for several reasons:
  58
  59 In nearly every scenario, work on the Celery queue will need to
  60 balance with the demands of other processes, and cannot proceed
  61 synchronously. This is a particularly relevant problem if you use
  62 MediaGoblin to host video content. Processing with Celery ought to be
  63 operationally separate from the MediaGoblin application itself, this
  64 simplifies management and support better workload distribution.
  65
  66 Basically, if you're doing anything beyond a trivial workload, such as
  67 image hosting for a small set of users, or have limited media types
  68 such as "ASCII art" or icon sharing, you will need to run ``celeryd``
  69 as a separate process.
  70
  71 Build an :ref:`init script <init-script>` around the following
  72 command::
  73
  74     CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd
  75
  76 Modify your existing MediaGoblin and application init scripts, if
  77 necessary, to prevent them from starting their own ``celeryd``
  78 processes.
  79
  80 Monitor exceptions
  81 ------------------
  82
  83 This is an example config using raven_ to report exceptions and
  84 :py:mod:`logging` messages to a sentry_ instance
  85
  86 .. _raven: http://raven.readthedocs.org/
  87 .. _sentry: https://github.com/getsentry
  88
  89 .. code-block:: ini
  90
  91     [pipeline:main]
  92     pipeline =
  93         errors
  94         raven
  95         routing
  96
  97     [loggers]
  98     keys = root, sentry
  99
 100     [handlers]
 101     keys = console, sentry
 102
 103     [formatters]
 104     keys = generic
 105
 106     [logger_root]
 107     level = INFO
 108     handlers = console, sentry
 109
 110     [logger_sentry]
 111     level = WARN
 112     handlers = console
 113     qualname = sentry.errors
 114     propagate = 0
 115
 116     [handler_sentry]
 117     class = raven.handlers.logging.SentryHandler
 118     args = ('http://public:secret@example.com/1',)
 119     level = WARNING
 120     formatter = generic
 121
 122     [filter:raven]
 123     use = egg:raven#raven
 124     dsn = http://71727ea2c69043e4bbcd793bb0115cd4:e9cedccb32d9482d81f99eeca8b1ad30@sentry.talka.tv/3
 125
 126 .. _init-script:
 127
 128 Use an Init Script
 129 ------------------
 130
 131 Look in your system's ``/etc/init.d/`` or ``/etc/rc.d/`` directory for
 132 examples of how to build scripts that will start, stop, and restart
 133 MediaGoblin and Celery. These scripts will vary by
 134 distribution/operating system.
 135
 136 These are scripts provided by the MediaGoblin community:
 137
 138 Debian
 139   * `GNU MediaGoblin init scripts
 140     <https://github.com/jwandborg/mediagoblin-init-scripts>`_
 141     by `Joar Wandborg <http://wandborg.se>`_
 142
 143 Arch Linux
 144   * `MediaGoblin - ArchLinux rc.d scripts
 145     <http://whird.jpope.org/2012/04/14/mediagoblin-archlinux-rcd-scripts>`_
 146     by `Jeremy Pope <http://jpope.org/>`_
 147   * `Mediagoblin init script on Archlinux
 148     <http://chimo.chromic.org/2012/03/01/mediagoblin-init-script-on-archlinux/>`_
 149     by `Chimo <http://chimo.chromic.org/>`_
 150
 151 .. TODO insert init script here
 152 .. TODO are additional concerns ?
 153    .. Other Concerns
 154    .. --------------