Commit | Line | Data |
---|---|---|
473a4431 CAW |
1 | .. MediaGoblin Documentation |
2 | ||
041c9363 | 3 | Written in 2011, 2012, 2013, 2014, 2015 by MediaGoblin contributors |
473a4431 CAW |
4 | |
5 | To the extent possible under law, the author(s) have dedicated all | |
6 | copyright and related and neighboring rights to this software to | |
7 | the public domain worldwide. This software is distributed without | |
8 | any warranty. | |
9 | ||
10 | You should have received a copy of the CC0 Public Domain | |
11 | Dedication along with this software. If not, see | |
12 | <http://creativecommons.org/publicdomain/zero/1.0/>. | |
13 | ||
b25b00d2 | 14 | ========================================= |
15 | Considerations for Production Deployments | |
16 | ========================================= | |
17 | ||
18 | This document contains a number of suggestions for deploying | |
19 | MediaGoblin in actual production environments. Consider | |
076bf0cf | 20 | ":doc:`deploying`" for a basic overview of how to deploy MediaGoblin. |
b25b00d2 | 21 | |
7d74b404 | 22 | Deploy with paste |
a085dda5 | 23 | ----------------- |
24 | ||
5471e08e JW |
25 | The MediaGoblin WSGI application instance you get with ``./lazyserver.sh`` is |
26 | not ideal for a production MediaGoblin deployment. Ideally, you should be able | |
9650aa39 | 27 | to use a Systemd service file or an init script to launch and restart the |
7d74b404 | 28 | MediaGoblin process. |
a085dda5 | 29 | |
9650aa39 | 30 | We will explore setting up MediaGoblin Systemd service files and init scripts, |
041c9363 JC |
31 | but first we need to create the directory that will store the MediaGoblin logs. |
32 | ||
7d74b404 JC |
33 | |
34 | .. _create-log-file-dir: | |
35 | ||
36 | Create the directory for your log file: | |
37 | --------------------------------------- | |
38 | ||
39 | Production logs for the MediaGoblin application are kept in the | |
40 | ``/var/log/mediagoblin`` directory. Create the directory and give it the | |
41 | proper permissions:: | |
42 | ||
43 | sudo mkdir -p /var/log/mediagoblin && sudo chown -hR mediagoblin:mediagoblin /var/log/mediagoblin | |
44 | ||
45 | ||
46 | .. _systemd-service-files: | |
47 | ||
9650aa39 | 48 | Use Systemd service files |
7d74b404 JC |
49 | ------------------------- |
50 | ||
9650aa39 | 51 | If your operating system uses Systemd, you can use Systemd ``service files`` |
7d74b404 JC |
52 | to manage both the Celery and Paste processes. Place the following service |
53 | files in the ``/etc/systemd/system/`` directory. | |
54 | ||
041c9363 | 55 | The first file should be named ``mediagoblin-celeryd.service``. Be sure to |
7d74b404 JC |
56 | modify it to suit your environment's setup: |
57 | ||
58 | .. code-block:: bash | |
59 | ||
60 | # Set the WorkingDirectory, Environment and ExecStart values to match your environment. | |
c9cdb036 | 61 | # If using Debian/*buntu, mkdir and chown are located in /bin/mkdir and /bin/chown, respectively. |
7d74b404 JC |
62 | # If using Fedora/CentOS/Red Hat, mkdir and chown are located in /usr/bin/mkdir and /usr/bin/chown, respectively. |
63 | ||
64 | [Unit] | |
9650aa39 | 65 | Description=MediaGoblin Celeryd |
7d74b404 JC |
66 | |
67 | [Service] | |
68 | User=mediagoblin | |
69 | Group=mediagoblin | |
70 | Type=simple | |
71 | WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin | |
1f23be2d CAW |
72 | # Start mg-celeryd process as root, then switch to mediagoblin user/group |
73 | # (This is needed to run the ExecStartPre commands) | |
74 | PermissionsStartOnly=true | |
7d74b404 | 75 | # Create directory for PID (if needed) and set ownership |
041c9363 JC |
76 | ExecStartPre=/bin/mkdir -p /run/mediagoblin |
77 | ExecStartPre=/bin/chown -hR mediagoblin:mediagoblin /run/mediagoblin | |
78 | # Celery process will run as the `mediagoblin` user after start. | |
f5e48d9e | 79 | Environment=MEDIAGOBLIN_CONFIG=/srv/mediagoblin.example.org/mediagoblin/mediagoblin.ini \ |
7d74b404 JC |
80 | CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery |
81 | ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/celery worker \ | |
82 | --logfile=/var/log/mediagoblin/celery.log \ | |
83 | --loglevel=INFO | |
84 | PIDFile=/run/mediagoblin/mediagoblin-celeryd.pid | |
85 | ||
86 | [Install] | |
87 | WantedBy=multi-user.target | |
88 | ||
89 | ||
041c9363 JC |
90 | The second file should be named ``mediagoblin-paster.service``: |
91 | ||
7d74b404 JC |
92 | |
93 | .. code-block:: bash | |
94 | ||
95 | # Set the WorkingDirectory, Environment and ExecStart values to match your environment. | |
c9cdb036 | 96 | # If using Debian/*buntu, mkdir and chown are located in /bin/mkdir and /bin/chown, respectively. |
7d74b404 JC |
97 | # If using Fedora/CentOS/Red Hat, mkdir and chown are located in /usr/bin/mkdir and /usr/bin/chown, respectively. |
98 | [Unit] | |
99 | Description=Mediagoblin | |
100 | ||
101 | [Service] | |
102 | Type=forking | |
103 | User=mediagoblin | |
104 | Group=mediagoblin | |
105 | Environment=CELERY_ALWAYS_EAGER=false | |
106 | WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin | |
107 | # Start mg-paster process as root, then switch to mediagoblin user/group | |
108 | PermissionsStartOnly=true | |
041c9363 JC |
109 | ExecStartPre=-/bin/mkdir -p /run/mediagoblin |
110 | ExecStartPre=/bin/chown -hR mediagoblin:mediagoblin /run/mediagoblin | |
7d74b404 JC |
111 | |
112 | ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ | |
113 | /srv/mediagoblin.example.org/mediagoblin/paste_local.ini \ | |
114 | --pid-file=/var/run/mediagoblin/mediagoblin.pid \ | |
115 | --log-file=/var/log/mediagoblin/mediagoblin.log \ | |
116 | --daemon \ | |
117 | --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543 | |
118 | ExecStop=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ | |
119 | --pid-file=/var/run/mediagoblin/mediagoblin.pid \ | |
120 | /srv/mediagoblin.example.org/mediagoblin/paste_local.ini stop | |
121 | PIDFile=/var/run/mediagoblin/mediagoblin.pid | |
122 | ||
123 | [Install] | |
124 | WantedBy=multi-user.target | |
125 | ||
126 | ||
041c9363 | 127 | |
7d74b404 JC |
128 | Enable these processes to start at boot by entering:: |
129 | ||
130 | sudo systemctl enable mediagoblin-celeryd.service && sudo systemctl enable mediagoblin-paster.service | |
131 | ||
132 | ||
041c9363 | 133 | Start the processes for the current session with:: |
7d74b404 | 134 | |
52628d64 JC |
135 | sudo systemctl start mediagoblin-paster.service |
136 | sudo systemctl start mediagoblin-celeryd.service | |
7d74b404 JC |
137 | |
138 | ||
139 | If either command above gives you an error, you can investigate the cause of | |
140 | the error by entering:: | |
141 | ||
142 | sudo systemctl status mediagoblin-celeryd.service or | |
143 | sudo systemctl status mediagoblin-paster.service | |
144 | ||
145 | The above ``systemctl status`` command is also useful if you ever want to | |
146 | confirm that a process is still running. If you make any changes to the service | |
041c9363 | 147 | files, you can reload the service files by entering:: |
7d74b404 JC |
148 | |
149 | sudo systemctl daemon-reload | |
150 | ||
151 | After entering that command, you can attempt to start the Celery or Paste | |
152 | processes again. | |
153 | ||
154 | .. _init-script: | |
155 | ||
156 | Use an init script | |
157 | ------------------ | |
158 | ||
9650aa39 | 159 | If your system does not use Systemd, you can use the following command as the |
7d74b404 | 160 | basis for an init script: |
5471e08e JW |
161 | |
162 | .. code-block:: bash | |
a085dda5 | 163 | |
076bf0cf WKG |
164 | CELERY_ALWAYS_EAGER=true \ |
165 | /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ | |
166 | /srv/mediagoblin.example.org/mediagoblin/paste.ini \ | |
167 | --pid-file=/var/run/mediagoblin.pid \ | |
168 | --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543 | |
a085dda5 | 169 | |
170 | The above configuration places MediaGoblin in "always eager" mode | |
171 | with Celery, this means that submissions of content will be processed | |
172 | synchronously, and the user will advance to the next page only after | |
173 | processing is complete. If we take Celery out of "always eager mode," | |
174 | the user will be able to immediately return to the MediaGoblin site | |
175 | while processing is ongoing. In these cases, use the following command | |
5471e08e JW |
176 | as the basis for your script: |
177 | ||
178 | .. code-block:: bash | |
a085dda5 | 179 | |
076bf0cf WKG |
180 | CELERY_ALWAYS_EAGER=false \ |
181 | /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ | |
182 | /srv/mediagoblin.example.org/mediagoblin/paste.ini \ | |
183 | --pid-file=/var/run/mediagoblin.pid \ | |
184 | --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543 | |
a085dda5 | 185 | |
7d74b404 JC |
186 | |
187 | Members of the MediaGoblin community have provided init scripts for the | |
188 | following GNU/Linux distributions: | |
189 | ||
190 | Debian | |
191 | * `GNU MediaGoblin init scripts | |
192 | <https://github.com/joar/mediagoblin-init-scripts>`_ | |
193 | by `Joar Wandborg <http://wandborg.se>`_ | |
194 | ||
195 | Arch Linux | |
196 | * `MediaGoblin - ArchLinux rc.d scripts | |
197 | <http://whird.jpope.org/2012/04/14/mediagoblin-archlinux-rcd-scripts>`_ | |
198 | by `Jeremy Pope <http://jpope.org/>`_ | |
199 | * `Mediagoblin init script on Archlinux | |
200 | <http://chimo.chromic.org/2012/03/01/mediagoblin-init-script-on-archlinux/>`_ | |
201 | by `Chimo <http://chimo.chromic.org/>`_ | |
202 | ||
203 | You can reference these scripts to create an init script for your own operating | |
204 | system. Similar scripts will be in your system's ``/etc/init.d/`` | |
205 | or ``/etc/rc.d/`` directory, but the specifics of an init script will vary from | |
206 | one distribution to the next. | |
207 | ||
208 | ||
209 | Separate celery | |
a085dda5 | 210 | --------------- |
b25b00d2 | 211 | |
5471e08e JW |
212 | MediaGoblin uses `Celery`_ to handle heavy and long-running tasks. Celery can |
213 | be launched in two ways: | |
214 | ||
215 | 1. Embedded in the MediaGoblin WSGI application [#f-mediagoblin-wsgi-app]_. | |
216 | This is the way ``./lazyserver.sh`` does it for you. It's simple as you | |
217 | only have to run one process. The only bad thing with this is that the | |
218 | heavy and long-running tasks will run *in* the webserver, keeping the user | |
219 | waiting each time some heavy lifting is needed as in for example processing | |
220 | a video. This could lead to problems as an aborted connection will halt any | |
221 | processing and since most front-end web servers *will* terminate your | |
222 | connection if it doesn't get any response from the MediaGoblin WSGI | |
223 | application in a while. | |
224 | ||
225 | 2. As a separate process communicating with the MediaGoblin WSGI application | |
226 | via a `broker`_. This offloads the heavy lifting from the MediaGoblin WSGI | |
227 | application and users will be able to continue to browse the site while the | |
228 | media is being processed in the background. | |
229 | ||
230 | .. _`broker`: http://docs.celeryproject.org/en/latest/getting-started/brokers/ | |
231 | .. _`celery`: http://www.celeryproject.org/ | |
232 | ||
b25b00d2 | 233 | |
5471e08e JW |
234 | .. [#f-mediagoblin-wsgi-app] The MediaGoblin WSGI application is the part that |
235 | of MediaGoblin that processes HTTP requests. | |
b25b00d2 | 236 | |
5471e08e | 237 | To launch Celery separately from the MediaGoblin WSGI application: |
b25b00d2 | 238 | |
5471e08e JW |
239 | 1. Make sure that the ``CELERY_ALWAYS_EAGER`` environment variable is unset or |
240 | set to ``false`` when launching the MediaGoblin WSGI application. | |
241 | 2. Start the ``celeryd`` main process with | |
b25b00d2 | 242 | |
5471e08e | 243 | .. code-block:: bash |
b25b00d2 | 244 | |
5471e08e | 245 | CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd |
b25b00d2 | 246 | |
9650aa39 | 247 | If you use our example Systemd ``service files``, Celery will be set to the |
7d74b404 JC |
248 | "CELERY_ALWAYS_EAGER=false" value by default. This will provide your users |
249 | with the best user experience, as all media processing will be done in the | |
250 | background. | |
251 | ||
f3f53028 | 252 | .. _sentry: |
f8107ccc | 253 | |
f3f53028 JW |
254 | Set up sentry to monitor exceptions |
255 | ----------------------------------- | |
f8107ccc | 256 | |
f3f53028 JW |
257 | We have a plugin for `raven`_ integration, see the ":doc:`/plugindocs/raven`" |
258 | documentation. | |
f8107ccc | 259 | |
f3f53028 | 260 | .. _`raven`: http://raven.readthedocs.org |
f8107ccc | 261 | |
f8107ccc | 262 | |
a085dda5 | 263 | .. TODO insert init script here |
076bf0cf | 264 | .. TODO are additional concerns ? |
a085dda5 | 265 | .. Other Concerns |
266 | .. -------------- |