e52a4b08db1f30463c9f62734246e0aed187229c
[mediagoblin.git] / docs / source / siteadmin / deploying.rst
1 .. MediaGoblin Documentation
2
3 Written in 2011, 2012, 2013 by MediaGoblin contributors
4
5 To the extent possible under law, the author(s) have dedicated all
6 copyright and related and neighboring rights to this software to
7 the public domain worldwide. This software is distributed without
8 any warranty.
9
10 You should have received a copy of the CC0 Public Domain
11 Dedication along with this software. If not, see
12 <http://creativecommons.org/publicdomain/zero/1.0/>.
13
14 .. _deploying-chapter:
15
16 =====================
17 Deploying MediaGoblin
18 =====================
19
20 GNU MediaGoblin is fairly new, and so at the time of writing there aren't
21 easy package-manager-friendly methods to install it. However, doing a basic
22 install isn't too complex in and of itself. Following this deployment guide
23 will take you step-by-step through setting up your own instance of MediaGoblin.
24
25 Of course, when it comes to setting up web applications like MediaGoblin,
26 there's an almost infinite way to deploy things, so for now, we'll keep it
27 simple with some assumptions. We recommend a setup that combines MediaGoblin +
28 virtualenv + FastCGI + Nginx on a .deb- or .rpm-based GNU/Linux distro.
29
30 Other deployment options (e.g., deploying on FreeBSD, Arch Linux, using
31 Apache, etc.) are possible, though! If you'd prefer a different deployment
32 approach, see our
33 `Deployment wiki page <http://wiki.mediagoblin.org/Deployment>`_.
34
35 .. note::
36
37 These tools are for site administrators wanting to deploy a fresh
38 install. If you want to join in as a contributor, see our
39 `Hacking HOWTO <http://wiki.mediagoblin.org/HackingHowto>`_ instead.
40
41 .. note::
42
43 Throughout the documentation we use the ``sudo`` command to indicate that
44 an instruction requires elevated user privileges to run. You can issue
45 these commands as the ``root`` user if you prefer.
46
47 If you need help configuring ``sudo``, see the
48 `Debian wiki <https://wiki.debian.org/sudo/>`_ or the
49 `Fedora Project wiki <https://fedoraproject.org/wiki/Configuring_Sudo/>`_.
50
51
52 Prepare System
53 --------------
54
55 Dependencies
56 ~~~~~~~~~~~~
57
58 MediaGoblin has the following core dependencies:
59
60 - Python 2.7 or Python 3.4+
61 - `python-lxml <http://lxml.de/>`_
62 - `git <http://git-scm.com/>`_
63 - `SQLite <http://www.sqlite.org/>`_/`PostgreSQL <http://www.postgresql.org/>`_
64 - `Python Imaging Library <http://www.pythonware.com/products/pil/>`_ (PIL)
65 - `virtualenv <http://www.virtualenv.org/>`_
66 - `nodejs <https://nodejs.org>`_
67
68 On a DEB-based system (e.g Debian, gNewSense, Trisquel, \*buntu, and
69 derivatives) issue the following command::
70
71 sudo apt-get install git-core python python-dev python-lxml \
72 python-imaging python-virtualenv npm nodejs-legacy automake \
73 nginx rabbitmq-server
74
75 On a RPM-based system (e.g. Fedora, RedHat, and derivatives) issue the
76 following command::
77
78 sudo yum install python-paste-deploy python-paste-script \
79 git-core python python-devel python-lxml python-imaging \
80 python-virtualenv npm automake nginx rabbitmq-server
81
82 (Note: MediaGoblin now officially supports Python 3. You may instead
83 substitute from "python" to "python3" for most package names in the
84 Debian instructions and this should cover dependency installation.
85 These instructions have not yet been tested on Fedora.)
86
87 (Note: you might have to include additional repositories to a RPM-
88 based system, because rabbitmq-server might be not included in
89 official repositories. As an alternative, you can try installing
90 redis-server and configure it as celery broker)
91
92 Configure PostgreSQL
93 ~~~~~~~~~~~~~~~~~~~~
94
95 .. note::
96
97 MediaGoblin currently supports PostgreSQL and SQLite. The default is a
98 local SQLite database. This will "just work" for small deployments.
99
100 For medium to large deployments we recommend PostgreSQL.
101
102 If you don't want/need PostgreSQL, skip this section.
103
104 These are the packages needed for Debian Jessie (stable)::
105
106 sudo apt-get install postgresql postgresql-client python-psycopg2
107
108 These are the packages needed for an RPM-based system::
109
110 sudo yum install postgresql postgresql-server python-psycopg2
111
112 An rpm-based system also requires that you initialize and start the
113 PostgreSQL database with a few commands. The following commands are
114 not needed on a Debian-based platform, however::
115
116 sudo /usr/bin/postgresql-setup initdb
117 sudo systemctl enable postgresql
118 sudo systemctl start postgresql
119
120 The installation process will create a new *system* user named ``postgres``,
121 which will have privileges sufficient to manage the database. We will create a
122 new database user with restricted privileges and a new database owned by our
123 restricted database user for our MediaGoblin instance.
124
125 In this example, the database user will be ``mediagoblin`` and the database
126 name will be ``mediagoblin`` too.
127
128 We'll add these entities by first switching to the *postgres* account::
129
130 sudo su - postgres
131
132 This will change your prompt to a shell prompt, such as *-bash-4.2$*. Enter
133 the following *createuser* and *createdb* commands at that prompt. We'll
134 create the *mediagoblin* database user first::
135
136 # this command and the one that follows are run as the ``postgres`` user:
137 createuser -A -D mediagoblin
138
139 Then we'll create the database where all of our MediaGoblin data will be stored::
140
141 createdb -E UNICODE -O mediagoblin mediagoblin
142
143 where the first ``mediagoblin`` is the database owner and the second
144 ``mediagoblin`` is the database name.
145
146 Type ``exit`` to exit from the 'postgres' user account.::
147
148 exit
149
150 .. caution:: Where is the password?
151
152 These steps enable you to authenticate to the database in a password-less
153 manner via local UNIX authentication provided you run the MediaGoblin
154 application as a user with the same name as the user you created in
155 PostgreSQL.
156
157 More on this in :ref:`Drop Privileges for MediaGoblin <drop-privileges-for-mediagoblin>`.
158
159
160 .. _drop-privileges-for-mediagoblin:
161
162 Drop Privileges for MediaGoblin
163 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
164
165 MediaGoblin does not require special permissions or elevated
166 access to run. As such, the preferred way to run MediaGoblin is to
167 create a dedicated, unprivileged system user for the sole purpose of running
168 MediaGoblin. Running MediaGoblin processes under an unprivileged system user
169 helps to keep it more secure.
170
171 The following command (entered as root or with sudo) will create a
172 system account with a username of ``mediagoblin``. You may choose a different
173 username if you wish.
174
175 If you are using a Debian-based system, enter this command::
176
177 sudo useradd -c "GNU MediaGoblin system account" -d /var/lib/mediagoblin -m -r -g www-data mediagoblin
178
179 If you are using an RPM-based system, enter this command::
180
181 sudo useradd -c "GNU MediaGoblin system account" -d /var/lib/mediagoblin -m -r -g nginx mediagoblin
182
183 This will create a ``mediagoblin`` user and assign it to a group that is
184 associated with the web server. This will ensure that the web server can
185 read the media files (images, videos, etc.) that users upload.
186
187 We will also create a ``mediagoblin`` group and associate the mediagoblin
188 user with that group, as well::
189
190 sudo groupadd mediagoblin && sudo usermod --append -G mediagoblin mediagoblin
191
192 No password will be assigned to this account, and you will not be able
193 to log in as this user. To switch to this account, enter::
194
195 sudo su mediagoblin -s /bin/bash
196
197 To return to your regular user account after using the system account, type
198 ``exit``.
199
200 .. _create-mediagoblin-directory:
201
202 Create a MediaGoblin Directory
203 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
204
205 You should create a working directory for MediaGoblin. This document
206 assumes your local git repository will be located at
207 ``/srv/mediagoblin.example.org/mediagoblin/``.
208 Substitute your preferred local deployment path as needed.
209
210 Setting up the working directory requires that we first create the directory
211 with elevated privileges, and then assign ownership of the directory
212 to the unprivileged system account.
213
214 To do this, enter the following command, changing the defaults to suit your
215 particular requirements. On a Debian-based platform you will enter this::
216
217 sudo mkdir -p /srv/mediagoblin.example.org && sudo chown -hR mediagoblin:www-data /srv/mediagoblin.example.org
218
219 On an RPM-based distribution, enter this command::
220
221 sudo mkdir -p /srv/mediagoblin.example.org && sudo chown -hR mediagoblin:nginx /srv/mediagoblin.example.org
222
223 .. note::
224
225 Unless otherwise noted, the remainder of this document assumes that all
226 operations are performed using this unprivileged account.
227
228
229 Install MediaGoblin and Virtualenv
230 ----------------------------------
231
232 We will now switch to our 'mediagoblin' system account, and then set up
233 our MediaGoblin source code repository and its necessary services.
234 You should modify these commands to suit your own environment.
235
236 Change to the MediaGoblin directory that you just created::
237
238 sudo su mediagoblin -s /bin/bash # to change to the 'mediagoblin' account
239 $ cd /srv/mediagoblin.example.org
240
241 Clone the MediaGoblin repository and set up the git submodules::
242
243 $ git clone git://git.savannah.gnu.org/mediagoblin.git -b stable
244 $ cd mediagoblin
245 $ git submodule init && git submodule update
246
247 .. note::
248
249 The MediaGoblin repository used to be on gitorious.org, but since
250 gitorious.org shut down, we had to move. We are presently on
251 Savannah. You may need to update your git repository location::
252
253 $ git remote set-url origin git://git.savannah.gnu.org/mediagoblin.git
254
255 Set up the hacking environment::
256
257 $ ./bootstrap.sh && ./configure && make
258
259 (Note that if you'd prefer to run MediaGoblin with Python 3, pass in
260 `--with-python3` to the `./configure` command.)
261
262 Create and set the proper permissions on the ``user_dev`` directory.
263 This directory will be used to store uploaded media files::
264
265 $ mkdir user_dev && chmod 750 user_dev
266
267 Assuming you are going to deploy with FastCGI, you should also install
268 flup::
269
270 $ ./bin/easy_install flup
271
272 (Note, if you're running Python 2, which you probably are at this
273 point in MediaGoblin's development, you'll need to run:)
274
275 $ ./bin/easy_install flup==1.0.3.dev-20110405
276
277 The above provides an in-package install of ``virtualenv``. While this
278 is counter to the conventional ``virtualenv`` configuration, it is
279 more reliable and considerably easier to configure and illustrate. If
280 you're familiar with Python packaging you may consider deploying with
281 your preferred method.
282
283 .. note::
284
285 What if you don't want an in-package ``virtualenv``? Maybe you
286 have your own ``virtualenv``, or you are building a MediaGoblin
287 package for a distribution. There's no need necessarily for the
288 virtualenv produced by ``./configure && make`` by default other
289 than attempting to simplify work for developers and people
290 deploying by hiding all the virtualenv and bower complexity.
291
292 If you want to install all of MediaGoblin's libraries
293 independently, that's totally fine! You can pass the flag
294 ``--without-virtualenv`` which will skip this step.
295 But you will need to install all those libraries manually and make
296 sure they are on your ``PYTHONPATH`` yourself! (You can still use
297 ``python setup.py develop`` to install some of those libraries,
298 but note that no ``./bin/python`` will be set up for you via this
299 method, since no virtualenv is set up for you!)
300
301 This concludes the initial configuration of the MediaGoblin
302 environment. In the future, when you update your
303 codebase, you should also run::
304
305 $ git submodule update && ./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate
306
307 .. note::
308
309 Note: If you are running an active site, depending on your server
310 configuration, you may need to stop it first or the dbupdate command
311 may hang (and it's certainly a good idea to restart it after the
312 update)
313
314
315 Deploy MediaGoblin Services
316 ---------------------------
317
318 Edit site configuration
319 ~~~~~~~~~~~~~~~~~~~~~~~
320
321 A few basic properties must be set before MediaGoblin will work. First
322 make a copy of ``mediagoblin.ini`` and ``paste.ini`` for editing so the original
323 config files aren't lost (you likely won't need to edit the paste configuration,
324 but we'll make a local copy of it just in case)::
325
326 $ cp -av mediagoblin.ini mediagoblin_local.ini && cp -av paste.ini paste_local.ini
327
328 Then edit mediagoblin_local.ini:
329 - Set ``email_sender_address`` to the address you wish to be used as
330 the sender for system-generated emails
331 - Edit ``direct_remote_path``, ``base_dir``, and ``base_url`` if
332 your mediagoblin directory is not the root directory of your
333 site.
334
335
336 Configure MediaGoblin to use the PostgreSQL database
337 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
338
339 If you are using PostgreSQL, edit the ``[mediagoblin]`` section in your
340 ``mediagoblin_local.ini`` and put in::
341
342 sql_engine = postgresql:///mediagoblin
343
344 if you are running the MediaGoblin application as the same 'user' as the
345 database owner.
346
347
348 Update database data structures
349 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
350
351 Before you start using the database, you need to run::
352
353 $ ./bin/gmg dbupdate
354
355 to populate the database with the MediaGoblin data structures.
356
357
358 Test the Server
359 ~~~~~~~~~~~~~~~
360
361 At this point MediaGoblin should be properly installed. You can
362 test the deployment with the following command::
363
364 $ ./lazyserver.sh --server-name=broadcast
365
366 You should be able to connect to the machine on port 6543 in your
367 browser to confirm that the service is operable.
368
369 The next series of commands will need to be run as a privileged user. Type
370 exit to return to the root/sudo account.::
371
372 exit
373
374 .. _webserver-config:
375
376
377 FastCGI and nginx
378 ~~~~~~~~~~~~~~~~~
379
380 This configuration example will use Nginx, however, you may
381 use any webserver of your choice as long as it supports the FastCGI
382 protocol. If you do not already have a web server, consider Nginx, as
383 the configuration files may be more clear than the
384 alternatives.
385
386 Create a configuration file at
387 ``/srv/mediagoblin.example.org/nginx.conf`` and create a symbolic link
388 into a directory that will be included in your ``nginx`` configuration
389 (e.g. "``/etc/nginx/sites-enabled`` or ``/etc/nginx/conf.d``) with
390 one of the following commands.
391
392 On a DEB-based system (e.g Debian, gNewSense, Trisquel, *buntu, and
393 derivatives) issue the following commands::
394
395 sudo ln -s /srv/mediagoblin.example.org/nginx.conf /etc/nginx/sites-enabled/
396 sudo systemctl enable nginx
397
398 On a RPM-based system (e.g. Fedora, RedHat, and derivatives) issue the
399 following commands::
400
401 sudo ln -s /srv/mediagoblin.example.org/nginx.conf /etc/nginx/conf.d/
402 sudo systemctl enable nginx
403
404 You can modify these commands and locations depending on your preferences and
405 the existing configuration of your Nginx instance. The contents of
406 this ``nginx.conf`` file should be modeled on the following::
407
408 server {
409 #################################################
410 # Stock useful config options, but ignore them :)
411 #################################################
412 include /etc/nginx/mime.types;
413
414 autoindex off;
415 default_type application/octet-stream;
416 sendfile on;
417
418 # Gzip
419 gzip on;
420 gzip_min_length 1024;
421 gzip_buffers 4 32k;
422 gzip_types text/plain application/x-javascript text/javascript text/xml text/css;
423
424 #####################################
425 # Mounting MediaGoblin stuff
426 # This is the section you should read
427 #####################################
428
429 # Change this to update the upload size limit for your users
430 client_max_body_size 8m;
431
432 # prevent attacks (someone uploading a .txt file that the browser
433 # interprets as an HTML file, etc.)
434 add_header X-Content-Type-Options nosniff;
435
436 server_name mediagoblin.example.org www.mediagoblin.example.org;
437 access_log /var/log/nginx/mediagoblin.example.access.log;
438 error_log /var/log/nginx/mediagoblin.example.error.log;
439
440 # MediaGoblin's stock static files: CSS, JS, etc.
441 location /mgoblin_static/ {
442 alias /srv/mediagoblin.example.org/mediagoblin/mediagoblin/static/;
443 }
444
445 # Instance specific media:
446 location /mgoblin_media/ {
447 alias /srv/mediagoblin.example.org/mediagoblin/user_dev/media/public/;
448 }
449
450 # Theme static files (usually symlinked in)
451 location /theme_static/ {
452 alias /srv/mediagoblin.example.org/mediagoblin/user_dev/theme_static/;
453 }
454
455 # Plugin static files (usually symlinked in)
456 location /plugin_static/ {
457 alias /srv/mediagoblin.example.org/mediagoblin/user_dev/plugin_static/;
458 }
459
460 # Mounting MediaGoblin itself via FastCGI.
461 location / {
462 fastcgi_pass 127.0.0.1:26543;
463 include /etc/nginx/fastcgi_params;
464
465 # our understanding vs Nginx's handling of script_name vs
466 # path_info don't match :)
467 fastcgi_param PATH_INFO $fastcgi_script_name;
468 fastcgi_param SCRIPT_NAME "";
469 }
470 }
471
472 The first four ``location`` directives instruct Nginx to serve the
473 static and uploaded files directly rather than through the MediaGoblin
474 process. This approach is faster and requires less memory.
475
476 .. note::
477
478 The user who owns the Nginx process, normally ``www-data`` or ``nginx``,
479 requires execute permission on the directories ``static``,
480 ``public``, ``theme_static`` and ``plugin_static`` plus all their
481 parent directories. This user also requires read permission on all
482 the files within these directories. This is normally the default.
483
484 Nginx is now configured to serve the MediaGoblin application. Perform a quick
485 test to ensure that this configuration works::
486
487 nginx -t
488
489 If you encounter any errors, review your Nginx configuration files, and try to
490 resolve them. If you do not encounter any errors, you can start your Nginx
491 server with one of the following commands (depending on your environment)::
492
493 sudo /etc/init.d/nginx restart
494 sudo /etc/rc.d/nginx restart
495 sudo systemctl restart nginx
496
497 Now start MediaGoblin. Use the following command sequence as an
498 example::
499
500 cd /srv/mediagoblin.example.org/mediagoblin/
501 su mediagoblin -s /bin/bash
502 ./lazyserver.sh --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543
503
504 Visit the site you've set up in your browser by visiting
505 <http://mediagoblin.example.org>. You should see MediaGoblin!
506
507 .. note::
508
509 The configuration described above is sufficient for development and
510 smaller deployments. However, for larger production deployments
511 with larger processing requirements, see the
512 ":doc:`production-deployments`" documentation.
513
514
515 Apache
516 ~~~~~~
517
518 Instructions and scripts for running MediaGoblin on an Apache server
519 can be found on the `MediaGoblin wiki <http://wiki.mediagoblin.org/Deployment>`_.
520
521
522 Should I Keep Open Registration Enabled?
523 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
524
525 Unfortunately, in this current release of MediaGoblin we are suffering
526 from spammers registering to public instances en masse. As such, you
527 may want to either:
528
529 a) Disable registration on your instance and just make
530 accounts for people you know and trust (eg via the `gmg adduser`
531 command). You can disable registration in your mediagoblin.ini
532 like so::
533
534 [mediagoblin]
535 allow_registration = false
536
537 b) Enable a CAPTCHA plugin. But unfortunately, though some CAPTCHA
538 plugins exist, for various reasons we do not have any general
539 recommendations we can make at this point.
540
541 We hope to have a better solution to this situation shortly. We
542 apologize for the inconvenience in the meanwhile.
543
544
545 Security Considerations
546 ~~~~~~~~~~~~~~~~~~~~~~~
547
548 .. warning::
549
550 The directory ``user_dev/crypto/`` contains some very
551 sensitive files.
552 Especially the ``itsdangeroussecret.bin`` is very important
553 for session security. Make sure not to leak its contents anywhere.
554 If the contents gets leaked nevertheless, delete your file
555 and restart the server, so that it creates a new secret key.
556 All previous sessions will be invalidated.
557
558 ..
559 Local variables:
560 fill-column: 70
561 End: