From 62a565e13b2a839f28c1382a72f17b165d546e4e Mon Sep 17 00:00:00 2001 From: Ben Sturmfels Date: Wed, 29 Apr 2020 02:19:16 +1000 Subject: [PATCH] Review and update the deploment docs for Debian 10 and CentOS 8 [#5593]. Mostly a thorough test and cleanup. The significant change is moving the systemd config into the deployment docs an the registration/security considerations to production considerations docs. --- Dockerfile-debian-python2-sqlite | 2 +- Dockerfile-debian-python3-sqlite | 2 +- Dockerfile-fedora-python3-sqlite | 9 +- docs/source/siteadmin/deploying.rst | 449 +++++++++++------- .../siteadmin/production-deployments.rst | 159 ++----- 5 files changed, 303 insertions(+), 318 deletions(-) diff --git a/Dockerfile-debian-python2-sqlite b/Dockerfile-debian-python2-sqlite index 155ceda5..977923b7 100644 --- a/Dockerfile-debian-python2-sqlite +++ b/Dockerfile-debian-python2-sqlite @@ -54,7 +54,7 @@ RUN groupadd --system mediagoblin --gid 1024 && adduser www-data mediagoblin USER www-data -RUN git clone --depth=1 git://git.savannah.gnu.org/mediagoblin.git -b master . +RUN git clone --depth=1 git://git.savannah.gnu.org/mediagoblin.git --branch master . RUN ./bootstrap.sh RUN VIRTUALENV_FLAGS='--system-site-packages' ./configure --without-python3 diff --git a/Dockerfile-debian-python3-sqlite b/Dockerfile-debian-python3-sqlite index 5c2c70d3..c6084345 100644 --- a/Dockerfile-debian-python3-sqlite +++ b/Dockerfile-debian-python3-sqlite @@ -138,7 +138,7 @@ USER www-data # version, so that local changes are immediately available to the running # container. Not as easy as it sounds. We have this working with docker-compose, # but still uses upstream MediaGoblin for the build. -RUN git clone --depth=1 git://git.savannah.gnu.org/mediagoblin.git -b master . +RUN git clone --depth=1 git://git.savannah.gnu.org/mediagoblin.git --branch master . RUN ./bootstrap.sh RUN VIRTUALENV_FLAGS='--system-site-packages' ./configure diff --git a/Dockerfile-fedora-python3-sqlite b/Dockerfile-fedora-python3-sqlite index ee337866..6cabcb1a 100644 --- a/Dockerfile-fedora-python3-sqlite +++ b/Dockerfile-fedora-python3-sqlite @@ -4,18 +4,17 @@ FROM fedora:31 -RUN yum -y update -RUN yum -y install \ +RUN dnf -y install \ automake \ gcc \ -git \ +git-core \ make \ nodejs \ npm \ python3-devel \ virtualenv -RUN yum -y install \ +RUN dnf -y install \ findutils \ python3-alembic \ python3-celery \ @@ -70,7 +69,7 @@ RUN chown root:www-data /var/www USER www-data -RUN git clone --depth=1 git://git.savannah.gnu.org/mediagoblin.git -b master . +RUN git clone --depth=1 git://git.savannah.gnu.org/mediagoblin.git --branch master . RUN ./bootstrap.sh RUN VIRTUALENV_FLAGS='--system-site-packages' ./configure diff --git a/docs/source/siteadmin/deploying.rst b/docs/source/siteadmin/deploying.rst index 40254e2c..d6dcd43c 100644 --- a/docs/source/siteadmin/deploying.rst +++ b/docs/source/siteadmin/deploying.rst @@ -1,6 +1,6 @@ .. MediaGoblin Documentation - Written in 2011, 2012, 2013 by MediaGoblin contributors + Written in 2011, 2012, 2013, 2020 by MediaGoblin contributors To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to @@ -17,20 +17,20 @@ Deploying MediaGoblin ===================== -GNU MediaGoblin is fairly new, and so at the time of writing there aren't -easy package-manager-friendly methods to install it. However, doing a basic -install isn't too complex in and of itself. Following this deployment guide -will take you step-by-step through setting up your own instance of MediaGoblin. +Following this deployment guide will take you step-by-step through +setting up your own instance of MediaGoblin. -Of course, when it comes to setting up web applications like MediaGoblin, -there's an almost infinite way to deploy things, so for now, we'll keep it -simple with some assumptions. We recommend a setup that combines MediaGoblin + -virtualenv + Waitress + Nginx on a .deb- or .rpm-based GNU/Linux distro. +MediaGoblin most likely isn't yet available from your operating +system's package manage, however, a basic install isn't too complex in +and of itself. We recommend a setup that combines +MediaGoblin, virtualenv, Waitress and Nginx on a .deb or .rpm-based +GNU/Linux distribution. -Other deployment options (e.g., deploying on FreeBSD, Arch Linux, using -Apache, etc.) are possible, though! If you'd prefer a different deployment -approach, see our -`Deployment wiki page `_. +Experts may of course choose other deployment options, including +Apache. See our `Deployment wiki page +`_ for for more details. +Please note that we are not able to provide support for these +alternative deployment options. .. note:: @@ -57,59 +57,73 @@ Dependencies MediaGoblin has the following core dependencies: -- Python 2.7 or Python 3.4+ +- Python 3.4+ (Python 2.7 is supported, but not recommended) - `python3-lxml `_ - `git `_ - `SQLite `_/`PostgreSQL `_ - `Python Imaging Library `_ (PIL) - `virtualenv `_ -- `nodejs `_ +- `Node.js `_ -On a DEB-based system (e.g Debian, gNewSense, Trisquel, Ubuntu, and -derivatives) issue the following command:: +On a DEB-based system (e.g Debian, Trisquel, Ubuntu and +derivatives) issue the following commands:: - sudo apt-get install git-core python python3-dev python3-lxml \ - python3-imaging python3-virtualenv npm nodejs-legacy automake \ - nginx rabbitmq-server + # Debian, Trisquel, Ubuntu and derivatives (Hereafter Debian and co.) + sudo apt update + sudo apt install automake git nodejs npm python3-dev python3-gi \ + python3-gst-1.0 python3-lxml python3-pil virtualenv -On a RPM-based system (e.g. Fedora, RedHat, and derivatives) issue the -following command:: + # Fedora, CentOS, RHEL and derivatives (Hereafter Fedora and co.) + dnf install automake gcc git-core make nodejs npm python3-devel \ + python3-lxml python3-pillow virtualenv - sudo yum install python3-paste-deploy python3-paste-script \ - git-core python python3-devel python3-lxml python3-imaging \ - python3-virtualenv npm automake nginx rabbitmq-server +.. note:: -(Note: MediaGoblin now uses Python 3 by default. To use Python 2, you may instead -substitute from "python3" to "python" for most package names in the -Debian instructions and this should cover dependency installation. -These instructions have not yet been tested on Fedora.) + MediaGoblin now uses Python 3 by default. To use Python 2, you may + instead substitute from "python3" to "python" for most package + names in the Debian instructions and this should cover dependency + installation. Python 2 installation has not been tested on Fedora + and co. -(Note: you might have to include additional repositories to a RPM- -based system, because rabbitmq-server might be not included in -official repositories. As an alternative, you can try installing -redis-server and configure it as celery broker) +For a production deployment, you'll also need Nginx as frontend web +server and RabbitMQ to store the media processing queue:: -Configure PostgreSQL -~~~~~~~~~~~~~~~~~~~~ + # Debian and co. + sudo apt install nginx-light rabbitmq-server + + # Fedora and co. + dnf install nginx rabbitmq .. note:: - MediaGoblin currently supports PostgreSQL and SQLite. The default is a - local SQLite database. This will "just work" for small deployments. + You might have to enable additional repositories under Fedora and + co. because rabbitmq-server might be not included in official + repositories. That looks like this for CentOS:: + + dnf install centos-release-rabbitmq-38.noarc - For medium to large deployments we recommend PostgreSQL. + As an alternative, you can try installing redis-server and + configure it as celery broker. + +Configure PostgreSQL +~~~~~~~~~~~~~~~~~~~~ + +.. note:: - If you don't want/need PostgreSQL, skip this section. + MediaGoblin currently supports PostgreSQL and SQLite. The default + is a local SQLite database. This will "just work" for small + deployments. For medium to large deployments we recommend + PostgreSQL. If you don't want/need PostgreSQL, skip this section. These are the packages needed for Debian Jessie (stable):: - sudo apt-get install postgresql postgresql-client python3-psycopg2 + sudo apt install postgresql python3-psycopg2 These are the packages needed for an RPM-based system:: - sudo yum install postgresql postgresql-server python3-psycopg2 + sudo dnf install postgresql postgresql-server python3-psycopg2 -An rpm-based system also requires that you initialize and start the +Fedora and co. also requires that you initialize and start the PostgreSQL database with a few commands. The following commands are not needed on a Debian-based platform, however:: @@ -123,29 +137,13 @@ new database user with restricted privileges and a new database owned by our restricted database user for our MediaGoblin instance. In this example, the database user will be ``mediagoblin`` and the database -name will be ``mediagoblin`` too. +name will be ``mediagoblin`` too. We'll first at the user:: -We'll add these entities by first switching to the *postgres* account:: - - sudo su - postgres - -This will change your prompt to a shell prompt, such as *-bash-4.2$*. Enter -the following *createuser* and *createdb* commands at that prompt. We'll -create the *mediagoblin* database user first:: - - # this command and the one that follows are run as the ``postgres`` user: - createuser -A -D mediagoblin + sudo --login --user=postgres createuser --no-createdb mediagoblin Then we'll create the database where all of our MediaGoblin data will be stored:: - createdb -E UNICODE -O mediagoblin mediagoblin - -where the first ``mediagoblin`` is the database owner and the second -``mediagoblin`` is the database name. - -Type ``exit`` to exit from the 'postgres' user account.:: - - exit + sudo --login --user=postgres createdb --encoding=UTF8 --owner=mediagoblin mediagoblin .. caution:: Where is the password? @@ -168,34 +166,35 @@ create a dedicated, unprivileged system user for the sole purpose of running MediaGoblin. Running MediaGoblin processes under an unprivileged system user helps to keep it more secure. -The following command (entered as root or with sudo) will create a -system account with a username of ``mediagoblin``. You may choose a different -username if you wish. +The following command will create a system account with a username of +``mediagoblin``. If you are using a Debian-based system, enter this command:: - sudo useradd -c "GNU MediaGoblin system account" -d /var/lib/mediagoblin -m -r -g www-data mediagoblin - -If you are using an RPM-based system, enter this command:: + # Debian and co. + sudo useradd --system --create-home --home-dir /var/lib/mediagoblin --group www-data --comment 'GNU MediaGoblin system account' mediagoblin - sudo useradd -c "GNU MediaGoblin system account" -d /var/lib/mediagoblin -m -r -g nginx mediagoblin + # Fedora and co. + sudo useradd --system --create-home --home-dir /var/lib/mediagoblin --group nginx --comment 'GNU MediaGoblin system account' mediagoblin This will create a ``mediagoblin`` user and assign it to a group that is associated with the web server. This will ensure that the web server can -read the media files (images, videos, etc.) that users upload. +read the media files that users upload (images, videos, etc.) -We will also create a ``mediagoblin`` group and associate the mediagoblin -user with that group, as well:: +Many operating systems will automatically create a group +``mediagoblin`` to go with the new user ``mediagoblin``, but just to +be sure:: - sudo groupadd mediagoblin && sudo usermod --append -G mediagoblin mediagoblin + sudo groupadd --force mediagoblin + sudo usermod --append --groups mediagoblin mediagoblin No password will be assigned to this account, and you will not be able to log in as this user. To switch to this account, enter:: - sudo su mediagoblin -s /bin/bash + sudo su mediagoblin --shell=/bin/bash To return to your regular user account after using the system account, type -``exit``. +``exit`` or ``Ctrl-d``. .. _create-mediagoblin-directory: @@ -211,19 +210,16 @@ Setting up the working directory requires that we first create the directory with elevated privileges, and then assign ownership of the directory to the unprivileged system account. -To do this, enter the following command, changing the defaults to suit your -particular requirements. On a Debian-based platform you will enter this:: - - sudo mkdir -p /srv/mediagoblin.example.org && sudo chown -hR mediagoblin:www-data /srv/mediagoblin.example.org - -On an RPM-based distribution, enter this command:: - - sudo mkdir -p /srv/mediagoblin.example.org && sudo chown -hR mediagoblin:nginx /srv/mediagoblin.example.org +To do this, enter the following commands, changing the defaults to suit your +particular requirements:: -.. note:: + # Debian and co. + sudo mkdir --parents /srv/mediagoblin.example.org + sudo chown --no-dereference --recursive mediagoblin:www-data /srv/mediagoblin.example.org - Unless otherwise noted, the remainder of this document assumes that all - operations are performed using this unprivileged account. + # Fedora and co. + sudo mkdir --parents /srv/mediagoblin.example.org + sudo chown --no-dereference --recursive mediagoblin:nginx /srv/mediagoblin.example.org Install MediaGoblin and Virtualenv @@ -233,41 +229,46 @@ We will now switch to our 'mediagoblin' system account, and then set up our MediaGoblin source code repository and its necessary services. You should modify these commands to suit your own environment. -Change to the MediaGoblin directory that you just created:: +Switch to the ``mediagoblin`` unprivileged user and change to the +MediaGoblin directory that you just created:: - sudo su mediagoblin -s /bin/bash # to change to the 'mediagoblin' account + sudo su mediagoblin --shell=/bin/bash $ cd /srv/mediagoblin.example.org +.. note:: + + Unless otherwise noted, the remainder of this document assumes that all + operations are performed using the unprivileged ``mediagoblin`` + account, indicated by the ``$`` prefix. + Clone the MediaGoblin repository and set up the git submodules:: - $ git clone https://git.savannah.gnu.org/git/mediagoblin.git -b stable + $ git clone --depth=1 https://git.savannah.gnu.org/git/mediagoblin.git --branch stable $ cd mediagoblin $ git submodule init && git submodule update -.. note:: - - The MediaGoblin repository used to be on gitorious.org, but since - gitorious.org shut down, we had to move. We are presently on - Savannah. You may need to update your git repository location:: +Set up the environment:: - $ git remote set-url origin https://git.savannah.gnu.org/git/mediagoblin.git + $ ./bootstrap.sh && VIRTUALENV_FLAGS='--system-site-packages' \ + ./configure && make -Set up the hacking environment:: - - $ ./bootstrap.sh && ./configure && make +.. note:: -(Note that if you'd prefer to run MediaGoblin with Python 2, pass in -``--without-python3`` to the ``./configure`` command.) + If you'd prefer to run MediaGoblin with Python 2, pass in + ``--without-python3`` to the ``./configure`` command. Create and set the proper permissions on the ``user_dev`` directory. This directory will be used to store uploaded media files:: - $ mkdir user_dev && chmod 750 user_dev + $ mkdir --mode='u=rwx,g=rx,o=' user_dev + $ chmod g+s user_dev This concludes the initial configuration of the MediaGoblin environment. In the future, when you update your codebase, you should also run:: + sudo su mediagoblin --shell=/bin/bash + $ cd /srv/mediagoblin.example.org $ git submodule update && ./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate .. note:: @@ -284,34 +285,30 @@ Deploy MediaGoblin Services Edit site configuration ~~~~~~~~~~~~~~~~~~~~~~~ -A few basic properties must be set before MediaGoblin will work. First -make a copy of ``paste.ini`` for editing so the original -config files aren't lost (you likely won't need to edit the paste configuration, -but we'll make a local copy of it just in case):: +Edit ``mediagoblin.ini`` and update ``email_sender_address`` to the +address you wish to be used as the sender for system-generated emails. - $ cp -av paste.ini paste_local.ini +.. note:: -``mediagoblin.ini`` does not need to be copied, because original config is -stored in ``mediagoblin.example.ini``. + If you're changing the MediaGoblin directories or URL prefix, you + may need to edit ``direct_remote_path``, ``base_dir``, and + ``base_url``. -Then edit ``mediagoblin.ini``: - - Set ``email_sender_address`` to the address you wish to be used as - the sender for system-generated emails - - Edit ``direct_remote_path``, ``base_dir``, and ``base_url`` if - your mediagoblin directory is not the root directory of your - site. +.. note:: + The default config is stored in ``mediagoblin.example.ini`` in case + you ever need it. Configure MediaGoblin to use the PostgreSQL database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are using PostgreSQL, edit the ``[mediagoblin]`` section in your -``mediagoblin.ini`` and put in:: +``mediagoblin.ini`` and remove the `# ` prefix on the line containing:: sql_engine = postgresql:///mediagoblin -if you are running the MediaGoblin application as the same 'user' as the -database owner. +This assumes you are running the MediaGoblin application under the +same system account and database account; both ``mediagoblin``. Update database data structures @@ -324,6 +321,16 @@ Before you start using the database, you need to run:: to populate the database with the MediaGoblin data structures. +Create an admin account +~~~~~~~~~~~~~~~~~~~~~~~ + +Create a MediaGoblin account with full administration access. Provide +your own email address and enter a secure password when prompted:: + + $ ./bin/gmg adduser --username you --email you@example.com + $ ./bin/gmg makeadmin you + + Test the Server ~~~~~~~~~~~~~~~ @@ -333,45 +340,45 @@ test the deployment with the following command:: $ ./lazyserver.sh --server-name=broadcast You should be able to connect to the machine on port 6543 in your -browser to confirm that the service is operable. +browser to confirm that the service is operable. You should also be +able to log in with the admin username and password. -The next series of commands will need to be run as a privileged user. Type -exit to return to the root/sudo account.:: +Type ``Ctrl-c`` to exit the above server test. - exit +The next series of commands will need to be run as a privileged user. +To return to your regular user account after using the system account, +type ``exit`` or ``Ctrl-d``. .. _webserver-config: -Waitress and nginx +Waitress and Nginx ~~~~~~~~~~~~~~~~~~ -This configuration example will use Nginx, however, you may -use any webserver of your choice. If you do not already have -a web server, consider Nginx, as the configuration files may -be more clear than the alternatives. +This configuration example will use Nginx, however, you may use any +webserver of your choice. If you do not already have a web server, +consider Nginx, as the configuration files may be more clear than the +alternatives. Create a configuration file at ``/srv/mediagoblin.example.org/nginx.conf`` and create a symbolic link into a directory that will be included in your ``nginx`` configuration -(e.g. "``/etc/nginx/sites-enabled`` or ``/etc/nginx/conf.d``) with -one of the following commands. - -On a DEB-based system (e.g Debian, Trisquel, Ubuntu and derivatives) -issue the following commands:: +(e.g. "``/etc/nginx/sites-enabled`` or ``/etc/nginx/conf.d``) with the +following commands:: - sudo ln -s /srv/mediagoblin.example.org/nginx.conf /etc/nginx/sites-enabled/ + # Debian and co. + sudo ln --symbolic /srv/mediagoblin.example.org/nginx.conf /etc/nginx/sites-enabled/mediagoblin.conf + sudo rm --force /etc/nginx/sites-enabled/default sudo systemctl enable nginx -On a RPM-based system (e.g. Fedora, RedHat, and derivatives) issue the -following commands:: - - sudo ln -s /srv/mediagoblin.example.org/nginx.conf /etc/nginx/conf.d/ + # Fedora and co. + sudo ln -s /srv/mediagoblin.example.org/nginx.conf /etc/nginx/conf.d/mediagoblin.conf sudo systemctl enable nginx -You can modify these commands and locations depending on your preferences and -the existing configuration of your Nginx instance. The contents of -this ``nginx.conf`` file should be modeled on the following:: +You can modify these commands and locations depending on your +preferences and the existing configuration of your Nginx instance. The +contents of this ``/srv/mediagoblin.example.org/nginx.conf`` file +should be modeled on the following:: server { ################################################# @@ -446,76 +453,150 @@ process. This approach is faster and requires less memory. Nginx is now configured to serve the MediaGoblin application. Perform a quick test to ensure that this configuration works:: - nginx -t + sudo nginx -t If you encounter any errors, review your Nginx configuration files, and try to resolve them. If you do not encounter any errors, you can start your Nginx -server with one of the following commands (depending on your environment):: +server (may vary depending on your operating system):: - sudo /etc/init.d/nginx restart - sudo /etc/rc.d/nginx restart sudo systemctl restart nginx -Now start MediaGoblin. Use the following command sequence as an -example:: +Now start MediaGoblin to test your Nginx configuration:: - cd /srv/mediagoblin.example.org/mediagoblin/ - su mediagoblin -s /bin/bash - ./lazyserver.sh --server-name=main + sudo su mediagoblin --shell=/bin/bash + $ cd /srv/mediagoblin.example.org/mediagoblin/ + $ ./lazyserver.sh --server-name=main -Visit the site you've set up in your browser by visiting -. You should see MediaGoblin! +You should be able to connect to the machine on port 80 in your +browser to confirm that the service is operable. If this is the +machine in front of you, visit or if it is a +remote server visit the URL or IP address provided to you by your +hosting provider. You should see MediaGoblin; this time via Nginx! .. note:: - - The configuration described above is sufficient for development and - smaller deployments. However, for larger production deployments - with larger processing requirements, see the - ":doc:`production-deployments`" documentation. + If you see an Nginx placeholder page, you may need to remove the + Nginx default configuration, or explictly set a ``server_name`` + directive in the Nginx config. -Apache -~~~~~~ +Type ``Ctrl-c`` to exit the above server test and ``exit`` or +``Ctrl-d`` to exit the mediagoblin shell. -Instructions and scripts for running MediaGoblin on an Apache server -can be found on the `MediaGoblin wiki `_. +.. _create-log-file-dir: -Should I Keep Open Registration Enabled? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Create the directory for your log file: +--------------------------------------- -Unfortunately, in this current release of MediaGoblin we are suffering -from spammers registering to public instances en masse. As such, you -may want to either: +Production logs for the MediaGoblin application are kept in the +``/var/log/mediagoblin`` directory. Create the directory and give it the +proper permissions:: -a) Disable registration on your instance and just make - accounts for people you know and trust (eg via the `gmg adduser` - command). You can disable registration in your mediagoblin.ini - like so:: + sudo mkdir --parents /var/log/mediagoblin + sudo chown --no-dereference --recursive mediagoblin:mediagoblin /var/log/mediagoblin - [mediagoblin] - allow_registration = false -b) Enable a CAPTCHA plugin. But unfortunately, though some CAPTCHA - plugins exist, for various reasons we do not have any general - recommendations we can make at this point. +.. _systemd-service-files: -We hope to have a better solution to this situation shortly. We -apologize for the inconvenience in the meanwhile. +Run MediaGoblin as a system service +----------------------------------- +To ensure MediaGoblin is automatically started and restarted in case of +problems, we need to run it as a system service. If your operating system uses +Systemd, you can use Systemd ``service files`` to manage both the Celery and +Paste processes. -Security Considerations -~~~~~~~~~~~~~~~~~~~~~~~ +Place the following service files in the ``/etc/systemd/system/`` +directory. The first file should be named +``mediagoblin-celeryd.service``. Be sure to modify it to suit your +environment's setup: + +.. code-block:: bash + + # Set the WorkingDirectory and Environment values to match your environment. + [Unit] + Description=MediaGoblin Celeryd + + [Service] + User=mediagoblin + Group=mediagoblin + Type=simple + WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin + Environment=MEDIAGOBLIN_CONFIG=/srv/mediagoblin.example.org/mediagoblin/mediagoblin.ini \ + CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery + ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/celery worker \ + --logfile=/var/log/mediagoblin/celery.log \ + --loglevel=INFO + + [Install] + WantedBy=multi-user.target + + +The second file should be named ``mediagoblin-paster.service``: + +.. code-block:: bash + + # Set the WorkingDirectory and Environment values to match your environment. + [Unit] + Description=Mediagoblin + + [Service] + Type=simple + User=mediagoblin + Group=mediagoblin + Environment=CELERY_ALWAYS_EAGER=false + WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin + ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ + /srv/mediagoblin.example.org/mediagoblin/paste.ini \ + --log-file=/var/log/mediagoblin/mediagoblin.log \ + --server-name=main + + [Install] + WantedBy=multi-user.target + + +Enable these processes to start at boot by entering:: -.. warning:: + sudo systemctl enable mediagoblin-celeryd.service && sudo systemctl enable mediagoblin-paster.service + + +Start the processes for the current session with:: + + sudo systemctl start mediagoblin-celeryd.service + sudo systemctl start mediagoblin-paster.service + + +If either command above gives you an error, you can investigate the cause of +the error by entering either of:: + + sudo systemctl status mediagoblin-celeryd.service + sudo systemctl status mediagoblin-paster.service + +The above ``systemctl status`` command is also useful if you ever want to +confirm that a process is still running. If you make any changes to the service +files, you can reload the service files by entering:: + + sudo systemctl daemon-reload + +After entering that command, you can attempt to start the Celery or Paste +processes again using ``restart`` instead of ``start``. + +Assuming the above was successful, you should now have a MediaGoblin +server that will continue to operate, even after being restarted. +Great job! + +.. note:: + + The configuration described above is sufficient for development and + smaller deployments. However, for larger production deployments + with larger processing requirements, see the + ":doc:`production-deployments`" documentation. + +.. note:: - The directory ``user_dev/crypto/`` contains some very - sensitive files. - Especially the ``itsdangeroussecret.bin`` is very important - for session security. Make sure not to leak its contents anywhere. - If the contents gets leaked nevertheless, delete your file - and restart the server, so that it creates a new secret key. - All previous sessions will be invalidated. + This configuration supports upload of images only, but MediaGoblin + also supports other types of media, such as audio, video, PDFs and + 3D models. For details, see ":doc:`media-types-chapter`". .. Local variables: diff --git a/docs/source/siteadmin/production-deployments.rst b/docs/source/siteadmin/production-deployments.rst index 681a3cda..511245f9 100644 --- a/docs/source/siteadmin/production-deployments.rst +++ b/docs/source/siteadmin/production-deployments.rst @@ -19,142 +19,48 @@ This document contains a number of suggestions for deploying MediaGoblin in actual production environments. Consider ":doc:`deploying`" for a basic overview of how to deploy MediaGoblin. -Deploy with paste ------------------ -The MediaGoblin WSGI application instance you get with ``./lazyserver.sh`` is -not ideal for a production MediaGoblin deployment. Ideally, you should be able -to use a Systemd service file or an init script to launch and restart the -MediaGoblin process. +Should I Keep Open Registration Enabled? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -We will explore setting up MediaGoblin Systemd service files and init scripts, -but first we need to create the directory that will store the MediaGoblin logs. +Unfortunately, in this current release of MediaGoblin we are suffering +from spammers registering to public instances en masse. As such, you +may want to either: +a) Disable registration on your instance and just make + accounts for people you know and trust (eg via the `gmg adduser` + command). You can disable registration in your mediagoblin.ini + like so:: -.. _create-log-file-dir: + [mediagoblin] + allow_registration = false -Create the directory for your log file: ---------------------------------------- +b) Enable a CAPTCHA plugin. But unfortunately, though some CAPTCHA + plugins exist, for various reasons we do not have any general + recommendations we can make at this point. -Production logs for the MediaGoblin application are kept in the -``/var/log/mediagoblin`` directory. Create the directory and give it the -proper permissions:: +We hope to have a better solution to this situation shortly. We +apologize for the inconvenience in the meanwhile. - sudo mkdir -p /var/log/mediagoblin && sudo chown -hR mediagoblin:mediagoblin /var/log/mediagoblin +Security Considerations +~~~~~~~~~~~~~~~~~~~~~~~ -.. _systemd-service-files: +.. warning:: -Use Systemd service files -------------------------- + The directory ``user_dev/crypto/`` contains some very + sensitive files. + Especially the ``itsdangeroussecret.bin`` is very important + for session security. Make sure not to leak its contents anywhere. + If the contents gets leaked nevertheless, delete your file + and restart the server, so that it creates a new secret key. + All previous sessions will be invalidated. -If your operating system uses Systemd, you can use Systemd ``service files`` -to manage both the Celery and Paste processes. Place the following service -files in the ``/etc/systemd/system/`` directory. - -The first file should be named ``mediagoblin-celeryd.service``. Be sure to -modify it to suit your environment's setup: - -.. code-block:: bash - - # Set the WorkingDirectory, Environment and ExecStart values to match your environment. - # If using Debian/Ubuntu, mkdir and chown are located in /bin/mkdir and /bin/chown, respectively. - # If using Fedora/CentOS/Red Hat, mkdir and chown are located in /usr/bin/mkdir and /usr/bin/chown, respectively. - - [Unit] - Description=MediaGoblin Celeryd - - [Service] - User=mediagoblin - Group=mediagoblin - Type=simple - WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin - # Start mg-celeryd process as root, then switch to mediagoblin user/group - # (This is needed to run the ExecStartPre commands) - PermissionsStartOnly=true - # Create directory for PID (if needed) and set ownership - ExecStartPre=/bin/mkdir -p /run/mediagoblin - ExecStartPre=/bin/chown -hR mediagoblin:mediagoblin /run/mediagoblin - # Celery process will run as the `mediagoblin` user after start. - Environment=MEDIAGOBLIN_CONFIG=/srv/mediagoblin.example.org/mediagoblin/mediagoblin.ini \ - CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery - ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/celery worker \ - --logfile=/var/log/mediagoblin/celery.log \ - --loglevel=INFO - PIDFile=/run/mediagoblin/mediagoblin-celeryd.pid - - [Install] - WantedBy=multi-user.target - - -The second file should be named ``mediagoblin-paster.service``: - - -.. code-block:: bash - - # Set the WorkingDirectory, Environment and ExecStart values to match your environment. - # If using Debian/*buntu, mkdir and chown are located in /bin/mkdir and /bin/chown, respectively. - # If using Fedora/CentOS/Red Hat, mkdir and chown are located in /usr/bin/mkdir and /usr/bin/chown, respectively. - [Unit] - Description=Mediagoblin - - [Service] - Type=forking - User=mediagoblin - Group=mediagoblin - Environment=CELERY_ALWAYS_EAGER=false - WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin - # Start mg-paster process as root, then switch to mediagoblin user/group - PermissionsStartOnly=true - ExecStartPre=-/bin/mkdir -p /run/mediagoblin - ExecStartPre=/bin/chown -hR mediagoblin:mediagoblin /run/mediagoblin - - ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ - /srv/mediagoblin.example.org/mediagoblin/paste_local.ini \ - --pid-file=/var/run/mediagoblin/mediagoblin.pid \ - --log-file=/var/log/mediagoblin/mediagoblin.log \ - --daemon \ - --server-name=main - ExecStop=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \ - --pid-file=/var/run/mediagoblin/mediagoblin.pid \ - /srv/mediagoblin.example.org/mediagoblin/paste_local.ini stop - PIDFile=/var/run/mediagoblin/mediagoblin.pid - - [Install] - WantedBy=multi-user.target - - - -Enable these processes to start at boot by entering:: - - sudo systemctl enable mediagoblin-celeryd.service && sudo systemctl enable mediagoblin-paster.service - - -Start the processes for the current session with:: - - sudo systemctl start mediagoblin-paster.service - sudo systemctl start mediagoblin-celeryd.service - - -If either command above gives you an error, you can investigate the cause of -the error by entering:: - - sudo systemctl status mediagoblin-celeryd.service or - sudo systemctl status mediagoblin-paster.service - -The above ``systemctl status`` command is also useful if you ever want to -confirm that a process is still running. If you make any changes to the service -files, you can reload the service files by entering:: - - sudo systemctl daemon-reload - -After entering that command, you can attempt to start the Celery or Paste -processes again. .. _init-script: -Use an init script ------------------- +Alternative init scripts +------------------------ If your system does not use Systemd, you can use the following command as the basis for an init script: @@ -187,11 +93,6 @@ as the basis for your script: Members of the MediaGoblin community have provided init scripts for the following GNU/Linux distributions: -Debian - * `GNU MediaGoblin init scripts - `_ - by `Joar Wandborg `_ - Arch Linux * `MediaGoblin - ArchLinux rc.d scripts `_ @@ -209,6 +110,9 @@ one distribution to the next. Separate celery --------------- +":doc:`deploying`" describes a configuration with a separate Celery process, but +the following section covers this in more detail. + MediaGoblin uses `Celery`_ to handle heavy and long-running tasks. Celery can be launched in two ways: @@ -251,6 +155,7 @@ background. .. _sentry: + Set up sentry to monitor exceptions ----------------------------------- -- 2.25.1