Reversing buildout and virtualenv instructions because I think

[mediagoblin.git] / docs / hackinghowto.rst

.. _hacking-howto:

===============
 Hacking HOWTO
===============


So you want to hack on GNU MediaGoblin?
=======================================

First thing to do is check out the `Web site
<http://mediagoblin.org/join/>`_ where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at :ref:`codebase-chapter` where
we've started documenting how GNU MediaGoblin is built and how to add
new things.

Third you'll need to :ref:`get the requirements
<get-requirements-section>`.

Fourth, you'll need to build a development environment.  For this step, there are two options: 

1. :ref:`buildout and bootstrap <hacking-with-buildout>` (easier) OR
2. :ref:`virtualenv <hacking-with-virtualenv>` (more flexible, but harder)

Pick one---don't do both!


.. _get-requirements-section:

Getting requirements
====================

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7  - http://www.python.org/

  You'll need Python as well as the dev files for building modules.

* python-lxml        - http://lxml.de/
* git                - http://git-scm.com/
* MongoDB            - http://www.mongodb.org/

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Mint or Ubuntu, running the following should install these
requirements::

    sudo apt-get install mongodb git-core python python-dev \
         python-lxml

.. YouCanHelp::

   If you have instructions for other GNU/Linux distributions to set
   up requirements, let us know!


.. _hacking-with-buildout:

How to set up and maintain an environment for hacking with buildout
===================================================================

.. Note::

   Either follow the instructions in this section OR follow the ones
   in :ref:`hacking-with-virtualenv`.  But don't do both!


**Requirements**

No additional requirements.


**Create a development environment**

After installing the requirements, follow these steps:

1. Clone the repository::

       git clone http://git.gitorious.org/mediagoblin/mediagoblin.git

2. Bootstrap and run buildout::

       cd mediagoblin
       python bootstrap.py && ./bin/buildout


That's it!  Using this method, buildout should create a ``user_dev``
directory, in which certain things will be stored (media, beaker
session stuff, etc).  You can change this, but for development
purposes this default should be fine.


**Updating for dependency changes**

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.  To do that, run::

    ./bin/buildout


**Updating for code changes**

You don't need to do anything---code changes are automatically
available.


**Deleting your buildout**

At some point, you may want to delete your buildout.  Perhaps it's to
start over.  Perhaps it's to test building development environments
with buildout.

To do this, do::

    rm -rf bin develop-eggs eggs mediagoblin.egg-info parts user_dev

Usually buildout works pretty great and is super easy, but if you get
problems with python-dateutil conflicts on your system, you may need
to use virtualenv instead.


.. _hacking-with-virtualenv:

How to set up and maintain an environment for hacking with virtualenv
=====================================================================

.. Note::

   Either follow the instructions in this section OR follow the ones
   in :ref:`hacking-with-buildout`.  But don't do both!


**Requirements**

* virtualenv: http://pypi.python.org/pypi/virtualenv
* virtualenv wrapper:
  http://www.doughellmann.com/projects/virtualenvwrapper/ (be sure to
  read the `install instructions
  <http://www.doughellmann.com/docs/virtualenvwrapper/install.html>`_)


**Create a development environment**

1. Clone the repository::

       git clone http://git.gitorious.org/mediagoblin/mediagoblin.git

2. Create a virtual environment::

       mkvirtualenv --no-site-packages mediagoblin

3. If that doesn't put you in the virutal environment you just
   created, then do::

       workon mediagoblin

4. Run::

       python setup.py develop

That's it!


**Activating a virtual environment**

When you want to work on GNU MediaGoblin, you need to activate the
virtual environment like this::

    workon mediagoblin


**Deactivating a virtual environment**

If you want to deactivate it, you can do this::

    deactivate


**Updating a virtual environment with dependency changes**

1. Enter the virtual environment.

2. Run::

      python setup.py develop


**Updating a virtual environment with code changes**

You don't need to do anything---code changes are automatically
available.


**Deleting a virtual environment**

At some point you may want to delete your virtual environment.
Perhaps it's to start over.  Perhaps it's so you can test building
development environments with virtualenv.

To do this, do::

    rmvirtualenv mediagoblin


Running the server
==================

If you did virtualenv, run::

    paster serve mediagoblin.ini --reload

If you did buildout, run::

    ./bin/paster serve mediagoblin.ini --reload


Running celeryd
===============

You need to do this if you want your media to process and actually
show up.  It's probably a good idea in development to have the web
server (above) running in one terminal and celeryd in another window.

If you did virtualenv, run::

    CELERY_CONFIG_MODULE=mediagoblin.celery_setup.from_celery celeryd

If you did buildout, run::

    CELERY_CONFIG_MODULE=mediagoblin.celery_setup.from_celery ./bin/celeryd


Running the test suite
======================

If you did virtualenv, run::

    nosetests

If you did buildout, run::

    ./bin/nosetests


Troubleshooting
===============

pymongo.errors.AutoReconnect: could not find master/primary
-----------------------------------------------------------

If you see this::

    pymongo.errors.AutoReconnect: could not find master/primary

then make sure mongodb is installed and running.

If it's installed, check the mongodb log.  On my machine, that's ``/var/log/mongodb/mongodb.log``.  If you see something like::

    old lock file: /var/lib/mongodb/mongod.lock.  probably means...

Then delete the lock file and relaunch mongodb.


Wiping your user data
=====================

.. Note::

   Unless you're doing development and working on and testing creating
   a new instance, you will probably never have to do this.  Will
   plans to do this work and thus he documented it.

.. YouCanHelp::

   If you're familiar with MongoDB, we'd love to get a `script that
   removes all the GNU MediaGoblin data from an existing instance
   <http://bugs.foocorp.net/issues/296>`_.  Let us know!


Quickstart for Django programmers
=================================

We're not using Django, but the codebase is very Django-like in its
structure.

* ``routing.py`` is like ``urls.py`` in Django
* ``models.py`` has mongokit ORM definitions
* ``views.py`` is where the views go

We're using MongoDB.  Basically, instead of a relational database with
tables, you have a big JSON structure which acts a lot like a Python
dict.


.. YouCanHelp::

   If there are other things that you think would help orient someone
   new to GNU MediaGoblin but coming from Django, let us know!


Bite-sized bugs to start with
=============================

**May 3rd, 2011**:  We don't have a list of bite-sized bugs, yet, but
this is important to us.  If you're interested in things to work on,
let us know on `the mailing list <http://mediagoblin.org/join/>`_ or
on the `IRC channel <http://mediagoblin.org/join/>`_.


Tips for people new to coding
=============================

Learning Python
---------------

GNU MediaGoblin is written using a programming language called `Python
<http://python.org/>`_.

There are two different incompatible iterations of Python which I'll
refer to as Python 2 and Python 3.  GNU MediaGoblin is written in
Python 2 and requires Python 2.6 or 2.7.  At some point, we might
switch to Python 3, but that's a future thing.

You can learn how to code in Python 2 from several excellent books
that are freely available on the Internet:

* `Learn Python the Hard Way <http://learnpythonthehardway.org/>`_
* `Dive Into Pyton <http://diveintopython.org/>`_
* `Python for Software Design <http://www.greenteapress.com/thinkpython/>`_
* `A Byte of Python <http://www.swaroopch.com/notes/Python>`_

These are all excellent texts.

.. YouCanHelp::

   If you know of other good quality Python tutorials and Python
   tutorial videos, let us know!


Learning Libraries GNU MediaGoblin uses
---------------------------------------

GNU MediaGoblin uses a variety of libraries in order to do what it
does.  These libraries are listed in the :ref:`codebase-chapter`
along with links to the project Web sites and documentation for the
libraries.

There are a variety of Python-related conferences every year that have
sessions covering many aspects of these libraries.  You can find them
at `Python Miro Community <http://python.mirocommunity.org>`_ [0]_.

.. [0] This is a shameless plug.  Will Kahn-Greene runs Python Miro
   Community.

If you have questions or need help, find us on the mailing list and on
IRC.


.. _hacking-howto-git:

Learning git
------------

git is an interesting and very powerful tool.  Like all powerful
tools, it has a learning curve.

If you're new to git, we highly recommend the following resources for
getting the hang of it:

* `Learn Git <http://learn.github.com/p/intro.html>`_ --- the GitHub
  intro to git
* `Pro Git <http://progit.org/book/>`_ --- fantastic book
* `Git casts <http://gitcasts.com/>`_ --- screencast covering git
  usage
* `Git Reference <http://gitref.org/>`_ --- Git reference that makes
  it easier to get the hang of git if you're coming from other version
  control systems


Learning other utilities
------------------------

The `OpenHatch <http://openhatch.org/>`_ site has a series of
`training missions <http://openhatch.org/missions/>`_ which are
designed to help you learn how to use these tools.

If you're new to tar, diff and patch, we highly recommend you sign up
with OpenHatch and do the missions.