READMEish.org

   1 GNU MediaGoblin
   2
   3 * About
   4
   5 What is MediaGoblin?  I'm shooting for:
   6
   7  - Initially, a place to store all your photos that's as awesome as,
   8    more awesome than, existing proprietary solutions
   9  - Later, a place for all sorts of media, such as video, music, etc
  10    hosting.
  11  - Federated, like statusnet/ostatus (we should use ostatus, in fact!)
  12  - Customizable
  13  - A place for people to collaborate and show off original and derived
  14    creations
  15  - Free, as in freedom.  Under the GNU AGPL, v3 or later.  Encourages
  16    free formats and free licensing for content, too.
  17
  18 Wow!  That's pretty ambitious.  Hopefully we're cool enough to do it.
  19 I think we can.
  20
  21 It's also necessary, for multiple reasons.  Centralization and
  22 proprietization of media on the internet is a serious problem and
  23 makes the web go from a system of extreme resilience to a system
  24 of frightening fragility.  People should be able to own their data.
  25 Etc.  If you're reading this, chances are you already agree though. :)
  26
  27 * Milestones
  28
  29 Excepting the first, not necessarily in this order.
  30
  31 ** Basic image hosting
  32 ** Multi-media hosting (including video and audio)
  33 ** API(s)
  34 ** Federation
  35
  36 Maybe this is 0.2 :)
  37
  38 ** Plugin system
  39
  40 * Technology
  41
  42 I have a pretty specific set of tools that I expect to use in this
  43 project.  Those are:
  44
  45  - *[[http://python.org/][Python]]:* because I love, and know well, the language
  46  - *[[http://www.mongodb.org/][MongoDB]]:* a "document database".  Because it's extremely flexible
  47    (and scales up well, but I guess not down well)
  48  - *[[http://namlook.github.com/mongokit/][MongoKit]]:* a lightweight ORM for mongodb.  Helps us define our
  49    structures better, does schema validation, schema evolution, and
  50    helps make things more fun and pythonic.
  51  - *[[http://jinja.pocoo.org/docs/][Jinja2]]:* for templating.  Pretty much django templates++ (wow, I
  52    can actually pass arguments into method calls instead of tediously
  53    writing custom tags!)
  54  - *[[http://wtforms.simplecodes.com/][WTForms]]:* for form handling, validation, abstraction.  Almost just
  55    like Django's templates,
  56  - *[[http://pythonpaste.org/webob/][WebOb]]:* gives nice request/response objects (also somewhat djangoish)
  57  - *[[http://pythonpaste.org/deploy/][Paste Deploy]] and [[http://pythonpaste.org/script/][Paste Script]]:* as the default way of configuring
  58    and launching the application.  Since MediaGoblin will be fairly
  59    wsgi minimalist though, you can probably use other ways to launch
  60    it, though this will be the default.
  61  - *[[http://routes.groovie.org/][Routes]]:* for URL routing.  It works well enough.
  62  - *[[http://jquery.com/][JQuery]]:* for all sorts of things on the javascript end of things,
  63    for all sorts of reasons.
  64  - *[[http://beaker.groovie.org/][Beaker]]:* for sessions, because that seems like it's generally
  65    considered the way to go I guess.
  66  - *[[http://somethingaboutorange.com/mrl/projects/nose/1.0.0/][nose]]:* for unit tests, because it makes testing a bit nicer.
  67  - *[[http://celeryproject.org/][Celery]]:* for task queueing (think resizing images, encoding
  68    video) because some people like it, and even the people I know who
  69    don't don't seem to know of anything better :)
  70  - *[[http://www.rabbitmq.com/][RabbitMQ]]:* for sending tasks to celery, because I guess that's
  71    what most people do.  Might be optional, might also let people use
  72    MongoDB for this if they want.
  73
  74 ** Why python
  75
  76 Because I (Chris Webber) know Python, love Python, am capable of
  77 actually making this thing happen in Python (I've worked on a lot of
  78 large free software web applications before in Python, including
  79 [[http://mirocommunity.org/][Miro Community]], the [[http://miroguide.org][Miro Guide]], a large portion of
  80 [[http://creativecommons.org/][Creative Commons' site]], and a whole bunch of things while working at
  81 [[http://www.imagescape.com/][Imaginary Landscape]]).  I know Python, I can make this happen in
  82 Python, me starting a project like this makes sense if it's done in
  83 Python.
  84
  85 You might say that PHP is way more deployable, that rails has way more
  86 cool developers riding around on fixie bikes, and all of those things
  87 are true, but I know Python, like Python, and think that Python is
  88 pretty great.  I do think that deployment in Python is not as good as
  89 with PHP, but I think the days of shared hosting are (thankfully)
  90 coming to an end, and will probably be replaced by cheap virtual
  91 machines spun up on the fly for people who want that sort of stuff,
  92 and Python will be a huge part of that future, maybe even more than
  93 PHP will.  The deployment tools are getting better.  Maybe we can use
  94 something like Silver Lining.  Maybe we can just distribute as .debs
  95 or .rpms.  We'll figure it out.
  96
  97 But if I'm starting this project, which I am, it's gonna be in Python.
  98
  99 ** Why mongodb
 100
 101 In case you were wondering, I am not a NOSQL fanboy, I do not go
 102 around telling people that MongoDB is web scale.  Actually my choice
 103 for MongoDB isn't scalability, though scaling up really nicely is a
 104 pretty good feature and sets us up well in case large volume sites
 105 eventually do use MediaGoblin.  But there's another side of
 106 scalability, and that's scaling down, which is important for
 107 federation, maybe even more important than scaling up in an ideal
 108 universe where everyone ran servers out of their own housing.  As a
 109 memory-mapped database, MongoDB is pretty hungry, so actually I spent
 110 a lot of time debating whether the inability to scale down as nicely
 111 as something like SQL has with sqlite meant that it was out.
 112
 113 But I decided in the end that I really want MongoDB, not for
 114 scalability, but for flexibility.  Schema evolution pains in SQL are
 115 almost enough reason for me to want MongoDB, but not quite.  The real
 116 reason is because I want the ability to eventually handle multiple
 117 media types through MediaGoblin, and also allow for plugins, without
 118 the rigidity of tables making that difficult.  In other words,
 119 something like:
 120
 121 #+BEGIN_SRC javascript
 122 {"title": "Me talking until you are bored",
 123  "description": "blah blah blah",
 124  "media_type": "audio",
 125  "media_data": {
 126      "length": "2:30",
 127      "codec": "OGG Vorbis"},
 128  "plugin_data": {
 129      "licensing": {
 130          "license": "http://creativecommons.org/licenses/by-sa/3.0/"}}}
 131 #+END_SRC
 132
 133 Being able to just dump media-specific information in a media_data
 134 hashtable is pretty great, and even better is having a plugin system
 135 where you can just let plugins have their own entire key-value space
 136 cleanly inside the document that doesn't interfere with anyone else's
 137 stuff.  If we were to let plugins to deposit their own information
 138 inside the database, either we'd let plugins create their own tables
 139 which makes SQL migrations even harder than they already are, or we'd
 140 probably end up creating a table with a column for key, a column for
 141 value, and a column for type in one huge table called "plugin_data" or
 142 something similar.  (Yo dawg, I heard you liked plugins, so I put a
 143 database in your database so you can query while you query.)  Gross.
 144
 145 I also don't want things to be too lose so that we forget or lose the
 146 structure of things, and that's one reason why I want to use MongoKit,
 147 because we can cleanly define a much structure as we want and verify
 148 that documents match that structure generally without adding too much
 149 bloat or overhead (mongokit is a pretty lightweight wrapper and
 150 doesn't inject extra mongokit-specific stuff into the database, which
 151 is nice and nicer than many other ORMs in that way).
 152
 153 ** Why wsgi minimalism / Why not Django
 154
 155 If you notice in the technology list above, I list a lot of components
 156 that are very [[http://www.djangoproject.com/][Django-like]], but not actually Django components.  What
 157 can I say, I really like a lot of the ideas in Django!  Which leads to
 158 the question: why not just use Django?
 159
 160 While I really like Django's ideas and a lot of its components, I also
 161 feel that most of the best ideas in Django I want have been
 162 implemented as good or even better outside of Django.  I could just
 163 use Django and replace the templating system with Jinja2, and the form
 164 system with wtforms, and the database with MongoDB and MongoKit, but
 165 at that point, how much of Django is really left?
 166
 167 I also am sometimes saddened and irritated by how coupled all of
 168 Django's components are.  Loosely coupled yes, but still coupled.
 169 WSGI has done a good job of providing a base layer for running
 170 applications on and [[http://pythonpaste.org/webob/do-it-yourself.html][if you know how to do it yourself]] it's not hard or
 171 many lines of code at all to bind them together without any framework
 172 at all (not even say [[http://pylonshq.com/][Pylons]], [[http://docs.pylonsproject.org/projects/pyramid/dev/][Pyramid]], or [[http://flask.pocoo.org/][Flask]] which I think are still
 173 great projects, especially for people who want this sort of thing but
 174 have no idea how to get started).  And even at this already really
 175 early stage of writing MediaGoblin, that glue work is mostly done.
 176
 177 Not to say I don't think Django isn't great for a lot of things.  For
 178 a lot of stuff, it's still the best, but not for MediaGoblin, I think.
 179
 180 One thing that Django does super well though is documentation.  It
 181 still has some faults, but even with those considered I can hardly
 182 think of any other project in Python that has as nice of documentation
 183 as Django.  It may be worth
 184 [[http://pycon.blip.tv/file/4881071/][learning some lessons on documentation from Django]], on that note.
 185
 186 I'd really like to have a good, thorough hacking-howto and
 187 deployment-howto, especially in the former making some notes on how to
 188 make it easier for Django hackers to get started.