docs/source/devel/storage.rst

   1 =========
   2  Storage
   3 =========
   4
   5 The storage systems attached to your app
   6 ----------------------------------------
   7
   8 Dynamic content: queue_store and public_store
   9 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  10
  11 Two instances of the StorageInterface come attached to your app. These
  12 are:
  13
  14 + **queue_store:** When a user submits a fresh piece of media for
  15   their gallery, before the Processing stage, that piece of media sits
  16   here in the queue_store. (It's possible that we'll rename this to
  17   "private_store" and start storing more non-publicly-stored stuff in
  18   the future...). This is a StorageInterface implementation
  19   instance. Visitors to your site probably cannot see it... it isn't
  20   designed to be seen, anyway.
  21
  22 + **public_store:** After your media goes through processing it gets
  23   moved to the public store. This is also a StorageInterface
  24   implelementation, and is for stuff that's intended to be seen by
  25   site visitors.
  26
  27 The workbench
  28 ~~~~~~~~~~~~~
  29
  30 In addition, there's a "workbench" used during
  31 processing...  it's just for temporary files during
  32 processing, and also for making local copies of stuff that
  33 might be on remote storage interfaces while transitionally
  34 moving/converting from the queue_store to the public store.
  35 See the workbench module documentation for more.
  36
  37 .. automodule:: mediagoblin.tools.workbench
  38    :members:
  39    :show-inheritance:
  40
  41
  42 Static assets / staticdirect
  43 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  44
  45 On top of all that, there is some static media that comes bundled with your
  46 application. This stuff is kept in:
  47
  48    mediagoblin/static/
  49
  50 These files are for mediagoblin base assets. Things like the CSS files,
  51 logos, etc. You can mount these at whatever location is appropriate to you
  52 (see the direct_remote_path option in the config file) so if your users
  53 are keeping their static assets at http://static.mgoblin.example.org/ but
  54 their actual site is at http://mgoblin.example.org/, you need to be able
  55 to get your static files in a where-it's-mounted agnostic way. There's a
  56 "staticdirector" attached to the request object. It's pretty easy to use;
  57 just look at this bit taken from the
  58 mediagoblin/templates/mediagoblin/base.html main template:
  59
  60     <link rel="stylesheet" type="text/css"
  61         href="Template:Request.staticdirect('/css/extlib/text.css')"/>
  62
  63 see? Not too hard. As expected, if you configured direct_remote_path to be
  64 http://static.mgoblin.example.org/ you'll get back
  65 http://static.mgoblin.example.org/css/extlib/text.css just as you'd
  66 probably expect.
  67
  68 StorageInterface and implementations
  69 ------------------------------------
  70
  71 The guts of StorageInterface and friends
  72 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  73
  74 So, the StorageInterface!
  75
  76 So, the public and queue stores both use StorageInterface implementations
  77 ... but what does that mean? It's not too hard.
  78
  79 Open up:
  80
  81     mediagoblin/storage.py
  82
  83 In here you'll see a couple of things. First of all, there's the
  84 StorageInterface class. What you'll see is that this is just a very simple
  85 python class. A few of the methods actually implement things, but for the
  86 most part, they don't. What really matters about this class is the
  87 docstrings. Each expected method is documented as to how it should be
  88 constructed. Want to make a new StorageInterface? Simply subclass it. Want
  89 to know how to use the methods of your storage system? Read these docs,
  90 they span all implementations.
  91
  92 There are a couple of implementations of these classes bundled in
  93 storage.py as well. The most simple of these is BasicFileStorage, which is
  94 also the default storage system used. As expected, this stores files
  95 locally on your machine.
  96
  97 There's also a CloudFileStorage system. This provides a mapping to
  98 [OpenStack's swift http://swift.openstack.org/] storage system (used by
  99 RackSpace Cloud files and etc).
 100
 101 Between these two examples you should be able to get a pretty good idea of
 102 how to write your own storage systems, for storing data across your
 103 beowulf cluster of radioactive monkey brains, whatever.
 104
 105 Writing code to store stuff
 106 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 107
 108 So what does coding for StorageInterface implementations actually look
 109 like? It's pretty simple, really. For one thing, the design is fairly
 110 inspired by [Django's file storage API
 111 https://docs.djangoproject.com/en/dev/ref/files/storage/]... with some
 112 differences.
 113
 114 Basically, you access files on "file paths", which aren't exactly like
 115 unix file paths, but are close. If you wanted to store a file on a path
 116 like dir1/dir2/filename.jpg you'd actually write that file path like:
 117
 118 ['dir1', 'dir2', 'filename.jpg']
 119
 120 This way we can be *sure* that each component is actually a component of
 121 the path that's expected... we do some filename cleaning on each component.
 122
 123 Your StorageInterface should pass in and out "file like objects". In other
 124 words, they should provide .read() and .write() at minimum, and probably
 125 also .seek() and .close().