Commit | Line | Data |
---|---|---|
340100ee E |
1 | ========= |
2 | Storage | |
3 | ========= | |
4 | ||
340100ee E |
5 | The storage systems attached to your app |
6 | ---------------------------------------- | |
7 | ||
8 | Dynamic content: queue_store and public_store | |
9 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
10 | ||
a2d94b0c | 11 | Two instances of the StorageInterface come attached to your app. These |
12 | are: | |
13 | ||
90e7fc67 CAW |
14 | + **queue_store:** When a user submits a fresh piece of media for |
15 | their gallery, before the Processing stage, that piece of media sits | |
16 | here in the queue_store. (It's possible that we'll rename this to | |
17 | "private_store" and start storing more non-publicly-stored stuff in | |
18 | the future...). This is a StorageInterface implementation | |
19 | instance. Visitors to your site probably cannot see it... it isn't | |
20 | designed to be seen, anyway. | |
21 | ||
22 | + **public_store:** After your media goes through processing it gets | |
23 | moved to the public store. This is also a StorageInterface | |
9650aa39 | 24 | implementation, and is for stuff that's intended to be seen by |
90e7fc67 | 25 | site visitors. |
a2d94b0c | 26 | |
340100ee E |
27 | The workbench |
28 | ~~~~~~~~~~~~~ | |
29 | ||
30 | In addition, there's a "workbench" used during | |
31 | processing... it's just for temporary files during | |
32 | processing, and also for making local copies of stuff that | |
33 | might be on remote storage interfaces while transitionally | |
34 | moving/converting from the queue_store to the public store. | |
35 | See the workbench module documentation for more. | |
36 | ||
37 | .. automodule:: mediagoblin.tools.workbench | |
38 | :members: | |
39 | :show-inheritance: | |
40 | ||
41 | ||
42 | Static assets / staticdirect | |
43 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
44 | ||
a2d94b0c | 45 | On top of all that, there is some static media that comes bundled with your |
9650aa39 | 46 | application. This stuff is kept in ``mediagoblin/static/``. |
a2d94b0c | 47 | |
9650aa39 | 48 | These files are for MediaGoblin base assets. Things like the CSS files, |
a2d94b0c | 49 | logos, etc. You can mount these at whatever location is appropriate to you |
50 | (see the direct_remote_path option in the config file) so if your users | |
51 | are keeping their static assets at http://static.mgoblin.example.org/ but | |
52 | their actual site is at http://mgoblin.example.org/, you need to be able | |
53 | to get your static files in a where-it's-mounted agnostic way. There's a | |
54 | "staticdirector" attached to the request object. It's pretty easy to use; | |
55 | just look at this bit taken from the | |
9650aa39 | 56 | mediagoblin/templates/mediagoblin/base.html main template:: |
a2d94b0c | 57 | |
58 | <link rel="stylesheet" type="text/css" | |
59 | href="Template:Request.staticdirect('/css/extlib/text.css')"/> | |
60 | ||
61 | see? Not too hard. As expected, if you configured direct_remote_path to be | |
62 | http://static.mgoblin.example.org/ you'll get back | |
63 | http://static.mgoblin.example.org/css/extlib/text.css just as you'd | |
64 | probably expect. | |
340100ee E |
65 | |
66 | StorageInterface and implementations | |
67 | ------------------------------------ | |
68 | ||
69 | The guts of StorageInterface and friends | |
70 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
71 | ||
a2d94b0c | 72 | So, the StorageInterface! |
73 | ||
74 | So, the public and queue stores both use StorageInterface implementations | |
75 | ... but what does that mean? It's not too hard. | |
76 | ||
9650aa39 | 77 | Open up ``mediagoblin/storage.py``. |
a2d94b0c | 78 | |
79 | In here you'll see a couple of things. First of all, there's the | |
80 | StorageInterface class. What you'll see is that this is just a very simple | |
81 | python class. A few of the methods actually implement things, but for the | |
82 | most part, they don't. What really matters about this class is the | |
83 | docstrings. Each expected method is documented as to how it should be | |
84 | constructed. Want to make a new StorageInterface? Simply subclass it. Want | |
85 | to know how to use the methods of your storage system? Read these docs, | |
86 | they span all implementations. | |
87 | ||
88 | There are a couple of implementations of these classes bundled in | |
89 | storage.py as well. The most simple of these is BasicFileStorage, which is | |
90 | also the default storage system used. As expected, this stores files | |
91 | locally on your machine. | |
92 | ||
93 | There's also a CloudFileStorage system. This provides a mapping to | |
9650aa39 | 94 | [OpenStack's Swift http://swift.openstack.org/] storage system (used by |
a2d94b0c | 95 | RackSpace Cloud files and etc). |
96 | ||
97 | Between these two examples you should be able to get a pretty good idea of | |
98 | how to write your own storage systems, for storing data across your | |
9650aa39 | 99 | Beowulf cluster of radioactive monkey brains, whatever. |
a2d94b0c | 100 | |
340100ee E |
101 | Writing code to store stuff |
102 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
a2d94b0c | 103 | |
104 | So what does coding for StorageInterface implementations actually look | |
105 | like? It's pretty simple, really. For one thing, the design is fairly | |
106 | inspired by [Django's file storage API | |
107 | https://docs.djangoproject.com/en/dev/ref/files/storage/]... with some | |
108 | differences. | |
109 | ||
110 | Basically, you access files on "file paths", which aren't exactly like | |
9650aa39 BS |
111 | Unix file paths, but are close. If you wanted to store a file on a path |
112 | like dir1/dir2/filename.jpg you'd actually write that file path like:: | |
a2d94b0c | 113 | |
9650aa39 | 114 | ['dir1', 'dir2', 'filename.jpg'] |
a2d94b0c | 115 | |
116 | This way we can be *sure* that each component is actually a component of | |
117 | the path that's expected... we do some filename cleaning on each component. | |
118 | ||
119 | Your StorageInterface should pass in and out "file like objects". In other | |
120 | words, they should provide .read() and .write() at minimum, and probably | |
121 | also .seek() and .close(). |