Commit | Line | Data |
---|---|---|
340100ee E |
1 | ========= |
2 | Storage | |
3 | ========= | |
4 | ||
340100ee E |
5 | The storage systems attached to your app |
6 | ---------------------------------------- | |
7 | ||
8 | Dynamic content: queue_store and public_store | |
9 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
10 | ||
a2d94b0c | 11 | Two instances of the StorageInterface come attached to your app. These |
12 | are: | |
13 | ||
90e7fc67 CAW |
14 | + **queue_store:** When a user submits a fresh piece of media for |
15 | their gallery, before the Processing stage, that piece of media sits | |
16 | here in the queue_store. (It's possible that we'll rename this to | |
17 | "private_store" and start storing more non-publicly-stored stuff in | |
18 | the future...). This is a StorageInterface implementation | |
19 | instance. Visitors to your site probably cannot see it... it isn't | |
20 | designed to be seen, anyway. | |
21 | ||
22 | + **public_store:** After your media goes through processing it gets | |
23 | moved to the public store. This is also a StorageInterface | |
24 | implelementation, and is for stuff that's intended to be seen by | |
25 | site visitors. | |
a2d94b0c | 26 | |
340100ee E |
27 | The workbench |
28 | ~~~~~~~~~~~~~ | |
29 | ||
30 | In addition, there's a "workbench" used during | |
31 | processing... it's just for temporary files during | |
32 | processing, and also for making local copies of stuff that | |
33 | might be on remote storage interfaces while transitionally | |
34 | moving/converting from the queue_store to the public store. | |
35 | See the workbench module documentation for more. | |
36 | ||
37 | .. automodule:: mediagoblin.tools.workbench | |
38 | :members: | |
39 | :show-inheritance: | |
40 | ||
41 | ||
42 | Static assets / staticdirect | |
43 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
44 | ||
a2d94b0c | 45 | On top of all that, there is some static media that comes bundled with your |
46 | application. This stuff is kept in: | |
47 | ||
48 | mediagoblin/static/ | |
49 | ||
50 | These files are for mediagoblin base assets. Things like the CSS files, | |
51 | logos, etc. You can mount these at whatever location is appropriate to you | |
52 | (see the direct_remote_path option in the config file) so if your users | |
53 | are keeping their static assets at http://static.mgoblin.example.org/ but | |
54 | their actual site is at http://mgoblin.example.org/, you need to be able | |
55 | to get your static files in a where-it's-mounted agnostic way. There's a | |
56 | "staticdirector" attached to the request object. It's pretty easy to use; | |
57 | just look at this bit taken from the | |
58 | mediagoblin/templates/mediagoblin/base.html main template: | |
59 | ||
60 | <link rel="stylesheet" type="text/css" | |
61 | href="Template:Request.staticdirect('/css/extlib/text.css')"/> | |
62 | ||
63 | see? Not too hard. As expected, if you configured direct_remote_path to be | |
64 | http://static.mgoblin.example.org/ you'll get back | |
65 | http://static.mgoblin.example.org/css/extlib/text.css just as you'd | |
66 | probably expect. | |
340100ee E |
67 | |
68 | StorageInterface and implementations | |
69 | ------------------------------------ | |
70 | ||
71 | The guts of StorageInterface and friends | |
72 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
73 | ||
a2d94b0c | 74 | So, the StorageInterface! |
75 | ||
76 | So, the public and queue stores both use StorageInterface implementations | |
77 | ... but what does that mean? It's not too hard. | |
78 | ||
79 | Open up: | |
80 | ||
81 | mediagoblin/storage.py | |
82 | ||
83 | In here you'll see a couple of things. First of all, there's the | |
84 | StorageInterface class. What you'll see is that this is just a very simple | |
85 | python class. A few of the methods actually implement things, but for the | |
86 | most part, they don't. What really matters about this class is the | |
87 | docstrings. Each expected method is documented as to how it should be | |
88 | constructed. Want to make a new StorageInterface? Simply subclass it. Want | |
89 | to know how to use the methods of your storage system? Read these docs, | |
90 | they span all implementations. | |
91 | ||
92 | There are a couple of implementations of these classes bundled in | |
93 | storage.py as well. The most simple of these is BasicFileStorage, which is | |
94 | also the default storage system used. As expected, this stores files | |
95 | locally on your machine. | |
96 | ||
97 | There's also a CloudFileStorage system. This provides a mapping to | |
98 | [OpenStack's swift http://swift.openstack.org/] storage system (used by | |
99 | RackSpace Cloud files and etc). | |
100 | ||
101 | Between these two examples you should be able to get a pretty good idea of | |
102 | how to write your own storage systems, for storing data across your | |
103 | beowulf cluster of radioactive monkey brains, whatever. | |
104 | ||
340100ee E |
105 | Writing code to store stuff |
106 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
a2d94b0c | 107 | |
108 | So what does coding for StorageInterface implementations actually look | |
109 | like? It's pretty simple, really. For one thing, the design is fairly | |
110 | inspired by [Django's file storage API | |
111 | https://docs.djangoproject.com/en/dev/ref/files/storage/]... with some | |
112 | differences. | |
113 | ||
114 | Basically, you access files on "file paths", which aren't exactly like | |
115 | unix file paths, but are close. If you wanted to store a file on a path | |
116 | like dir1/dir2/filename.jpg you'd actually write that file path like: | |
117 | ||
118 | ['dir1', 'dir2', 'filename.jpg'] | |
119 | ||
120 | This way we can be *sure* that each component is actually a component of | |
121 | the path that's expected... we do some filename cleaning on each component. | |
122 | ||
123 | Your StorageInterface should pass in and out "file like objects". In other | |
124 | words, they should provide .read() and .write() at minimum, and probably | |
125 | also .seek() and .close(). |