Better structure for upload folders for q2a (human readable)

Question

I once brought up the issue: Improving the image upload folder-file-structure which found no upvotes yet :)

In one of my projects I needed to write a custom upload plugin where filenames survived.

What I did to have unique folders and human readable structure:

/uploadfolder/USERID/timestamp_filename.ext

I think this is the way to go for the q2a core, each user gets his own folder and therein the files get stored.
Then we finally have something human readable.

Feel free to give your suggestions, thanks.
Kai

-----

Update:

Since I could not wait, I wrote a plugin that implements another upload structure: https://github.com/q2apro/q2apro-better-upload-folders

Comments

"Have small directories (files in deeply nested small directories are much faster to access than in a flatter structure, due to the time it takes to read the contents of a big directory)." -- https://stackoverflow.com/a/2148018/1066234

Answer 1

The thing is "150/15015929639481993982" is actually human readable. In fact, you even know how the directory names and the blobids are generated.

I think this is the way to go for the q2a core, each user gets his own folder and therein the files get stored.

Although technically possible, there are a few things to mention. On one side, this doesn't make it more human readable, it just groups them by user. Whether that is better is considerably arguable. You seem to have a business rule or requirement that would be satisfied by using that structure but other user could have different requirements such as grouping by question id.

So, in my opinion, you're trying to solve an application level issue at an operative system level. In your particular case, it might be better to develop the management of the images separately by means of a plugin. Then provide the appropriate UI to navigate the "virtual filesystem" in the structure that suits your need.

Your approach also has the downside that is against one of the purposes of the current approach, which seems to be a fairly even file distribution across directories. If you group them by user id then you could have a user who uploads too many files while, in the general case, users who don't upload anything. So the directories for some users would be close to empty while the ones for the most active users would be full of files. As you know, too many files in a directory might bring performance issues.

Taking it a step further, the "userid" approach seems not to consider anonymous users. Would they all be placed under a "/anonymous" directory? This would, again, intensify the performance issue explained before.

Taking it a second step further, when using external users you could have users with userids such as slashes. Using them as directory names would mean unintentionally going deep in a directory structure. Even worse, you could have characters that are not valid in the filesystem the images are stored, e.g. %^$*(){}[]~`|, and that wouldn't allow the directory to be created.

Conclusion: IMO it is a no go.

Comments

Okay, the userid approach can be neglected. But what about the date approach. I think it is much better than the "which seems to be a fairly even file distribution across directories." - it would also do so. E.g. 2015/05/01 or 2015/05/02. One folder for each year, 12 subfolders per year, max. 31 subfolders per month. Looks good to me, at least :)

Ah, please see here: http://www.question2answer.org/qa/36400/improving-the-image-upload-folder-file-structure