- Compatible XF Versions
- 2.1 , 2.2
- Visible Branding
- No
XenForo Version Compatibility
The download is only compatible with XenForo 2.1 and above.
Why this guide?
Since XenForo 2.0.0 we have supported remote file storage using an abstracted file system named Flysystem. It's called an abstracted file system as it adds a layer of abstraction between the code and a file system. It means that it provides a consistent API for performing file system operations so that whether the file system is a local disk-based file system or a distributed and remotely accessible file system, our code calls the same functions and Flysystem takes care of the rest.
As useful as that is, it isn't the most obvious or straightforward thing to set up so this guide and accompanying add-on will help.
So, if you're planning to make use of the video uploads function in XF 2.1 or above and you're worried about increased disk space requirements this will help.
Making the required files available
Although it is possible for you to download the files and set up things like the autoloader yourself, you will probably prefer to simply download the add-on that is attached to this resource. You can install the add-on in the usual fashion.
Before you start
If you're setting this up on an existing site, you will need to manually move your existing files over. There's a section about that at the end. While you are moving the existing files, setting things up and testing, we recommend closing the forum first.
Setting up DigitalOcean Spaces
We'll cover this first as it is the most straightforward to set up. If you'd prefer to use Amazon S3 skip ahead to the Setting up Amazon S3 section below.
Now we need to create some API credentials. To do this:
Configuring XF to use DigitalOcean Spaces
We now need to configure XF to use DigitalOcean Spaces for file storage. We'll start with what usually goes into the data directory first. This generally includes attachment thumbnails and avatars.
Open your
First we need to configure the Amazon S3 client (the DigitalOcean Spaces API is compatible with the Amazon AWS SDK).
We will do this using a
Note that the
Next we need to set up the actual Flysystem adapter to use the S3 client:
Finally, we need to ensure that avatar and attachment thumbnail URLs are prepended with the correct URL. This requires the endpoint URL you noted down earlier, again:
At this point, everything should be working in terms of new uploads. Don't be alarmed if you notice that avatars and thumbnails are missing; if you have existing files, they will need to be moved over manually which we'll go through later.
First, we need to test that the configuration works. Simply go and upload a new avatar. The avatar will now be stored and served remotely!
If you check your DigitalOcean Spaces account now, you should see that new folders have been created containing your new avatar:
Success! But we're only half way there!
We now need to add support for the internal_data directory stuff too. Generally, this is attachments and any other stuff that should be "private". Back to config.php and the code to add is very similar:
Now try to upload an attachment to a post and, much like before, you should now see additional files and folders in your Spaces file browser.
Setting up Amazon S3
Configuring XF to use Amazon S3
We now need to configure XF to use Amazon S3 for file storage. We'll start with what usually goes into the
Open your
We will do this using a
Note that the
Next we need to set up the actual Flysystem adapter to use the S3 client:
Finally, we need to ensure that avatar and attachment thumbnail URLs are prepended with the correct URL:
At this point, everything should be working in terms of new uploads. Don't be alarmed if you notice that avatars and thumbnails are missing; if you have existing files, they will need to be moved over manually which we'll go through later.
First, we need to test that the configuration works. Simply go and upload a new avatar. The avatar will now be stored and served remotely!
If you check your bucket file browser now, you should see that new folders have been created containing your new avatar:
Success! But we're only half way there!
We now need to add support for the
Now try to upload an attachment to a post and, much like before, you should now see additional files and folders in your bucket file browser.
The download is only compatible with XenForo 2.1 and above.
Why this guide?
Since XenForo 2.0.0 we have supported remote file storage using an abstracted file system named Flysystem. It's called an abstracted file system as it adds a layer of abstraction between the code and a file system. It means that it provides a consistent API for performing file system operations so that whether the file system is a local disk-based file system or a distributed and remotely accessible file system, our code calls the same functions and Flysystem takes care of the rest.
As useful as that is, it isn't the most obvious or straightforward thing to set up so this guide and accompanying add-on will help.
So, if you're planning to make use of the video uploads function in XF 2.1 or above and you're worried about increased disk space requirements this will help.
Making the required files available
Although it is possible for you to download the files and set up things like the autoloader yourself, you will probably prefer to simply download the add-on that is attached to this resource. You can install the add-on in the usual fashion.
Before you start
If you're setting this up on an existing site, you will need to manually move your existing files over. There's a section about that at the end. While you are moving the existing files, setting things up and testing, we recommend closing the forum first.
Setting up DigitalOcean Spaces
We'll cover this first as it is the most straightforward to set up. If you'd prefer to use Amazon S3 skip ahead to the Setting up Amazon S3 section below.
- Go to the DigitalOcean Cloud page and sign up or log in.
- At this point, if you're new to DigitalOcean, you may need to set up billing.
- You will now be able to create a new project.
- Click the "Start using Spaces" link.
- Choose your datacenter region (I have chosen Amsterdam).
- Leave "Restrict File Listing" selected.
- Choose a unique name (I have chosen "xftest")
- Click "Create a space"
Now we need to create some API credentials. To do this:
- Click "Manage" in the left sidebar.
- Click "API".
- In the "Spaces access keys" section click "Generate New Key".
- Type a name for the key (Again, I have chosen "xftest") and save.
Configuring XF to use DigitalOcean Spaces
We now need to configure XF to use DigitalOcean Spaces for file storage. We'll start with what usually goes into the data directory first. This generally includes attachment thumbnails and avatars.
Open your
src/config.php
file.First we need to configure the Amazon S3 client (the DigitalOcean Spaces API is compatible with the Amazon AWS SDK).
We will do this using a
closure
so that we can reuse the same code and we only have to type it out once:
PHP:
$s3 = function()
{
return new \Aws\S3\S3Client([
'credentials' => [
'key' => 'ABC',
'secret' => '123'
],
'region' => 'ams3',
'version' => 'latest',
'endpoint' => 'https://ams3.digitaloceanspaces.com'
]);
};
Note that the
key
and secret are what you noted down after setting up the "Spaces access key" earlier. The region can be inferred from the endpoint URL you noted down earlier
. It's the part after the first . in the URL, in my case it is ams3
. The endpoint
is the same endpoint URL minus the unique name you chose.Next we need to set up the actual Flysystem adapter to use the S3 client:
PHP:
$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'data');
};
PHP:
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xftest.ams3.digitaloceanspaces.com/data/' . $externalPath;
};
At this point, everything should be working in terms of new uploads. Don't be alarmed if you notice that avatars and thumbnails are missing; if you have existing files, they will need to be moved over manually which we'll go through later.
First, we need to test that the configuration works. Simply go and upload a new avatar. The avatar will now be stored and served remotely!
If you check your DigitalOcean Spaces account now, you should see that new folders have been created containing your new avatar:
Success! But we're only half way there!
We now need to add support for the internal_data directory stuff too. Generally, this is attachments and any other stuff that should be "private". Back to config.php and the code to add is very similar:
PHP:
$config['fsAdapters']['internal-data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'internal_data');
};
Setting up Amazon S3
- Go to the AWS Management Console page and sign up or log in.
- In the "AWS services" section type "S3" to go to the "S3 Console".
- Click "Create bucket".
- Choose a bucket name (I have chosen xftest).
- Choose a region (I have chosen EU London).
- Accept any further default options until the bucket is created.
- You now need to go to the "IAM" console.
- Click "Add user".
- Pick a username (yep, I used xftest again ).
- Set the access type to "Programmatic".
- To set permissions, click the "Attach existing policies directly" tab followed by the "Create policy" button.
- IAM and the various policies and permissions can be fairly daunting. We can make it a bit easier, though you may have different requirements. On this page there is a tab called "JSON". Paste the following in there, replacing YOUR-BUCKET-NAME with the bucket name you chose earlier:
JSON:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:putObject",
"s3:putObjectAcl",
"s3:ReplicateObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME",
"arn:aws:s3:::YOUR-BUCKET-NAME/*"
]
}
]
}
- Click "Review policy" give it a name and save.
- Go back to the previous "Add user" page, click the "Refresh" button and search for the policy you just created.
- Click "Next", followed by "Create user".
Configuring XF to use Amazon S3
We now need to configure XF to use Amazon S3 for file storage. We'll start with what usually goes into the
data
directory first. This generally includes attachment thumbnails and avatars.Open your
src/config.php
file.We will do this using a
closure
so that we can reuse the same code and we only have to type it out once:
PHP:
$s3 = function()
{
return new \Aws\S3\S3Client([
'credentials' => [
'key' => 'ABC',
'secret' => '123'
],
'region' => 'eu-west-2',
'version' => 'latest',
'endpoint' => 'https://s3.eu-west-2.amazonaws.com'
]);
};
Note that the
key
and secret
are what you noted down after setting up the IAM user earlier. The region
can be inferred from the S3 endpoint URL.Next we need to set up the actual Flysystem adapter to use the S3 client:
PHP:
$config['fsAdapters']['data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'data');
};
PHP:
$config['externalDataUrl'] = function($externalPath, $canonical)
{
return 'https://xftest.s3.eu-west-2.amazonaws.com/data/' . $externalPath;
};
First, we need to test that the configuration works. Simply go and upload a new avatar. The avatar will now be stored and served remotely!
If you check your bucket file browser now, you should see that new folders have been created containing your new avatar:
Success! But we're only half way there!
We now need to add support for the
internal_data
directory stuff too. Generally, this is attachments and any other stuff that should be "private". Back to config.php and the code to add is very similar:
PHP:
$config['fsAdapters']['internal-data'] = function() use($s3)
{
return new \League\Flysystem\AwsS3v3\AwsS3Adapter($s3(), 'xftest', 'internal_data');
};