Configure S3
Connect SmartAGT to a private S3 bucket. S3 is required in containers and can be chosen on VMs, according to the storage mode.
Connect SmartAGT to a private S3 bucket. In a cluster, S3 is required: every Backend node shares the same bucket, from which each one reads the documents uploaded through the others. See the storage mode.
Prerequisites
- An S3-compatible service that meets the requirements below, on premises or with a cloud provider.
- A private bucket, created by the storage administrator: SmartAGT never creates the bucket.
- An S3 access key restricted to this bucket, with the four permissions below.
| Requirement | Expected |
|---|---|
| API | S3 with Signature V4 and ListObjectsV2 |
| Access | HTTPS from every Backend host, with a certificate trusted by these hosts |
| Addressing | Path style, https://host/bucket/key, the default of on-premises services.On AWS, except PrivateLink, SmartAGT uses virtual-hosted style, https://bucket.host/key |
| Bucket | Dedicated to this installation, shared by all its nodes, with no other use |
| Permissions | s3:ListBucket on the bucket. s3:GetObject, s3:PutObject and s3:DeleteObject on its objects |
| Object Lock | Not supported |
| Versioning | Allowed, with a lifecycle rule that expires noncurrent versions |
Access policy of the key
Standard S3 policy format. On a service that grants permissions per key and per bucket, grant read and write on the bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::smartagt-documents"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::smartagt-documents/*"]
}
]
}
Procedure
1. Prepare TLS trust
With a private authority, install its trust chain on every Backend host before initialization. In containers, add it to the trust store of the image.
sudo install -m 644 s3-ca.crt /usr/local/share/ca-certificates/smartagt-s3.crt
sudo update-ca-certificates
In PowerShell as administrator:
Import-Certificate -FilePath '.\s3-ca.crt' -CertStoreLocation Cert:\LocalMachine\Root
2. Connect the bucket
The S3 access keys are secrets: they never appear in the YAML. Like the PostgreSQL password and the KEK, they are supplied through the environment.
-
In the delivered Backend YAML, fill in the following block with the bucket settings, without the keys:
smartagt: mediastore: backend: s3 s3: endpoint_url: "https://storage.example.org" bucket: "smartagt-documents" region: "eu-west-1"endpoint_urlis the service address, without the bucket. On AWS:https://s3.<region>.amazonaws.com.regionisus-east-1when the service declares none. -
At initialization, add the keys to the session before running
init:read -rsp 'S3 access key: ' SMARTAGT_MEDIASTORE_S3_ACCESS__KEY; echo read -rsp 'S3 secret: ' SMARTAGT_MEDIASTORE_S3_SECRET__KEY; echo export SMARTAGT_MEDIASTORE_S3_ACCESS__KEY SMARTAGT_MEDIASTORE_S3_SECRET__KEYAlso add
SMARTAGT_MEDIASTORE_S3_ACCESS__KEY,SMARTAGT_MEDIASTORE_S3_SECRET__KEYto the--preserve-envlist of theinitcommand.$env:SMARTAGT_MEDIASTORE_S3_ACCESS__KEY = [Net.NetworkCredential]::new('', (Read-Host 'S3 access key' -AsSecureString)).Password $env:SMARTAGT_MEDIASTORE_S3_SECRET__KEY = [Net.NetworkCredential]::new('', (Read-Host 'S3 secret' -AsSecureString)).Password -
When creating the services, add the same keys to their environment:
In
/etc/smartagt/backend.env, after the password and the KEK:SMARTAGT_MEDIASTORE_S3_ACCESS__KEY=<S3 access key> SMARTAGT_MEDIASTORE_S3_SECRET__KEY=<S3 secret>Before the creation loop, read the keys:
$s3Key = [Net.NetworkCredential]::new('', (Read-Host 'S3 access key' -AsSecureString)).Password $s3Secret = [Net.NetworkCredential]::new('', (Read-Host 'S3 secret' -AsSecureString)).PasswordThen add them to the
Environmentvalue of each service:Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$($service.Name)" -Name Environment -Type MultiString -Value "SMARTAGT_CORE_DATABASE_PASSWORD=$password", "SMARTAGT_KEK=$kek", "SMARTAGT_MEDIASTORE_S3_ACCESS__KEY=$s3Key", "SMARTAGT_MEDIASTORE_S3_SECRET__KEY=$s3Secret"Before the creation loop, read the keys:
$s3Key = [Net.NetworkCredential]::new('', (Read-Host 'S3 access key' -AsSecureString)).Password $s3Secret = [Net.NetworkCredential]::new('', (Read-Host 'S3 secret' -AsSecureString)).PasswordThen add them to the environment of each service:
nssm set $service.Name AppEnvironmentExtra "SMARTAGT_CORE_DATABASE_PASSWORD=$password" "SMARTAGT_KEK=$kek" "SMARTAGT_MEDIASTORE_S3_ACCESS__KEY=$s3Key" "SMARTAGT_MEDIASTORE_S3_SECRET__KEY=$s3Secret"
In containers, inject both variables from the secrets manager of your infrastructure.
Expected result: init logs Stockage S3 vérifié and the services start.
The connection is declared during the Backend installation, identically on every node: Install the Backend.
-
In the shared YAML of the Backend nodes, fill in the following block, without the access keys:
smartagt: mediastore: backend: s3 s3: endpoint_url: "https://storage.example.org" bucket: "smartagt-documents" region: "eu-west-1"endpoint_urlis the service address, without the bucket. On AWS:https://s3.<region>.amazonaws.com.regionisus-east-1when the service declares none. -
Provide the keys through the environment, in the
initsession and then in the environment of the services of every node:SMARTAGT_MEDIASTORE_S3_ACCESS__KEYfor the access key,SMARTAGT_MEDIASTORE_S3_SECRET__KEYfor the secret.
Expected result: on every node, the services start without an S3 connection error.
Note
init and the API check the bucket and the four permissions at every start: they upload, read back and delete a smartagt-probe/… witness object. Workers check access to the bucket.
Special cases
Refusal at startup
init or the API stops with a message that names the cause. Messages are printed in French.
| Message | Action |
|---|---|
Bucket S3 « … » introuvable |
Create the bucket, or correct its name |
Accès refusé au bucket S3 « … » |
Check the key, then the s3:ListBucket permission |
… refusé, droit s3:… manquant |
Grant the named permission |
Bucket S3 « … » refusé (HTTP 301, 307 ou 400) |
Correct region or endpoint_url |
Un bucket sous Object Lock … n'est pas pris en charge |
Use a bucket without Object Lock |
Stockage S3 injoignable à l'adresse … |
Check the network, the proxy and certificate trust. The nature follows the address: tls_rejected, dns_unresolved, connection_refused, timeout |
Proxy and firewall
When the service is reachable only through a proxy, declare HTTPS_PROXY, and NO_PROXY if needed, in the init session and in the environment of the services. On Windows, the WinHTTP proxy of the machine also works. A proxy entered later in the console takes precedence.
On AWS, SmartAGT reaches the bucket host, <bucket>.s3.<region>.amazonaws.com: a proxy exception or a firewall rule is written *.s3.<region>.amazonaws.com.
Versioned bucket
Deleted objects and startup witnesses remain as noncurrent versions. A lifecycle rule that expires them keeps the bucket from growing endlessly.
Reset
init --reset, with the services stopped on every node, deletes the SmartAGT objects from the bucket. Any other object is kept and reported in the log: Purge S3 terminée (supprimés=…, conservés=…).
Storage outage in service
Users read "File storage is unreachable", "did not respond in time" or "refused the operation". The cause is in the API log: S3 <opération> en échec (…) or S3 <opération> refusé (HTTP …). SmartAGT recovers by itself when the service is back, without a restart. A document is never removed from the database until the service proves it is absent.