troubleshooting-s3-files
aws/agent-toolkit-for-aws
Diagnose and resolve Amazon S3 Files mount failures, permissions, sync, and performance issues.
What is troubleshooting-s3-files?
Troubleshoots Amazon S3 Files problems including mount failures, IAM permission errors, synchronization delays, conflict resolution, and performance degradation. Use this skill when an S3 file system fails to mount, returns access denied, doesn't sync changes to S3, shows files in lost+found, or performs slower than expected.
- Verify AWS CLI and S3 Files client dependencies
- Classify issues into 8 categories: client installation, network/security groups, IAM permissions, synchronization, conflict resolution, performance, encryption/KMS, and VPC DNS
- Diagnose mount failures and connection timeouts via security group and network validation
- Resolve IAM permission errors for mount, write, and root access
- Monitor and fix S3 synchronization issues and export errors
- Recover files from .s3files-lost+found conflict directory
How to install troubleshooting-s3-files
npx skills add https://github.com/aws/agent-toolkit-for-aws --skill troubleshooting-s3-files- AWS CLI v2 installed and configured with valid credentials
- S3 Files client (amazon-efs-utils v3.0.0+) installed on compute instances
- Access to AWS account with permissions to describe S3 Files resources and security groups
- VPC and security group configuration details for the affected resources
How to use troubleshooting-s3-files
- 1.Verify AWS CLI version and S3 Files client availability on the instance
- 2.Identify the symptom and use the classification table to determine the issue category
- 3.Follow the category-specific troubleshooting steps (A through H)
- 4.For mount failures, check security groups and network connectivity using provided AWS CLI commands
- 5.For permission errors, validate IAM role trust policy and required s3files actions
- 6.For sync issues, check file status with getfattr and monitor PendingExports CloudWatch metric
- 7.Enable debug logging in /etc/amazon/efs/s3files-utils.conf if needed
- 8.Collect logs with the provided tar command and share with AWS Support if issue persists
Use cases
- Mount fails with 'command not found' or connection timeout errors
- Access denied errors during mount or file operations
- Files written to S3 Files don't appear in S3 bucket within expected time
- Files appear in .s3files-lost+found directory due to sync conflicts
- Read or write performance is slower than expected
- AWS infrastructure engineers troubleshooting S3 Files deployments
- DevOps teams managing S3 Files mounts on EC2 or ECS
- System administrators diagnosing permission and network issues
- Developers debugging S3 Files synchronization problems
troubleshooting-s3-files FAQ
S3 Files is a different product from Mountpoint for S3, S3 File Gateway, and File Cache. S3 Files uses the aws s3files CLI, s3files: IAM actions, and mount -t s3files. Do not confuse the tools or IAM permissions between them.
S3 Files does not validate IAM role permissions at creation time. If the compute role lacks required permissions or the trust policy is misconfigured, the file system gets stuck in 'creating' with an access denied message. Fix the IAM role and delete/recreate the file system.
Writes sync within approximately 60 seconds. You can check the sync status using getfattr with the user.s3files.status attribute. If files don't appear, check for ExportError values like S3AccessDenied, S3BucketNotFound, or EncryptionKeyInaccessible.
Files in .s3files-lost+found-{fs-id} indicate a sync conflict where the file was modified both via the file system and S3 simultaneously. S3 version wins and the file system version is moved to lost+found.
Connection timeout is the most common mount failure, almost always caused by security group misconfiguration. The mount target security group must allow inbound TCP 2049 from the compute instance security group. Verify with nc -zv and fix with aws ec2 authorize-security-group-ingress.
Full instructions (SKILL.md)
Source of truth, from aws/agent-toolkit-for-aws.
name: troubleshooting-s3-files description: > Diagnoses and resolves Amazon S3 Files issues including mount failures, permission errors, synchronization problems, and performance issues. Use when the user has an S3 file system that is not mounting, returning access denied, not syncing changes to S3, showing files in lost+found, or performing slower than expected. version: 1
Troubleshooting S3 Files
Overview
Diagnoses and resolves Amazon S3 Files issues: mount failures, IAM permissions, synchronization, conflict resolution, and performance.
For authoritative guidance, see S3 Files Troubleshooting.
Common Tasks
0. Verify Dependencies
- You MUST verify
awsCLI is available withs3filessubcommand support - You MUST confirm valid AWS credentials
- You MUST ONLY check for tool existence and version — MUST NOT execute destructive or mutating commands during verification
- You MUST inform the user if any required tools are missing
- You MUST respect the user's decision to abort if tools are unavailable
- You SHOULD explain steps before executing and wait for user confirmation on write commands
1. Classify the Issue
| Symptom | Category |
|---|---|
| mount.s3files: command not found | A: Client Installation |
| Connection timed out during mount | B: Network/Security Group |
| Mount hangs indefinitely (no timeout) | B: Network/Security Group |
| Access denied during mount | C: IAM Permissions |
| File system stuck in "creating" | C: IAM Permissions |
| Permission denied on file operations | C: IAM Permissions |
| Files not appearing in S3 after write | D: Synchronization |
| Files in .s3files-lost+found directory | E: Conflict Resolution |
| Slow reads or high latency | F: Performance |
| NFS server error | G: Encryption/KMS |
| DNS name resolution fails | H: VPC DNS |
2. Category A — Client Installation
mount.s3files: command not found means amazon-efs-utils is missing or < v3.0.0.
sudo yum -y install amazon-efs-utils # Amazon Linux
3. Category B — Network/Security Group
Connection timeout is the #1 mount failure — almost always security groups.
Verify mount target exists in the instance's AZ:
aws s3files list-mount-targets --file-system-id fs-ID --region REGION
Cross-AZ mounting works but adds latency.
Verify security groups — most common fix:
- Mount target SG MUST have inbound TCP 2049 from compute SG
- Compute SG MUST have outbound TCP 2049 to mount target SG
- Fix:
aws ec2 authorize-security-group-ingress --group-id sg-MT --protocol tcp --port 2049 --source-group sg-COMPUTE
Test connectivity:
nc -zv az-ID.fs-ID.s3files.REGION.on.aws 2049
Note: These SG troubleshooting steps also apply to EFS — use
aws efs describe-mount-targetsinstead.
Mount hangs in isolated VPC: If the VPC has no internet access, S3 Files requires a CloudWatch Logs VPC endpoint (com.amazonaws.REGION.logs) for mount to complete.
4. Category C — IAM Permissions
File system stuck in "creating" status:
S3 Files does NOT validate IAM role permissions at creation time. Wrong trust policy or missing permissions → stuck in creating with access denied in statusMessage.
Check status:
aws s3files get-file-system --file-system-id fs-ID --region REGION
Check statusMessage. If access denied, fix the IAM role and delete/recreate.
Mount access denied: Compute role needs s3files:ClientMount. For dev/test only, AmazonS3FilesClientFullAccess is acceptable — avoid in production.
Write permission denied: Compute role needs s3files:ClientWrite
Root access denied: Compute role needs s3files:ClientRootAccess. ⚠️ Bypasses POSIX permissions — prefer access points with scoped POSIX users.
Check file system policy:
aws s3files get-file-system-policy --file-system-id fs-ID --region REGION
5. Category D — Synchronization
Files not appearing in S3: Writes sync within ~60 seconds. Check status:
getfattr -n "user.s3files.status;$(date -u +%s)" filename --only-values
Common ExportError values:
| Error | Fix |
|---|---|
| S3AccessDenied | File system IAM role lacks S3 write permissions |
| S3BucketNotFound | Bucket deleted or renamed |
| RoleAssumptionFailed | Trust policy misconfigured |
| EncryptionKeyInaccessible | KMS key disabled or permissions revoked |
| PathTooLong | File path exceeds 1,024 byte S3 key limit |
Monitor: PendingExports CloudWatch metric. Growing = exceeds 800 files/sec rate.
6. Category E — Conflict Resolution
Files in .s3files-lost+found-{fs-id} = sync conflict (modified via FS and S3 simultaneously). S3 wins; FS version moved to lost+found.
7. Category F — Performance
First access latency: Normal — first directory access imports metadata.
Intelligent read routing not working: Compute role needs s3:GetObject on the bucket.
Slow writes: If PendingExports growing, distribute across multiple file systems.
8. Category G — Encryption/KMS
NFS server error with encrypted FS = KMS issue. Verify key is enabled and role has KMS permissions.
9. Category H — VPC DNS
DNS resolution failure = VPC DNS settings disabled.
aws ec2 describe-vpc-attribute --vpc-id vpc-ID --attribute enableDnsHostnames
aws ec2 describe-vpc-attribute --vpc-id vpc-ID --attribute enableDnsSupport
Both MUST be true. If not:
aws ec2 modify-vpc-attribute --vpc-id vpc-ID --enable-dns-hostnames Value=true
aws ec2 modify-vpc-attribute --vpc-id vpc-ID --enable-dns-support Value=true
Troubleshooting
AWS CLI endpoint URL cannot be resolved
CLI is too old for S3 Files. Run aws --version — if v1.x, upgrade to AWS CLI v2: Installing the AWS CLI.
ECS task fails with DNS resolution error
Used efsVolumeConfiguration instead of s3filesVolumeConfiguration. Fix: use fileSystemArn in S3 Files-specific volume config.
S3 Files vs other products confusion
S3 Files is NOT Mountpoint for S3, S3 File Gateway, or File Cache. Uses aws s3files CLI, s3files: IAM actions, mount -t s3files.
Enable Debug Logs
Set logging_level = DEBUG in /etc/amazon/efs/s3files-utils.conf. Logs at /var/log/amazon/efs/mount.log.
Collect Logs for AWS Support
sudo tar -czf /tmp/s3files-logs.tar.gz /var/log/amazon/efs/ /etc/amazon/efs/s3files-utils.conf
Security Considerations
- When diagnosing IAM issues, verify least-privilege — avoid FullAccess as a shortcut
- Without a file system policy, any VPC client can mount
- Restrict
/var/log/amazon/efs/access — logs contain S3 key names
Additional Resources
Related skills
More from aws/agent-toolkit-for-aws and the wider catalog.
amazon-bedrock
Build generative AI applications on Amazon Bedrock with model invocation, RAG, agents, guardrails, and AgentCore deployment.
aws-amplify
Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 using TypeScript code-first approach.
aws-billing-and-cost-management
Analyze AWS costs, optimize spending, manage budgets, and right-size resources with Compute Optimizer and Cost Optimization Hub.
aws-cdk
Author, deploy, and troubleshoot AWS infrastructure with CDK in TypeScript or Python.
axiom-sre
Expert SRE investigator for incidents using hypothesis-driven methodology and Axiom observability queries.
building-dashboards
Design and build Axiom dashboards via API with APL/MPL queries, SmartFilters, and chart configuration.