Recovering SSH Access to Running EC2 Instances: EBS Volume Mounting and User Management Approaches

Problem Context and Challenges

In the AWS EC2 environment, key pairs serve as the core authentication mechanism for SSH access to instances. When administrators cannot obtain the original key pair files used during instance launch, they face the challenge of being unable to access running instances. This situation commonly occurs during personnel changes, key loss, or permission transfers.

EC2 Key Management Mechanism Analysis

During initial boot, EC2 instances write the specified public key material to the ~/.ssh/authorized_keys file. This mechanism ensures that only users holding the corresponding private key can establish SSH connections to the instance. However, AWS does not support directly replacing key pairs for running instances, creating technical challenges for key recovery.

EBS Volume Mounting Repair Method

For instances using EBS-boot AMIs, filesystem repair can be achieved by mounting the root volume to a secondary instance. The core concept of this approach involves mounting the "hard drive" of the problematic instance to a healthy instance for operations.

Detailed Operational Steps

First, identify the target instance and corresponding EBS volume:

instance_a=i-XXXXXXXX
volume=$(ec2-describe-instances $instance_a |
  egrep '^BLOCKDEVICE./dev/sda1' | cut -f3)

Select a secondary instance in the same availability zone as the repair platform:

instance_b=i-YYYYYYYY

Execute volume migration operations:

ec2-stop-instances $instance_a
ec2-detach-volume $volume
ec2-attach-volume --instance $instance_b --device /dev/sdj $volume

Mount and access the filesystem on the secondary instance:

ssh ...instance b...
sudo mkdir -p /vol-a
sudo mount /dev/sdj /vol-a

File Repair Operations

After successful mounting, the complete filesystem of the target instance becomes available under the /vol-a directory. Multiple repair operations can be performed:

• Add new SSH public keys to /vol-a/home/ubuntu/.ssh/authorized_keys
• Fix erroneous configurations in /vol-a/etc/sudoers
• Check error messages in system logs /vol-a/var/log/syslog
• Back up important data files

Note: User IDs may differ between instances, requiring special attention to permission issues when operating on non-root user files.

Recovery Operations

After completing repairs, unmount the volume and remount it to the original instance:

sudo umount /vol-a
sudo rmdir /vol-a
ec2-detach-volume $volume
ec2-attach-volume --instance $instance_a --device /dev/sda1 $volume
ec2-start-instances $instance_a

Multi-User Key Management Method

If instance access is available through existing keys, creating new users with independent key configurations provides an alternative approach.

User Creation and Configuration

First, log into the instance using existing keys:

ssh -i my_orig_key.pem ubuntu@111.111.11.111

Create a new user and set permissions:

sudo adduser john
sudo su -
passwd john
visudo

Add permission configuration to the sudoers file:

john   ALL = (ALL)    ALL

Key Generation and Configuration

Switch to the new user and generate a key pair:

su john
cd /home/john/
ssh-keygen -b 1024 -f john -t dsa
mkdir .ssh
chmod 700 .ssh
cat john.pub > .ssh/authorized_keys
chmod 600 .ssh/authorized_keys

Key Transfer and Testing

Transfer the private key file to the local machine:

sudo cp john /home/ubuntu/
sudo chmod 777 /home/ubuntu/john
scp -i my_orig_key.pem ubuntu@111.111.11.111:/home/ubuntu/john john
chmod 600 john

Test the availability of the new key:

ssh -i john john@111.111.11.111

Technical Points and Best Practices

When implementing the above methods, several key technical considerations require attention:

Availability Zone Consistency

EBS volumes can only be mounted between instances within the same availability zone, requiring careful selection of secondary instances to ensure zone matching.

Data Persistence Considerations

The EBS volume mounting method preserves all persistent data, but data in ephemeral storage will be lost. Data risks must be assessed before operations.

Permission Management Security

In the multi-user approach, proper permission allocation is crucial. The principle of least privilege should be followed to avoid excessive authorization.

Key Security Management

Newly generated keys should be promptly cleaned from instances, and private key files should have strict access permissions (600) set locally.

Application Scenario Analysis

These two methods suit different operational scenarios:

EBS Volume Mounting Method Applications

• Emergency recovery when instance access is completely unavailable
• Repair of corrupted system configuration files
• Large-scale key rotation operations
• Diagnosis and repair of system-level failures

Multi-User Key Management Method Applications

• Permission separation in team collaboration
• Granting temporary access permissions
• Regular key rotation
• Audit and access tracking requirements

Conclusion

Through EBS volume mounting and multi-user key management approaches, administrators can effectively resolve SSH access permission loss for EC2 instances without interrupting production services. These methods each have distinct advantages: EBS volume mounting suits complete access permission recovery, while multi-user key management better fits team collaboration and permission management scenarios. In practical applications, appropriate methods should be selected based on specific requirements and environmental conditions, while strictly adhering to security best practices.