[AWS] EC2에 EBS mount 쉽게 하기(feat. amzlinux2, nvme)

Intro

AWS의 EC2를 생성할때 EBS를 생성하는데, root EBS가 아닌 별도의 EBS를 추가로 생성하게되면 os내부에서 별도의 mount작업을 진행해야합니다. 저는 기존에 /etc/fstab 설정을 수정하여 작업을 자주 했었는데요. 오늘은 systemd에 서비스로 등록하여 마운트 하는 방법을 알아보도록 하겠습니다.

AWS의 EC2 instance type의 5세대 이상부터 nitro system이 도입되면서 리눅스 os내 device명도 바뀌었습니다. (t 타입은 3세대부터)

기존 4세대 이전에는 linux os device 네이밍 규칙이 /dev/xvda, /dev/xvdb, /dev/xvdc, /dev/xvdd 와 같이 붙혀졌고 5세대 이후부터는 /dev/sda1, /dev/sdb, /dev/sdc, /dev/sdd 과 같이 붙혀지게 됩니다.

하지만 실제 os내부에서 device name을 검색해보면 (ls -al /dev/) /dev/nvme0n1p1, /dev/nvme1n1, /dev/nvme2n1, /dev/nvme3n1 과 같이 설정이 되어있습니다.

쉽게 마운트 설정을 하기 위해 /dev/sda1, /dev/sdb, /dev/sdc, /dev/sdd 로 심볼릭 링크를 생성할 예정입니다. /usr/loca/bin 위치에 아래와 같이 ebsnvme-id 파일과 nvme-to-block-mapping.sh 파일 생성하고 이를 동작시켜서 심볼릭 링크를 생성합니다.

그리고 os를 재부팅 했을 경우에도 동작하도록 systemd 에 심볼릭링크 생성, 파일 시스템 포맷, EBS마운트 서비스로 등록하는 작업을 진행해보도록 하겠습니다.

OS는 cenos7 또는 amazonlinux2 입니다.

심볼릭 링크 생성

/usr/loca/bin 위치에 ebsnvme-id 파일과 nvme-to-block-mapping.sh 파일 생성 하고 nvme-to-block-mapping.sh 파일 실행 ./nvme-to-block-mapping.sh

symlink created: /dev/nvme0n1 to /dev/sda1                            
symlink created: /dev/nvme1n1 to /dev/sdb 

ebsnvme-id

#!/usr/bin/env python2.7

# Copyright (C) 2017 Amazon.com, Inc. or its affiliates.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#    http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the
# specific language governing permissions and limitations under the
# License.

"""
Usage:
Read EBS device information and provide information about
the volume.
"""

import argparse
from ctypes import *
from fcntl import ioctl
import sys

NVME_ADMIN_IDENTIFY = 0x06
NVME_IOCTL_ADMIN_CMD = 0xC0484E41
AMZN_NVME_VID = 0x1D0F
AMZN_NVME_EBS_MN = "Amazon Elastic Block Store"

class nvme_admin_command(Structure):
    _pack_ = 1
    _fields_ = [("opcode", c_uint8),      # op code
                ("flags", c_uint8),       # fused operation
                ("cid", c_uint16),        # command id
                ("nsid", c_uint32),       # namespace id
                ("reserved0", c_uint64),
                ("mptr", c_uint64),       # metadata pointer
                ("addr", c_uint64),       # data pointer
                ("mlen", c_uint32),       # metadata length
                ("alen", c_uint32),       # data length
                ("cdw10", c_uint32),
                ("cdw11", c_uint32),
                ("cdw12", c_uint32),
                ("cdw13", c_uint32),
                ("cdw14", c_uint32),
                ("cdw15", c_uint32),
                ("reserved1", c_uint64)]

class nvme_identify_controller_amzn_vs(Structure):
    _pack_ = 1
    _fields_ = [("bdev", c_char * 32),  # block device name
                ("reserved0", c_char * (1024 - 32))]

class nvme_identify_controller_psd(Structure):
    _pack_ = 1
    _fields_ = [("mp", c_uint16),       # maximum power
                ("reserved0", c_uint16),
                ("enlat", c_uint32),     # entry latency
                ("exlat", c_uint32),     # exit latency
                ("rrt", c_uint8),       # relative read throughput
                ("rrl", c_uint8),       # relative read latency
                ("rwt", c_uint8),       # relative write throughput
                ("rwl", c_uint8),       # relative write latency
                ("reserved1", c_char * 16)]

class nvme_identify_controller(Structure):
    _pack_ = 1
    _fields_ = [("vid", c_uint16),          # PCI Vendor ID
                ("ssvid", c_uint16),        # PCI Subsystem Vendor ID
                ("sn", c_char * 20),        # Serial Number
                ("mn", c_char * 40),        # Module Number
                ("fr", c_char * 8),         # Firmware Revision
                ("rab", c_uint8),           # Recommend Arbitration Burst
                ("ieee", c_uint8 * 3),      # IEEE OUI Identifier
                ("mic", c_uint8),           # Multi-Interface Capabilities
                ("mdts", c_uint8),          # Maximum Data Transfer Size
                ("reserved0", c_uint8 * (256 - 78)),
                ("oacs", c_uint16),         # Optional Admin Command Support
                ("acl", c_uint8),           # Abort Command Limit
                ("aerl", c_uint8),          # Asynchronous Event Request Limit
                ("frmw", c_uint8),          # Firmware Updates
                ("lpa", c_uint8),           # Log Page Attributes
                ("elpe", c_uint8),          # Error Log Page Entries
                ("npss", c_uint8),          # Number of Power States Support
                ("avscc", c_uint8),         # Admin Vendor Specific Command Configuration
                ("reserved1", c_uint8 * (512 - 265)),
                ("sqes", c_uint8),          # Submission Queue Entry Size
                ("cqes", c_uint8),          # Completion Queue Entry Size
                ("reserved2", c_uint16),
                ("nn", c_uint32),            # Number of Namespaces
                ("oncs", c_uint16),         # Optional NVM Command Support
                ("fuses", c_uint16),        # Fused Operation Support
                ("fna", c_uint8),           # Format NVM Attributes
                ("vwc", c_uint8),           # Volatile Write Cache
                ("awun", c_uint16),         # Atomic Write Unit Normal
                ("awupf", c_uint16),        # Atomic Write Unit Power Fail
                ("nvscc", c_uint8),         # NVM Vendor Specific Command Configuration
                ("reserved3", c_uint8 * (704 - 531)),
                ("reserved4", c_uint8 * (2048 - 704)),
                ("psd", nvme_identify_controller_psd * 32),     # Power State Descriptor
                ("vs", nvme_identify_controller_amzn_vs)]  # Vendor Specific

class ebs_nvme_device:
    def __init__(self, device):
        self.device = device
        self.ctrl_identify()

    def _nvme_ioctl(self, id_response, id_len):
        admin_cmd = nvme_admin_command(opcode = NVME_ADMIN_IDENTIFY,
                                       addr = id_response,
                                       alen = id_len,
                                       cdw10 = 1)

        with open(self.device, "rw") as nvme:
            ioctl(nvme, NVME_IOCTL_ADMIN_CMD, admin_cmd)

    def ctrl_identify(self):
        self.id_ctrl = nvme_identify_controller()
        self._nvme_ioctl(addressof(self.id_ctrl), sizeof(self.id_ctrl))

        if self.id_ctrl.vid != AMZN_NVME_VID or self.id_ctrl.mn.strip() != AMZN_NVME_EBS_MN:
            raise TypeError("[ERROR] Not an EBS device: '{0}'".format(self.device))

    def get_volume_id(self):
        vol = self.id_ctrl.sn

        if vol.startswith("vol") and vol[3] != "-":
            vol = "vol-" + vol[3:]

        return vol

    def get_block_device(self, stripped=False):
        dev = self.id_ctrl.vs.bdev

        if stripped and dev.startswith("/dev/"):
            dev = dev[5:]

        return dev

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Reads EBS information from NVMe devices.")
    parser.add_argument("device", nargs=1, help="Device to query")

    display = parser.add_argument_group("Display Options")
    display.add_argument("-v", "--volume", action="store_true",
            help="Return volume-id")
    display.add_argument("-b", "--block-dev", action="store_true",
            help="Return block device mapping")
    display.add_argument("-u", "--udev", action="store_true",
            help="Output data in format suitable for udev rules")

    if len(sys.argv) < 2:
        parser.print_help()
        sys.exit(1)

    args = parser.parse_args()

    get_all = not (args.udev or args.volume or args.block_dev)

    try:
        dev = ebs_nvme_device(args.device[0])
    except (IOError, TypeError) as err:
        print >> sys.stderr, err
        sys.exit(1)

    if get_all or args.volume:
        print "Volume ID: {0}".format(dev.get_volume_id())
    if get_all or args.block_dev or args.udev:
        print dev.get_block_device(args.udev)

nvme-to-block-mapping.sh

#!/bin/bash

# for details:
# https://russell.ballestrini.net/aws-nvme-to-block-mapping/

# this will create a symlinks like:
#
#     /dev/nvme1n1 -> /dev/xvdh
#
# these ebs block device paths are set by stacker and assumed by ansible.
#
# if the device is non ebs, it will use the following mapping:
non_ebs_mapping=("/dev/sdb1" "/dev/sdc1" "/dev/sdd1" "/dev/sde1" "/dev/sdf1" "/dev/sdg1")

# nvme0n1 uses ${non_ebs_mapping[0]} (the 0 index item in the array)
#
#     /dev/nvme0n1 -> /dev/sdb1
#
# nvme3n1 uses ${non_ebs_mapping[3]} (the 3 index item in the array)
#
#     /dev/nvme3n1 -> /dev/sde1
#

# why we only iterate from 0 to 26:
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html
for i in `seq 0 26`; do
    nvme_block_device="/dev/nvme${i}n1"

    # skip any nvme paths which don't exist.
    if [ -e  $nvme_block_device ]; then

        # get ebs block mapping device path set by stacker (or base AMI).
        mapping_device=$(/usr/local/bin/ebsnvme-id ${nvme_block_device} --block-dev)

        # if the mapping_device is empty, it isn't an EBS device so
        # we will use the non_ebs_mapping to translate the device.
        if [[ -z "$mapping_device" ]]; then
            mapping_device="${non_ebs_mapping[$i]}"
        fi

        # if block mapping device path does not start with /dev/ fix it.
        if [[ "$mapping_device" != /dev/* ]]; then
            mapping_device="/dev/${mapping_device}"
        fi

        # if the block mapping device path already exists, skip it.
        if [ -e $mapping_device ]; then
            echo "path exists: ${mapping_device}"

        # otherwise, create a symlink from nvme block device to mapping device.
        else
            echo "symlink created: ${nvme_block_device} to ${mapping_device}"
            ln -s $nvme_block_device $mapping_device
        fi
    fi
done

systemd 서비스 등록

/etc/systemd/system/ 경로에 아래 3가지 파일 생성 합니다.

• nvme-to-block-mapping.service : 심볼릭 링크생성
• format-dev-sdb.service : 파일시스템 포맷
• mount-data.mount : data 폴더로 마운트 (data폴더는 예시이므로 커스터마이징 하시기바랍니다)

nvme-to-block-mapping.service

[Unit]
Description=Symblic link nvme device name

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c 'sh /usr/local/bin/nvme-to-block-mapping.sh'

[Install]
WantedBy=multi-user.target

format-dev-sdb.service

[Unit]
Description=Formats /dev/sdb volume
After=nvme-to-block-mapping.service
Requires=nvme-to-block-mapping.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c 'blkid /dev/sdb || (wipefs -fa /dev/sdb && mkfs.ext4 /dev/sdb)'

[Install]
WantedBy=multi-user.target

mount-data.mount

[Unit]
Description=Mount /dev/sdb to /data
After=format-dev-sdb.service
Requires=format-dev-sdb.service

[Mount]
What=$(readlink /dev/sdb)
Where=/data
Type=ext4

[Install]
WantedBy=multi-user.target

서비스 등록 및 실행

systemctl daemon-reload

systemctl enable nvme-to-block-mapping.service format-dev-sdb.service mount-data.mount 

systemctl start nvme-to-block-mapping.service format-dev-sdb.service mount-data.mount