[ChatOps] Slack과 AWS Lambda 를 이용한 AWS EC2 제어

Intro

AWS EC2를 슬랙 채팅창의 명령어를 통해 쉽게 컨트롤 하는 방법을 알려드리도록 하겠습니다.

slack의 slash command 기능을 사용하고, AWS Lambda의 python 코딩이 필요하며, AWS API Gateway, AWS SSM이 사용됩니다.

본문 내용은 해당 블로그를 참고 하였습니다 :) [https://aws-diary.tistory.com/107]

ChatOps 아키텍처

slack workspace 및 slack api 만들기

workspace 생성

slack 메일 인증

slack api설정

• https://api.slack.com/apps
• create new app - 이름입력, 위에서만든 workspace 입력

Slack credential 토큰 인증값 확인

• https://api.slack.com/apps
• 해당 workspace- Basic information - App Credentials
• 토큰값 확인
  

  

AWS Lambda 함수 생성

slack-auth

• Python3.6
• Layer 구성 필요 :

  slack-slash-command-75453822-7715-4fb1-9229-26c4742392a8 (1).zip

  8.9 kB

import json
import boto3

def check_token(token):
    token_list = ["Slack Verify Token"]
    if token in token_list:
        return True
    else:
        return False

def lambda_handler(event, context):
    print(event)
    token = event['token']
    client = boto3.client('lambda')
    if check_token(token):
        response = client.invoke_async(
            FunctionName='slack-slash-command-'+event['command'][1:].lower(),
            InvokeArgs=json.dumps(event)
        )
        return {
            # "response_type": "in_channel",
            "text": "Check the Slack Token...\nSuccess Authentication!\nIt takes few seconds to process..."
        }
    else:
        return {
            # "response_type": "in_channel",
            "text": "Check the Slack Token...\nFailed Authentication\nPlease Check the Slash Command App Setting"
        }

slack-slash-command-ec2

• Python3.6
• Layer 구성 필요 : 위 파일 참고

import json
import boto3
import slackweb

def slack_send(url, message):
    slack = slackweb.Slack(url=url)
    # slack.notify(response_type="in_channel", text=message)
    slack.notify(text=message)

def get_client(token):
    ssm = boto3.client('ssm')
    response = ssm.get_parameter(Name=token, WithDecryption=True)
    key = response['Parameter']['Value']
    key = key.split(',')
    return boto3.client(
        'ec2',
        aws_access_key_id=key[0],
        aws_secret_access_key=key[1]
    )

def status(client, tag_key, value):
    info = ""
    ec2_list = client.describe_instances()
    for ec2 in ec2_list['Reservations']:
        name = ""
        resource_env = ""
        ec2_info = ec2['Instances'][0]
        try:
            for tag in ec2_info['Tags']:
                if tag['Key'] == "Name":
                    name = tag['Value']
                elif tag['Key'] == 'Env':
                    resource_env = tag['Value']
            if tag_key == "all":
                info += name + "\t" + ec2_info['State']['Name'] + "\t" + ec2_info['PrivateIpAddress'] + "\t" + ec2_info['InstanceType'] + "\t" + resource_env +"\n"
            else:
                for tag in ec2_info['Tags']:
                    if tag_key.lower() == tag['Key'].lower():
                        if value.lower() in tag['Value'].lower():
                            info += name + "\t" + ec2_info['State']['Name'] + "\t" + ec2_info['PrivateIpAddress'] + "\t" + ec2_info['InstanceType'] + "\t" + resource_env +"\n"
        except :
            continue
    print(info)
    return info

def stop(client, tag_key, value):
    response = client.describe_instances()
    ec2_list = []
    for ec2 in response['Reservations']:
        for instance in ec2['Instances']:
            if tag_key == "all":
                ec2_list.append(instance['InstanceId'])
            else:
                try:
                    for tags in instance['Tags']:
                        if tags['Key'].upper() == tag_key.upper():
                            if value.upper() in tags['Value'].upper():
                                print(tags['Value'] + ':' + instance['InstanceId'])
                                ec2_list.append(instance['InstanceId'])
                except :
                    continue
    stop_response = client.stop_instances(InstanceIds=ec2_list)
    stop_instances = stop_response['StoppingInstances']
    result = ""
    for stop_instance in stop_instances:
        result += stop_instance['InstanceId'] + " " + stop_instance['CurrentState']['Name'] + "\n"
    return result

def start(client, tag_key, value):
    response = client.describe_instances()
    ec2_list = []
    for ec2 in response['Reservations']:
        for instance in ec2['Instances']:
            if tag_key == "all":
                ec2_list.append(instance['InstanceId'])
            else:
                try:
                    for tags in instance['Tags']:
                        if tags['Key'].upper() == tag_key.upper():
                            if value.upper() in tags['Value'].upper():
                                print(tags['Value'] + ':' + instance['InstanceId'])
                                ec2_list.append(instance['InstanceId'])
                except:
                    continue
    start_response = client.start_instances(InstanceIds=ec2_list)
    start_instances = start_response['StartingInstances']
    result = ""
    for start_instance in start_instances:
        result += start_instance['InstanceId'] + " " + start_instance['CurrentState']['Name'] + "\n"
    return result

def help():
    message = "***EC2 Commnand 사용법***\n"\
    "명령어 문법은 다음과 같습니다 -> '/ec2 command tag-key tag-value'\n"\
    "ex1) '/ec2 status env dev' -> Env 태그 값에 dev가 포함된 EC2 List를 가져옵니다.\n"\
    "ex2) '/ec2 start name test' -> Name 태그 값에 test가 포함된 EC2 서버들을 구동시킵니다.\n"\
    "ex3) '/ec2 status(or start) all -> 모든 EC2 List를 가져옵니다. (or 구동시킵니다.)\n"\
    "status tag value : 해당 tag의 값에 value값이 포함된 EC2 List를 가져옵니다.\n"\
    "start tag value : 해당 tag의 값에 value값이 포함된 EC2 서버들을 구동시킵니다.\n"\
    "stop tag value : 해당 tag의 값에 value값이 포함된 EC2 서버들을 정지시킵니다."
    print(message)
    return message

def do_act(token, act):
    client = get_client(token)
    function = act[0]
    if len(act) >= 2:
        tag_key = act[1]
        if tag_key == "all":
            value = "all"
        elif tag_key == "help":
            value = ""
        else:
            value = act[2]
    if function == "status":
        return status(client, tag_key, value)
    elif function == "start":
        return start(client, tag_key, value)
    elif function == "stop":
        return stop(client, tag_key, value)
    elif function == "help":
        return help()
    else:
        return "Please Check the function"

def lambda_handler(event, context):
    act = event['text'].split(' ')
    print(act)
    result = do_act(event['token'], act)
    slack_send(event['response_url'], result)

slack-slash-command-ag

• Python3.6
• Layer 구성 필요 : 위 파일 참고

import json
import boto3
import slackweb

def slack_send(url, message):
    slack = slackweb.Slack(url=url)
    # slack.notify(response_type="in_channel", text=message)
    slack.notify(text=message)

def get_client(service_name, token):
    ssm = boto3.client('ssm', region_name="ap-northeast-2")
    response = ssm.get_parameter(Name=token, WithDecryption=True)
    key = response['Parameter']['Value']
    key = key.split(',')

    return boto3.client(
        service_name,
        aws_access_key_id=key[0],
        aws_secret_access_key=key[1],
        region_name="ap-northeast-2"
    )

def ec2_describe(token, ec2_id_list):
    info = ""
    client = get_client('ec2', token)
    ec2_list = client.describe_instances(InstanceIds=ec2_id_list)
    for ec2 in ec2_list['Reservations']:
        name = ""
        resource_env = ""
        ec2_info = ec2['Instances'][0]
        try:
            for tag in ec2_info['Tags']:
                if tag['Key'] == "Name":
                    name = tag['Value']
                elif tag['Key'] == 'Env':
                    resource_env = tag['Value']
            info += name + "\t" + ec2_info['State']['Name'] + "\t" + ec2_info['PrivateIpAddress'] + "\t" + ec2_info['InstanceType'] + "\t" + resource_env +"\n"
        except :
            continue
    print("EC2 List\n" + info)
    return info

def describe(client, token):
    info = ""
    eks_ng_name = ""
    ag_list = client.describe_auto_scaling_groups()
    for ag in ag_list['AutoScalingGroups']:
        ag_name = ag['AutoScalingGroupName']
        ag_des_capacity = ag['DesiredCapacity']
        ag_min_capacity = ag['MinSize']
        ag_max_capacity = ag['MaxSize']
        ag_tags = ag['Tags']
        try:
            for tags in ag_tags:
                print( "tags key:"+tags['Key'])
                print( "tags value:"+tags['Value'])
                if tags['Key'] == 'eks:nodegroup-name' :
                     eks_ng_name = tags['Value']

            #info += ec2_describe(token, ec2_id_list)
        except :
            continue

        info += ag_name + "\t(" +eks_ng_name + ")\t" + str(ag_des_capacity) + "\t" + str(ag_min_capacity) + "\t" + str(ag_max_capacity) + "\n"
        ec2_list = ag['Instances']
        try:
            ec2_id_list=[]
            for ec2 in ec2_list:
                ec2_id_list.append(ec2['InstanceId'])
            #info += ec2_describe(token, ec2_id_list)
        except :
            continue
    print(info)
    return info

def all_ag_name(client, token):
    info = ""
    ag_list = client.describe_auto_scaling_groups()
    ag_name_list = []

    for ag in ag_list['AutoScalingGroups']:
        ag_name = ag['AutoScalingGroupName']
        ag_name_list.append(ag_name)
    return ag_name_list

def update(client, name, desire_capacity, min_capacity, max_capacity, token):

    if name == 'all' :
        asg_names = all_ag_name(client, token)
        count = 0
        message = ""

        message = message + "- Current autuscaling gruops : " + str(asg_names) +"\n\n"+ "- Total ausotscaling groups count : "+ str(len(asg_names)) + "\n\n\n"

        for asg in asg_names :

            count = count + 1

            response = client.update_auto_scaling_group(
            AutoScalingGroupName=asg,
            MinSize=int(min_capacity),
            MaxSize=int(max_capacity),
            DesiredCapacity=int(desire_capacity)
            )

            if response['ResponseMetadata']['HTTPStatusCode'] == 200:
                message = message + "["+str(count)+"/"+str(len(asg_names))+"] Success "+ asg + " Update Autoscaling Group \n-> " + "Min:" + str(min_capacity) + ", Max:" + str(max_capacity) + ", Desire: "+ str(desire_capacity)+"\n\n"
            else:
                message = message + "[" +str(count)+"/"+str(len(asg_names))+"] Failed "+ asg + " Update Autoscaling Group\n"

        print(message)

        return message

    else:
        response = client.update_auto_scaling_group(
            AutoScalingGroupName=name,
            MinSize=int(min_capacity),
            MaxSize=int(max_capacity),
            DesiredCapacity=int(desire_capacity)
        )

        if response['ResponseMetadata']['HTTPStatusCode'] == 200:
            return "Success "+ name + " Update Autoscaling Group \n-> " + "Min:" + str(min_capacity) + ", Max:" + str(max_capacity) + ", Desire:"+ str(desire_capacity)
        else:
            return "Failed "+ name + " Update Autoscaling Group"

def help():
    message = "*** AG Commnand 사용법 ***\n"\
    "명령어 문법은 다음과 같습니다 -> '/ag command params'\n"\
    "ex1) '/ag describe' -> Auto Scailing Group을 나열합니다.\n"\
    "ex2) '/ag update eks-f6b7904b-daee-949a-2805-e807f1c1bac4 2 2 6' -> Auto Scailing Group의 용량을 목표2 최소2 최대6 수정\n"\
    "describe : 모든 Auto Scailing Group과 포함된 EC2 List를 가져옵니다.\n"\
    "update name desire_capacity min_capacity max_capacity : 오토스케일링 그룹을 업데이트 합니다."
    #print(message)
    return message

def do_act(token, act):

    client = get_client('autoscaling', token)

    function = act[0]


    if function == "describe":
        return describe(client, token)
    elif function == "update":
        return update(client, act[1], act[2], act[3], act[4], token)
    elif function == "help":
        return help()
    else:
        return "Please Check the function"

def lambda_handler(event, context):
    act = event['text'].split(' ')

    result = do_act(event['token'], act)

    slack_send(event['response_url'], result)

AWS SSM 생성

AWS Systems Manager - Parameter Store - Create parameter

• Name : 슬랙 토큰 값
• Value : aws_access_key_id,aws_secret_access_key

API Gateway 생성

Create API

Create resource

Create Method - POST

실행 시킬 Lambda함수 입력

Integration Request - Mapping Templates

Add mapping template

• ResfulAPI 요청을 할때 요청 형식 변환하는 작업
• Content Type : application/x-www-form-urlencoded

## convert HTML POST data to JSON

## get the raw post data from the AWS built-in variable and give it a nicer name
#set($rawAPIData = $input.path('$'))

## first we get the number of "&" in the string, this tells us if there is more than one key value pair
#set($countAmpersands = $rawAPIData.length() - $rawAPIData.replace("&", "").length())

## if there are no "&" at all then we have only one key value pair.
## we append an ampersand to the string so that we can tokenise it the same way as multiple kv pairs.
## the "empty" kv pair to the right of the ampersand will be ignored anyway.
#if ($countAmpersands == 0)
 #set($rawPostData = $rawAPIData + "&")
#end

## now we tokenise using the ampersand(s)
#set($tokenisedAmpersand = $rawAPIData.split("&"))

## we set up a variable to hold the valid key value pairs
#set($tokenisedEquals = [])

## now we set up a loop to find the valid key value pairs, which must contain only one "="
#foreach( $kvPair in $tokenisedAmpersand )
 #set($countEquals = $kvPair.length() - $kvPair.replace("=", "").length())
 #if ($countEquals == 1)
  #set($kvTokenised = $kvPair.split("="))
  #if ($kvTokenised[0].length() > 0)
   ## we found a valid key value pair. add it to the list.
   #set($devNull = $tokenisedEquals.add($kvPair))
  #end
 #end
#end

## next we set up our loop inside the output structure "{" and "}"
{
#foreach( $kvPair in $tokenisedEquals )
  ## finally we output the JSON for this pair and append a comma if this isn't the last pair
  #set($kvTokenised = $kvPair.split("="))
 "$util.urlDecode($kvTokenised[0])" : #if($kvTokenised[1].length() > 0)"$util.urlDecode($kvTokenised[1])"#{else}""#end#if( $foreach.hasNext ),#end
#end
}

참고 - Content Type 이란

• RestAPI의 경우 보통 Json타입으로 요청하고, 요청 받음
• 그래서 Content-type이 application/json 타입으로 설정하는게 대부분임
• 그러나 그렇지 않은 경우도 있음
• html form태그를 사용하여 POST방식으로 요청하거나, jQuery의 ajax 등의 요청을 할때는 application/x-www-form-urlencoded 임
• application/x-www-form-urlencoded

Name=dragon+kim&Age=100

• application/json

{
"Name" : "dragon kim",
"Age" : "100"
}

API deploy

Slack API에 등록

slack api - slach commands -Create New Command