Bash Script to Schedule the Creation Snapshots for EC2 Volumes

Glenn's Wimpy Disclaimer

Usually I code in either Java, Groovy or Scala, however this was written in bash.  There was one reason for this:  No one else in my company can support any of the languages above easily.   

And as a DevOp I thought this could be useful to others.  With that being said please add comments on what it would take to make the script a little more professional.

What does it do?

In short this script should be launched daily by CRON and it will look for tagged volumes backup the volume based on the date associated with the tag.

Initializing the code

#!/bin/bash

echo "starting volume backup"

#The interval between each backup

dayCount=7

#The date the backups should removed.

EXPIRE_DATE=$(date -d "+$dayCount day"  +%Y%m%d)

#The backup activate date.  i.e. any volume with a date older than this will be backed up.

START_DATE=$(date -d "-$dayCount day"  +%Y%m%d)

#Todays Date.

CURRENT_DATE=$(date +%Y%m%d);

The code above is fairly simple initialize number of days between backups as well as establish the backup expire date and the date the volume should be backed up.

Primary Loop

# get all the volumes and review each one searching for the backupdate tag

ec2-describe-volumes | while read line

do

#get volume id

 volume=$(echo $line|cut -f3 -d' ')

#Get the backup key (if present)

 key=$(echo $line|cut -f4 -d' ')

#get the backup date

 value=$(echo $line|cut -f5 -d' ')

#if a backup tag is present for the volume check to see if its date signifies whether we should backup or not.

 if [ "$key" == "backupdate" ]

 then

  let vart=$value-$START_DATE

# should we backup the volume

  if [ $vart -lt 0 ]

  then

    echo "Backing up $volume"

    echo "for date $value"

    echo "new date $DATE"

#create the snapshot (backup) of the volume

    snapresult=$(ec2-create-snapshot $volume -d \"backup$DATE\")

#get the snapshot Id

    snapshot=$(echo $snapresult|cut -f2 -d' ')

#update the backupdate flag with the current date on the volume

    ec2-create-tags $volume --tag backupdate="$CURRENT_DATE"

#tag snapshot with metadata with the information on when to remove the snapshot

    ec2-create-tags $snapshot --tag backupdate="$EXPIRE_DATE"

    ec2-create-tags $snapshot --tag Name=$EXPIRE_DATE

  fi

 fi

# if no more entries exit the loop

 if [ -z "$line" ]

 then

   exit

 fi

done

echo "Finished volume backup" 

The comments in the code above should explain how volumes are acquired as well as how snapshots are created and tagged.

Once I finished the code above I added it to a CRON job that runs at 5:00 am every morning.

Hope this helps you out a little,
--Glenn

Sending Email Attachments using Spring Integration

Introduction

Here is a scenario I ran across recently, we had a customer who wanted us to process a series of files and place the contents into a relational database.  One of the requirements was that they wanted was to have a notification sent to their monitoring system (no big deal right).  When asked how they wanted us to send the notification, they said that they wanted it as an email notification with the result info in the attachment.  I was in a little shock assuming they would want it as an SNMP trap or Syslog etc. So as coding began there were no real discussion on email attachments using Spring Integration, so I decided to create one :).

The Message Flow

Figure1: Email Transformer and outbound mail adapter

So here we go, we see that 2 messageToMail transformers were created, one for success messages and the other for failed messages.  These could have been consolidated to one transformer, but I was experimenting.  Anyhow, we assume the processing of the file is complete and a status message has been sent to one of the messageToMail transformers.  The job of the transformer is to extract out the file name, status of the message and time finished then build up the email message.  Once the email message has been built it places it on to the outboundMail Channel in which the outbound mail adapter is listening.

For those who love the context xml here ya go:

<bean id="mailTransformerBean" class="com.concurrent.integration.EmailMessageTransformer">

<property name="mailTo" value="${email.mailTo}" />

<property name="mailFrom" value="${email.mailFrom}" />

<property name="mailSubject" value="${email.mailSubject}" />

<property name="mailHost" value="${email.mailHost}" />

<property name="attachmentName" value="${email.attachmentName}"/>

<property name="customer" value="${email.customer}"/>

<property name="processType" value="${email.processType}"/>

<property name="processName" value="${email.processName}"/>

<property name="processScript" value="${email.processScript}"/>

<property name="site" value="${email.site}"/>

</bean>

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">

<property name="host" value="${email.mailHost}" />

<property name="port" value="${email.mailPort}" />

<property name="username" value="${email.mailUserName}" />

<property name="password" value="${email.password}" />

<property name="javaMailProperties">

<props>

<prop key="mail.smtp.auth">true</prop>

<prop key="mail.smtp.starttls.enable">true</prop>

</props>

</property>

</bean>

 <si:channel id="outboundMail" />

 <!-- Here's the Spring Integration flow -->

<si:channel id="mailSuccessNotice"/>

<si:channel id="mailFailNotice"/>

 <si:transformer id="messageToMailXFormSuccess" ref="mailTransformerBean" input-channel="mailSuccessNotice"

 method="transform" output-channel="outboundMail" />

 <si:transformer id="messageToMailXFormFail" ref="mailTransformerBean" input-channel="mailFailNotice"

 method="transform" output-channel="outboundMail" />

The Transformer Code!

So now to the important part, the transformer code that builds the email and inserts the attachement.

1. The code is broken into 5 sections:
2. Get the Message 
3. Build the MIME message
4. Extract message data
5. Build email and attachment
6. Send!!!!

Get the message
The transformer is associated with the mailSuccessNotice -or- mailFailNotice channel.  In either case the transform method is called.  Its first job is to extract the message payload and retrieve the mail text.

public MailMessage transform(Message<?> message) {

   Object payload = message.getPayload();

   String mailText = payload.toString();

Build the MIME message
The second thing the transformer does is to create the mime message object that will house the contents of the email.

JavaMailSenderImpl sender = new JavaMailSenderImpl();

sender.setHost(mailHost);

MimeMessage mimeMsg = sender.createMimeMessage();

MimeMailMessage msg = new MimeMailMessage(mimeMsg);

try {

   msg = new MimeMailMessage(new MimeMessageHelper(mimeMsg, true));

} catch (Exception e) {

   logger.warn("An email creation error occured because: "+          e.getMessage());

}

Extract message data

Now lets get the basic data from the mail text.  I created a method that would retrieve the information from the payload and generate a status message to be placed in the email. The code following is method signature:

String processStatus = parseMessagePayload(mailText,statusData);

This is the body of the method:

CSVParser parser = new CSVParser(new StringReader(payload));

String processStatus = null;

try {

String [] results = parser.getLine();

processStatus = results[0];

startTime = results[1];

endTime = results[2];

if(processStatus.equals("FAIL")){

     statusData.statusElement.description = "Process ended with "+results[3] 

             +"Warnings";

}else{

     statusData.statusElement.description = "Process ended Successfully"; }

} catch (Exception e) {

logger.error("Unable to parse message from processor.  Incomplete Email will be sent.");

}

return processStatus;

Build email and attachment

Now the transformer will build up the data for the mime message.

/* Create the mime message using spring.framework.mail packages */

MimeMessage mimeMsg = sender.createMimeMessage();

MimeMailMessage msg = new MimeMailMessage(mimeMsg);

try {

msg = new MimeMailMessage(new MimeMessageHelper(mimeMsg, true));

} catch (Exception e) {

logger.warn("An email creation error occured because: "+ e.getMessage());

}

msg.setTo(mailTo);

msg.setFrom(mailFrom);

msg.setSubject(mailSubject);

msg.setSentDate(new Date());

StatusData statusData = new StatusData();

/*Once the status message has been generated we create a Notification POJO and initialize it with the data.  

*/

Notification notification = new Notification();

notification.customer = customer;

notification.messageType = processType;

notification.site = site;

notification.processType = processType;

notification.processName = processName;

notification.processScript = processScript;

notification.processStartTime = startTime;

notification.processEndTime = endTime;

notification.processStatus = processStatus;

ProcessData processData = new ProcessData();

if(processStatus.equals("FAIL")){

processData.failData = statusData;

}else{

processData.successData = statusData;

}

notification.processData = processData;

String notificationString = "";

/** the user wants the data in XML format since it is being ingested by their ticket tracking system. */

try {

final JAXBContext jaxbContext = JAXBContext

.newInstance(Notification.class);

        StringWriter writer = new StringWriter();

jaxbContext.createMarshaller().marshal(notification, writer);

notificationString = writer.toString();

} catch (Exception ex) {

logger.error("Unable to create the attachment.  Incomplete Email will be sent.");

}

/** the body of the email contains the mail text plus the serialized content notification.

*/

msg.setText(mailText + " " + notificationString);

/**

The user has requested that we add the message as an attachment to the email.

*/

try {

msg.getMimeMessageHelper().addAttachment(attachmentName,

new ByteArrayResource(notificationString.getBytes()));

} catch (Exception e) {

logger.error("Unable to append attachment to email.  Incomplete Email will be sent.");

}

Send!!!
Now the message has been created.  It is routed to the outbound mail channel where it is then handled by the mailSender object as enumerated below.

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">

<property name="host" value="${email.mailHost}" />

<property name="port" value="${email.mailPort}" />

<property name="username" value="${email.mailUserName}" />

<property name="password" value="${email.password}" />

<property name="javaMailProperties">

<props>

<prop key="mail.smtp.auth">true</prop>

<prop key="mail.smtp.starttls.enable">true</prop>

</props>

</property>

</bean>

Hope this helps out a little.   Please leave feedback.
Thanks,
--Glenn