Amazon EC2 cloud setup: Difference between revisions

From genomewiki
Jump to navigationJump to search
m (minor edit)
m (snapshot pending)
 
(7 intermediate revisions by the same user not shown)
Line 30: Line 30:
Verify regions available with the toolset command:
Verify regions available with the toolset command:
  $ ec2-describe-regions
  $ ec2-describe-regions
Which responds:
  REGION  eu-west-1      eu-west-1.ec2.amazonaws.com
  REGION  eu-west-1      eu-west-1.ec2.amazonaws.com
  REGION  us-east-1      us-east-1.ec2.amazonaws.com
  REGION  us-east-1      us-east-1.ec2.amazonaws.com
Line 51: Line 49:


  $ grep getting-started.manifest describe-images.txt  
  $ grep getting-started.manifest describe-images.txt  
Responds:
  IMAGE  ami-3c47a355    ec2-public-images/getting-started.manifest.xml ... etc...
  IMAGE  ami-3c47a355    ec2-public-images/getting-started.manifest.xml ... etc...


Note the identification ami-3c47a355, used in this command:
Note the identification ami-3c47a355, used in this command:
  $ ec2-run-instances ami-3c47a355 -k gsg-keypair
  $ ec2-run-instances ami-3c47a355 -k gsg-keypair
Which responds:
  RESERVATION    r-cee26ea7      891631566226    default
  RESERVATION    r-cee26ea7      891631566226    default
  INSTANCE        i-32a7c65b      ami-3c47a355                    pending gsg-keypair    0              m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc
  INSTANCE        i-32a7c65b      ami-3c47a355                    pending gsg-keypair    0              m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc
Line 66: Line 60:
Use in this command to display the status of the instance:
Use in this command to display the status of the instance:
  $ ec2-describe-instances i-32a7c65b
  $ ec2-describe-instances i-32a7c65b
Which responds:
  RESERVATION    r-cee26ea7      891631566226    default
  RESERVATION    r-cee26ea7      891631566226    default
  INSTANCE        i-32a7c65b      ami-3c47a355    ec2-72-44-35-57.compute-1.amazonaws.com domU-12-31-39-03-46-52.compute-1.internal      running gsg-keypair    0m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce  ari-a51cf9cc
  INSTANCE        i-32a7c65b      ami-3c47a355    ec2-72-44-35-57.compute-1.amazonaws.com domU-12-31-39-03-46-52.compute-1.internal      running gsg-keypair    0m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce  ari-a51cf9cc
Line 75: Line 67:
Open SSH and HTTP ports:
Open SSH and HTTP ports:
  $ ec2-authorize default -p 22
  $ ec2-authorize default -p 22
$ ec2-authorize default -p 80
responds:
  GROUP          default
  GROUP          default
  PERMISSION              default ALLOWS  tcp    22      22      FROM    CIDR  0.0.0.0/0
  PERMISSION              default ALLOWS  tcp    22      22      FROM    CIDR  0.0.0.0/0
$ ec2-authorize default -p 80
  GROUP          default
  GROUP          default
  PERMISSION              default ALLOWS  tcp    80      80      FROM    CIDR  0.0.0.0/0
  PERMISSION              default ALLOWS  tcp    80      80      FROM    CIDR  0.0.0.0/0
Line 87: Line 77:
To see SSH keys:
To see SSH keys:
  $ ec2-get-console-output i-32a7c65b
  $ ec2-get-console-output i-32a7c65b
Responds:
  ec2: 2048 92:81:4f:19:e6:5b:23:e6:78:b7:e2:5d:be:ea:70:1d /etc/ssh/ssh_host_key.pub
  ec2: 2048 92:81:4f:19:e6:5b:23:e6:78:b7:e2:5d:be:ea:70:1d /etc/ssh/ssh_host_key.pub
  ec2: 2048 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc /etc/ssh/ssh_host_rsa_key.pub
  ec2: 2048 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc /etc/ssh/ssh_host_rsa_key.pub
Line 116: Line 104:
</PRE>
</PRE>


There is your login prompt.  You are now a system administrator.
There is your login prompt.  You are now a Linux system administrator.
 
==Mounting snapshots==
 
Creating a snapshot based on the Ensembl image, identification found at: [http://aws.amazon.com/publicdatasets/ Public Datasets]
$ ec2-create-volume --snapshot snap-c78360ae -z us-east-1b
VOLUME  vol-399d7d50    550    snap-c78360ae  us-east-1b      creating      2009-04-03T17:19:02+0000
 
Attach that volume to a running image, as device /dev/sdh:
$ ec2-attach-volume vol-399d7d50 -i i-4bc1a022 -d /dev/sdh
ATTACHMENT      vol-399d7d50    i-4bc1a022      /dev/sdh        attaching      2009-04-03T17:25:04+0000
 
Taking a look at those:
$ ec2-describe-volumes
VOLUME  vol-399d7d50    550    snap-c78360ae  us-east-1b      in-use  2009-04-03T17:19:02+0000
ATTACHMENT      vol-399d7d50    i-4bc1a022      /dev/sdh        attached      2009-04-03T17:25:04+0000
 
Examining instances:
$ ec2-describe-instances | grep running
INSTANCE        i-4bc1a022      ami-3c47a355    ec2-75-101-203-203.compute-1.amazonaws.com      ip-10-251-126-22.ec2.internal  running gsg-keypair    0      m1.small 2009-04-03T17:03:17+0000        us-east-1b      aki-a71cf9ce    ari-a51cf9cc
 
Logged into the instance, mounting that volume:
 
# mkdir /mnt/ensembl
# mount /dev/sdh /mnt/ensembl
# cd /mnt/ensembl
# df -h .
/dev/sdh              546G  535G  11G  99% /mnt/ensembl
 
After it was mounted:
$ ec2-describe-volumes
VOLUME  vol-b49d7ddd    550    snap-c78360ae  us-east-1c      in-use  2009-04-03T17:57:54+0000
ATTACHMENT      vol-b49d7ddd    i-bac4a5d3      /dev/sdh        attached      2009-04-03T17:59:17+0000
 
==Creating a snapshot==
 
Create a 1000 Gb volume:
$ ec2-create-volume -s 1000 -z us-east-1c
VOLUME  vol-5e9e7e37    1000            us-east-1c      creating        2009-04-03T18:09:45+0000
 
Attach that volume to a running image, as device /dev/sdi:
$ ec2-attach-volume vol-5e9e7e37 -i i-bac4a5d3 -d /dev/sdi
 
Make a new filesystem on this volume, logged into the instance:
 
# yes | mkfs -t ext3 /dev/sdi
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
 
That took about 15 minutes, 8000 inode tables was the longest bit of the process.
Create a mount point and mount it:
# mkdir /mnt/ucsc
# mount /dev/sdi /mnt/ucsc
 
Load your data into that filesystem.  It is going into the S3 storage system
as that takes place.  When it is complete, create a snapshot:
$ ec2-create-snapshot vol-5e9e7e3
SNAPSHOT        snap-a8a352c1  vol-5e9e7e37    pending 2009-04-03T23:48:55+0000
 
==Turning Off an Instance==
 
You need to make sure you turn these instances off, they consume accounting
charges if they are left on:
 
$  ec2-terminate-instances i-32a7c65b
INSTANCE        i-32a7c65b      running shutting-down
 
[[Category:Cloud Computing]]

Latest revision as of 23:50, 3 April 2009

Initial Set Up

Following the instructions with Amazon documentation: Getting Started Guide

You have your account setup, and credit card submitted.

Going to use the Java toolset, lookup where java is installed on hgwdev, set environment:

JAVA_HOME=/usr/java/jre1.6.0_11

From your Amazon account management screen, save the cert-*.pem and pk-*.pem files and save in your home directory ~/.ec2/ directory, chmod 700 ~/.ec2/ for security, and set the files at chmod 600 to be read-only for yourself.

Set environment:

EC2_PRIVATE_KEY=$HOME/.ec2/pk-*.pem
EC2_CERT=$HOME/.ec2/cert-*.pem

Record your Amazon account number for later reference, it looks like: 1234-5678-9012

Pick up Java toolset from: Java Toolset

Unpack in some directory, and set environment:

EC2_HOME=/some/directory/ec2-api-tools-1.3-34128

Add $EC2_HOME/bin to your PATH

And to protect Java from the huge memory on hgwdev, set environment:

EC2_JVM_ARGS=-Xmx256M

Verify regions available with the toolset command:

$ ec2-describe-regions
REGION  eu-west-1       eu-west-1.ec2.amazonaws.com
REGION  us-east-1       us-east-1.ec2.amazonaws.com

Set environment:

EC2_URL=https://us-east-1.ec2.amazonaws.com

Create SSH keys:

$ cd $HOME/.ssh
$ ec2-add-keypair gsg-keypair > id_rsa.gsg-keypair
$ chmod 600 id_rsa-gsg-keypair

Sample AMI Images

Initial sample images are listed with:

$ ec2-describe-images -o self -o amazon > describe-images.txt

And, we want to use this one for a test first:

$ grep getting-started.manifest describe-images.txt 
IMAGE   ami-3c47a355    ec2-public-images/getting-started.manifest.xml ... etc...

Note the identification ami-3c47a355, used in this command:

$ ec2-run-instances ami-3c47a355 -k gsg-keypair
RESERVATION     r-cee26ea7      891631566226    default
INSTANCE        i-32a7c65b      ami-3c47a355                    pending gsg-keypair     0               m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce    ari-a51cf9cc

Means it is running, Note the instance identification: i-32a7c65b

Use in this command to display the status of the instance:

$ ec2-describe-instances i-32a7c65b
RESERVATION     r-cee26ea7      891631566226    default
INSTANCE        i-32a7c65b      ami-3c47a355    ec2-72-44-35-57.compute-1.amazonaws.com domU-12-31-39-03-46-52.compute-1.internal       running gsg-keypair    0m1.small        2009-04-02T22:47:17+0000        us-east-1c      aki-a71cf9ce   ari-a51cf9cc

Note the name of the machine in that output: ec2-72-44-35-57.compute-1.amazonaws.com

Open SSH and HTTP ports:

$ ec2-authorize default -p 22
GROUP           default
PERMISSION              default ALLOWS  tcp     22      22      FROM    CIDR   0.0.0.0/0
$ ec2-authorize default -p 80
GROUP           default
PERMISSION              default ALLOWS  tcp     80      80      FROM    CIDR   0.0.0.0/0

You can now see the Apache WEB server at: http://ec2-72-44-35-57.compute-1.amazonaws.com/

To see SSH keys:

$ ec2-get-console-output i-32a7c65b
ec2: 2048 92:81:4f:19:e6:5b:23:e6:78:b7:e2:5d:be:ea:70:1d /etc/ssh/ssh_host_key.pub
ec2: 2048 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc /etc/ssh/ssh_host_rsa_key.pub
ec2: 1024 4f:18:83:90:7b:11:be:c3:b4:c5:7a:a3:11:9d:26:76 /etc/ssh/ssh_host_dsa_key.pub

Login via ssh to this system:

$ ssh -i $HOME/.ssh/id_rsa.gsg-keypair root@ec2-72-44-35-57.compute-1.amazonaws.com

Verify the displayed RSH key is one of those from the console-output command:

RSA key fingerprint is 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc.
Are you sure you want to continue connecting (yes/no)? yes

         __|  __|_  )  Fedora 8
         _|  (     /    32-bit
        ___|\___|___|

 Welcome to an EC2 Public Image
                       :-)

    Getting Started

 --[ see /etc/ec2/release-notes ]--

[root@domU-12-31-39-03-46-52 ~]# 

There is your login prompt. You are now a Linux system administrator.

Mounting snapshots

Creating a snapshot based on the Ensembl image, identification found at: Public Datasets

$ ec2-create-volume --snapshot snap-c78360ae -z us-east-1b
VOLUME  vol-399d7d50    550     snap-c78360ae   us-east-1b      creating       2009-04-03T17:19:02+0000

Attach that volume to a running image, as device /dev/sdh:

$ ec2-attach-volume vol-399d7d50 -i i-4bc1a022 -d /dev/sdh
ATTACHMENT      vol-399d7d50    i-4bc1a022      /dev/sdh        attaching      2009-04-03T17:25:04+0000

Taking a look at those:

$ ec2-describe-volumes
VOLUME  vol-399d7d50    550     snap-c78360ae   us-east-1b      in-use  2009-04-03T17:19:02+0000
ATTACHMENT      vol-399d7d50    i-4bc1a022      /dev/sdh        attached       2009-04-03T17:25:04+0000

Examining instances:

$ ec2-describe-instances | grep running
INSTANCE        i-4bc1a022      ami-3c47a355    ec2-75-101-203-203.compute-1.amazonaws.com      ip-10-251-126-22.ec2.internal   running gsg-keypair     0      m1.small 2009-04-03T17:03:17+0000        us-east-1b      aki-a71cf9ce    ari-a51cf9cc

Logged into the instance, mounting that volume:

# mkdir /mnt/ensembl
# mount /dev/sdh /mnt/ensembl
# cd /mnt/ensembl
# df -h .
/dev/sdh              546G  535G   11G  99% /mnt/ensembl

After it was mounted:

$ ec2-describe-volumes
VOLUME  vol-b49d7ddd    550     snap-c78360ae   us-east-1c      in-use  2009-04-03T17:57:54+0000

ATTACHMENT vol-b49d7ddd i-bac4a5d3 /dev/sdh attached 2009-04-03T17:59:17+0000

Creating a snapshot

Create a 1000 Gb volume:

$ ec2-create-volume -s 1000 -z us-east-1c
VOLUME  vol-5e9e7e37    1000            us-east-1c      creating        2009-04-03T18:09:45+0000

Attach that volume to a running image, as device /dev/sdi:

$ ec2-attach-volume vol-5e9e7e37 -i i-bac4a5d3 -d /dev/sdi

Make a new filesystem on this volume, logged into the instance:

# yes | mkfs -t ext3 /dev/sdi
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

That took about 15 minutes, 8000 inode tables was the longest bit of the process. Create a mount point and mount it:

# mkdir /mnt/ucsc
# mount /dev/sdi /mnt/ucsc

Load your data into that filesystem. It is going into the S3 storage system as that takes place. When it is complete, create a snapshot:

$ ec2-create-snapshot vol-5e9e7e3
SNAPSHOT        snap-a8a352c1   vol-5e9e7e37    pending 2009-04-03T23:48:55+0000

Turning Off an Instance

You need to make sure you turn these instances off, they consume accounting charges if they are left on:

$  ec2-terminate-instances i-32a7c65b
INSTANCE        i-32a7c65b      running shutting-down