The true power (and efficiency) of the cloud can only be realised if you orchestrate infrastructure as and when needed. This is usually done at scale through Kubernetes (K8s). However for small projects K8 is expensive and an overkill.
Auto AWS is a lightweight python library that can create your AWS infrastructure, starting with EC2. The code can be found on my Github
Usage
|
|
In case you are wondering the config file looks like this
[EC2]
tag=name
ami=ami-005a087ff43033c77
key_pair=someKey
security_group=ec2-sg
max_bid=1.0
type=r5.2xlarge
region=us-east-1
product_description=Linux/UNIX
username=ubuntu
public_ip_address=x.x.x.x
iam_role=arn:aws:iam::888888888:instance-profile/name
valid_hours=6
To run jobs on start up of the machine, save bash
commands in userdata.txt
(in the same folder as your script)
Eg.
|
|
When you launch it, it will tell you the spot price you are paying, along with helpful logs.
Spot price $0.1956
Spot request created, status: open
Waiting for spot provisioning
Waiting... None
Waiting... None
Instance allocated ,Id: i-006071b23e23207f4
You can see your running instances too
|
|
InstanceId PublicIpAddress InstanceType state name
0 i-099707305b10aa727 x.x.x.x t2.micro running bastion
Once you are done with your job, you can destroy it in the same session.
|
|
Next steps
More EC2 automation.
- Pass
userdata.txt
as string inec2.create()
rather than having to save to disk. - Automatically stop instance when done. Eg. monitor a S3 file which can signal to AutoEC2 that the job is complete and so it can kill the instance.