Jenkins is a continuous integration tool. It is used to build and deploy applications. Normally, there are number of build projects preconfigured in a Jenkins Server. However by default, all builds will be executed on the same instance that Jenkins is server up & running. This results to reduce your performance of the Jenkins Server. To avoid above problems, Jenkins provides the capability to execute builds on external hosts (called build agents). To fulfill above requiremnt, we are going to use the EC2 Fleet Plugin which launches EC2 Spot or On Demand instances as worker nodes for Jenkins server, automatically scaling the capacity with the load. This plugin helps us to:

1. Execute jobs in the spot instances

2. Scales instances as needed

3. Reduce our cost


  • Jenkins up and running
  • You need a Jenkins server up and running. This will be our master server. If you haven't Jenkins server, you can follow this tutorial and set up your Jenkins Master.

    Create Policy

    Before configuring the EC2 Fleet Jenkins Plugin, we need to create AWS IAM User for our EC2 Fleet Plugin. To create IAM User, you can follow the below instructions.

  • Go to AWS Console
  • Click on IAM
  • Then Click on Policies which is listed under the Access management from left side menu
  • Then Click on Create Policy
  • In next window, switch to JSON view and paste below code in it

  • 1
  • Then Click on Next: Tags and add tag for your created policy
  • Then Click on Next: Review
  • Then give a name for your policy and click on Create policy
  • Now you can use this policy to create IAM User.

    Create IAM User

  • Then Click on Users which is listed under the Access management from left side menu
  • Then Click on Add User
  • Then give a name to your user and check on Access key - Programmatic access. Then click on Next:Permission
  • Then select Attach existing policies directly. Then select the policy which created in previous step and click on Next:Tag and add tag for your created user
  • Click on Next: Review
  • Click on Create User
  • Download the .csv file for later use. Click on Close
  • Create a launch template

    This is the template that will be used to launch the EC2 instances.
  • Go to EC2 Service
  • Click on Launch Instance which is listed under the Instances from left side menu
  • Then Click on Create launch template
  • Then name the Launch Template and add small description
  • In Launch Template Contents section, choose the amzn2-ami-hvm-2.0.20220719.0-x86_64-gp2 (you can choose any image acording to your preferences)
  • In Instance type section, choose a t3.small (you can choose any type acording to your preferences
  • In Key pair (login) section, choose an existing key pair or create a new key pair that will be used to connect to these spot instances
  • In Network settings, Section choose VPC and security group for the instances
  • In Advanced details, Section check Request Spot Instances and add below script in User data

  • 1
    #!/bin/bash -xe
    sudo amazon-linux-extras install epel -y
    sudo yum update -y
    sudo yum remove java-1.7.0-openjdk
    sudo yum install java-1.8.0-openjdk -y
    sudo yum install git -y
    sudo yum -y update aws-cli
  • Finally click on Create Launch template
  • Create a Auto Scaling Group

    To create Auto Scaling Group
  • Click on Auto Scaling Groups
  • Then click on Create Auto Scaling group
  • Then name your Auto Scaling Group and choose the previously created Launch Temaple and click on Next button
  • Then choose desire VPC and all Availability Zones and subnets under choosen VPCpreviously created Launch Temaple and click on Next button
  • Then under Configure group size and scaling policies add Desired capacity = 1, Minimum capacity = 1, Maximum Capacity = 4
  • Then Review and create Auto Scaling Group
  • Now you can see one of the instances already spinning up.

    Install EC2-Fleet Plugin

  • Navigate to your Jenkins Dashboard and click on Manage Jenkins
  • Then goto Manage Plugins and select Available option and search for EC2 Fleet Jenkins Plugin and install it
  • Click on restart Jenkins after installation is complete
  • Setup EC2-Fleet Plugin

  • Go to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds
  • Choose Amazon EC2 Fleet from Add a new Cloud drop-down menu
  • Then give a name
  • In the AWS Credentials section, click on Add
  • Then choose Kind as AWS Credentials, Scope as Jenkins and nodes only, add name for id and paste created access key and secret key and finally click on Add
  • Then choose created credential as AWS Credentials from drop-down menu
  • Then choose region to your respective one
  • Choose created Auto Scaling Group for EC2 Fleet section
  • Click on Test Connection

  • In the Launcher Section

  • Change Launcher drop down to Launch Agents via SSH
  • Then choose Kind as SSH Username with Private Key, Scope as Jenkins and nodes only, add name for id and add ec2-user as username and paste content on our.pem file, we created on Launch template step and finally click on Add
  • Then choose created credential from drop-down menu
  • For Host Key Verification Strategy choose Non verifying verification strategy
  • Check Private IP option
  • Change label to spot-agents
  • Then add Number of executors: 1, Max Idle Minutes before scaledown: 5, Minimum Cluster Size: 1, Maximum Cluster Size: 5, Maximum Total Uses: -1 (default)
  • Leave the rest default. Click on Save
  • Conclusion

    In this tutorial, we learned how to configure EC2-Fleet plugin in Jenkins. If you have any issue regarding this tutorial, mention your issue in comment section or reach me through my E-mail.

    Happy Coding