30 April 2020

Deploying IaC using Terraform

TERRAFORM is used to automate deployment in infrastructure across multi providers in both public and private clouds. Provisioning of infrastructure through 'software' to achieve 'consistent' and 'predictable' environments is Infrastructure as code

               IaC - In simple words replacement of SOPs which is automated on top of it.

core concepts to achieve this:
  • Defined in Code: Iac should be defined in code weather in form of json yaml or hcl.
  • Stored in source control: the code should be stored somewhere in version source control repository like GitHub.
  • Declarative & Imperative: In imperative I am going to tell software each and everything which it needs to do the job. In declarative software already have some sort of Idea or a predefined routine, what it is going to do with taking some references.so terraform is an example of declarative approach to deploy IaC
  • Idempotent & consistence: once a job is done, and if again I get a request to do the same job it is Idempotent behavior of terraform to not repeat the steps done while fulfilling this job, instead will say there is no change in configuration and current requirement is same as the desired one so no changes needs to be made. otherwise in non-idempotent world each time this job comes it gonna repeat the same steps again and again to fulfil the requirement which is already in place.
  • Push & pull : terraform works on the principle of push mechanism where it pushes the configuration to its target.
Key benefit here is - everything is documented in code, which makes you understand about your infrastructure in more details.
key terraform components
In this exercise I am trying to demonstrate as how you can quickly deploy t2.micro instance of amazon linux without login into aws console by just writing a terraform plan
to being with you need to fulfill a prerequisite:
                          
                             "you should have an IAM user with AWS CLI access"

and to form a terraform configuration file with .tf as extension, below are few blocks which terraform tend to use to define things

#VARIABLES - input variables ca be declared here
#PROVIDER - AWS, google like providers can be declared here
#DATA - data from provider collected here in form of data source
#RESOURCE - feeding info about resources from provider here
#OUTPUT - data is outputted when apply is called

defining variables in terraform can be achieved in multiple ways, so here I tried to define variable from an external file

To persist variable values, create a file and assign variables within this file. Create a file named terraform.tfvars with the following contents:

aws_access_key = "AKIA5O3GDEMOVOBSE4RA"
aws_secret_key = "+bh/vVqoDEMOErxv7YlrSs/sdRwN9ZzeKDtAjCP" key_name = "tfkeypair"
private_key_path = "C:\\tfkeypair.pem"

For all files which match terraform.tfvars or *.auto.tfvars present in the config directory terraform reads the variable

So in this exercise I tried to deploy an t2.micro instance on Amazon EC instance with nginx up and running on it.

and at the end your terraform configuration files structure will look like:


let start composing a terraform configuration file

#VARIABLES

first we are going to define set of variables here, that are used during the configuration. I have defined keypairs so that we can SSH to our AWS instance, with default region where my instance will be deployed

#VARIABLES
variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "key_name" {} variable "private_key_path" {} variable "region" { default = "us-east-1" }


#PROVIDER

here we are defining our providers, and feeding information about our key details defined in our variable section with syntax var.variableName

#PROVIDER
provider "aws" {
access_key = var.aws_access_key secret_key = var.aws_secret_key region = var.region }


#DATA

here in data source block we are pulling data from the provider, in this exercise we are using amazon as provider and using linux ami for our EC2 instance

#DATA
data "aws_ami" "aws-linux" {
most_recent = true owners = ["amazon"] filter { name = "name" values = ["amzn-ami-hvm*"] } filter { name = "root-device-type" values = ["ebs"] } filter { name = "virtualization-type" values = ["hvm"] } }

#RESOURCE

In this block we can define more then one resources, here I have used default VPC, so that it will not be deleted on destroying the instance. Next we have defined security group so that we can SSH to our instance which is going to run nginx in this example, and opening port 80 & 22 ; and for that we need to define VPC id so that it will create the security group.

#RESOURCE
resource "aws_default_vpc" "default" {
} resource "aws_security_group" "allow_ssh" { name = "nginx_demo" description = "allow ports for nginx demo" vpc_id = aws_default_vpc.default.id

# to allow traffic from outside to inside
ingress {
from_port = 22 to_port = 22 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } ingress { from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] }

# to allow traffic from inside to outside i.e. from instance to internet
egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } }


# in this block we are actually defining out instance which will be nginx with t2.micro as resource type
resource "aws_instance" "nginx" {
ami = data.aws_ami.aws-linux.id instance_type = "t2.micro" key_name = var.key_name vpc_security_group_ids = [aws_security_group.allow_ssh.id]


# since we are doing SSH so we need to define a connection in resource block, so that terraform understand where to connect with connection { type = "ssh" host = self.public_ip user = "ec2-user" private_key = file(var.private_key_path) } # since we want to remotely exec command so provisioner "remote-exec" { inline = [ "sudo yum install nginx -y" , "sudo service nginx start" ] } }

#OUTPUT

this block will help to give you the output of your configuration

#OUTPUT
output "aws_instance_public_dns" {
value = aws_instance.nginx.public_dns }

#END

now to deploy above configuration, terraform deployment process follows a cycle:

Initialization 
Planning
Application 
destruction

$ terraform init
this initializes the terraform configuration and check for provider modules/plugins if its already not available and downloads the modules as shown below

$ terraform plan -out ami.tfplan
it looks for configuration file in pwd and loads any variables if found in form of terraform.tfvars file , and stores out the plan as shown below

$ terraform apply "ami.tfplan" 
it performs the configuration you created as code, applies it to provider and does the magic
now test the configuration by hitting above highlighted url and you will see result as:
and validate from your aws console you will see this:


now if you dont want the configs to be active and charge you money you can destroy it

$ terraform destroy
lastly from your config folder you can destroy the config you applied and it will destroy everything corresponding to your config


Br,
Punit


03 April 2020

Google Cloud Platform Commands sheet

Google as a cloud provider gives your various web services to build your infrastructure and automate it for seamless delivery of your application in production environment in a highly secure way.
This post lists out all the gcloud command which you can apply in your google cloud operations. 

GCP comprises of 3 core components:

Network          ¬Moving
Compute         ¬Processing
Storage           ¬Remembering

GCP Basics 
FunctionCommand
Check version
settings
gcloud version
gcloud info
gcloud components list                                         
Init profile
gcloud init
list services
list enabled services
 gcloud services list
 gcloud services list --enabled
Upgrade local SDKgcloud components update
gcloud components update --version 219.0.1
List all sql instances gcloud sql instances list
List all zonesgcloud compute zones list                 

Project configs
FunctionCommand
List projectsgcloud projects list, gcloud config list                     
Show project infogcloud compute project-info describe
Switch project           gcloud config set project <project-id>
create project & set as defaultgcloud projects create mygcp-project-777 --name mygcp-project --set-as-default                       
set a default projectgcloud config set core/project mygcp-project-777                
set default compute regions & zonegcloud config set compute/region europe-west6
 gcloud config set compute/zone europe-west-6-a

Bucket Basics, gsutil=gcloud storage
FunctionCommand
List all buckets and filesgsutil lsgsutil ls -lh gs://<bucket-name>
Download filegsutil cp gs://<bucket-name>/<dir-path>/package-1.1.tgz .
Upload filegsutil cp <filename> gs://<bucket-name>/<directory>/
Cat filegsutil cat gs://<bucket-name>/<filepath>/
Delete filegsutil rm gs://<bucket-name>/<filepath>
Move filegsutil mv <src-filepath> gs://<bucket-name>/<directory>/<dest-filepath>
Copy foldergsutil cp -r ./conf gs://<bucket-name>/
Show disk usagegsutil du -h gs://<bucket-name/<directory>
Create bucketgsutil mb gs://<bucket-name>
Make all files readablegsutil -m acl set -R -a public-read gs://<bucket-name>/
Config auth    gsutil config -a
Grant bucket access                 gsutil iam ch user:pporwal@gmail.com:objectCreator,objectViewer gs://<bucket-name>       
Remove bucket access gsutil iam ch -d user:pporwal@gmail.com:objectCreator,objectViewer gs://<bucket-name>
Calculate file sha1sumgsha1sum syslog-migration-10.0.2.tgz, shasum syslog-migration-10.0.2.tgz
Gsutil helpgsutil help
 gsutil help options

Image & Containers
FunctionCommand
List all imagesgcloud compute images list     
List all container clustersgcloud container clusters list                                           
Set kubectl contextgcloud container clusters get-credentials <cluster-name>

GKE
FunctionCommand
Set the active accountgcloud config set account <ACCOUNT>                                      
Set kubectl contextgcloud container clusters get-credentials <cluster-name>
Change regiongcloud config set compute/region us-west
Change zonegcloud config set compute/zone us-west1-b
List all container clustersgcloud container clusters list

IAM
FunctionCommand
Authenticate client                   gcloud auth activate-service-account --key-file <key-file>
list of credentialed accounts  gcloud auth list                                                       
Set the active accountgcloud config set account <ACCOUNT>         
Auth to GCP container registrygcloud auth configure-docker
Print token for active accountgcloud auth print-access-tokengcloud auth print-refresh-token                          
Revoke generated credentialgcloud auth <application-default> revoke

Compute Instance
FunctionCommand
List all instances                                gcloud compute instances listgcloud compute instance-templates list
Show instance infogcloud compute instances describe "<instance-name>" --project "<project-name>" --zone "us-west2-a"
Stop an instancegcloud compute instances stop myinstance   
Start an instancegcloud compute instances start myinstance
Create an instancegcloud compute instances create vm1 --image image1 --tags test --zone "<zone>" --machine-type f1-micro    
SSH to instancegcloud compute ssh --project "<project-name>" --zone "<zone-name>" "<instance-name>"
Download filesgcloud compute copy-files example-instance:~/REMOTE-DIR ~/LOCAL-DIR --zone us-central1-a
Upload filesgcloud compute copy-files ~/LOCAL-FILE-1 example-instance:~/REMOTE-DIR --zone us-central1-a

Compute Columes/Disk
Function                              Command
List all disksgcloud compute disks list
List all disk typesgcloud compute disk-types list
List all snapshotsgcloud compute snapshots list
Create snapshotgcloud compute disks snapshot <diskname> --snapshotname <name1> --zone $zone

Compute Network
FunctionCommand
List all networksgcloud compute networks list                                               
Detail of one network                      gcloud compute networks describe <network-name> --format json                                        
Create network with auto subnetgcloud compute networks create <network-name>
Create n/w with subnetgcloud compute networks subnets create subnet1 --network my-vcp --range 192.168.0.0/24
Get a static ipgcloud compute addresses create --region us-west2-a vpn-1-static-ip
List all ip addressesgcloud compute addresses list
Describe ip addressgcloud compute addresses describe <ip-name> --region us-central1
List all routesgcloud compute routes list

DNS
FunctionCommand
List of all record-sets in my zonegcloud dns record-sets list --zone my_zone                                                     
List first 10 DNS recordsgcloud dns record-sets list --zone my_zone --limit=10

Compute Firewall
FunctionCommand
List all firewall rules                gcloud compute firewall-rules list
List all forwarding rulesgcloud compute forwarding-rules list
Describe one firewall rulegcloud compute firewall-rules describe <rule-name>
Create one firewall rulegcloud compute firewall-rules create my-rule --network default --allow tcp:9200 tcp:3306
Update one firewall rulegcloud compute firewall-rules update default --network default --allow tcp:9200 tcp:9300

Compute Services
FunctionCommand
List my backend servicesgcloud compute backend-services list
List all my health check endpointsgcloud compute http-health-checks list                                                              
List all URL mapsgcloud compute url-maps list

some points to remember in VPC

there are two modes of VPC
1. AUTO MODE
2. CUSTOM MODE

To create VPC, GCP API should be enabled.
A VPC network is global whereas Subnets are regional.
By default in VPC there is 1 subnet for all regions.
Each subnet is region comes up with 4 firewall rules.
    rule1 allow ICMP (ping)
    rule2 allow for internal use in CIDR
    rule3 allow TCP:3389 (RDP)
    rule4 allow TCP:22 (SSH)
all above rules are ingress type rules.
Firewall rules are global, can be applied by instance-level-tag/service account.
By default it blocks all the data coming in, & allows all the data going out.

to automatically create a subnet in every region:

Subnets have a */20 CIDR range (e.g. 192.168.0.0/20).
Get all subnets of a VPC network

$ gcloud compute networks subnets list --filter="network:my-vpc"

Filter syntax

Create a compute instance with a specific machine type
$ gcloud compute instances create i1 --machine-type=n1-standard-2

Machine type

Default machine type is n1-standard-1 (1 CPU, 3.75 GB RAM)
Instance name argument can be repeated to create multiple instances
Create a compute instance in a specific VPC network and subnet

$ gcloud compute instances create i1 --network my-vpc --subnet my-subnet-1

Default VPC network is default

If --network is set to a VPC network with “custom” subnet mode, then --subnet must also be specified
Instance name argument can be repeated to create multiple instances

Create a compute instance with a specific OS image

$ gcloud compute instances create i1 --image-family ubuntu-1804-lts --image-project ubuntu-os-cloud

Images

Default image family is debian-9
User either --image-family (uses latest image of this family) or --image (a concrete image)
--image-project serves as a namespace for --image and --image-family(may have multiple images/image families with same name in multiple projects)

List all available images (including projects and families) with:
$ gcloud compute images list

Get the VPC network and subnet of a compute instance
{
$ gcloud compute instances describe i1 --format "value(networkInterfaces.network)" | sed 's|.*/||'
$ gcloud compute instances describe i1 --format "value(networkInterfaces.subnetwork)" |sed 's|.*/||'
}


Format syntax
Get the names of all compute instances
$ gcloud compute instances list --format="value(name)"

Can be used, for example, for deleting all existing compute instances:
$ gcloud compute instances delete $(gcloud compute instances list --format="value(name)")

Allow ingress traffic to a VPC network
$ gcloud compute firewall-rules create my-vpc-allow-ssh-icmp --network my-vpc --allow tcp:22,icmp --source-ranges 0.0.0.0/0

0.0.0.0/0 is the default for --source-ranges and could be omitted.

This allows incoming ICMP and SSH (TCP port 22) traffic to any instances in the VPC network from any source (e.g. from the public Internet).

After creating this firewall rule, you’re able to:
Ping instances in the VPC network:
Ping EXTERNAL_IP
SSH to instances in the VPC network: 
$ gcloud compute ssh i1

Note that a newly created VPC network has no firewall rules applied and instances cannot be reached at all (not even from inside the VPC network). 
You have to create firewall rules to make compute instances reachable.
Create a regional static IP address
$ gcloud compute addresses create addr-1 --region=europe-west6

Regional IP addresses can be attached to compute instances, regional load balancers, etc. in the same region as the IP address.
The name argument can be repeated to create multiple addresses

One of --global or --region must be specified.
Create a global static IP address
$ gcloud compute addresses create addr-1 --global

Global IP addresses can only be attached to global HTTPS, SSL proxy, and TCP proxy load balancers.
The name argument can be repeated to create multiple addresses.

One of --global or --region must be specified.

keep clouding!!