ShauryaAg

Featured.
How I designed and deployed a scalable microservice architecture with limited resources
ShauryaAg on Apr 03, 2021
Setting up CI/CD Pipeline for a Monolithic Service
ShauryaAg on Apr 03, 2021
How I designed a CI/CD setup for a Microservice Architecture at zero cost
ShauryaAg on Apr 03, 2021
Blogs.

How I designed and deployed a scalable microservice architecture with limited resources

ShauryaAg on Apr 03, 2021

Ok, so let’s start with what’s a microservice.

Basically, there are types are two types of services: a monolithic and a microservice.

Monolithic Architecture:

A monolith architecture is just one big service deployed on one instance, where if one part of the service faces an issue, the whole site goes down.

Microservice Architecture:

A microservice architecture has the whole service divided into multiple smaller services (called microservices). Each service is deployed separately and doesn’t affect the deployment of the other in any way.

Even if one service goes down, the other services are not affected in any way.

Let’s get started with setting up our microservice architecture…

At the moment we have three services (there will be more), an authentication service that authenticates users viva OTP, one service to upload files to AWS S3, and a GraphQL service for all our other needs at the moment.

Note: This is not a tutorial on how to write a NodeJS API

Let’s get started…

Our directory structure is as following

├── appspec.yml
├── docker-compose.yml
├── env
│   └── ...
├── nginx
│   ├── Dockerfile
│   └── ...
├── nodeauth
│   ├── Dockerfile
│   └── ...
├── nodeupload
│   ├── Dockerfile
│   └── ...
└── scripts
    └── ...

In order to make the setup easy (on the deployment side), I containerized all the setup using docker.

Here’s our **nodeservice/Dockerfile** for all our NodeJS services.

Note: The exposed PORT should be different for different services

Now, we need to setup nginx as our API gateway

For that we need the **_nginx/Dockerfile_** and nginx configuration files

# Pull Nginx image from DockerHub
FROM nginx
# Copying general purpose files
COPY example.com/ /etc/nginx/example.com/
# Replace default.conf with our nginx.conf file
COPY nginx.conf /etc/nginx/nginx.conf
view raw Dockerfile hosted with ❤ by GitHub

and the **_nginx.conf_** file is as follows

events {
worker_connections 1024;
}
http {
upstream nodeauth {
least_conn;
server nodeauth:8000;
}
upstream nodeupload {
least_conn;
server nodeupload:8001;
}
server {
listen 80;
server_name example.com sub.example.com;
location /api/v1/auth {
include example.com/proxy.conf;
proxy_pass http://nodeauth;
}
location /api/v1/upload {
include example.com/proxy.conf;
proxy_pass http://nodeupload;
}
}
}
view raw nginx.conf hosted with ❤ by GitHub

Now, we have our three containers ready, we just need a docker-compose file to start them all with a single command.

**_docker-compose.yml_**

version: "3.8"
services:
nodeauth:
build:
context: ./nodeauth
ports:
- "8000:8000"
nodeupload:
build:
context: ./nodeupload
ports:
- "8001:8001"
nginx:
restart: always
build:
context: ./nginx
depends_on:
- nodeauth
- nodeupload
ports:
- "80:80"
view raw docker-compose.yml hosted with ❤ by GitHub

Viola! You got all your containers ready, now you just need to deploy those to the server. Package ’em all up and push to a git repository of your choice.

I used an AWS EC2 instance to deploy.

Go to your AWS EC2 Console and start up a new instance. I used a t2.micro instance with ubuntu:20.04 image, (as it is covered under the free tier).

SSH into your EC2 instance, and clone the repository that you just made onto the machine using git clone <repo-link> .

Now, you need to install docker on your instance sudo snap install docker .

Once you have cloned the repo onto the machine and installed the dependencies, it’s time to build and start the docker containers. Well, that’s where containerizing everything helps, you only need to run one command to run everything.

sudo docker-compose up --build

Make sure you have changed the directory into your cloned repository

You got it done! Now you just need to set the ANAME for your domain and you can make requests to your domain.

But wait…

The requests are only going to http instead of https . You need to set up https for your domain now.

In order to setup https for our server, we need to generate an SSL certificate. We can always do that stuff manually, but that’s no good when there are other people who have done this stuff for you.

I used staticfloat/nginx-certbot docker image to do this stuff for me.

We need to listen on PORT 443 for https instead of PORT 80 in case of http , and specify sslcertificate, sslcertificate_key in your **_nginx.conf_**.

# Auth Service
upstream nodeauth {
least_conn;
server nodeauth:8000;
}
# Upload Service
upstream nodeupload {
least_conn;
server nodeupload:8001;
}
server {
listen 443 ssl;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
location /api/v1/auth {
include example.com/proxy.conf;
proxy_pass http://nodeauth;
}
location /api/v1/upload {
include example.com/proxy.conf;
proxy_pass http://nodeupload;
}
# Include general purpose files
include example.com/security.conf;
include example.com/general.conf;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
}
view raw nginx.conf hosted with ❤ by GitHub

And your **nginx/Dockerfile** changes to…

FROM staticfloat/nginx-certbot
ENV CERTBOT_EMAIL=info@example.com
COPY conf.d/ /etc/nginx/user.conf.d
COPY example.com/ /etc/nginx/example.com
EXPOSE 443
view raw Dockerfile hosted with ❤ by GitHub

You also need to make changes in the **docker-compose.yml** to map the letsencrypt/ directory of your container to that of your host machine.

version: "3.8"
services:
nodeauth:
build:
context: ./nodeauth
env_file:
- ./env/.env
ports:
- "8000:8000"
nodeupload:
build:
context: ./nodeupload
env_file:
- ./env/.env
ports:
- "8001:8001"
nginx:
restart: always
build:
context: ./nginx
environment:
CERTBOT_EMAIL: info@example.com
ports:
- "80:80"
- "443:443"
depends_on:
- nodeauth
- nodeupload
volumes:
- ./nginx/conf.d:/etc/nginx/user.conf.d:ro
- letsencrypt:/etc/letsencrypt
volumes:
letsencrypt:
view raw docker-compose.yml hosted with ❤ by GitHub

You are all done! Just push these changes to your git repository, and pull them on your instance.

Configure the security group for your instance and enable HTTPS.

Now, all you gotta do is run sudo docker-compose up --build and you should get services running on https .

But think about it…

Do you really want to pull every change that you make on your service manually and restart the service again? No, right?

Neither do I, that’s why I set up CI/CD pipelines to deploy and restart the services automatically every time a new commit is pushed to the git repo.

Let’s get to that stuff in PART 2….

Read Next:

Setting up CI/CD Pipeline for a Monolithic Service

ShauryaAg on Apr 03, 2021

Read first:

As mentioned in the last part, we have already setup and deployed our three services, but we don’t want to keep pulling the changes every time we make a small change in the codebase, that’s why we need to setup our CI/CD Pipeline.

Let’s get started…

Let’s first treat all our services as Monolithic, and deploy them to the instance.

Create IAM Roles for CodeDeploy and EC2 Instance.

  • Go to IAM → Roles in your AWS Console
  • Create an IAM Role with AmazonEC2RoleforAWSCodeDeploy and AutoScalingNotificationAccessRole policies.
  • Let’s name this IAM Role as CodeDeployInstanceRole
  • Create another IAM Role with AWSCodeDeployRole policy.
  • Let’s name this one as CodeDeployServiceRole

Configure EC2 Instance for the application

Make sure you already have an instance running.

You just gotta modify the tags to let the CodeDeploy Agent know which instances to deploy the code on.

  • Let’s put two tags: env: production and name: my-application

Create S3 Bucket for application revision

This bucket will be used to store our revised application before it is deployed to the instance.

Note: Creating this bucket is necessary if you want to add some files to the codebase that you couldn’t store in the Github repository, such as _.env_ files.

If you don’t have any such thing, then you can skip this step.

  • You may name the bucket whatever you like.
  • Make sure Block all public access option is checked.

Configure CodeDeploy

  • Navigate to CodeDeploy in AWS Management Console.
  • Create an application and give it a name.
  • Under Compute Platform choose EC2/On-premises
  • Create Application.
  • After creating the application, create a deployment group
  • Name it whatever you’d like, and under Service Role choose CodeDeployServiceRole.
  • Under Deployment Type choose In-place.
  • For Environment Configuration choose Amazon EC2 instances and specify the tags that we specified previously for our instances: env: production and name: my-application
  • For Deployment Settings choose CodeDeployDefault.OneAtATime.
  • Deselect Enable Load Balancing
  • Create Deployment Group.

Phew, we are done with setting up on the AWS side, now let’s get to the good stuff.

Setting up on the Code Repository Side

Create an appspec.yml file, and place it in the root of the directory.

version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/my-application
hooks:
BeforeInstall:
- location: ./scripts/init.sh
timeout: 300
runas: root
ApplicationStart:
- location: ./scripts/start_app.sh
timeout: 300
runas: root
ApplicationStop:
- location: ./scripts/cleanup.sh
timeout: 300
runas: root
view raw appspec.yml hosted with ❤ by GitHub

Let’s setup our CI workflow now. I am using Github Actions for my CI/CD setup.

Create a .github/workflow/deploy.yml file

name: CI/CD Deployment
on: [push]
jobs:
buildAndTest:
name: CI Pipeline
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.x']
steps:
- uses: actions/checkout@v2
# Initialize Node.js
- name: Install Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
# Install project dependencies and test
- name: Install dependencies
run: npm install
- name: Run tests
run: npm run test
deploy:
name: Deploy
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
node-version: ['12.x']
appname: ['my-application-codedeploy']
deploy-group: ['production']
s3-bucket: ['my-application-codedeploys']
s3-filename: ['prod-aws-codedeploy-${{ github.sha }}']
needs: buildAndTest
steps:
- name: Checkout
uses: actions/checkout@v2
# Configure AWS Credentials
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
# Fetch secrets from S3 Bucket
- name: Configure Secrets
run: |
aws s3 cp s3://my-application-secrets/production/env/ ./env/ --recursive
# Deploy push to AWS S3
- name: AWS Deploy push
run: |
aws deploy push \
--application-name ${{ matrix.appname }} \
--description "Revision for the ${{ matrix.appname }}-${{ github.sha }}" \
--no-ignore-hidden-files \
--s3-location s3://${{ matrix.s3-bucket }}/${{ matrix.s3-filename }}.zip \
--source .
# Create deployment using AWS CodeDeploy
- name: AWS Create Deployment
run: |
aws deploy create-deployment \
--application-name ${{ matrix.appname }} \
--deployment-config-name CodeDeployDefault.OneAtATime \
--deployment-group-name ${{ matrix.deploy-group }} \
--file-exists-behavior OVERWRITE \
--s3-location bucket=${{ matrix.s3-bucket }},key=${{ matrix.s3-filename }}.zip,bundleType=zip \
view raw deploy.yml hosted with ❤ by GitHub

Note: In the _Configure Secrets_ step, we are fetching our secrets from a AWS S3 Bucket where we have stored to _.env_ files, as those can not be stored on the Github repository.

If you don’t have any secrets to configure, you can deploy directly from Github repository. Instead _s3-location_, you'd need to specify _github-location_: reference

Now, we just have to configure AWS Credentials that we used above.

Setting up CodeDeploy IAM User

  • Go to IAM -> Users in your AWS Console.
  • Create a new IAM User, let’s name it CodeDeployUser, and give it Programmatic Access
  • We need 2 sets of permissions: AmazonS3FullAccess and AWSCodeDeployDeployerAccess
  • Create the user, and save the user’s credentials ACCESS_KEY_ID and SECRET_ACCESS_KEY

Set those secrets in your Github Repository and you are all good to go!

Great! now, every push you do to your repository will be deployed to your EC2 instance.

But, wait. If you push to your repository after modifying just one of the services, all of them need to be restarted, that’s not how a Microservice Architecture works.

We need to decouple all our services from each other for all of them to operate separately.

Let’s get to that stuff in PART 3…

Read Next:

How I designed a CI/CD setup for a Microservice Architecture at zero cost

ShauryaAg on Apr 03, 2021

Read first:

In the last part we set up our pipeline for a Monolithic Architecture, but that’s not what I promised in the first part of the series.

Let’s get going…

Off to decoupling our services, so that they can live freely once again.

Well, that’s easy. Create a separate repository for each service, copy the workflow files to each of them. You are done! Ok, bye.

No, definitely not.

You can use that setup if all you have to do is run unit tests, but what about integration testing. Can’t do that on the production instance, and you are a struggling startup; you don’t have enough resources to spin up more instances just to run integration tests.

Well?

Ok, we will be putting each service in its own separate remote repository, but we will have one parent repository that refers to all the services’ repositories.

Let’s get started with git submodules…

Git Submodules are a way of adding a git repository inside a git repository. All submodules point to a commit in the remote repository.

The original intention behind git submodules was to keep a copy of a certain commit (or release) locally on which our project might depend.

You just need to run:

git submodule add <my-remote> <optional-path>

However, we need them to keep up-to-date with our services’ repository, that’s why it’s a good thing that we can make them point to a certain branch instead too.

git config -f .gitmodules submodule.<submodule-name>.branch <branch-name>

Now, you can keep all your submodules up-to-date with just one command

git submodule update --remote

Now, that git submodules are out of the way, let’s get to the actual “good” stuff.

Ok, let’s talk about the actual workflow that I follow:

  • Each service’s repository contains its unit test
  • After a commit is pushed to the service’s repository, it runs its unit tests.
  • If all the unit test pass, then we commit it to the parent repository.

_<submodule>/.github/workflows/test-and-push.yml_

name: CI/CD Deployment
on: [push]
jobs:
buildAndTest:
name: CI Pipeline
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ['12.x']
steps:
- uses: actions/checkout@v2
# Initialize Node.js
- name: Install Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
# Install project dependencies and test
- name: Install dependencies
run: npm install
- name: Run tests
run: npm run test
push:
name: Deploy
runs-on: ubuntu-latest
needs: buildAndTest
steps:
- name: Checkout
uses: actions/checkout@v2
# Clone the parent repo
- name: Clone Parent
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
git clone --recursive <parent-repo-remote>
# Confifure username and email
- name: Config username
run: |
git config --global user.name '<your-name>'
git config --global user.email '<your-email>'
# Commit and push changes
- name: Commit and push
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
git add .
git commit -m "Updated Tracking ${{ github.sha }}"
git push origin master
view raw test-and-push.yml hosted with ❤ by GitHub

  • The parent repo lists the submodules where the changes were made since the last push and only deploys those services again.

<parent-repo>/.github/workflows/deploy.yml

name: CI/CD Deployment
on: [push]
jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8]
node-version: ['12.x']
appname: ['my-application-codedeploy']
deploy-group: ['prod']
s3-bucket:
['my-application-codedeploys']
s3-filename:
['prod-aws-codedeploy-${{ github.sha }}']
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
submodules: 'true'
token: ${{ secrets.PAT }} # Defining PAT to fetch private submodules
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}
- name: AWS Deploy
env:
AWS_APP_NAME: ${{ matrix.appname }}
AWS_DEPLOY_GROUP: ${{ matrix.deploy-group }}
AWS_BUCKET_NAME: ${{ matrix.s3-bucket }}
AWS_FILENAME: ${{ matrix.s3-filename }}
GITHUB_EVENT_BEFORE: ${{ github.event.before }}
GITHUB_SHA: ${{ github.sha }}
run: |
sudo chmod +x ./scripts/deploy.sh
./scripts/deploy.sh
view raw deploy.yml hosted with ❤ by GitHub

This way all the other services keep running without any disturbance, while one of the service is updated.

<parent-repo>/scripts/deploy.sh

# /bin/bash
# Getting all the submodules(directories)/files where changes were made
temp=("$(git diff-tree --submodule=diff --name-only ${GITHUB_EVENT_BEFORE} ${GITHUB_SHA})")
echo $temp
# Keeping only distinct values in the array
UNIQ_SUBS=($(echo "${temp[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
for SUB in ${UNIQ_SUBS[@]}
do
if [ -d "$SUB" ]
then
cd ${SUB}
if [ ! -f "appspec.yml" ]
then
echo $PWD
echo "AppSpec.yml not found in ${SUB}"
continue
fi
else
continue
fi
chmod +x ../scripts/get_env.sh
../scripts/get_env.sh ${SUB}
aws deploy push \
--application-name ${AWS_APP_NAME} \
--description "Revision for the ${SUB}-${AWS_APP_NAME}" \
--no-ignore-hidden-files \
--s3-location s3://${AWS_BUCKET_NAME}/${SUB}-${AWS_FILENAME}.zip \
--source .
aws deploy create-deployment \
--application-name ${AWS_APP_NAME} \
--deployment-config-name CodeDeployDefault.OneAtATime \
--deployment-group-name ${SUB}-${AWS_DEPLOY_GROUP} \
--file-exists-behavior OVERWRITE \
--s3-location bucket=${AWS_BUCKET_NAME},key=${SUB}-${AWS_FILENAME}.zip,bundleType=zip
cd ..
done
view raw deploy.sh hosted with ❤ by GitHub

I also created separate Deployment Groups for each service of the name: _<service-name>-prod_, i.e. for _nodeauth_ service, I created a _nodeauth-prod_ deployment group, with rest of the configuration same as we did in the previous part.

We also do need to modify the appspec.yml and our scripts.

  • Since each service is a separate deployment, we need to put the appspec.yml in each service's repository.

version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/my-application/my-submodule-1
hooks:
BeforeInstall:
- location: ./scripts/init.sh
timeout: 300
runas: root
ApplicationStart:
- location: ./scripts/start_app.sh
timeout: 300
runas: root
ApplicationStop:
- location: ./scripts/cleanup.sh
timeout: 300
runas: root
view raw appspec.yml hosted with ❤ by GitHub

  • Since we have decoupled all the services from each other, we no longer run them using sudo docker-compose up, each service has to be started individually.

<submodule>/scripts/start_app.sh

SERVICE_NAME=${DEPLOYMENT_GROUP_NAME%-*}
# Change directory into the service folder
cd /home/ubuntu/tastebuds-backend/${SERVICE_NAME}/
# Read env variables from .env file
export $(cat .env | xargs)
# Remove the previous container
sudo docker stop ${SERVICE_NAME}
sudo docker rm ${SERVICE_NAME}
# Create a default network to connect the services
sudo docker network create ${NETWORK}
# Build the docker image
sudo docker build -t ${SERVICE_NAME} .
# Run the container
if [ -z ${VOLUME} ]
then
sudo docker run --rm -d -p ${PORT}:${PORT} --network ${NETWORK} --name ${SERVICE_NAME} ${SERVICE_NAME}
else
sudo docker run --rm -d -p ${PORT}:${PORT} -v $(pwd)/${VOLUME} --network ${NETWORK} --name ${SERVICE_NAME} ${SERVICE_NAME}
fi
# unset env variables
unset $(grep -v '^#' .env | sed -E 's/(.*)=.*/\1/' | xargs)
view raw start_app.sh hosted with ❤ by GitHub

  • I wrote a bit complex script so that we can use the same set of scripts in all our services instead of writing them again and again as new services are added.
  • The scripts can also be added as a git submodule to all the services’ repositories, which makes it easier to maintain (but they weren’t in my setup at the moment of writing this blog).
  • init.sh contains code to install docker on the instance (if not already present).
  • cleanup.sh contains code to remove the previous unused containers.

That’s it. You are finally done. You’ve got your own weird-ass setup to test and deploy a Microservice Architecture at zero cost. You can also keep the previous docker-compose.yml to maintain a local development setup.

The single instance’s cost is covered under AWS free-tier.

disclaimer: Yes, there is probably a better way of doing this. I hope there is.

How to approach image overlay problems

ShauryaAg on Jun 05, 2020

Every image has three channels: R, G, B, that is, Red, Green, and Blue which is used to define the pixel value at any point in the image, where the value of red, green, or blue lies between 0–255.

For example: a pixel value of [255, 0, 0] would be all RED, and [255, 255, 0] would be a mix of RED and GREEN, which gives a YELLOW color.

But, if the image is read using OpenCV, it yields the image in BGR format, that is,[255, 0, 0] would be BLUE and so on.

Installing OpenCV

OpenCV is an open-source library for image manipulation in Python or C language.

For Python OpenCV can be downloaded using pip install opencv-python.

Reading an image in OpenCV

Any image can be read in opencv using cv2.imread() command. However, OpenCV doesn’t support HEIC images yet, you may have to use another library like Pillow to read HEIC images (or convert them into .JPEG first).

import cv2image = cv2.imread(‘image.jpg’)

After reading the image, it can be converted into RGB format from BGR if necessary, using cv2.cvtColor() command.

image\_rgb = cv2.cvtColor(image, cv2.COLOR\_BGR2RGB)image\_gray = cv2.cvtColor(image, cv2.COLOR\_BGR2GRAY)

Overlays

Images are nothing but a bunch of pixel values stored in a matrix-like format. The value of any pixel can be changed independently of the others.

Let’s say there’s an image

image_1

Read the image using opencv:

image\_1 = cv2.imread(‘image\_1.jpg’)print(image\_1)

This gives a bunch of pixel values in matrix form.

array(\[\[\[107, 108, 106\],\[107, 108, 106\],\[107, 108, 106\],…,\[ 77, 78, 76\],\[ 77, 78, 76\],\[ 76, 77, 75\]\],…,\[\[ 93, 88, 87\],\[ 93, 88, 87\],\[ 92, 87, 86\],…,\[ 52, 62, 62\],\[ 52, 62, 62\],\[ 52, 62, 62\]\]\], dtype=uint8)

If you’d just change the pixel values of a certain area of the images to let’s say [0, 0, 0], that area of the image would become BLACK as that’s the pixel values of the color BLACK. Similarly, if you’d change the pixel values to [255, 0, 0], that area would become BLUE (OpenCV reads the images in BGR format).

image\_1\[50: 100, 50:100\] = \[255, 0, 0\]

Similarly, those pixel values can be replaced by another image, just by using the pixel values of that image.

In order to do that, you must reshape the overlaying image to the size whose pixels values you want to replace.

This can be done by using cv2.resize() function.

image\_2 = cv2.imread(‘image\_2.jpg’)resized\_image\_2 = cv2.resize(image\_2, dsize=(100, 100))

Here, dsize accepts the dimensions to which the image is to be resized

Now, the second image can be overlayed on the top of the first image.

image\_1\[50:150, 50:150\] = resized\_image\_2

Overlaying PNG Images

Unlike JPEG images, PNG (Portable Network Graphics) images can also have a 4th channel in them, that defines the ALPHA (opacity) at the given pixel.

OpenCV reads the PNGs in the same way as JPEGs (that is, with 3 channels) unless specified otherwise.

To read a PNG image with its Alpha values, we need to specify the flag cv2.IMREAD_UNCHANGED while reading an image.

Now, the image read has 4 channels: BGRA.

image\_3 = cv2.imread(‘image\_3.png’, cv2.IMREAD\_UNCHANGED)print(image\_3)array(\[\[\[0 0 0 0\]\[0 0 0 0\]\[0 0 0 0\]…\[0 0 0 0\]\[0 0 0 0\]\[0 0 0 0\]\]…\[\[0 0 0 0\]\[0 0 0 0\]\[0 0 0 0\]…\[0 0 0 0\]\[0 0 0 0\]\[0 0 0 0\]\]\], dtype=uint8)

(**Note:** The values printed are all 0s because the starting and ending of the image are blanks)

However, this image has 4 channels but our JPEG image has only 3 channels so, those values can not simply be replaced.

We need to add a dummy channel in our JPEG image.

For this, we will use a numpy. It can be installed using pip install numpy .

numpy offers a function numpy.dstack() to stack values against along the depth.

First, we need a dummy array of the same size as of the image.

To create a dummy channel we can use numpy.ones() function to create an array of ones.

import numpy as npones = np.ones((image\_1.shape\[0\], image\_1.shape\[1\]))\*255image\_1 = np.dstack(\[image\_1, ones\])

We are multiplying the array of ones with 255 because the value of the alpha channel also exists between 0–255.

Now, you can replace the pixel values of the image with the PNG image.

image\_1\[150:250, 150:250\] = image\_3

However, it will not give the desired result as we are also changing the value of the alpha channel to zero.

We only need to replace those pixel values that have a non zero value.

To do that, you can always brute-force your way by checking each pixel value and replacing the non-zero ones, but that's time-consuming.

So, there’s a better way.

You can take the alpha values of the image that is to be overlayed.

alpha\_image\_3 = image\_3\[:, :, 3\] / 255.0

We are dividing the pixel values by 255.0 in order to keep the values between 0–1.

The sum of alpha of image_1 and image_3 needs to be equal to 255.

So, you can create another array which contains the values of the needed alpha to create the sum to be equal to 255.

alpha_image = 1 — alpha_image_3

Now, you can simply take element-wise product of the alpha values and the image pixel values of each channel of each image and take their sum.

for c in range(0, 3): image\_1\[150:250, 150:250, c\] = ((alpha\_image\*image\_1\[150:250,      150:250, c\]) + (alpha\_image\_3\*image\_3\[:, :, c\]))

Voilà! The image has now been overlayed on top of the other. ez pz :)

Introduction to Git

ShauryaAg on Sep 12, 2019

Git is a version control system. A version control system is a tool that helps in maintaining different versions of a project. If you made a project and then realized you need another feature in the project, but while coding the new feature you screwed up and need to know what mistake did you make, you can look at the previous version of the code. It also helps in collaborating with multiple people on a project.

You can download Git through here.

Git v/s GitHub/GitLab:

A lot of people get confused between Git and GitHub. So, Git is the version control system that manages the different versions of the files while GitHub is a platform (website) where you can host your remote repositories. There are multiple hosting platforms available like GitHub, GitLab, BitBucket, etc that can be used to host your remote repository, GitHub is just one of them. Using GitHub is not at all necessary to use Git. Git can also be used to maintain records of your personal work on your own PC.

Some Basic Terminologies:

Repository: The folder of the project.

Remote Repository: The project folder that is available online.

Cloning: Downloading the code from the remote repository.

Forking: means copying the repository in your account, so you can make some changes into it.

Commit: It’s like a checkpoint which you can go back to when you screw up.

Branches: Suppose you are working on a feature in the project, while your partner decides to work on another feature in the same project, so he can make a new branch in the project and work separately without disrupting the other person’s work, and they can merge the two branches together when both are done.

Push: Making the changes onto the remote repository after you have modified the code on the local repository.

Pull: Making the changes onto the local repository (the one that is present on your PC, not online) from the remote repository after someone else pushed some changes onto the remote repository.

Pull Request: Asking the owner the original repository to merge your changes into their repository. (You can make a pull request on GitHub [called Merge Request on GitLab] after you have done some changes in your forked repository or on some other branch in the same repository)

Git Command Line:

Command Line Interface is the environment where you can interact with your PC only using the textual commands (that is, without the use of mouse). You need to memorise the commands (or look them up every time) in order to use the command line properly. There is a list of some basic Git commands below that will help you get started, and here’s a link to a Git cheat sheet.

It is not at all necessary to use Git Command Line Interface (CLI) with Git. You can also use Git Graphical User Interface (GUI) to use all the Git features (no one uses that, though).

You can also use other GUIs like GitHub Desktop to use Git features.

Most Basic Commands:

git clone [url]: Downloads the local from the remote repository (only to be used when starting a project)

git init: Initializes the git repository (A repository has to be initialized as a git repository before it can start tracking the changes).

git remote add [remote-name] [remote-url]: remote-name is the name by which you refer the remote repository URL (like the URL of the GitHub/Gitlab repository)

git add [file-name]: adds the files to the staging area (git does not commit all the files every time. Unlike some other version control tools, only the ones that are added to the staging area)

git commit -m “[message]”: commits the added files

git push [remote-name] [branch-name]: pushes the files onto the remote repository.

git fetch [remote-name]: downloads the files from the remote repository.

git merge [branch-name]: merges the current local branch with the specified branch-name.

git pull [remote-name] [branch-name]: basically fetching and merging in one command.

git checkout [branch-name]: to go to the specified branch

Contact

@

© Shaurya Agarwal 2021