Month: February 2021

In this blog, you will learn everything about setting up an upstream for a git repository, a branch and the --set-upstream-to command usage

You will also learn about different scenarios where git upstream is used and how to apply it in your git development workflow.

Have you wondered how to keep the forked repo in sync with the main repo? It’s done by adding the main repo as the upstream. It is explained in the below sections.

What is Upstream in Git?

Note: We are following the name “main” instead of “master” considering the removal of terms like “master” and “slave” in the IT community.

Like the literal meaning, upstream refers to something which points to where it originated from.

In the git world, upstream refers to the original repo or a branch. For example, when you clone from Github, the remote Github repo is upstream for the cloned local copy.

Let’s take a look at different types of git upstreams.

Git Repository Upstream

Whenever you clone a git repository, you get a local copy in your system. So, for your local copy, the actual repository is upstream.

Git Forked repository Upstream

When you clone a Forked repository to your local, the forked repository is considered as the remote origin, and the repository you forked from is upstream.

This workflow is common in open-source development and even in normal project developments.

For example, If you wanted to contribute to an open-source project, this is what you do.

1. You would fork the actual project repository.
2. Then you clone from the forked repository.
3. Meanwhile, to keep up with the main open-source repository, you pull the changes from it through your git upstream config.
4. You push your changes to a branch in the forked repository.
5. Then you would raise a PR to the actual project repository from your forked repository.
   

Tip: With Jenkins multibranch pipelines, you can easily set up the hooks for pull requests from a Forked branch.

Setup Git Upstream For a Repository

Let’s say you are working on a forked project and you want to sync changes from the main project repo.

You can add the actual repo as an upstream to your local copy. This way you can pull all the changes happening in the main project repo.

Add an Upstream Repo

Before adding upstream, you can take a look at your .git/config file and you will see something like below with a remote origin. Which is a clone of the forked repo.

Now lets the add the upstream to our repo with the main project repo URL using the add upstream command.

git remote add upstream https://github.com/devopscube/vagrant-examples.git

Now if you check the .git/config, you will see an upstream entry as shown below.

You can also check out the remote origin and upstream branches using the following command.

git branch --remotes

Here is a sample output.

[email protected]:~/vagrant-examples$ git branch --remotes
  origin/HEAD -> origin/main
  origin/dev
  origin/main
  upstream/dev
  upstream/main

Sync Changes From The Upstream repo

To sync an upstream repo, first, you need to fetch the upstream changes.

git fetch upstream

Then, merge the changes from the upstream branch to the local branch. In this example, its the main upstream branch.

git merge upstream/main

Setup Git Upstream For a Branch

But when we talk about an upstream branch, it refers to the specific upstream of a branch in the remote respository.

For example, let’s say you clone a git repository with two branches, main and dev.

When you clone the repository both the branches, main, and dev will have the respective upstream branches (remote branches) because these branches are present in both the local copy as well the remote or upstream repository. Also, the tracking is enabled by default.

Now let’s look at scenarios where you want to set up git upstream using --set-upstream-to command.

Important Note: In git version update, 1.8.0 the --set-upstream the command is changed to --set-upstream-to

Add Git Upstream Using –set-upstream-to

When you create a branch (named foo) in your local repository and want to add an upstream branch for tracking, you can use the following command.

git push -u origin foo

Important Note: Here -u is the shorthand for --set-upstream-to

When you push a local branch with the upstream command, it automatically creates the remote branch and adds tracking to your local branch.

Add Git Upstream To Existing Remote Branch

Sometimes, when you create a local branch, you might push the remote repository changes without adding the upstream tag. For example,

git push origin foo

It will create the remote branch without tracking. Meaning, when someone pushes any change to the branch foo in the remote branch, and when you try to pull the changes, it will throw the following error.

There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details

    git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

    git branch --set-upstream-to=origin/<branch> testing

To avoid this, you can set git upstream using the following command. It will enable tracking, and you will be able to pull the changes.

git branch --set-upstream-to origin/foo

Add Git Upstream Using Git HEAD

In git, HEAD refers to the currently active branch.

For example, if you have checked out foo branch, your HEAD now is foo.

So when you create a new local branch named foo, you can create the upstream using the HEAD as shown below.

git push -u origin HEAD

Git set-upstream FAQ’s

Design is a very important aspect when it comes to kubernetes cluster setup.

This blog will talk about the 10 high-level things that have to the taken into consideration when setting up a Kubernetes cluster.

1. Kubernetes Networking (Cloud, Hybrid, or On-Prem):

Kubernetes network has to be designed in such a way that it can accommodate future cluster and application requirements.

One common mistake organizations do is using CIDR ranges that are not part of the organization’s network. In the future when they want the clusters to be in a hybrid network, it ends up in migration.

It is better to discuss with the organization’s network team before finalizing the network design. This way, you can carve out and reserve an IP range even if you are not part of the hybrid network.

Each cloud provider gives multiple options for Node and Pod networking. For example, GKE offers multi-cluster services, VPC native clusters with routable pod IPs from the same VPC and the peered VPCs.

But if you don’t want to expose your pod IPs, you might need to use something like an IP masquerading agent in your cluster so that the outgoing traffic will always have the Node IP as the source identity.

Also, Ingress and egress traffic design are essential. There could be API gateways, on-prem systems, and third-party APIs that you need to connect from the cluster apps.

Your design should include all the access requirements so that you won’t face any access restrictions during implementation.

2. Kubernetes Security, Compliance & Benchmarks

Following are the generic security considerations.

1. Understand the compliance requirements and security benchmarks as per your organization’s policy. If you are using managed services, make sure it complies with the organization’s compliance policies.
2. Will there be any PCI/PII data apps?. If yes, then segregate these apps in terms of access and storage.
3. Implement the standard pod security policies ( disabling container root access, privileged access, read-only file system, etc).
4. Accessing the container registries securely.
5. Network policies to control pod to pod traffic and isolate apps as per access requirements.
6. A well-designed pipeline to make sure only approved container images are deployed in the cluster. Containers should be scanned for vulnerabilities and fail the CI process if the image scan fails to meet the security requirements,

3. Kubernetes Cluster Access

It is very important to design and document the way the kubernetes cluster is accessed.

Following are the key considerations.

1. Restricting manual cluster-admin access. Instead, cluster-admin access should only be allowed through automation.
2. Implement RBAC’s authorization.
3. Allow kubernetes API access via service accounts with limited privileges.
4. Implement Open Policy Agent for fine-grained access controls.
5. Consider options for openID connect
6. Have a good audit mechanism for checking the roles and removing unused users, roles, service accounts, etc.

Design the access levels so that you can hand off responsibilities to other teams using the cluster. It would save time for everyone, and you can focus more on the engineering par rather than working on repeated tasks.

4. Kubernetes High Availability & Scaling

High availability another key factor in the kubernetes cluster.

Here you need to consider the worker node availability across different availability zones.

Also, consider Pod Topology Spread Constraints to spread pods in different availability zones.

When we talk about scaling, it’s not just autoscaling of instances or pods.

It’s about how gracefully you can scale down and scale up the apps without any service interruptions.

Depending on the type of apps that needs to be hosted on kubernetes, you can design deployments to evict the pods gracefully during scale down and patching activities.

Also, consider chaos engineering experiments before production to check the cluster and app stability.

5. Kubernetes Ingress

Ingress is an essential component of Kubernetes clusters. There are many ways to set up a kubernetes ingress.

Also, there are different types of ingress controllers.

You can try out the best option that will be suitable for your organization’s compliance policies and scaling requirements.

Few considerations,

1. Have separate ingress controllers for the platform-tools.
2. SSL management for ingress endpoints.
3. Do not try to route all the apps through the same ingress. If your apps grow day by day, they could end up in a big configuration file creating issues.

6. Kubernetes Backup & Restore Strategy

Whether it is a managed service or custom kubernetes implementation, it is essential to back up the cluster.

When we say backup, it is primarily backing up etcd.

You should have a very good design to automate the backup of the kubernetes cluster and its associated components.

Also, a design to restore the cluster if required.

There are also options to take the dump of existing objects in JSON format. You can use dump to restore the objects in the same or a different cluster.

7. Kubernetes Node & Container Image Patching & Lifecycle Management

Patching is a repeated process.

When it comes to kubernetes, there is node and container patching.

Make sure you implement DevSecOps principles in your CI/CD pipelines.

Here are some key considerations,

1. An automated pipeline integrated with container scanning tools to patch container images on a monthly schedule.
2. An automated pipeline to perform node patching without downtime.
3. An automated pipeline to manage the lifecycle of container images. You don’t want to keep so many versions in your registry that are outdated.

8. Kubernetes Cluster Upgrades

Generally, you can perform a cluster updgrade in two ways

1. Upgrading the existing cluster.
2. Create a new cluster and migrate the apps to the new cluster.

You need a very good automated pipeline design to perform a cluster upgrade.

There could be Networking, DNS, and other component changes during an upgrade. It all depends on the design & organizational policies.

9. Kubernetes Cluster Capacity & Storage

Cluster capacity a very important topic of discussion.

You need to decide on the number of clusters you need to run.

Some organization prefers running multiple clusters to reduce the blast radius and easy maintenance. While others prefer a big cluster with a large number of worker nodes or less number of nodes with huge instance capacity.

You can decide on the cluster capacity based on your needs and the size of the team to manage the clusters.

Next comes the storage part.

Plan how you want to attach volumes to containers. Follow all the standard storage security practices on kubernetes.

When it comes to the cloud, there is out of the box support for provisioning storage,

If you are planning to run stateful sets, it is very important to design the storage to get high throughputs and maximum availability.

10. Kubernetes Logging & Monitoring

Most of the organizations will have a centralized logging and monitoring system and they prefer to integrate kubernetes with these systems.

Here are the key considertions.

1. How much log data will be generated.
2. Mechanisms to ingest Kubernetes logs into the logging systems considering huge data volume.
3. Scaling logging and monitoring components deployed in the cluster.
4. Data retention as per organizations policy.
5. Define and document the KPIs for monitoring.

Conclusion

These are some of the key considerations which often get missed while setting up a kubernetes cluster.

Missing these aspects while implementing kubernetes could lead to issues in the overall cluster and might impose compromises for the business.

Ideally, the Solution/ Technical architect should keep all the mentioned Items (there could be many but worth considering) as a checklist while designing the cluster architecture to make sure they are implemented during the IaaC development.