AWS / Azure / GCP Cloud | Database | Data Analytics | LakeHouse | Machine Learning | Shrenik Parekh

Mesosphere DC/OS Concept: DC/OS stands for Data Center Operating System It is a distributed operating system, abstracts the cluster hardware, software resources, and provides container orchestration,package management, networking, logging and metrics, storage and volumes, and identity management. DC/OS has system space / kernel space and user space . System space is a protected area, not accessible to users, which involves low-level operations such as resource allocation, security, and process isolation User space includes user applications, jobs, and services. Built-in package manager can be used to install services into user space. Each DC/OS node also has a host operating system which manages the underlying machine Made up of many components - distributed systems kernel and a container orchestration engine Marathon. DC/OS runs on a cluster of nodes, instead of a single machine. The kernel of DC/OS is based on Apache Mesos distributed system kernel A cluster manager...

Master-to-master replication: MySQL offers parameters auto_increment_increment, and auto_increment_offset for master to master replication. Most of the time folks believe when we set auto_increment_offset to odd no., such as 1 for server 1(Master 1),  and even no. such as 2, for server 2 (Master 2), we are all set. But according to Oracle these two parameter  " can be used to control the operation of AUTO_INCREMENT columns ." Means columns which are not using AUTO_INCREMENT for primary key, master to master replication could break at some point, and require human intervention. Default is 1 for these two parameters. Ref. https://dev.mysql.com/doc/refman/8.0/en/replication-options-master.html For auto_increment_increment - 10, and auto_increment_offset   auto_increment_increment: This parameter controls the interval between successive column values. Use of auto_increment_increment in table definition - col_name INT NOT NULL AUTO_INCREMENT PRIMARY KEY. When you SET @@auto_...

Install Mesos, Mesosphere, Marathon, Zookeeper, Chronos on Red Hat Linux: Identify repository rpm for Mesos, Mesosphere, Marathon, Zookeepter, for which is mesosphere-el-repo-7-3.noarch.rpm . Install following software on each master.   Update Repository: # rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-3.noarch.rpm Install mesos, and marathon from mesosphere repository:   # yum -y install --enablerepo=mesosphere mesos marathon Check installation and version of mesos: # yum list installed | grep mesos mesos.x86_64   1.9.0-2.0.1.el7 @mesosphere mesosphere-el-repo.noarch  7-3  installed # yum list installed | grep marathon marathon.noarch  1.9.109-1.el7 @mesosphere-noarch Install mesosphere-zookeeper: # yum install mesosphere-zookeeper Check installation and version of Zookkeeper: # yum list installed | grep zoo* mesosphere-zookeeper.x86_64  3.4.6-0.1.20141204175332.centos7 Install Chronos: #yum i...

Kubernetes Master (Control Plane): Kubernetes master is a collection of small specialized services, API server, the cluster store, the controller manager, and the scheduler which makes the control plane of the cluster, for production environment multi-master is must have. EKS-Elastic  Kubernetes Service from Amazon, GCP- Google Cloud Platform from Google,  Azure from Microsoft are deploying highly available masters. Most of the time application work load runs on nodes and not on Masters. The API Server: API server is the brain of the cluster, the front end to into the Kubernetes Masters, Expose RESTFUL API which consume JSON, user POST manifest file to the API  server which get validated and then deployed to the cluster. API Server is central management entity and it is the only one entity directly communicate with etcd distributed storage component. Ref.: https://www.openshift.com/blog/kubernetes-deep-dive-api-server-part-1 The Cluster Store: Cluster stor...

Kubernetes: Kubernetes word derived from Greek word meaning Helmsman – person who steers a ship. Open-sourced handed over to the Cloud Native Computing Foundation (CNCF), written in Go (Golang) Kubernetes is an orchestration, orchestration of containerized apps., often written as k8s It is leading container orchestrator lets us manage containerized apps and or micro service apps. Micro service apps made of lots of small and independents parts. You say hey Kubernetes, here is my app, consist of these parts, just run it for me, and Kubernetes run it.   You package the app as containers, give them declarative manifest, and let Kubernetes runt it. It is platform agnostic, runs on bare metal, VMs, cloud instances (private and public), OpenStack, anything with Linux. Does scaling, self-healing, load balancing, rolling updates and more. Lives on Github at kubernetes/kubernetes, Twitter - @kubernetesio How Kubernetes relate to Docker: Docker is a low level...

Linux Daily Command: Linux A to Z commands - https://www.tecmint.com/linux-commands-cheat-sheet/ Archive directory using tar: tar -cvf <project_name.tar> /mysql/<project_anme> /mysql/mysqlbakcup/<project_name.tar> Extract directory using tar: tar -xvf <File_Name.tar> <Target_Location> tar - https://www.freecodecamp.org/news/tar-in-linux-example-tar-gz-tar-file-and-tar-directory-and-tar-compress-commands/ Copy files to remote server: rsync -avz  *.sql <id>@<remote_server>:</remote_server_path/backup/> --progress Copy directory dir_2 to remote server: rsync -avz /dir_1/dir_2 <id>@<remote_server>:</remote_server_path/backup/> --progress Copy remote file to local server: rsync -avzh <id>@<remote_server>:</remote_server_path/backup/> </local_server_path/local_dir> Copy remote file to local /tmp directory server over SSH: rsync -avzhe ssh <id>@<remote_server>:</remote_server_path/bac...

Install MySQL 8.0.19 Community edition: Remove Maria DB packages: # yum remove mariadb-libs.x86_64* Install MySQL 8.0.19 rpm packages: # yum install mysql-community-common-8.0.19-1.el7.x86_64.rpm # yum install mysql-community-libs-8.0.19-1.el7.x86_64.rpm # yum install mysql-community-libs-compat-8.0.19-1.el7.x86_64.rpm # yum install mysql-community-client-8.0.19-1.el7.x86_64.rpm # yum install mysql-community-embedded-compat-8.0.19-1.el7.x86_64.rpm # yum install mysql-community-devel-8.0.19-1.el7.x86_64.rpm # yum install mysql-community-server-8.0.19-1.el7.x86_64.rpm # yum install mysql-shell-8.0.19-1.el7.x86_64.rpm At this point mysql daemon is up and running at default location /var/lib/mysql at default port 3306. If you would like to customize location and port and other configuration parameters, make sure to remove directory /var/lib/mysql and kill the existing mysqld process running. Disable Firewall: # systemctl stop firewalld # systemctl disable firewall...