99% of developers don't get Docker.

You've seen the meme. A developer tells

a project manager, "It works on my

machine." And the PM responds, "Great.

We'll ship your machine." Many

developers think Docker is just a way to

wrap their messy code and dependencies

in a box. But they're still wondering

why it breaks in production. You're

probably building 2 GB images for a

simple node app, hard- coding

environment variables, and treating

containers like lightweight virtual

machines. But a container isn't a

virtual machine. It doesn't need a

hypervisor or a bloated guest OS. It's a

process shared directly with the host

kernel through namespaces and control

groups. Today, we're stripping away the

blackbox mystery. We're going to do a

deep dive of how Docker engines, images,

and layers actually work so you can stop

shipping bloat and start shipping

infrastructure. Starting with the

foundations, hardware versus OS

virtualization. To understand Docker, we

have to understand the evolution of

isolation and where its predecessor, the

virtual machine, fell short. Hardware

virtualization is a technology that uses

a software layer called a hypervisor

like ESXi or KVM to simulate physical

hardware CPU memory storage and network.

Each virtual machine runs its own

independent operating system and

applications acting like an independent

computer unaware of other virtual

machines on the same hardware or

physical machine. The hypervisor sits on

the physical hardware and carves it into

multiple isolated virtual computers,

abstracting software from the underlying

infrastructure to allow virtual machines

to operate independently. To the guest

operating system sitting on top, it

thinks it is running on real bare metal

hardware. This can be full

virtualization where the guest operating

system is unaware it's virtualized or

para virtualization where the operating

system is modified to communicate with

the hypervisor for better performance.

But why were virtual machines not

enough? The first is due to resource

tax. Because every virtual machine

carries its own full kernel, you lose

significant RAM and CPU just to keep the

OS alive. Running 10 VMs means 10 copies

of the Linux kernel and 10 sets of

background drivers, wasting resources.

There's also startup latency. Booting a

full operating system takes minutes. In

a modern microservices world, waiting

minutes to scale up for a traffic spike

is unacceptable. And finally, size. A

virtual machine image is usually several

gigabytes, making it heavy to store and

slow to move across a network. Now,

let's discuss the shift to operating

system virtualization. Docker

virtualizes the operating system

instead. Virtualizing the operating

system means using containerization to

isolate applications by sharing the host

machine's OS kernel rather than

virtualizing full hardware like a

virtual machine. Containers act as

isolated user spaces, allowing multiple

lightweight applications to run

independently on one OS. Containers

create isolated user spaces or name

spaces for processes, libraries, and

dependencies, making them faster and

more resource efficient than VMs.

They're also highly portable. The

application and its dependencies are

packaged together, ensuring they run

consistently across any environment.

Docker also provides resource efficiency

because they don't require a separate

OS. Containers use significantly less

RAM and CPU compared to full hardware

virtualization. The lineage of isolation

is built on concepts that existed long

before Docker. There was chroot in 1979.

This is the oldest ancestor. It's short

for change route. It allows you to

change the apparent root directory for a

process and its children. While it

isolates the file system, effectively

creating a chroot jail where a process

cannot see files outside its assigned

directory, it is leaky. It doesn't

isolate networking users or process ids

and a root user can easily break out of

a chroot environment using a second

chroot call. And then there were free

BSD jails in 2000. This was a massive

leap forward that introduced the concept

of OS level virtualization. Jails didn't

just isolate the file system. They

partitioned the network stack giving

each jail its own IP, the user subsystem

and the process tree. Each jail has its

own root user and host name, but they

all share the same FreeBSD kernel. It

proved that you could have high density

isolation without the overhead of a VM.

Docker took these concepts to the Linux

kernel to create a workflow where a

container carries exactly what it needs

like a specific version of Node.js16 and

OpenSSL without touching the host's

global libraries. So what is Docker

exactly? At its core, Docker is a

platform designed to package,

distribute, and run applications in

standardized units called containers. It

acts as the translation layer between

your code and the infrastructure,

providing a consistent interface for

managing the software life cycle. When

people say Docker, they are usually

referring to four distinct things.

First, the Docker engine, also known as

the heart of Docker. It is a client

server application consisting of a

longunning background demon process

dockerd APIs that specify interfaces for

programs to talk to the demon and a CLI

client. Then there is the docker file.

This is a textbased manifest that

defines the source of truth for your

environment. It documents every step

required for your app to run making your

environment infrastructure as code.

Third is images. This is the blueprint

of your application. An image is a

readonly executable package that

includes everything needed to run an

application code, runtime, libraries,

environment variables, and config files.

When you run an image, it becomes a

container. And fourth, Docker Hub. This

is a centralized registry similar to

GitHub, but for binary images that hosts

official security scanned images for

databases like PostgreSQL, web servers

like EngineX, and runtimes like NodeJS.

Before we look at how the Linux kernel

actually draws these isolation

boundaries, let's talk about another

place you're quietly bleeding

engineering cycles, your CI/CD pipeline.

Just like how wrestling with environment

drift is a massive waste of your time,

so is staring at a terminal waiting 20

minutes for a Docker image to build.

That context switching completely

destroys developer momentum. If you are

building multi-architecture container

images in standard CI environments like

GitHub actions, you already know the

pain of relying on slow buggy software

emulation like QMU. That's why you need

to look at depot, the sponsor of this

video. Depot is a drop-in replacement

for your Docker builds that makes them

up to 40 times faster. Instead of

wrestling with standard CI runners,

Depot routes your builds to remote

machines equipped with native Intel and

ARM processors, completely eliminating

the need for slow emulation. But the

real secret weapon is their caching.

Instead of wasting time saving,

compressing, and loading cache layers

over a network, which sometimes takes

longer than the build itself, depot uses

a shared, blazingly fast NVMe cache that

is instantly available across all of

your parallel builds. There is no

complex migration. You literally just

swap Docker Build for Depot build in

your workflow and your pipeline is

instantly faster. Stop burning money on

CI minutes and shattering your team's

focus. Go to depot.dev to get your time

back. Now, back to how the kernel makes

Docker possible. Let's talk about the

anatomy of isolation and kernel

primitives. The Linux kernel draws these

boundaries using two secret weapons.

First is namespaces. We're going to talk

about isolation or what a process can

see. Namespaces provide the virtual

reality goggles for a process. In Linux,

many resources are global like the list

of all running processes or the network

card. Namespaces wrap these global

resources in an abstraction so that a

process inside a namespace thinks it has

its own private isolated instance of

that resource. First, let's talk about

the PID namespace. On your host, your

app might be process ID4502.

But inside the container's P namespace,

the app sees itself as P1, the system's

innit process. It cannot see or interact

with any processes outside its own

bubble. Second, there's the net

namespace. It provides a private network

stack. This includes its own IP address,

routing table, and firewall rules. This

is why you can run three separate

containers all listening on port 80

without a port already in use error.

Each is on its own private network. Then

we have the mnt or mount namespace. This

isolates the mount points. The process

sees a completely different file system

root or slash than the host machine. It

cannot see the hosts/etc/

shadow or/home folders unless you

specifically mount them. And then we

have the UTS namespace. This allows the

container to have its own host name and

domain name separate from the host

machine. A container host name is just a

unique label assigned to a container

within a network identifying it for

intercontainer communication typically

defaulting to the container ID in

Docker. And then we have control groups

or croups. If namespaces are about what

you can see, Croups are about what you

can use. They set hard limits on RAM and

CPU which prevent a single buggy

container from crashing your entire prod

node. Now let's talk about the union

file system. A Docker image is tiny

because of the union file system. Images

are composed of layers. Each instruction

in a Docker file creates a new layer.

These are stacked and treated as one

system. If you have three apps based on

Ubuntu 24.04, Docker stores one readonly

layer of Ubuntu on your disk shared by

everyone. When a container is launched,

Docker adds a thin unique writable layer

on top of the immutable image layers.

All changes made within that specific

container are written to this top layer.

This is the copy on write strategy. If

the app changes a file, Docker copies it

to the top layer and modifies it there

leaving the base image untouched. Let's

dive a little bit deeper into the copy

on write mechanism. Firstly for reading.

If an application needs to read a file,

Docker first looks in the writable

layer. If it's not there, it accesses

the file from the shared readonly layers

below. And then we have writing. The

first time an application modifies an

existing file from a lower layer.

Docker's storage driver performs a copy

up operation, copying the file to the

container's unique writable layer. The

modification then occurs on this copied

file in the writable layer, leaving the

original file in the readonly layer

untouched and available for other

containers to use. This ensures each

container has its own isolated data

state while maximizing efficiency. And

this in turn provides reduced storage

consumption because images are shared.

Launching 10 containers based on a 1

GBTE image does not require 10 GB of

storage. It only requires the base 1

GBTE plus small individual write layers.

It also provides fast container startup.

Containers start almost instantly

because they do not need to copy the

entire file system image, only create a

new empty writable layer. Now, let's

talk about designing the build or

optimizing the Docker file. A

professional Docker file is optimized

via layer caching. Docker caches each

line and if you want to change a line,

Docker invalidates the cache for that

line and everything below it. Let's look

at this sample Docker file. The from

instruction sets the base image. This is

the foundation of your stack. Choosing a

small base like Alpine significantly

reduces your image size and attack

surface. The work dur sets the execution

context. Any following run cmd or copy

instructions will happen inside this

folder ensuring your file structure is

predictable. Copy moves files from the

host machine into the image. By copying

package.json JSON before the rest of the

code. We ensure npm install only reruns

if our dependencies actually change. Run

executes commands in a new layer on top

of the current image and commits the

results. This is used to install

packages or build your application.

Envals production sets an environment

variable named nodeen to the value

production within the image. This is a

standard practice for Node.js JS

applications as it triggers several

optimizations such as improved

performance, disabled development only

warnings and loggings, and the omission

of development dependencies during the

installation process, and this produces

a smaller final image size. Expose 3000

informs Docker that the container is

expected to listen for network traffic

on TCP port 3000. CMD provides the

default for an executing container.

Unlike run which executes during the

build, cmd is the entry point of the app

once the container is actually launched.

In this case, it just runs the server.js

file as a NodeJS program. Now, let's

talk about data persistence and

orchestration. Starting with Docker

volumes. By default, data inside a

container is ephemeral. If the container

is deleted, the data is gone. Volumes

are the preferred mechanism for

persisting data generated by and used by

Docker containers. They are stored on

the host but managed by Docker allowing

you to swap out containers without

losing your database records or uploads.

For orchestration, let's touch on Swarm

versus K8. As you scale from one

container to hundreds, you need

orchestration. Docker Swarm is Docker's

native orchestration tool. It's easier

to set up and great for smaller, simpler

clusters. Kubernetes, also known as K8s,

is the industry standard. It is highly

complex but offers immense power for

automated scaling, self-healing, and

managing massive distributed workloads.

There's also the question of Docker

versus Podman. Docker uses a client

server architecture with a centralized

demon known as Docker D. Podman on the

other hand is demonless. It uses a fork

exec model where the CLI directly

interacts with the OCI or open container

initiative runtime. For Docker, the

demon typically runs as root, posing a

single point of failure and higher

security risk. Podman, on the other

hand, is rootless by default,

significantly reducing the potential

attack surface. For Docker, all

containers are managed by a single

persistent background service, while for

Podman, each container runs as a regular

isolated user process manageable with

standard Linux tools like systemd.

Docker primarily uses Docker Compose for

multicontainer orchestration which

requires separate YAML file creation.

Podman is built with Kubernetes in mind

and it supports native pods and can

automatically generate Kubernetes

compatible YAML files from local

workloads. But what are the actual next

steps in your Docker journey? First is

to realize that Docker is not just a

tool. It's a mental shift from managing

servers to managing artifacts. By

mastering namespaces, layers, and

volumes, you gain total predictability

over your software's life cycle. If you

want to begin your journey in becoming a

10x engineer, I highly recommend

checking out Code Crafters, where you'll

learn how to build Git, Docker, Reddus

from scratch. They are hands down the

best project-based coding platform out

there. Check out my link below for 40%

off. As always, thank you very much for

watching this video and happy coding.