[Linux] 05 Linux Process & Service

in Study on Linux


Ubuntu

Table of Content

🎯 Learning Objectives - lesson 1

  • Distinguish Program vs. Process πŸ”„
    • Understand and explain the foundational difference between a program and a process in a single sentence.
    • Clearly define and distinguish between a PID (Process ID) and a PPID (Parent Process ID).
  • Master Three Observation Tools πŸ› οΈ
    • Differentiate and utilize three core process monitoring utilities based on the situation:
      • ps: Provides a static snapshot of current processes.
      • pstree: Displays process relationships in a hierarchical tree structure.
      • top: Offers a live, dynamic real-time resource usage view.
  • Identify Top Resource-Consuming Processes πŸ“Š
    • Successfully locate the top processes consuming system CPU and memory resources.
    • Comprehensively explain the core operational column definitions:
      • %CPU: Percentage of CPU time used.
      • %MEM: Percentage of physical memory utilized.
      • RSS: Resident Set Size (the actual physical memory allocated to the process).

Program vs Process πŸ”„

The core difference between a program and a process boils down to a single question: β€œIs it currently running?”

πŸ’Ύ Program (Static Β· 정적)

  • Definition: An executable file stored safely on disk (HDD/SSD).
  • State: A dormant chunk of code resting within the file system.
  • πŸ“š Analogy: β€œA cookbook sitting quietly on a bookshelf.”

βš™οΈ Process (Dynamic Β· 동적)

  • Definition: An active instance loaded into system memory (RAM) and running.
  • State: An operational entity receiving CPU resources to execute its instructions.
  • πŸ”₯ Analogy: β€œThe actual act of cooking a meal inside the kitchen.”

πŸ’‘ Core Takeaway

A single program (1 recipe book) can generate multiple processes running simultaneously in memory.

  • Example: Opening 3 separate tabs in Google Chrome triggers multiple individual chrome process instances concurrently in RAM.

PID, PPID & Process Status Codes πŸ†”πŸ“Š

πŸ†” Process Identification: PID & PPID

Every process running in Linux is assigned unique numeric identifiers by the kernel to trace its lineage:

  • PID (Process ID): A unique numeric identifier assigned to a process when it is created.
  • PPID (Parent Process ID): The PID of the parent process that created it.
    • Example: If you execute a command in the bash terminal, bash acts as the parent process, and the executed command becomes the child process.

πŸ“Š Process Status Codes (STAT)

The execution state of a process is represented by a specific code flag:

  • R (Running / Runnable) πŸƒβ€β™‚οΈ
    • The process is actively running or waiting in the CPU run queue to be executed.
  • S (Interruptible Sleep) 😴
    • The process is resting/waiting for an event or resource to become available (e.g., waiting for user input). It wakes up immediately upon receiving a signal.
  • D (Uninterruptible Sleep) 🧱
    • The process is waiting deeply for hard physical I/O operations (like disk read/write). It completely ignores external signals while in this state.
  • T (Stopped) πŸ›‘
    • The process has been suspended by a job control signal (such as hitting Ctrl+Z) or by a debugger.
  • Z (Zombie) 🧟
    • The process has completed its execution, but its exit status remains uncollected because the parent process has not read it yet. It occupies no memory but still holds its slot in the PID table.

ps command πŸ“Έ

The ps (Process Status) command is a static tool used to look at snapshots of currently running processes on the system.

βš™οΈ Practical Syntaxes & Usage Modes

The ps command supports options written in different styles (UNIX/BSD/GNU). Three primary combinations are commonly used depending on what you need to track:

  1. ps -ef (Standard Format Check) πŸ“‹
    • Displays every process running on the entire system in full detail.
    • Commonly paired with grep to isolate a specific service.
    • Example: ps -ef | grep nginx πŸ”
  2. ps aux (Resource Allocation Check) πŸ“Š
    • Displays all system processes with a heavy focus on resource consumption (such as %CPU and %MEM).
  3. ps -o pid,ppid,comm (Custom Column Selection) πŸ› οΈ
    • Displays only specific, user-selected tracking columns.
    • Perfect for cleanly evaluating a process tree lineage (like showing just PID, Parent PID, and Command Name).

πŸ” Output Column Definitions

When running ps, understanding the column headers is essential for troubleshooting:

  • UID / USER: The user account that owns and initiated the process.
  • PID: Process ID (The unique numeric ID of the current process).
  • PPID: Parent Process ID (The identifier of the process that launched this one).
  • %CPU / %MEM: Percentage of CPU time and physical RAM used by the process.
  • STAT: Process operational status flag (e.g., R, S, D, T, Z).
  • COMMAND: The exact command name or execution string that triggered the process.

pstree command 🌳

The pstree (Process Tree) command visually maps running processes into a tree layout, revealing parent-child links that might be hidden in flat layouts.

βš™οΈ Main Options & Parameters

pstree handles clear parameters to quickly analyze operational lineages without processing raw metadata manually:

  • p (Show PIDs) πŸ†”
    • Displays the numeric Process ID (PID) directly inside parentheses alongside each process name.
    • Essential for isolating exactly which specific sub-worker needs to be target-managed or ended.
  • a (Show Arguments) ⌨️
    • Displays the exact execution parameters, path scripts, or configuration flags used when launching each process.
  • g (Show Process Groups) πŸ‘₯
    • Groups associated child workers together by their unified Process Group ID (PGID).
  • s (Show Parents) πŸ—ΊοΈ
    • Traces backward from a specified PID to reveal only its direct lineage chain up to the root ancestor.

πŸ” Practical Output Logic Examples

Rather than displaying long columns, pstree uses a nested structure:

# 1. Standard structural visual check
$ pstree

systemd───sshd───sshd───bash───pstree

top command πŸ“Ή

While ps provides a static photo snapshot, top acts as a live, real-time video stream of the system. It refreshes its interface automatically every 3 seconds, letting you immediately spot the top resource-consuming processes at a glance.

⌨️ In-App Hotkeys (Interactive Mode)

Once top is running inside your terminal, you can control it dynamically using these single-character keys:

  • P (Uppercase): Sorts the process list in descending order by %CPU usage.
  • M (Uppercase): Sorts the process list in descending order by %MEM (Memory) usage.
  • h or ? (Lowercase): Opens the interactive help menu to display all available shortcut controls.
  • q (Lowercase): Safely exits the top program and returns you to the standard shell prompt.

βš™οΈ Command-Line Execution Options

  • top -d N: Modifies the default interface refresh interval to N seconds.
    • Example: top -d 1 configures the view to update every single second.

πŸ” Core Memory Column Concepts

To correctly pinpoint exactly what is draining system memory, keep an eye on these distinct metrics:

  • RES / RSS (Resident Set Size) πŸ’Ύ
    • Measures the actual physical memory currently allocated to and used by a process in Kilobytes (KB).
  • %MEM πŸ“Š
    • Displays the percentage of total available system RAM that the process is using.

πŸ’‘ Tip: Both RES and %MEM are essential counters used side-by-side to find the primary memory-hogging tasks running on a server.

Learning Objectives - lesson 2 🎯

  • Master Foreground vs. Background Operations πŸ”„
    • Understand the functional difference between jobs running in the foreground and tasks executing in the background.
    • Differentiate how standard output and terminal controls behave for each mode.
  • Control Interactive Jobs Flexibly πŸ•ΉοΈ
    • Learn and apply essential execution controls to manage ongoing jobs smoothly:
      • & (Ampersand): Immediately launch commands into the background.
      • Ctrl + Z: Safely suspend an active foreground job.
      • jobs: View the list of currently active or suspended tasks.
      • bg: Resume a suspended process to run in the background.
      • fg: Bring a background or suspended process back to the foreground.
  • Manage Process Priorities (Nice Values) πŸ“ˆ
    • Define what a process Nice value is and how it influences execution priority.
    • Explain the full range of legal Priority and Nice value parameters.
    • Understand the specific privilege restrictions applied when modifying process priorities (e.g., standard users vs. superuser capabilities).

Foreground vs Background πŸ”„

Within a single shell session, running tasks are divided into two distinct processing regions based on which process controls terminal inputs (the keyboard).

⌨️ 1. Foreground Processes (ν¬κ·ΈλΌμš΄λ“œ)

  • Definition: An interactive task that directly occupies both your keyboard input and terminal screen output.
  • Constraints: A shell session can run only one foreground task at any given time.
  • Behavior: The command prompt is locked away while this process runs. You must wait for the task to fully finish executing before you can input your next command.
  • πŸ“ž Analogy: β€œA person you are actively speaking with over the phone right now.”

🎡 2. Background Processes (λ°±κ·ΈλΌμš΄λ“œ)

  • Definition: A detached task that is isolated from your keyboard inputs, running quietly in the rear of the terminal environment.
  • Capabilities: You can execute multiple background jobs simultaneously within the same shell instance.
  • Behavior: The shell drops the command prompt back down to you immediately after launch, letting you input other commands while the background job works.
  • πŸ“» Analogy: β€œA radio playing background music while you work on other things.”

πŸ•ΉοΈ State Shifting & Job Control Logic

The lecture note establishes three core pathways to pass tasks back and forth between these environments:

  1. Direct Background Injection (&) 🟒
    • Appending an ampersand (&) to the end of a line boots the process immediately into the background upon startup.
  2. Foreground Suspension (Ctrl + Z) 🟑
    • Pressing Ctrl + Z instantly halts an ongoing foreground process and pushes it into a paused background slot.
  3. Active Region Swapping (fg / bg) πŸ”΅
    • fg: Pulls a hidden background job or paused task back into the active, interactive foreground.
    • bg: Instructs a paused background task to resume running while staying safely out of the way in the background.

Job ID vs PID πŸ†”πŸ·οΈ

A single process running in Linux can be identified by two different tracking numbers, depending entirely on which component assigned the number.

πŸ“Š The Code Comparison breakdown

  • [1] (Job ID) 🐚
    • Who assigns it: Assigned locally by the Shell session.
    • 3421 (PID) 🐧
    • Who assigns it: Assigned globally by the Linux Kernel.
$ sleep 100 &       # Starts a process directly in the background
[1] 3421            # The shell outputs both assigned numbers instantly!
  • Analysis: [1] and 3421 point to the exact same running process instance.

πŸ” Key Differences Between the Numbers

AttributeJob ID (%N) 🐚Process ID (PID) 🐧
ScopeLocal: Only valid within that specific shell window. If you close the terminal, it disappears.Global: System-wide unique identifier. Visible and accessible from any other shell window.
Notations%1, %2, %+ (current job), %- (previous job)Plain integers (e.g., 3421, 2841)
Target CommandsBuilt-in job management utilities:Β 
jobs Β· fg Β· bgGlobal system-wide binaries:Β 
ps Β· top Β· killΒ Β 

πŸ’‘ Core Rule of Thumb: > Use Job IDs (prepended with a % sign like %1) when you are managing internal terminal multi-tasking workspace tasks via fg or bg. Use PIDs when troubleshooting or terminating items via global tools like kill 3421.

Sending to Background πŸ“₯

πŸ› οΈ The Two Pathways to the Background

  • 1. Starting in the Background (&) 🟒
    • Method: Append an ampersand (&) symbol to the very end of your command string.
    • Behavior: The command launches directly behind the scenes from its inception, giving you back an active interactive prompt immediately.
    • Example: sleep 100 &
  • 2. Interrupted Shift Mid-execution (Ctrl + Z) 🟑
    • Method: Press the Ctrl + Z shortcut key while a foreground program is actively occupying the screen.
    • Behavior: This instantly halts/pauses the active foreground task and sends it into the background in a Stopped status. You can later choose to resume it using bg or fg.

πŸ” Secondary Comparison & Output Identifiers

  • Ctrl + C (Immediate Terminate) πŸ›‘
    • Pushing Ctrl + C kills the active foreground command right away. This is provided as a reference comparison and does not move anything to the background.
  • The Shell Notification Format: [N] PID 🏷️
    • When a process goes to the background, the shell outputs a confirmation line mapping its shorthand index to its kernel tracker:
      • [N]: The assigned Job ID (local to this terminal screen).
      • PID: The global Process ID assigned by the system kernel.

Managing Active Jobs πŸ•ΉοΈ

πŸ“‹ Core Job Control Commands

  • jobs (View Summary) πŸ‘€
    • Lists all active background and suspended tasks initialized within the current shell session.
    • Displays the shorthand Job ID, operational Status, and the original execution Command string.
  • jobs -l (View Detailed Layout) πŸ”
    • Functions identically to the standard jobs command but appends a tracking column showing the exact system-wide PID (Process ID) for each entry.
  • fg %N (Bring to Foreground) πŸ“₯
    • Pulls Job number N out of its background position or suspended state and loads it straight into the active, interactive foreground.
  • bg %N (Resume in Background) πŸ“€
    • Instructs a Stopped (paused) job number N to safely resume execution behind the scenes while remaining in the background.

πŸ’‘ Syntax Reference Check

When targeting jobs using fg or bg, remember to append the percent symbol (%) right before the digit to explicitly specify you are calling a Job ID rather than a standard system PID.

  • Example: fg %1 hooks onto terminal job #1, whereas kill 12345 references a system-wide kernel process ID.

Persistence β€” nohup & disown ⏳

πŸ’₯ The Problem: SIGHUP (Hangup Signal)

By default, when a terminal window or SSH session is closed, the shell automatically transmits a SIGHUP (Signal 1, Hangup) broadcast to all active child processes running under its lineage. This signal instructs those processes to terminate immediately.

To keep scripts or server software running seamlessly, you must bypass or sever this signal chain using two primary tools:

πŸ›‘οΈ 1. The nohup Command (No Hangup)

  • Core Function: Bypasses the hangup signal from the start. It explicitly tells the launched process to completely ignore any incoming SIGHUP commands.

  • Standard Syntax:Bash

      $ nohup python3 script.py &
  • Key Operational Behaviors:
    • Because it is detached from an active terminal context, it automatically redirects the program’s standard output (stdout) and standard error (stderr) streams to a file named nohup.out in the current workspace directory.
    • It is strictly a preventative tool used at the initial moment you launch a task.

βœ‚οΈ 2. The disown Command (Shell Lineage Removal)

  • Core Function: Severs the relationship between the active shell and a process that is already running. It modifies the shell’s active task table so that the specified background job is no longer considered a child process of that terminal session.

  • Standard Syntax:Bash

      $ sleep 500 &      # 1. Started normally in background
  [1] 4512
  $ disown %1        # 2. Removes Job #1 from the shell's tracking table
  • Key Operational Behaviors:
    • Unlike nohup, disown is a reactive tool applied mid-execution to an active background job.
    • Once disowned, typing jobs will yield nothing for that task. When the shell exits, it does not send a SIGHUP to the process because it no longer recognizes it as a child.

πŸ’‘ Comparison Summary

Featurenohup πŸ›‘οΈdisown βœ‚οΈ
When to useAt the exact moment of startup.After a process is already running in the background.
How it worksTells the process to ignore SIGHUP.Removes the process from the shell’s job tracking list entirely.
Output LoggingAutomatically creates a nohup.out log file.Keeps the original output tracking context intact.

Learning Objectives - lesson 3 🎯

πŸ“‹ Core Learning Objectives

  • Understand the Linux Signal Mechanism πŸ“‘
    • Define what a Signal is and explain how it acts as an asynchronous communication method used to control process behavior.
    • Understand the concept of custom Signal Handlers and how a process intercepts, catches, handles, or blocks incoming operational instructions.
  • Master the Core Termination Signals πŸ›‘
    • Identify and explain the functional differences between the most common system signals:
      • 1 (SIGHUP): Hangup signal; typically sent when a terminal finishes or closes.
      • 2 (SIGINT): Interrupt signal; generated via Ctrl + C for graceful interactive stops.
      • 9 (SIGKILL): Forceful kill signal; terminates a process instantly at the kernel level and cannot be caught, ignored, or blocked.
      • 15 (SIGTERM): Standard termination signal; the default signal used to ask a program to clean up and exit safely.
      • 19 (SIGSTOP): Pause signal; suspends process execution immediately.
  • Acquire Practical Termination Skills πŸ› οΈ
    • Learn to use kill and killall commands to transmit specific signals to targeted processes.
    • Differentiate termination scope by applying commands using either a single system-wide PID or matching by generic Process Names.
    • Fully grasp the essential safety workflow: always try a soft termination first (SIGTERM) before resorting to a forced stop (SIGKILL).

Signals β€” Messages to a Process πŸ“‘

πŸ’‘ Core Concept: What is a Signal?

  • A Standard Message: A signal is an asynchronous message or notification sent directly to a running process to instruct it to perform a specific action (such as pausing, reloading, or shutting down).
  • Interception (Signal Handling): When a process receives a signal, it can catch it and execute a custom piece of code known as a Signal Handler to perform cleanups before reacting, unless the signal is designed to bypass this layout entirely.

🟒 1. Gentle Warnings (λΆ€λ“œλŸ¬μš΄ κ²½κ³ )

Gentle signals communicate an intent to terminate or modify the process, allowing the program the chance to save its state, finish pending tasks, or clean up temporary resources.

  • 1 (SIGHUP) β€” Hangup πŸ”Œ
    • Sent when its controlling terminal is closed or disconnected. Processes often use this signal to reload their configuration files without shutting down.
  • 2 (SIGINT) β€” Interrupt ⌨️
    • Triggered interactively when a user presses Ctrl + C in the terminal. It politely requests the foreground task to halt.
  • 15 (SIGTERM) β€” Terminate πŸšͺ
    • The default signal sent by the standard kill command. It formally asks the process to shut down cleanly and gracefully.

πŸ”΄ 2. Absolute Force (μ ˆλŒ€ ꢌλ ₯)

Forceful signals act abruptly at the kernel level. The targeted program is given no advance notice and cannot protect itself.

  • 9 (SIGKILL) β€” Kill πŸ’€
    • Instantly destroys the process. Because it operates directly via the system kernel, it cannot be caught, ignored, handled, or blocked by any custom signal handler.
  • 19 (SIGSTOP) β€” Stop πŸ›‘
    • Instantly freezes/pauses the process mid-execution. Like SIGKILL, it forces the state transition immediately and cannot be intercepted by the application.

🧠 Best Practice Workflow Summary:

Always attempt a clean shutdown using a Gentle Warning like SIGTERM (15) first. Only resort to Absolute Force like SIGKILL (9) if the program becomes completely unresponsive or turns into a runaway process that refuses to close.

The 5 Main Signals πŸ“‘

Linux relies heavily on a core set of standard signals to control and coordinate process states. This page highlights the 5 most frequently used signals, their numeric shortcuts, and how the system behaves when they are received.

πŸ“‹ The 5 Essential System Signals

  • 1 β€” SIGHUP (Hangup) πŸ”Œ
    • Default Action: Terminates the target process.
    • Trigger Event: Sent automatically when the controlling terminal window or SSH connection is closed.
    • Special Use Case: Daemon processes (like web servers) often intercept this signal to reload their configuration files safely without performing a full shutdown.
  • 2 β€” SIGINT (Interrupt) ⌨️
    • Default Action: Terminates the target process.
    • Trigger Event: Generated instantly when a user presses the Ctrl + C keyboard shortcut.
    • Behavior: Politely requests the active foreground program to stop what it is doing and exit cleanly.
  • 9 β€” SIGKILL (Kill) πŸ’€
    • Default Action: Forced immediate termination.
    • Trigger Event: Triggered manually by system administrators to deal with frozen or uncooperative software.
    • Behavior: Bypasses the application entirely and acts directly at the kernel level. It cannot be blocked, caught, handled, or ignored by the process.
  • 15 β€” SIGTERM (Terminate) πŸšͺ
    • Default Action: Soft/Graceful termination.
    • Trigger Event: The standard fallback signal transmitted when running a plain kill <PID> command without specifying an option flag.
    • Behavior: Formally requests the program to clean up its state, close open file descriptors, release sockets, and exit safely. This should always be your first choice for stopping a service.
  • 19 β€” SIGSTOP (Stop) πŸ›‘
    • Default Action: Pauses process execution.
    • Trigger Event: Sent to temporarily halt a process without destroying it.
    • Behavior: Like SIGKILL, this signal is absolute and cannot be blocked or handled by the process. (Note: This is functionally identical to what SIGTSTP/Ctrl+Z requests, but SIGSTOP cannot be ignored).

Command: kill 🎯

Despite its aggressive name, the kill command is fundamentally a signal delivery tool used to send any standard message (not just termination requests) to a running process.

βš™οΈ Command Syntax & Formats

The kill command requires you to target a process using its unique PID (Process ID). You can specify the signal you want to send using either its numeric code or its text name (with or without the SIG prefix).

  • Syntax Pattern:

      $ kill -[SIGNAL] [PID]

πŸ› οΈ Common Operational Examples

  • 1. The Standard, Polite Request (Default) πŸšͺBash

      $ kill 1234
    • Signal Sent: 15 (SIGTERM)
    • Behavior: Requests process 1234 to clean up its files and shut down gracefully. This is always the safest option to try first.

  • 2. The Forceful Shutdown (Kernel Level) πŸ’€Bash

      $ kill -9 1234
  $ kill -SIGKILL 1234
    • Signal Sent: 9 (SIGKILL)
    • Behavior: Instantly terminates the process via the kernel. Use this only when a program is completely frozen or unresponsive to a standard kill command.

  • 3. Reloading Configuration Files πŸ”„Bash

      $ kill -1 1234
  $ kill -HUP 1234
    • Signal Sent: 1 (SIGHUP)
    • Behavior: Often used for background servers (like web servers) to tell them to reread their config files without a full restart.

πŸ’‘ Core Safety Rule of Thumb

kill -15 (SIGTERM) ➑️ kill -9 (SIGKILL)

Always attempt a standard kill first to give the application a chance to save your data and close down properly. Only escalate to -9 if the process refuses to exit.

Termination by Name πŸ“›

While the standard kill command requires you to look up and target specific numeric PIDs, Linux provides dedicated utilities to signal multiple processes simultaneously using their Process Names.

πŸ› οΈ Core Commands: killall and pkill

  • killall (Exact Match Termination) 🎯
    • Behavior: Sends a signal to every running instance that exactly matches the specified process name.
    • Syntax: killall [options] [process_name]

    • Example:Bash

        $ killall chrome
      • Impact: Instantly targets and terminates all open Google Chrome processes or tabs across the system.
  • pkill (Pattern Match Termination) πŸ”
    • Behavior: A more flexible tool that matches processes based on extended criteria, such as partial name strings or specific ownership.
    • Syntax: pkill [options] [pattern]

    • Example:Bash

        $ pkill -u spiderman
      • Impact: Forcefully signals and terminates every single process currently owned by the user account spiderman.

⚠️ Critical Comparison & Safety Warning

Metrickill [PID] 🎯killall [Name] πŸ’£
Target PrecisionSurgical: Targets a single, specific process instance.Broad: Targets all running instances sharing that exact name.
Risk LevelLow: High safety, assuming you verified the PID.High: High risk of accidental collateral damage if unintended scripts share the name.

πŸ›‘ The β€œNuclear” Danger of killall:

Be exceptionally careful when switching between different operating systems. In Linux, killall kills processes matching a name. However, in Solaris / UNIX, running killall does exactly what it literally saysβ€”it terminates absolutely every process on the system, completely shutting down the operating server. Always verify your environment before running it!

Command: nice & renice πŸ“ˆ

Linux manages CPU scheduling using process priority values. The nice and renice commands allow you to influence how much CPU time a process receives by adjusting its β€œniceness” relative to other tasks.

πŸ“Š The Core Mechanics: Nice Values vs. Priority

  • The Scale: Nice values range from 20 (Highest Priority) to 19 (Lowest Priority). The default value for a standard process is 0.
  • The β€œNice” Logic:
    • A high nice value (e.g., 19) means the process is being very β€œnice” to the system, stepping aside and letting other tasks take the CPU first.
    • A negative/low nice value (e.g., 20) means the process is selfish, demanding more CPU cycles and higher scheduling priority.

  • The Math: The system’s internal Priority value ($PR$) is typically calculated using the base priority modified by the nice value: \[PR = 20 + \text{nice}\]

πŸ› οΈ Setting Priority at Startup: nice

The nice command is used to launch a brand new process with a modified priority from the start.

  • Syntax: nice -n [NICE_VALUE] [COMMAND]

  • Example (Lowering Priority):Bash

      $ nice -n 10 python3 heavy_script.py &
    • Impact: Launches the script with a nice value of 10, preventing it from hogging the CPU and keeping the system responsive for other interactive tasks.

πŸ”„ Modifying Running Processes: renice

The renice command is used to alter the priority of a process that is already running in memory. Unlike nice, it targets the process using its PID.

  • Syntax: renice [NICE_VALUE] -p [PID]

  • Example (Raising Priority):Bash

      $ sudo renice -5 -p 4321
    • Impact: Instantly bumps the priority of process 4321 to 5, giving it a higher share of CPU resources mid-execution.

πŸ›‘ Strict Privilege Restrictions

To prevent standard users from disrupting system stability, the Linux kernel enforces strict permissions regarding priority adjustments:

  • Regular Users:
    • Can only increase the nice value (i.e., make their own processes less important).
    • Cannot assign negative nice values.
    • Cannot modify processes owned by other users.
  • Superuser (root):
    • Has total control. Can decrease nice values (giving processes higher priority) and modify any process on the system using sudo.

Learning Objectives - lesson 4 🎯

πŸ“‹ Core Learning Objectives

  • Define and Recognize Daemon Processes πŸ‘»
    • Clearly explain what a Daemon is and how it differs from a standard user-initiated background process.
    • Identify distinct daemon naming conventions (such as the trailing d in process names like sshd, httpd, or systemd).
    • Understand the lifecycle of a daemonβ€”running continuously in the background from system boot to shutdown to handle persistent system requests.
  • Master systemd Service Management βš™οΈ
    • Understand the role of systemd as the first process (PID 1) that initializes the Linux system space.
    • Learn to use the systemctl command-line utility to control the operational states of system services.
    • Successfully execute core management workflows using specific service verbs:
      • start / stop: Immediately activate or terminate a service in the current session.
      • status: Inspect live diagnostic logs and the runtime state of a specific daemon.
      • enable / disable: Configure whether a service automatically registers to launch at system boot time.

πŸ” Quick Concept Preview

ConceptCore FocusCommon Example
Daemon πŸ‘»The background worker itself that waits silently to perform tasks.sshd (Secure Shell Daemon)
systemd πŸ‘‘The master supervisor system that manages and monitors those daemons.systemctl status sshd

Daemon & Init β€” PID 1 πŸ‘»πŸ‘‘

This page bridges the gap between individual user tasks and system-level execution, explaining how Linux manages persistent background services from the moment the machine boots up.

πŸ‘» 1. What is a Daemon?

A Daemon is a specialized system process that runs continuously in the background to handle persistent service requests, completely detached from any interactive user terminal.

  • The Naming Rule: You can easily spot a daemon because its process name almost always ends with the letter d.
    • Examples: sshd (handles incoming SSH connections), httpd (Apache web server), crond (handles scheduled tasks).
  • The Lifecycle: Unlike user tasks (like sleep 100) which close when a terminal exits, daemons are initialized during the system boot phase and remain alive until the machine shuts down.

πŸ‘‘ 2. The Init Process & PID 1

When a Linux system boots up, the operating system kernel mounts the filesystems and initializes exactly one master user-space process. This is the ancestor of all other processes on the system.

  • PID 1: This initial process is permanently assigned Process ID 1.
  • The Mother of All Processes: Every single process that runs on a Linux machine is a direct or indirect child of PID 1. If you trace the parent-child lineage (PPID) of any process far enough back using pstree, it will always lead back to PID 1.

βš™οΈ 3. The Evolution: systemd

Historically, Linux used a standard called SysV init to manage startup scripts sequentially. Modern Linux distributions have replaced this with systemd.

  • Role: On modern setups, systemd is the specific program running as PID 1.
  • Core Responsibilities:
    1. Spawns and orchestrates all system daemons in parallel to speed up boot times.
    2. Acts as the system’s primary supervisor, automatically restarting critical services if they crash.
    3. Adopts the ultimate cleanup role: if a parent process dies before its child, PID 1 (systemd) steps in to adopt the orphan process, keeping the process tree organized.

systemd Unit βš™οΈπŸ“¦

Modern Linux systems do not view background tasks as simple, isolated shell scripts. Instead, systemd organizes system resources, services, and devices into standardized, manageable objects called Units.

πŸ’‘ What is a systemd Unit?

A unit is a configuration file that encapsulates both the properties and the execution logic of a specific system resource. Rather than writing custom complex bash scripts to handle a daemon’s startup, logging, and crash recovery, administrators define these behaviors using declarative .unit configuration files.

πŸ“‚ Common systemd Unit Types

Units are categorized by their file extensions, each representing a different type of system resource managed by the kernel:

  • .service (Service Units) βš™οΈ
    • The most common unit type. Represents a daemon or background application that can be started, stopped, or reloaded (e.g., sshd.service, nginx.service).
  • .socket (Socket Units) πŸ”Œ
    • Manages network sockets or filesystem FIFOs. systemd can listen on these sockets and automatically wake up the corresponding .service unit only when network traffic arrives.
  • .device (Device Units) πŸ–₯️
    • Represents physical hardware devices recognized by the Linux kernel (via udev). Useful for triggering scripts when a specific device (like a storage drive) is plugged in.
  • .mount (Mount Units) πŸ’Ύ
    • Controls filesystem mount points inside the directory tree, serving as a modern, dynamic alternative to editing the static /etc/fstab file.
  • .target (Target Units) 🎯
    • Used to group multiple units together to define system states (known historically as runlevels). For example, multi-user.target groups all services required for a standard command-line server setup.

πŸ“ Structure of a Unit File

A typical unit file (usually located in /lib/systemd/system/ or /etc/systemd/system/) is structured into distinct, easy-to-read sections:

  1. [Unit]: Defines generic metadata about the unit (e.g., a brief description) and establishes activation dependencies (e.g., β€œRun this unit After the network is online”).
  2. [Service]: Specific to .service units. Dictates the exact command to run on startup (ExecStart), how to shut it down (ExecStop), and its recovery behavior (e.g., Restart=on-failure).
  3. [Install]: Contains installation rules used when you enable or disable the service to boot automatically (e.g., WantedBy=multi-user.target).

systemctl status check 🩺

The systemctl status command is the primary diagnostic tool used to inspect the exact, real-time health and operational state of a systemd service.

βš™οΈ Command Syntax

To check a service, you append the service name (the .service extension is optional but good practice) to the status verb:

Bash

$ systemctl status sshd.service

πŸ” Anatomy of a Status Output

Running this command returns a rich block of diagnostic metadata. Key indicators to check include:

  • Loaded Line (Loaded: ...) πŸ“‚
    • Shows the absolute file path of the unit configuration file on disk and tells you if the service is enabled (will launch automatically at boot) or disabled (requires manual start).
  • Active Line (Active: ...) ⚑
    • active (running) (Green): The service is healthy and actively executing.
    • inactive (dead) (White/Gray): The service is cleanly stopped.
    • failed (Red): The service crashed or failed to initialize, often appending an error exit code.
  • Main PID (Main PID: ...) πŸ†”
    • Displays the principal Process ID assigned to the master daemon process by the system kernel.
  • Tasks & CGroup (CGroup: ...) 🌲
    • Visualizes the process tree hierarchy, showing all child processes or worker threads currently operating underneath the main service wrapper.
  • Recent Logs (Journal Entries) πŸ“
    • Automatically pipes the trailing last few lines of standard log outputs associated with that specific service (sourced directly from journald), allowing you to instantly debug a boot failure without digging through external system log files.

Service Lifecycle Management πŸ”„

⚑ 1. Active Runtime Management (Immediate Effect)

These commands control whether a service is running right now in the current session. They do not alter what happens when the computer restarts.

  • systemctl start [service_name] 🟒
    • Action: Instantly launches and activates the service in the background.
  • systemctl stop [service_name] πŸ›‘
    • Action: Safely terminates the running service and cleans up its active processes.
  • systemctl restart [service_name] πŸ”„
    • Action: Forcefully stops the service and immediately boots it back up. Commonly used to apply updates after editing an application’s backend files.

πŸ’Ύ 2. Boot Configuration Management (Persistent Effect)

These commands dictate whether a service should automatically launch when the Linux operating system turns on. They do not affect the service’s current running state.

  • systemctl enable [service_name] βš™οΈ
    • Action: Configures the service to automatically start at system boot time.
    • Behind the Scenes: systemd creates a symbolic link (symlink) pointing from its auto-start target directories (e.g., multi-user.target.wants/) to the actual unit file location.
  • systemctl disable [service_name] βœ–οΈ
    • Action: Prevents the service from starting automatically when the system boots up. It must be turned on manually using start.
    • Behind the Scenes: Removes the symlink from the auto-start target directories.

πŸ’‘ Quick Summary Matrix

GoalCommandImmediate Change?Survives Reboot?
Run it right nowsystemctl startYesNo
Kill it right nowsystemctl stopYesNo
Set to auto-start on bootsystemctl enableNoYes
Stop auto-starting on bootsystemctl disableNoYes

🧠 Common Workflow Tip:

If you install a new daemon and want it running immediately and on every subsequent boot, you must execute both actions:

systemctl start myservice followed by systemctl enable myservice.

Automatic Start on Boot Mechanism βš™οΈπŸ”—

πŸ” What Happens Under the Hood?

When you configure a service’s boot behavior, systemd alters the file structure inside the system configuration directories (typically /etc/systemd/system/):

  • Enabling a Service (systemctl enable) πŸ”—
    • The Mechanism: systemd reads the [Install] section of the service’s unit file to find its target dependency (usually WantedBy=multi-user.target).
    • The Action: It automatically creates a new Symbolic Link inside a directory named after that target (e.g., /etc/systemd/system/multi-user.target.wants/). This symlink points directly back to the master unit file stored in /lib/systemd/system/.
    • The Result: When the system boots up and loads the multi-user environment, it scans that .wants/ folder and initializes every service linked inside it.
  • Disabling a Service (systemctl disable) βœ‚οΈ
    • The Mechanism: systemd reverses the registration process.
    • The Action: It physically deletes the symbolic link out of the corresponding .wants/ dependency directory.
    • The Result: The master unit file remains completely safe and untouched on your drive, but because the boot target no longer has a link pointing to it, the service will sit silently and sleep during the next system startup.

πŸ“‹ File System Mapping Summary

StateCommandSymbolic Link StatusBoot Behavior
Enabled 🟒systemctl enable sshdCreated inside .../multi-user.target.wants/sshd.serviceStarts automatically on boot.
Disabled βšͺsystemctl disable sshdRemoved from the .wants/ directory entirely.Stays inactive until manually started.

⚠️ Key Takeaway: > Running systemctl enable or disable changes zero configurations inside the actual application code or the service file itself. It simply manages the filesystem shortcuts that tell systemd what to fetch during the system startup sequence.

Learning Objectives - lesson 5 🎯

  • Master Custom systemd Unit Configuration πŸ“
    • Understand the internal file structures and key sections required to build a valid systemd unit file from scratch ([Unit], [Service], and [Install]).
    • Learn how to define custom application pathways using ExecStart variables to launch self-made scripts or binary files as official system daemons.
  • Understand the Linux Cron Automation Framework ⏰
    • Define Cron and explain how the system uses the crond daemon to execute recurring, automated tasks backgrounded in the operating system.
    • Master the Cron Expression Syntaxβ€”the 5-field asterisk notation used to pinpoint specific scheduling windows:
      • Minute βž” Hour βž” Day of Month βž” Month βž” Day of Week
    • Differentiate between system-wide automation configuration folders (like /etc/crontab) and individual user-level task schedules.
  • Manage Scheduled Tasks Interactively πŸ› οΈ
    • Learn and apply essential command flags to safely handle user-specific automation schedules via the crontab utility:
      • crontab -e: Safely open, edit, or append new recurring rules in the user’s active configuration file.
      • crontab -l: Print a clean summary list of all currently configured automated tasks.
      • crontab -r: Completely erase and remove the user’s active task tracking schedule.

Writing Custom Unit Files πŸ“πŸ› οΈ

This page provides a practical guide on how to create and structure your own custom systemd unit file to manage a self-made script or application as an official system service.

πŸ“‚ Location and Structure

Custom, user-defined unit files are typically saved in the /etc/systemd/system/ directory (e.g., /etc/systemd/system/my_service.service). The file uses an INI-style format divided into three distinct, mandatory sections:

🧱 1. The [Unit] Section (Metadata & Dependencies)

This section contains general information about the service and defines its relationship with other system components.

  • Description=: A short, human-readable name or summary explaining what the service does. This text appears when you run systemctl status.
  • After=: Defines startup sequencing. For example, After=network.target ensures this service waits to launch until the system’s network interface is fully up and running.

βš™οΈ 2. The [Service] Section (Execution Logic)

This is the core operational block that dictates exactly how the background process behaves.

  • ExecStart=: Specifies the absolute file path to the command, script, or binary file that systemd should run to start the service.
    • Example: ExecStart=/usr/bin/python3 /home/user/app.py
  • Restart=: Configures the automatic recovery policy. Setting this to on-failure tells systemd to automatically reboot the program if it crashes with a non-zero exit code.
  • User=: Defines which system user account should execute the process (e.g., User=nobody or User=ubuntu for security isolation, preventing it from running as root by default).

πŸ”— 3. The [Install] Section (Boot Configuration)

This section tells systemd how to register the service when you run the systemctl enable command.

  • WantedBy=: Specifies the target milestone that should trigger this service during boot.
    • Using multi-user.target is the standard configuration for regular console services, meaning it will load during a normal, multi-user command-line system startup.

πŸ’‘ Example of a Minimal Unit File

Ini, TOML

[Unit]
Description=My Custom Python Web Service
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/myscript.py
Restart=on-failure
User=www-data

[Install]
WantedBy=multi-user.target

⚠️ Important Step: After creating or editing a unit file, you must run sudo systemctl daemon-reload to force systemd to scan the directory and notice your new configuration before you can start or enable it.

Service Types β€” simple vs. oneshot βš™οΈπŸ”€

When writing the [Service] block of a systemd unit file, the Type= directive tells systemd how the application behaves and how it should determine if the service has successfully started. This page focuses on the two most fundamental types: simple and oneshot.

🟒 1. Type=simple (The Default Continuous Daemon)

This is the standard type used for traditional, long-running background services that stay active indefinitely.

  • How it Works: systemd spawns the process specified in ExecStart and immediately considers the service fully operational and active.
  • Startup Sequence: It does not wait for the application to finish initializing or loading into memory; as long as the initial process fork doesn’t fail right away, systemd moves on to loading the next dependent service.
  • Operational State: The service remains in an active (running) state for its entire lifecycle until it is manually stopped or crashes.
  • Typical Use Case: Web servers, database systems, or persistent network handlers (e.g., Nginx, SSH, custom APIs).

🟑 2. Type=oneshot (The Finite Task / Script)

This type is designed for short-lived scripts or initialization tasks that run to completion and exit.

  • How it Works: systemd launches the ExecStart command but blocks subsequent services and waits. It considers the service active only after the process exits cleanly with an exit code of 0.
  • Startup Sequence: If another service depends on this one, systemd will explicitly wait for this task to finish completely before spinning up the dependent service.
  • Operational State: Once the process finishes, its runtime state transitions to active (exited). It is technically no longer running in memory, but systemd remembers that it successfully ran.
  • Typical Use Case: One-time setup scripts, clearing out /tmp directories on boot, applying database migrations, or running backup routines.

🧱 3. Tracking Lifetime: RemainAfterExit=yes

By default, when a oneshot process exits, systemd treats it as finished. If you check its status, it might show as inactive. To change this behavior, you can pair it with RemainAfterExit:

  • Purpose: When added to a Type=oneshot service, it tells systemd to keep the service marked as active (exited) even after the main process has died.
  • Why use it?: It is incredibly useful for configuration steps (like setting up a firewall rule or mounting a virtual drive). This way, systemd registers that the state is still technically β€œon” or β€œactive,” allowing you to gracefully tear it down later using systemctl stop.

βš–οΈ Quick Comparison Summary

MetricType=simple 🟒Type=oneshot 🟑
LifecycleContinuous daemon; runs forever.Short-lived; runs to completion and exits.
Systemd ReadinessConsidered up immediately upon process launch.Considered up only after the process exits cleanly.
Boot BlockingParallel; doesn’t block other boot services.Sequential; halts dependent services until finished.
Typical Runtime Stateactive (running)active (exited) (when successful)

Custom Unit File Deployment Workflow πŸ”„πŸ› οΈ

πŸ“₯ Step 1: Create the Unit Configuration File

You must write your declarative configuration parameters into a dedicated .service file.

  • Target File Path: /etc/systemd/system/my_service.service
  • Administrative Requirement: Because /etc/ belongs to system-wide configurations, you must use superuser privileges (e.g., sudo vim or sudo nano) to write this file.

πŸ”„ Step 2: Reload the systemd Daemon Manager

Whenever you create a brand-new unit file or modify an existing one on disk, systemd will not automatically recognize the changes. You must force it to re-scan the configuration directory.

  • The Command:Bash

      $ sudo systemctl daemon-reload

  • Why it matters: Skipping this step will cause subsequent execution attempts to throw an error, as systemd will still be looking for its old cached configuration in memory.

🟒 Step 3: Trigger Immediate Execution

Once systemd has registered your newly created service object, you can launch it immediately into the active system background.

  • The Command:Bash

      $ sudo systemctl start my_service.service

  • Behavior: This executes the instruction specified in your file’s ExecStart= directive.

🩺 Step 4: Verify Runtime Health & Diagnostic Integrity

Always check whether your service initialized correctly or cleanly completed its task without running into runtime crashes.

  • The Command:Bash

      $ systemctl status my_service.service

  • Verification Targets: Check that the status shows a green active (running) marker for traditional services, or an active (exited) state if you deployed a standard Type=oneshot initialization script. Review the trailing journal logs appended at the bottom for any script warnings or failures.

πŸ’Ύ Step 5: Enable Automatic Boot Persistence (Optional)

If the service needs to survive system maintenance cycles or machine reboots, register it to launch automatically during the system initialization process.

  • The Command:Bash

      $ sudo systemctl enable my_service.service

  • Behavior: This reads the [Install] block of your file and generates the necessary symbolic links within the system’s boot target directories.

Oneshot & Service Dependency πŸ§©β›“οΈ

This page explains how to orchestrate complex boot sequences by combining short-lived initialization scripts (Type=oneshot) with formal systemd dependency directives to ensure tasks execute in a strict, reliable order.

πŸ”— 1. Orchestrating Sequences with Requires= and After=

When building complex systems, certain daemons cannot start until a prerequisite setup task has finished completely. systemd handles this using two distinct directives in the [Unit] section:

  • Requires=[prerequisite.service] (The Hard Dependency)
    • Defines a strict activation link. If you start your main service, systemd will automatically trigger the prerequisite service as well. If the prerequisite service fails to start, your main service will refuse to run.
  • After=[prerequisite.service] (The Ordering Constraint)
    • Dictates the exact startup timeline. It explicitly forces your main service to wait until the prerequisite service is fully up and running before initializing itself.

⚑ 2. The Power of Pairing Dependencies with Type=oneshot

The true strength of systemd orchestration comes from pairing these unit dependencies with a Type=oneshot initialization task.

  • The Problem with Type=simple: If your setup script is configured as Type=simple, systemd considers it β€œready” the exact millisecond the process forks. It will immediately launch the next dependent daemon in parallel, causing that daemon to crash because the initialization environment isn’t fully ready yet.
  • The oneshot Solution: When the prerequisite is configured as Type=oneshot, systemd shifts into a sequential blocking mode:
    1. systemd triggers the oneshot initialization script.
    2. It explicitly holds back and blocks the main dependent service.
    3. The script executes its finite task (e.g., creating a directory, setting up network routes, or clearing logs) and exits cleanly (exit 0).
    4. Only after the oneshot process terminates does systemd mark it as active and safely launch the main application daemon.

πŸ“ Practical Architecture Example

Imagine a main Web Application daemon (webapp.service) that requires a database folder setup script (db-init.service) to run completely before it can launch.

Prerequisite Setup Script:

# /etc/systemd/system/db-init.service
[Unit]
Description=Initialize Database Directories

[Service]
Type=oneshot
ExecStart=/usr/local/bin/setup-db-folders.sh
# Because it's oneshot, systemd waits until this script finishes exiting

Main Application Daemon:

# /etc/systemd/system/webapp.service
[Unit]
Description=Main Web Application
Requires=db-init.service
After=db-init.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/webapp/server.py

🎯 Key Architectural Rule:

Use Type=oneshot for any blocking configuration step that must run to completion before your main application starts up. Combine it with Requires= and After= on your main daemon to create an absolute, crash-proof boot sequence.

Managing Scheduled Tasks with crontab β°πŸ“

While cron is the background daemon that executes automated tasks, crontab (cron table) is the command-line utility and configuration file used to manage individual user automation schedules.

πŸ› οΈ Core Command-Line Flags

The crontab utility includes safety flags to interactively handle your active automation files.

  • crontab -e (Edit) πŸ“
    • Behavior: Opens your personal cron configuration file inside the system’s default text editor (e.g., nano or vim).
    • Safety Tip: Always use this command to modify your schedule instead of editing system files directly; crontab -e automatically checks your syntax for errors before saving changes.
  • crontab -l (List) πŸ“‹
    • Behavior: Prints a clean, read-only summary of all automated tasks currently registered under your active user account. It is a quick way to audit your schedules without risking accidental edits.
  • crontab -r (Remove) ❌
    • Behavior: Instantly deletes your entire personal crontab file, erasing all scheduled tasks for your user account.
    • Warning: Use this flag with extreme caution, as it typically deletes the file immediately without asking for a confirmation prompt.

πŸ—‚οΈ Configuration Storage: User vs. System

The Linux operating system separates user-defined automated tasks from critical system-level tasks by storing them in entirely different locations:

                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                  β”‚   Where Cron Tasks Live       β”‚
                  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                  β”‚
         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
         β–Ό                                                 β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚     User-Level Crontabs         β”‚               β”‚     System-Wide Crontabs        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€               β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β€’ Managed via 'crontab -e'      β”‚               β”‚ β€’ Managed by Root/Administratorsβ”‚
β”‚ β€’ Path: /var/spool/cron/crontabsβ”‚               β”‚ β€’ Path: /etc/crontab            β”‚
β”‚ β€’ Runs with user permissions    β”‚               β”‚ β€’ Path: /etc/cron.d/            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜               β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  • User-Level Schedules (/var/spool/cron/crontabs/) πŸ‘€
    • This directory holds individual configuration files named after specific system user accounts (e.g., a file named ubuntu or john).
    • Tasks written here execute automatically using the specific privileges of the user who created them. Regular users do not need sudo access to manage their own files.
  • System-Wide Schedules (/etc/crontab & /etc/cron.d/) βš™οΈ
    • These root-owned files are reserved for core operating system tasks, maintenance scripts, and packages installed by system package managers (like apt).
    • Unlike user files, system-wide crontabs include an extra configuration field that explicitly dictates which system user account should execute the script.

Learning Objectives - lesson 6 🎯

  • Understand the Linux Logging Infrastructure πŸ“‘
    • Grasp how the operating system captures, organizes, and stores system messages, application errors, and kernel warnings.
    • Differentiate between traditional text-based logging mechanisms and modern binary logging daemons.
  • Master Log Inspection with journald & journalctl πŸ”
    • Understand the role of systemd-journald as the centralized system service responsible for collecting data from the kernel, system services, and boot routines.
    • Learn to use the journalctl command-line utility to query and filter the centralized system journal database.
    • Successfully filter operational log data using specific runtime flags:
      • Filter logs based on boot cycles to isolate errors from a previous restart.
      • Target specific services to isolate diagnostic output from a single daemon.
      • View live, real-time log updates as they hit the system disk.
  • Audit System Boot History and Failures πŸ₯Ύ
    • Leverage system utilities to track how long it takes for a machine to complete its startup sequence.
    • Analyze and isolate specific bottlenecks, delayed startup scripts, or crashing daemons that slow down or prevent a clean system boot.

Command: journalctl πŸ”

The journalctl command is the primary tool used to query and view logs generated by systemd-journald, the centralized, binary logging daemon in modern Linux systems. Because logs are stored in a structured binary format rather than plain text, journalctl provides powerful filtering capabilities to quickly isolate specific system events.

πŸ› οΈ Core Command-Line Flags & Workflows

Instead of manually running grep on massive text files, you can use built-in flags to slice through system logs by time, service, or priority:

  • 1. Real-Time Log Tailing (Follow Mode) ⏱️Bash

      $ journalctl -f
    • Behavior: Behaves like the traditional tail -f command. It displays the most recent log entries and leaves the terminal open, printing new log lines in real time as they occur.

  • 2. Filtering by Specific Service Unit βš™οΈBash

      $ journalctl -u sshd.service
    • Behavior: Restricts the log output strictly to messages generated by a particular systemd unit (e.g., sshd.service). This is highly effective for debugging a single crashing daemon without system noise.

  • 3. Viewing Logs from the Current Boot Only πŸ₯ΎBash

      $ journalctl -b
    • Behavior: Filters out all historical data and displays only the entries recorded since the most recent system startup.
    • Tip: You can check logs from previous boots using negative integers (e.g., journalctl -b -1 shows logs from the second-to-last boot cycle).

  • 4. Jumping Direct to the End πŸ“„Bash

      $ journalctl -e
    • Behavior: Immediately scrolls to the very bottom of the log file inside the pager, allowing you to see the most recent events right away.

πŸ’‘ Advanced Combinations for Fast Debugging

Because these flags are modular, you can chain them together to create highly specific diagnostic queries:

# View live, real-time updates specifically for your custom web application
$ journalctl -f -u webapp.service

# Jump straight to the latest boot logs for the Nginx web server
$ journalctl -b -u nginx.service -e

/var/log & logrotate πŸ“πŸ”„

While modern systemd systems utilize binary journals, Linux continues to maintain traditional plain-text log files for backward compatibility and human-readable auditing. This page focuses on where these logs live and how the system prevents them from consuming all available disk space.

πŸ“ 1. The /var/log/ Directory

In the Linux filesystem hierarchy, the /var/log/ directory is the dedicated, centralized repository for persistent, system-wide text logs.

  • Characteristics: Unlike binary journals, these logs are standard text files. You can inspect them using traditional command-line utilities like cat, less, tail, or grep.
  • Key Log Files Found Here:
    • /var/log/syslog (or /var/log/messages): The general system log that captures a broad range of global system activity and non-critical application messages.
    • /var/log/auth.log (or /var/log/secure): Tracks security authentication events, including both successful and failed user login attempts.
    • /var/log/nginx/ or /var/log/apache2/: Dedicated subdirectories where specific third-party applications store their operational access and error files.

πŸ”„ 2. The logrotate Utility

Because servers run continuously for months or years, active text logs will grow indefinitely. If left unchecked, a single chatty application could generate massive files that completely fill up the system’s hard drive, causing critical database crashes or system-wide failure.

To prevent this, Linux utilizes a background utility called logrotate.

How Log Rotation Works:

logrotate is executed automatically as a scheduled background task (usually via a daily cron job). It handles log files using a cyclical shifting process:

  1. Rotation: The active log file (e.g., syslog) is closed and renamed to a historical file (e.g., syslog.1). A brand-new, empty syslog file is created immediately in its place so applications can keep writing without interruption.
  2. Compression: Older historical files (e.g., syslog.2, syslog.3) are compressed into .gz zip archives (e.g., syslog.2.gz) to drastically reduce the amount of physical disk storage they occupy.
  3. Purging (Retention Limit): The system only keeps a pre-configured number of historical archives (for example, 4 weeks of logs). When a new rotation occurs and exceeds this limit, the oldest archive (syslog.4.gz) is permanently deleted from the disk.

πŸ“ Example of a logrotate Configuration

The behavior of logrotate is governed by configuration files located in /etc/logrotate.conf and /etc/logrotate.d/. A typical service configuration block looks like this:

/var/log/nginx/*.log {
    daily           # Rotate the log files every single day
    rotate 7        # Keep exactly 7 days of historical archives before deleting
    compress        # Compress historical logs into .gz files to save space
    delaycompress   # Wait until the next rotation cycle to compress (.1 stays text)
    missingok       # If the log file is missing, move on without throwing an error
    notifempty      # Do not rotate the log file if it is completely empty
}

Linux Boot Flow Sequence πŸ₯ΎπŸ—️

The Linux boot flow is the precise, multi-stage initialization sequence that transitions a machine from completely powered-off hardware to a fully operational, multi-user operating system environment.

🧱 The 5 Core Stages of the Boot Sequence

1. Hardware Initialization: BIOS / UEFI πŸ–₯️

  • The Action: The moment power hits the motherboard, the hardware firmwareβ€”either the legacy BIOS (Basic Input/Output System) or modern UEFI (Unified Extensible Firmware Interface)β€”initializes.
  • The Mission: It runs a POST (Power-On Self-Test) to verify core hardware integrity (RAM, CPU, storage drives). Once verified, it looks at the system’s boot order configuration to locate and read the primary boot sector of the selected storage drive.

2. The OS Selector: GRUB Bootloader 🚨

  • The Action: The firmware hands control over to the bootloader, which on modern Linux systems is typically GRUB (Grand Unified Bootloader).
  • The Mission: GRUB displays the system boot menu (allowing you to select different kernel versions or recovery modes). Its primary job is to locate the compiled Linux kernel file on disk, load it into system memory, and pass execution control to it.

3. The Core Safe Space: Kernel & initramfs 🧠

  • The Action: The Linux Kernel decompresses itself into memory and takes total control of the hardware.
  • The Mission: Because the kernel doesn’t yet have access to the full hard drive, it mounts a temporary, microscopic filesystem in RAM called initramfs (Initial RAM Filesystem). This temporary space contains just enough essential hardware drivers to let the kernel safely find, mount, and access the true root filesystem (/) on your physical hard drive.

4. The Master Initializer: systemd (PID 1) πŸ‘‘

  • The Action: Once the real root filesystem is mounted, the kernel cleans up the temporary RAM disk and immediately launches the very first user-space process.
  • The Mission: This process is systemd, permanently assigned PID 1. It takes over as the master orchestrator, evaluating system unit files and preparing to bring up the rest of the operating system space in parallel.

5. Target Achievement: Environment Runlevels 🎯

  • The Action: systemd looks at its default boot configuration to determine which Target Unit it needs to reach.
  • The Mission: It begins spawning all background daemons, network interfaces, and filesystems required to satisfy that target environment:
    • multi-user.target: Initializes a standard, command-line server environment with networking and multi-user login prompts.
    • graphical.target: Loads everything in multi-user.target plus the heavy display managers and graphical user interfaces (GUI) for desktop use.

⏱️ Boot Performance Auditing: systemd-analyze

Modern systemd environments include built-in tools to measure exactly how long this boot process takes, helping you optimize server startup speeds:

  • systemd-analyze
    • Output: Prints a single-line summary of total boot duration, broken down by how long was spent in the kernel versus user-space startup.
  • systemd-analyze blame
    • Output: Prints a detailed list of every single running service, sorted in descending order by exactly how many seconds it took to initialize. This allows you to instantly pinpoint a lagging script or hanging daemon that is bottle-necking your boot speed.

Runlevel ↔ Target Mapping πŸ—ΊοΈπŸ”„

Historically, traditional Unix and Linux systems used an initialization framework called SysV init to manage the operating system state using numeric Runlevels (numbered 0 through 6). Modern Linux systems driven by systemd have replaced these numbers with descriptive, object-oriented Target Units (.target).

πŸ—ΊοΈ The Runlevel-to-Target Mapping Table

Each classic numeric runlevel maps directly to a specific, modern systemd target file. While you can still use the old runlevel numbers in many commands for backward compatibility, systemd translates them into targets under the hood:

RunlevelClassic SysV Init PurposeModern systemd Target UnitDescription & Operational State
0Haltpoweroff.targetShuts down the operating system and cleanly powers off the machine’s hardware components.
1Single-User Moderescue.targetA minimal, isolated maintenance mode. No network, no daemon services, drops straight into a root shell for emergency repairs.
2, 3, 4Multi-User Mode (Console)multi-user.targetStandard non-graphical server environment. Full networking services are active, and multiple users can log in concurrently via CLI.
5Multi-User Mode (Graphical)graphical.targetStandard desktop environment. Includes everything in Level 3, plus loads the display manager and graphical user interface (GUI).
6Rebootreboot.targetCleanly terminates all active processes, unmounts filesystems, and restarts the machine hardware.

βš™οΈ Managing System Targets with systemctl

Instead of the historical init [0-6] commands, modern administrators use specific systemctl operations to audit or dynamically shift the system’s operational environment:

  • Check the Current Default Target πŸ‘οΈBash

      $ systemctl get-default
    • Output Example: multi-user.target (tells you the system will boot into standard command-line mode by default).

  • Change the Default Boot Environment πŸ’ΎBash

      $ sudo systemctl set-default graphical.target
    • Impact: Permanently changes the boot configuration. On the next hardware restart, the machine will automatically load the graphical desktop GUI.

  • Switch Targets Instantly (Isolate Mode) ⚑Bash

    $ sudo systemctl isolate multi-user.target

    • Impact: Dynamically switches the system state in real time without a full reboot. This closes down the desktop interface, terminates active GUI applications, and drops the system down into the pure command-line multi-user environment. Use with caution, as it stops any running services not explicitly required by the new target.



Β© 2017. by isme2n

Powered by aiden