Learning Awk Is Essential For Linux Users

Main ideas / lessons

• Awk is an essential Linux text-processing tool: you provide text, and awk can extract:
  • specific fields/columns
  • specific rows/lines
  • search for patterns
  • replace or transform text
• Field/column concept: awk treats fields as column-like segments, typically separated by whitespace by default. You can change what separates fields using field separators (e.g., -F ':' or FS=...).
• Awk is both a command-line utility and a scripting language:
  • It supports printing, pattern matching, conditional filtering, string functions, length checks, simple arithmetic, loops, and range/line counting.

---

Methodology / “how-to” instructions shown

1) Extract a specific column from command output

1. Run a command that produces tabular-ish output (example: ps).
2. Pipe into awk and print a chosen field:
   • awk '{ print $1 }' → prints the first field of each line
   • awk '{ print $2 }' → prints the second field

Notes:

• awk uses space by default as the field separator.

---

2) Print everything (or default behavior)

• Use: awk '{ print $0 }'
• Meaning:
  • $0 represents the entire line (the whole record).

---

3) Use a different field separator (e.g., colon-delimited files)

For files like /etc/passwd where fields are separated by ::

• Print the first field:
  • awk -F ':' '{ print $1 }' /etc/passwd
• Print multiple fields:
  • awk -F ':' '{ print $1, $6, $7 }' /etc/passwd

Formatting for readability:

• Add spaces between columns, or
• Use tabs as separators:
  • (Conceptually) print $1 "\t" $6 "\t" $7

---

4) Control both input separator and output separator (FS and OFS)

• Example concept:
  • BEGIN { FS=":"; OFS="-" }
• Then print fields:
  • awk 'BEGIN{FS=":"; OFS="-"} { print $1, $6, $7 }' /etc/passwd

Result:

• The output includes the chosen separator (here -) between fields.

---

5) Print the last field of each line

• Use:
  • awk -F '/' '/pattern/ { print $NF }' file

Concept shown:

• From /etc/shells, treat / as the delimiter.
• Match lines beginning with / (pattern uses the beginning-anchor concept).
• Print $NF to get the last field (the shell name without the full path).

Remove duplicates (pipeline concept):

• Pipe into:
  • sort | uniq

---

6) Filter rows based on whether lines contain a substring

• Use a pattern inside /.../ to select only lines that match.
• Example concept:
  • df | awk '/\/dev\/loop/ { ...print selected fields... }'

Then print specific fields from the matched lines.

---

7) Perform arithmetic on fields

• Example shown:
  • print $1 "\t" ($2 + $3)

Also discussed:

• You can use other operators like subtraction.

---

8) Filter lines by the length of the record

• Use length($0):
  • Keep lines where length($0) > 7
  • Example concept:
    • awk 'length($0) > 7' /etc/shells

Reverse/alternate filter:

• length($0) < 8

---

9) Filter based on the last field’s value (e.g., process name)

• Example logic from ps -ef:
  • Use $NF (last field)
  • Compare it to a target string (e.g., /bin/fish)
• If it matches, print the whole line ($0).

---

10) Use BEGIN and for loops to generate data

• Example concept:
  • awk 'BEGIN { for (i=1; i<=10; i++) print "sqrt of", i, "is", sqrt(i) }'

This demonstrates awk can generate output without input files.

---

11) Print lines whose first character matches a set

• Example concept (from .bashrc):
  • Match records where the first column/first character begins with a or b or c
  • Print the entire line for matching records.

---

12) Extract substring portions (skip first N characters)

• Alternative to column selection:
  • Use substr($0, startIndex)
• Example described:
  • Remove the first three characters so the “second column” becomes visible without line numbers.

---

13) Match patterns and compute index position with match()

• Example concept:
  • For each line, check whether a letter exists:
    • match($0, "o")
• Then print the line plus the character index where it appears.

---

14) Print a range of lines using record number (NR)

• Example from piping df output into awk:
  • Select line numbers from 7 through 11:
    • NR==7, NR==11 { ... }

To include line numbers:

• Print NR along with the content field (as described).

---

15) Count total lines/records (single or multiple files)

• For one file:
  • awk 'END { print NR }' file
  • (Subtitles describe awk 'END print NR' to get the total line count.)
• For multiple files:
  • Run awk across multiple inputs; NR accumulates totals across files
  • Example shown with /etc/shells plus /etc/passwd.

---

Sources / speakers featured (as named in subtitles)

• Video narrator/speaker (unnamed): the main presenter demonstrating awk commands.
• Special thanks / producers and patrons named:
  • absi gabe james mitchell akami allen chuck david dilling gregory eryon paul polytech scott stevens finn west willie
• No other named guests speaking on camera are explicitly indicated beyond the thank-you list.