SysAdmin - Operation of running systems

In

Linux Foundation Certified SysAdmin (LFCS): Operation of running systems

Boot, reboot, and shut down a system safely

  • Power off
    1
    2
    3
    4
    # power off, default option
    shutdown -p
    # hard shutdown in 5 minutes
    shutdown -h +5m
  • Reboot
    1
    2
    3
    4
    5
    6
    7
    8
    # power off, default option
    shutdown -p
    # reboot
    shutdown -r
    # reboot with time: now
    shutdown -r now
    # time with time: in 1 hour
    shutdown -r +1h
  • Uptime
    1
    2
    # how long have you been up?
    uptime

Bootloaders

Operating modes

System V run levels

Level Explanation
0 halt, complete shutdown
1 single user mode
2 multiuser mode without networking (serial ports)
3 multiuser mode wit networking
4 Not used/user-definable
5 graphic mode with networking and x-window
6 reboot 
1
2
3
4
# current system V run level
runlevel
# hold down [[Shift]] while rebooting to go to GRUB, to choose run level
# add the number to the linux line at the bottom

Install and configure

  • GRUB2 - boot/grub/grub.cfg
    • It is automatically generated by updateGRUB process from /etc/grub.d -> those can be changed
    • Run update-grub after changes
  • Example of grub Linux file: 15_angelesLinuxBoot (number is priority)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    #!/bin/sh -e
    # what is happening when we run our update grub, for debug
    echo "displayed when update-grub is run"
    cat << EOF
    # test to show on the menu itself
    menuentry "Other Linux Partition" {
     # hard drive device, partition (starts by 1, not 0)
      set root=(hd0,1)
      # boot options, depends on SO
      linux /boot/vmlinuz
      initrd /boot/initrd.img
    }
    # end of file
    EOF
    • Make it runnable by changing its permissions
      1
      sudo chmod +x 15_angelesLinuxBoot
  • Example of grub Windows file: 16_angelesWindowsBoot (number is priority)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    #!/bin/sh -e
    # what is happening when we run our update grub, for debug
    echo "adding Windows partition to grub menu"
    cat << EOF
    # test to show on the menu itself
    menuentry "Windows Partition" {
     # hard drive device, partition (starts by 1, not 0)
      set root=(hd2,1)
      # boot options, depends on SO
     chainloader (hd2,1)+1
    }
    # end of file
    EOF
    • Make it runnable by changing its permissions
      1
      sudo chmod +x 16_angelesWindowsBoot
  • Install and fix
    1
    2
    3
    4
    # install (somewhere)
    grub-install /dev/hda
    # fix easily if broken, by using a live cd
    grub-install --root-directory=/mnt /dev/sda

Processes: diagnose and manag

  • Tools

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # scalate privileges
    sudo su
    # see what is happening on a table
    # 20 is lowest priority, -20 is highest priority
    top
    # prettier version
    htop
    # classical plain way
    ps
    # ps showing everything
    ps aux
    # pipe 'ps aux' to filter it
    ps aux | grep cron
    # get pids
    pgrep bash
    # see processes tree
    ps acjf

  • End process

    1
    2
    3
    4
    5
    # kill pid
    kill 731
    # list all signals
    kill -l
    # fix overactive process

  • Start a copy of a process and change priority

    1
    2
    3
    4
    5
    6
    # lowest prority for a bash process
    nice -n 20 /bin/bash
    # find the pid
    ps aus | grep /bin/bash
    # we find the process pid is 2871, so we renice it
    renice 10 2871

Log files

1
2
3
4
5
6
# go to the log directory
cd /var/log
# use less or grep to find specific things
sudo less messages
# Shift+G to go to the very end
grep "Error" messages

Scheduler

  • Cronjobs: create tasks to run at a set date and time

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # list the crontabs
    crontab -l
    # lets supose we have this script
    cat ../script/moveAndBackup.sh
    # we want to run it at 5am everyday of the week
    # -e for edit
    crontab -e
    # table: [minutes, hour, dayOfMonth, month, dayOfWeek (0-6), command]
    # 5am everyday of the week
    # 0 5 * * 1 tar -zcf /var/backupd/home.tgz /home/
    # 2am everyday of the week
    # 0 2 * * 1 tar /home/user1/script/moveAndBackup.sh
    # crtl+x to save and exit
    crontab -l
    # think of the permission levels

  • Verify completion of scheduled jobs

    1
    2
    3
    4
    # go to log directory
    cd var/log
    # look for CRONs
    cat syslog | grep CRON

Update software

  • Ubuntu/Debian (deb)

    • dpkg
      1
      2
      3
      4
      5
      6
      # dpkg: ulist installed packages
      sudo dpkg - l
      # install a local package 'gedit'
      sudo dpkg -i gedit
      # check which version is installed for xauth
      sudo dpkg - l | grep xauth
    • aptitude
      • visual
        1
        2
        # load the front-end like
        aptitude
      • apt CLI
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        # update list of pending packages
        sudo apt-get update
        # do the previously listed update
        sudo apt-get upgrade
        # upgrade kernel, distribution
        sudo apt-get dist-upgrade
        # delete the no-longer needed packages
        sudo apt-get autoremove
        # list the packages names for installed packages
        sudo apt-cache pkgnames
        # search all additional text for anything that has xterm in it
        sudo apt-cache search xterm
        # get additional information for the nmap package
        sudo apt-cache show nmap
        # install gedit from repository
        sudo apt-get install gedit
        # remove application, keep config
        sudo apt-get remove gedit
        # remove application, config, etc
        sudo apt-get purge gedit
        # older systems: apt-get remove --purge gedit
        # download but not install, you may use dpkg after this
        sudo apt-get download gedit
        # get the changelog for a package
        sudo apt-get changelog gedit
        # check broken dependencies
        apt-get check
        # build dependency tree fpr nmap package
        apt-get build-dep nmap
        # clean the cache
        apt-get autoclean

  • CentOS/Redhat (.rpm)

    • yum
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      # check plus update
      yum update
      # get info about a package
      yum list openssh-server
      # deeper information
      yum search openssh-server
      #install
      yum install openssh-server
      # install saying 'yes' to everything
      yum install -y openssh-server
      # remove but keep config
      yum remove openssh-server
      # remove all
      yum purge openssh-server
      # package info
      yum info nmap
      # list package, find nmaps
      yum list | grep nmap
      # similar to previous line
      yum search nmap
      #show if installed on the system
      yum list installed | grep nmap
      # groups or "sets of applications"
      yum group list
      # you may install a whole group
      yum group install 'File and Print server'
      # show repos in use
      yum repolist
      # show all known (enabled or not)
      yum repolist all
      #enable a repository
      yum --enable-repo=extras-source/7 pkgname
      # clean unused packages
      yum clean all
      # what happened recently
      yum history
    • yumtils
      1
      2
      yum install -u yum-utils
      yumdonwloader xterm
    • rpm
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      # query dependencies
      rpm -qpR xterm
      # query is xterm installed
      rpm -q xterm
      # every single file installed as part of xterm
      rpm -ql xterm
      #check recently altered packages
      rpm -qa --last
      # documentation
      rpm -qdf xterm
      # verified packages
      rpm -Va
      # query public keys
      rpm -qa gpg-pubkey*
      # a
      rpm
      • Fix db
        1
        2
        cd var/lib/rpm
        rpm --rebuilddb

Verify the integrity and availability

  • Resources (HD, RAM)
    • With live CD
      1
      2
      3
      4
      5
      6
      # list block devices
      lsblk
      # unmunt the device
      sudo umount /mnt
      # check and clean
      sudo fsck -y /dev/xvf1
    • cron it before running
      1
      2
      3
      # just add the file
      sudo touch fsck
      # after you reboot, it will be deleted
    • use memcheck from GRUB (hold [[Shift]])
  • Key processes
    • ps
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      # fix better on widescreen
      ps au
      # more info for a certain process
      ps aux | grep cron
      # list 
      ps ef
      # process for the root user
      ps -fu root
      # process for "user" group
      ps -fG user
      # show process tree
      ps -e --forest
    • top (live, updated)
      1
      2
      3
      4
      top
      # alternative with cleaner interface, has mouse support
      sudo apt install htop
      htop

Change kernel runtime parameters, persistent and non-persistent

  • parameters are files
    1
    2
    3
    4
    5
    6
    7
    8
    9
    # parameters are files here
    cd /proc/sys
    ls -la
    # is ip forward on?
    cat /proc/sys/net/ipv4/ip_forward
    # set the value as off
    sudo echo 0 > /proc/sys/net/ipv4/ip_forward
    # apply the changes you just did
    sysctl -p
  • system control
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    # show kernel runtime parameters
    sudo sysctl -a
    # everything is a file
    sudo sysctl -a | wc -l
    # is ip forward on?
    sysctl net.ipv4.ip_forward
    # set it as on
    sysctl -w net.ipv4.ip_forward=1
    # do what you just did, but with sudo
    sudo !!
    # apply the changes you just did
    sysctl -p
  • persist (normally they would be lost after reboot
    1
    2
    3
    cd /etc/sysctl.d
    nano 10-network-security.conf
    # add comment and set value 'net.ipv4.ip_forward=1'

Scripting

  • Shell scripting for system maintenance tasks

    1. create a shell script with nano test.sh 
      1
      2
      3
      4
      #!bin/bash
      echo "Line 1"
      echo "Line 2"
      ls -la
    2. check that you can execute it: chmod +x /test.sh
    3. launch test.sh
    4. you may add a directory to your path EXPORT PATH=$PATH:/home/user/scripts

  • Scripting conditionals and loops

    1
    2
    3
    4
    5
    6
    7
    8
    #!bin/bash
    DIRECTORY="/home/user/test"
    # conditional
    if [ -d $DIRECTORY ] ; then
      echo "the directory exists"
    else
      echo "the directory does not exist"
    fi
    1
    2
    3
    4
    5
    6
    #!bin/bash
    # loop for
    for COUNT in 1 2 3 4 5 6 7 8 9
    do
      echo "This is line # $COUNT"
    done  
    1
    2
    3
    4
    5
    6
    7
    8
    9
    #!bin/bash
    # supose a file 'myhosts' contains the following IPs
    #8.8.8.8
    #8.8.4.4

    # loop while
    while read $HOST; do
      echo "This is line # $COUNT"
    done < myhosts

Manage the startup process and services (in services configuration)

  • upstart (classic in Debian)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    cd /etc/init
    # know pid for process
    status cron
    # cron pid=1404
    ps aux | 1404
    # change status
    stop cron
    start cron
    restart cron
    # override
    ls -ls cron.conf
    echo manual | sudo tee /etc/init/cron.override
    ls -la cron*
    # undo
    cron.override
  • systemd (current in Ubuntu)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    cd /etc/systemd
    # check status
    systemctl status cron
    # requests authentication for changing status
    systemctl stop cron
    systemctl start cron
    # synch status server, remove softlink
    systemd-sysv-install disable cron
    # restart
    systemctrl enable cron

List and identify SELinux/AppArmor file and process contexts

  • SELinux
    1
    2
    3
    4
    5
    # all security context files on SELinux
    # grep it for a certain value
    sudo semanage fcontext -l
    # example of context
    cd /etc/rc.d
  • AppArmor
    1
    2
    3
    4
    # status
    sudo aa-status
    # check profiles on directory
    cd /etc/apparmor.d
  • Common tools
    1
    2
    3
    4
    # show security context for files
    ls -Z
    # processes with security context
    ps auxZ

Identify the component of a Linux distribution that a file belongs to

  • RedHat
    • rpm
      1
      rpm -qf /bin/znew
    • yum
      1
      yum whatprovides /bin/znew
  • Debian
    • dpkg
      1
      2
      3
      dpkg -S /usr/bin/zdump
      # all files installed by a particular package
      dpkg -L libc-bin