Angeles' code notepad

Coding notes

Angeles' code notepad

In

Linux Foundation Certified SysAdmin (LFCS): Networking

Configure networking and hostname resolution statically or dynamically

  • Older Debian
    • etc/network/interfaces contains loopback asks for further configuration in interfaces.d
      1
      2
      3
      4
      cd etc/network
      cat interfaces.d
      # change configuration for ethernet0
      nano eth0.cfg
    • file content example
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      # bring this up automatically
      auto eth0
      ## This commented configuration is for dinamic
      ## keyword (iface), id
      # iface eth0 inet dhcp
      ## This uncommented configuration is for static
      iface eth0 inet static
      address 10.9.8.7
      netmask 255.255.255.0
      gateway 10.9.8.1
      dns-search mydomain.com
      dns-nameservers 8.8.8.8.8.8.4.4
    • restart after changes to enforce them
      1
      sudo ifup eth0
  • New Ubuntu

    • etc/network/interfaces contains loopback asks for further configuration in interfaces.d

      1
      2
      3
      4
      5
      cd etc/network
      ifconfig -a
      cat interfaces.d
      # change configuration for ethernet0
      nano 50-cloud-init.cfg

    • everything else from the older Debian machine applies

  • CentOS 7
    • etc/sysconfig/network-scripts most interesting files are ifconfig-eth0 and ifcfg-lo
      1
      2
      3
      4
      5
      cd etc/sysconfig/network-scripts
      cat interfaces.d
      # ethernet=ifconfig-eth0, loopback=ifcfg-lo
      # change configuration for ethernet0
      nano ifconfig-eth0.cfg
    • file content example
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      ## dynamic address
      BOOTPROTO=dhcp
      DEVICE=eth0
      HWADDR=0a:67:42:8f:24:9e
      ONBOOT=yes
      TYPE=Ethernet
      USERCTL=no
      ## static address
      # BOOTPROTO=none
      # DEVICE=eth0
      # HWADDR=0a:67:42:8f:24:9e
      # ONBOOT=yes
      # TYPE=Ethernet
      # IPADDR=10.9.8.7
      ## subnet/mask in CENTOS-> number of 1s begore we hit our 1st 0
      # PREFIX=24
      # GATEWAY=10.9.8.1
      # DNS1=10.9.8.53
      # DNS2=8.8.8.8
      # DNS3=8.8.4.4
    • restart after changes to enforce them
      1
      systemctl restart network
    • you may also use the old school ifconfig commands
      1
      ifconfig eth1 10.1.0.122 netmask 255.255.255.0 && ifconfig eth1 up

      When setting a static route for a network or IP address on your system, you are bypassing the default gateway

Configure network services to start automatically at boot

Example with telnet: old, not recommended, now we use SSH

  • systemd
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    # is it already installed?
    sudo systemctl | grep telnet
    # if not, install it
    sudo apt install telnet
    sudo apt install telnetd
    # management platform for network connections
    sudo apt install xinetd
    # enable that service, is it running?
    sudo /etc/init.d/inet status
    sudo /etc/services
    telnet localhost
    sudo sysctl enable xinetd
  • System V init (sysvinit)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    # install it, it brings inet with it
    sudo yum install telnet-server
    sudo yum install telnet
    # enable that service, is it running?
    chkconfig xinetd
    chkconfig telnet on
    chkconfig
    service xinet start
    # go for it
    telnet localhost
    # disable
    sudo chkconfig telnet off
    sudo chkconfig
    sudo service xinet stop

Implement packet filtering

You will need a second machine to check the results (ping)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
## check chain policies
iptables -L
# get blank slate
iptables --flush
# change the forward policy to accept
# we will not be filtering anything
iptables -P FORWARD ACCEPT
## create a filter to prevent access
# check your interfaces
ifconfig
# reject on ethernet0 the icmp protocol traffic -> get error
ipdatebles -A INPUT --protocol icmp --in-interface eth0 -j reject
# drop on ethernet0 the icmp protocol traffic -> not return anything at all
ipdatebles -A INPUT --protocol icmp --in-interface eth0 -j drop

Start, stop, and check the status of network services

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#protocol information, sockets...
netstat
# sudo gets additional information
# ports in which you are listening now
netstat -at
# UDP ports in which you are listening now
netstat -ua
# everything connected: -u for UDP, -x for UNIX
netstat -l
# check protocols
netstat -s | more
# display service names with their pids
netstat -tp
# display interfaces and loopback address
netstat -i
# kernel interface table, almost identical to ifconfig -a
netstat -ie
# groups
netstat -g
# find ssh listening programs
netstat -ap | grep ssh
# get the statistics
netstat --statistics-raw

Statically route IP traffic

1
2
3
4
5
6
7
8
9
10
11
12
13
# all Ip adresses, you may use ipconfig too
ip a
## what if it is not a local package (123.123.123.xxx)
## learn the routes, 'route -n' gets the same value
ip route list
# how IP addresses of each hub on the way (if not blocked)
traceroute 8.8.8.8
## make it go through a certain device first
# send all packages starting with 8.8, static protocol
# metric -> kernel selects the one with lower metric, via inet <ip>, device <id>
sudo ip route add 8.8.0.0/16 proto static metric 10 via inet 10.9.185.143 dev eth0
## remove jump
sudo ip route del 8.8.0.0/16 proto static metric 10 via inet 10.9.185.143 dev eth0

Synchronize time using other network peers

Same time in severla machines is important (e.g. using docker)

1
2
3
4
5
6
7
# the most important is ntp, ntpd if you want it to run in boot
sudo apt install ntp
# edit configuration
sudo nano /etc/ntp.conf
## www.pool.ntp.org -> select an active ntp server constantly update
# restart
sudo /etc/init.d/ntp restart

In

Linux Foundation Certified SysAdmin (LFCS): Users and groups management

User accounts

  • Create user
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # old method
    # create user
    # specify directoryUserName and userName
    useradd -d /home/testuser1 testuser1
    # give the user a password, it will prompt the value
    passwd testuser1
    # you should also copy files from bash, etc

    # new method, all the previous in 1 line
    adduser testuser2
  • Change directory owner 
    1
    2
    3
    4
    5
    # create a folder and pass the ownership to user1
    mkdir testuser1Dir
    chown testuser1:testuser1 testuser1Dir
    # switch user, to act as testuser1
    su - testuser1
  • Remove user
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # manual
    cd /home
    userdel testuser2
    # its data is still there so we change teh ownsership
    chown -R user1:user1 testuser1
    # delete all
    rm -r testuser2

    # delete all related to testuser2
    userdel -R testuser2

Local groups and group memberships

  • Add group
    1
    2
    3
    4
    5
    # see groups
    cat /etc/group
    # groupName:x:groupId:userName
    addgroup testGroup1
    groupadd test2
  • add a user to group
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
      nano /etc/group
      # write down on the file
      # groupName:x:groupId:userName
      # test1:x:1002:user1
      # after that you can check the groups you are in with
      groups
      # other option: add user1 to docker group
      usermod -aG docker user1
      ```  
    * change password for group
      ```bash  
      sudo gpasswd test1
      # add the new password
      # the switch to that group
      newgrp test1
      # type password

System-wide environment profiles

  • Session variables (usually in .bashrc, .profile or .bash_profile`)

    .bash_profile contents can be executed any time the user logged into the system

    1
    2
    3
    4
    5
    6
    7
    8
    # check environment variables
    env
    # define env var for the session
    export PAPAS_VAR="papasconchoco"
    # checkenv var value
    echo $PAPAS_VAR
    #remove env var
    unset PAPAS_VAR
  • Remote session variables location
    1
    2
    cd /etc/environment
    cd /etc/profile
  • Unset
    1
    2
    3
    4
    # execute to be in absence of environment variables
    env -i
    # unset via null value
    export PAPAS_VAR=''

Template user environment

  • Skeleton directory
    1
    2
    3
    4
    5
    # the templates for new users are here
    cd /etc/skel
    # add environment variables
    sudo nano .bashrc
    # type the values, valid for users created from now on

Configure user resource limits

Configuration on limits.conf

1
sudo nano /etc/security/limits.conf
  • columns on the file: [domain] [type] [item] [value]
  • domain (user, group, wildcards (*, %)) -> no limit with wildcard for root
  • soft and hard (only increased by root) limits
    • ulimit for more resources with soft, until reaching hard limit value
  • item (core, data, cpu, memlock, nproc, as (address space limit), maxlogins, maxsyslogins)

Manage user privileges

Configuration on access.conf when login for privileges

1
sudo nano /etc/security/access.conf
  • Can he log-in? -> example ‘root’ should not be able to login remotely
  • Not everyone should be able to sudo
    1
    2
    3
    4
    # change the command "mycommand" to only be used for certain groups
    sudo chgrp adm mycommand
    # eliminate the execute bit for "other users" 
    sudo chmod 754

Configure PAM

  • Plugable Authetication Modules -> separate the authentication from the privilege grating software
  • Privilege grating software attaches to the PAM API
  • /etc/pam.conf (mostly ignored, backup) and /etc/pam.d (takes precedence)
    1
    2
    # check auth for common services
    less common-account
  • select the authentication scheme you prefer (permit, paranoid)
    1
    2
    3
    @include common-account
    @include common-session
    @include common-password
    • types of management
      • authentication
      • account
      • session
      • password
    • modules
      • pam-access (grant access)
      • cracklib (password against dictionary)
      • debug (login)
      • deny (prevent access)
      • echo (write messages)

In

Linux Foundation Certified SysAdmin (LFCS): Operation of running systems

Boot, reboot, and shut down a system safely

  • Power off
    1
    2
    3
    4
    # power off, default option
    shutdown -p
    # hard shutdown in 5 minutes
    shutdown -h +5m
  • Reboot
    1
    2
    3
    4
    5
    6
    7
    8
    # power off, default option
    shutdown -p
    # reboot
    shutdown -r
    # reboot with time: now
    shutdown -r now
    # time with time: in 1 hour
    shutdown -r +1h
  • Uptime
    1
    2
    # how long have you been up?
    uptime

Bootloaders

Operating modes

System V run levels

Level Explanation
0 halt, complete shutdown
1 single user mode
2 multiuser mode without networking (serial ports)
3 multiuser mode wit networking
4 Not used/user-definable
5 graphic mode with networking and x-window
6 reboot 
1
2
3
4
# current system V run level
runlevel
# hold down [[Shift]] while rebooting to go to GRUB, to choose run level
# add the number to the linux line at the bottom

Install and configure

  • GRUB2 - boot/grub/grub.cfg
    • It is automatically generated by updateGRUB process from /etc/grub.d -> those can be changed
    • Run update-grub after changes
  • Example of grub Linux file: 15_angelesLinuxBoot (number is priority)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    #!/bin/sh -e
    # what is happening when we run our update grub, for debug
    echo "displayed when update-grub is run"
    cat << EOF
    # test to show on the menu itself
    menuentry "Other Linux Partition" {
     # hard drive device, partition (starts by 1, not 0)
      set root=(hd0,1)
      # boot options, depends on SO
      linux /boot/vmlinuz
      initrd /boot/initrd.img
    }
    # end of file
    EOF
    • Make it runnable by changing its permissions
      1
      sudo chmod +x 15_angelesLinuxBoot
  • Example of grub Windows file: 16_angelesWindowsBoot (number is priority)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    #!/bin/sh -e
    # what is happening when we run our update grub, for debug
    echo "adding Windows partition to grub menu"
    cat << EOF
    # test to show on the menu itself
    menuentry "Windows Partition" {
     # hard drive device, partition (starts by 1, not 0)
      set root=(hd2,1)
      # boot options, depends on SO
     chainloader (hd2,1)+1
    }
    # end of file
    EOF
    • Make it runnable by changing its permissions
      1
      sudo chmod +x 16_angelesWindowsBoot
  • Install and fix
    1
    2
    3
    4
    # install (somewhere)
    grub-install /dev/hda
    # fix easily if broken, by using a live cd
    grub-install --root-directory=/mnt /dev/sda

Processes: diagnose and manag

  • Tools

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # scalate privileges
    sudo su
    # see what is happening on a table
    # 20 is lowest priority, -20 is highest priority
    top
    # prettier version
    htop
    # classical plain way
    ps
    # ps showing everything
    ps aux
    # pipe 'ps aux' to filter it
    ps aux | grep cron
    # get pids
    pgrep bash
    # see processes tree
    ps acjf

  • End process

    1
    2
    3
    4
    5
    # kill pid
    kill 731
    # list all signals
    kill -l
    # fix overactive process

  • Start a copy of a process and change priority

    1
    2
    3
    4
    5
    6
    # lowest prority for a bash process
    nice -n 20 /bin/bash
    # find the pid
    ps aus | grep /bin/bash
    # we find the process pid is 2871, so we renice it
    renice 10 2871

Log files

1
2
3
4
5
6
# go to the log directory
cd /var/log
# use less or grep to find specific things
sudo less messages
# Shift+G to go to the very end
grep "Error" messages

Scheduler

  • Cronjobs: create tasks to run at a set date and time

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # list the crontabs
    crontab -l
    # lets supose we have this script
    cat ../script/moveAndBackup.sh
    # we want to run it at 5am everyday of the week
    # -e for edit
    crontab -e
    # table: [minutes, hour, dayOfMonth, month, dayOfWeek (0-6), command]
    # 5am everyday of the week
    # 0 5 * * 1 tar -zcf /var/backupd/home.tgz /home/
    # 2am everyday of the week
    # 0 2 * * 1 tar /home/user1/script/moveAndBackup.sh
    # crtl+x to save and exit
    crontab -l
    # think of the permission levels

  • Verify completion of scheduled jobs

    1
    2
    3
    4
    # go to log directory
    cd var/log
    # look for CRONs
    cat syslog | grep CRON

Update software

  • Ubuntu/Debian (deb)

    • dpkg
      1
      2
      3
      4
      5
      6
      # dpkg: ulist installed packages
      sudo dpkg - l
      # install a local package 'gedit'
      sudo dpkg -i gedit
      # check which version is installed for xauth
      sudo dpkg - l | grep xauth
    • aptitude
      • visual
        1
        2
        # load the front-end like
        aptitude
      • apt CLI
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        # update list of pending packages
        sudo apt-get update
        # do the previously listed update
        sudo apt-get upgrade
        # upgrade kernel, distribution
        sudo apt-get dist-upgrade
        # delete the no-longer needed packages
        sudo apt-get autoremove
        # list the packages names for installed packages
        sudo apt-cache pkgnames
        # search all additional text for anything that has xterm in it
        sudo apt-cache search xterm
        # get additional information for the nmap package
        sudo apt-cache show nmap
        # install gedit from repository
        sudo apt-get install gedit
        # remove application, keep config
        sudo apt-get remove gedit
        # remove application, config, etc
        sudo apt-get purge gedit
        # older systems: apt-get remove --purge gedit
        # download but not install, you may use dpkg after this
        sudo apt-get download gedit
        # get the changelog for a package
        sudo apt-get changelog gedit
        # check broken dependencies
        apt-get check
        # build dependency tree fpr nmap package
        apt-get build-dep nmap
        # clean the cache
        apt-get autoclean

  • CentOS/Redhat (.rpm)

    • yum
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      # check plus update
      yum update
      # get info about a package
      yum list openssh-server
      # deeper information
      yum search openssh-server
      #install
      yum install openssh-server
      # install saying 'yes' to everything
      yum install -y openssh-server
      # remove but keep config
      yum remove openssh-server
      # remove all
      yum purge openssh-server
      # package info
      yum info nmap
      # list package, find nmaps
      yum list | grep nmap
      # similar to previous line
      yum search nmap
      #show if installed on the system
      yum list installed | grep nmap
      # groups or "sets of applications"
      yum group list
      # you may install a whole group
      yum group install 'File and Print server'
      # show repos in use
      yum repolist
      # show all known (enabled or not)
      yum repolist all
      #enable a repository
      yum --enable-repo=extras-source/7 pkgname
      # clean unused packages
      yum clean all
      # what happened recently
      yum history
    • yumtils
      1
      2
      yum install -u yum-utils
      yumdonwloader xterm
    • rpm
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      # query dependencies
      rpm -qpR xterm
      # query is xterm installed
      rpm -q xterm
      # every single file installed as part of xterm
      rpm -ql xterm
      #check recently altered packages
      rpm -qa --last
      # documentation
      rpm -qdf xterm
      # verified packages
      rpm -Va
      # query public keys
      rpm -qa gpg-pubkey*
      # a
      rpm
      • Fix db
        1
        2
        cd var/lib/rpm
        rpm --rebuilddb

Verify the integrity and availability

  • Resources (HD, RAM)
    • With live CD
      1
      2
      3
      4
      5
      6
      # list block devices
      lsblk
      # unmunt the device
      sudo umount /mnt
      # check and clean
      sudo fsck -y /dev/xvf1
    • cron it before running
      1
      2
      3
      # just add the file
      sudo touch fsck
      # after you reboot, it will be deleted
    • use memcheck from GRUB (hold [[Shift]])
  • Key processes
    • ps
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      # fix better on widescreen
      ps au
      # more info for a certain process
      ps aux | grep cron
      # list 
      ps ef
      # process for the root user
      ps -fu root
      # process for "user" group
      ps -fG user
      # show process tree
      ps -e --forest
    • top (live, updated)
      1
      2
      3
      4
      top
      # alternative with cleaner interface, has mouse support
      sudo apt install htop
      htop

Change kernel runtime parameters, persistent and non-persistent

  • parameters are files
    1
    2
    3
    4
    5
    6
    7
    8
    9
    # parameters are files here
    cd /proc/sys
    ls -la
    # is ip forward on?
    cat /proc/sys/net/ipv4/ip_forward
    # set the value as off
    sudo echo 0 > /proc/sys/net/ipv4/ip_forward
    # apply the changes you just did
    sysctl -p
  • system control
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    # show kernel runtime parameters
    sudo sysctl -a
    # everything is a file
    sudo sysctl -a | wc -l
    # is ip forward on?
    sysctl net.ipv4.ip_forward
    # set it as on
    sysctl -w net.ipv4.ip_forward=1
    # do what you just did, but with sudo
    sudo !!
    # apply the changes you just did
    sysctl -p
  • persist (normally they would be lost after reboot
    1
    2
    3
    cd /etc/sysctl.d
    nano 10-network-security.conf
    # add comment and set value 'net.ipv4.ip_forward=1'

Scripting

  • Shell scripting for system maintenance tasks

    1. create a shell script with nano test.sh 
      1
      2
      3
      4
      #!bin/bash
      echo "Line 1"
      echo "Line 2"
      ls -la
    2. check that you can execute it: chmod +x /test.sh
    3. launch test.sh
    4. you may add a directory to your path EXPORT PATH=$PATH:/home/user/scripts

  • Scripting conditionals and loops

    1
    2
    3
    4
    5
    6
    7
    8
    #!bin/bash
    DIRECTORY="/home/user/test"
    # conditional
    if [ -d $DIRECTORY ] ; then
      echo "the directory exists"
    else
      echo "the directory does not exist"
    fi
    1
    2
    3
    4
    5
    6
    #!bin/bash
    # loop for
    for COUNT in 1 2 3 4 5 6 7 8 9
    do
      echo "This is line # $COUNT"
    done  
    1
    2
    3
    4
    5
    6
    7
    8
    9
    #!bin/bash
    # supose a file 'myhosts' contains the following IPs
    #8.8.8.8
    #8.8.4.4

    # loop while
    while read $HOST; do
      echo "This is line # $COUNT"
    done < myhosts

Manage the startup process and services (in services configuration)

  • upstart (classic in Debian)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    cd /etc/init
    # know pid for process
    status cron
    # cron pid=1404
    ps aux | 1404
    # change status
    stop cron
    start cron
    restart cron
    # override
    ls -ls cron.conf
    echo manual | sudo tee /etc/init/cron.override
    ls -la cron*
    # undo
    cron.override
  • systemd (current in Ubuntu)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    cd /etc/systemd
    # check status
    systemctl status cron
    # requests authentication for changing status
    systemctl stop cron
    systemctl start cron
    # synch status server, remove softlink
    systemd-sysv-install disable cron
    # restart
    systemctrl enable cron

List and identify SELinux/AppArmor file and process contexts

  • SELinux
    1
    2
    3
    4
    5
    # all security context files on SELinux
    # grep it for a certain value
    sudo semanage fcontext -l
    # example of context
    cd /etc/rc.d
  • AppArmor
    1
    2
    3
    4
    # status
    sudo aa-status
    # check profiles on directory
    cd /etc/apparmor.d
  • Common tools
    1
    2
    3
    4
    # show security context for files
    ls -Z
    # processes with security context
    ps auxZ

Identify the component of a Linux distribution that a file belongs to

  • RedHat
    • rpm
      1
      rpm -qf /bin/znew
    • yum
      1
      yum whatprovides /bin/znew
  • Debian
    • dpkg
      1
      2
      3
      dpkg -S /usr/bin/zdump
      # all files installed by a particular package
      dpkg -L libc-bin

In

Linux Foundation Certified SysAdmin (LFCS): Essential commands

Log into local and remote consoles

  • Connect via SSH (text mode)

    1
    2
    3
    4
    5
    # no userName takes the one you are logged with on your machine
    # machine can be an IP address or a domain
    ssh username@machine
    # ECDSA key is added the first time
    # the  insert password

  • Connect via VNC (grahical mode - VNC viewer)

    • Give the network address
    • Check ECDSA key (fingerprint) to connect
    • Add username and password

Warning!
The server we want to connect to must have xwindows installed, plus VNC server

Search for files

Linux is case sensitive

  • Find

    • Basic
      1
      2
      3
      4
      5
      6
      7
      touch etc/test.txt
      # find on current directory
      find -name "test.txt"
      # check what is on our current directory
      ls -la
      # find starting from root
      find / -name "test.txt"
    • Advanced
      • by name
        1
        2
        3
        4
        5
        6
        # find starting from etc
        find /etc -name "test.txt"
        # -i to ignore case sensitive
        find /etc -iname "test.txt"
        # reverse search
        find /etc -not -name "test.txt"
      • by type
        1
        2
        3
        4
        5
        6
        7
        8
        # find for type (e.g. character devices)
        find /etc -type c
        # find symbolic links
        find /etc -type l
        # find symbolic directories
        find /etc -type d
        # find files whose name ends in ''.log'
        find /etc -type -f "*.log"
      • by size
        1
        2
        # files whose size is greater than 27K
        find /usr/bin -size +27000
      • by creation date
        1
        2
        3
        4
        # created more than 1 day ago
        find /usr/bin -type f -mtime 1
        # created less than 1 day ago
        find /usr/bin -type f -mtime -1
      • by user creation date
        1
        2
        # owned by myUser, pipe it through more to pause on each screen
        find /usr/bin -user myUser | more
      • by permision
        1
        2
        3
        4
        5
        # with permissions 755
        find /usr/bin -perm 755
        # you may do things which what you find on the same line
        # '{} \;' are placeholders
        find /usr/bin -name "test.txt" -exec chmod 700 {} \;

  • Check petitions

    1
    2
    3
    4
    # df stands for disk-free
    # - h stands for human readable
    df -h
    # root, mounted devices, temporary file systems...

  • Find executable in our path

    1
    2
    which python
    which ifconfig

  • Manual

    1
    man find

  • Locate

    1
    2
    3
    # Less powerful, depends on database which should be updated
    updatedb
    locate test.txt

Basic file system features and options

  • Definitions
    • block device: set of addressable blocks used to store and retrieve data
    • filesystem: where a computer system persists general data for users and/or applications

Filesystems can affect

  • Performance pf the system
  • Efficiency of the media
  • Compatibility with other systems
  • Jounaling
    • journal prevent data corruption, records metadata
    • add a bit of overhead to write files
    • some high performance servers might not need it
    • often not used on removable media
  • Current standard
    • EXT
      • EXT (EXTended File System)
      • EXT2 (extended file attributes)
      • EXT3 (journaling)
      • EXT4(add backward compatibility)
  • Coming standard
    • BtrFS (substitute to EXT by Oracle)
      • B-tree File system
      • drive pooling, snapshots, compression, online defragmentation
  • Common alternatives
    • ReiserFS
      • New features that wouldn’t be implemented by EXT (efficiencies for small text files)
      • Unlikely to continue development
    • ZFS (Sun, for Solaris, acquired by Oracle - opensourced not compatible with Linux kernel license)
      • drive pooling, snapshits, dynamic disk striping
      • each file has a checksum
      • (Ubuntu 16.04 supports it) uses containers by default
    • XFS
      • similar to Ext4
      • can be enlarged (but not shrunk) on the fly (good for backup servers)
      • bad for many small files (like web servers)
    • JFS (Journaled, by IBM)
      • low CPU usage
      • partitions can be enlarged (but not shrunk)
      • support in most every major distribution
      • not widely tested
  • Crossplatform
    • FAT (FAT16, FAT32, exFAT)
      • Microsoft File Allocation Table, not journaled
      • USB drives on Windows and Apple hardware
  • Swap space
    • virtual memory ()scratch space)
    • hibernating (analogous to Windows Paging File)

Compare and manipulate file content and I/O redirection

  • List all files
    1
    2
    # list files
    ls -la
  • Read (send to default output) and concatenate
    1
    2
    3
    4
    5
    6
    7
    8
    # catenate
    cat shopingListfile.txt
    # catenate various files as a single file
    cat shopingListfile.txt anotherList.txt
    # catenate, pipe it to paginate it with 'more'
    cat shopingListfile.txt | more
    # pip it to navegate using the up and down arrow keys
    cat shopingListfile.txt | less
  • Sort
    1
    2
    3
    4
    # sort the text content in alphabetical order
    sort shopingListfile.txt | more
    # sort the text content in reverse alphabetical order
    sort -r shopingListfile.txt | more
  • Create a new output file with >
    1
    cat shopingListfile.txt anotherList.txt | sort > combined.txt
  • Format (check man)
    1
    2
    # fix spacing issues on a file
    fmt -u format.txt
  • Numbered lines
    1
    2
    # If it si not redirected to other file, the numbers won't be stored
    nl format.txt > numberedFormat.txt
  • Remove delimiters
    1
    2
    3
    # -d means delimited
    # -f1 means first part
    cut -d ";" -f1 delimited.txt
  • Redirection
    1
    2
    3
    4
    # store on a new file, or overwrite if it already exists
    ls -la > directory.txt
    # append to that file
    ls -ls >> directory.txt

Basic Regular Expressions

  • Anchor characters (^, $)

    1
    2
    3
    4
    # starts the line with A
    ^A
    # ends the line with Z
    Z$

  • Selectors

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    # Find all the lines that start with 'The'
    grep '^The' alice.txt
    # Find line with a single character - '.' means any character
    grep '^.$' alice.txt
    # Find line which starts with a character
    grep '^[0123456789]' alice.txt
    # Find line which starts with a character, short version
    grep '^[0-9]' alice.txt
    # With '*' A character appear 0 or more times (e.g at least has 2 digits)
    grep '^[0-9]*[0-9][0-9]' alice.txt
    # Between 4 and 8 lowercase characters
    grep '[a-z]\{4,8\}' alice.txt
    # Search for a word with either capital or lowercase letter
    grep '\<[tT]he>' alice.text
    # back-reference: any 2 lowerkey letters adjoined (e.g. aa, baac)
    # [a-z][a-z] won't work for that pairs
    grep '\([a-z]\)\1' alice.txt

Archive, backup, compress, unpack, and decompress files

  • archive and backup (tar)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    cd /home/user
    #backup on dir
    # create dir
    mkdir datab_kup
    cd data_bakup
    # recursive copy of the '/data' folder
    cp -r /data .
    # undo hat was done previously
    rm *
    rm -rf *
    # backup the 'data' folder with tar (from 'tape archives')
    # concatenate, no compression
    # -c create new file
    # -v for verbose
    # -f for file
    tar cvf databkup.tar data
    # -t lists content of tar file
    tar tvf databkup.tar
    # search files on tar for a particular the text `wpa_supplicant`
    tar tvf databkup.tar | grep wpa_supplicant
  • compress (zip)
    1
    gzip databkup.tar
  • archive plus compress
    1
    2
    # add the 'z' directive
    tar cvfz databkup.tar.gz /data
  • extract or unzip/decompress
    1
    2
    # extract everything on a new directory
    tar xvfz databkup.tar.gz

Create, delete, copy, and move files and dirs

  • create
    1
    2
    3
    4
    5
    6
    7
    8
    9
    # file
    touch testFile
    # edit the file with pico or nano
    nano testFile
    # directory
    mkdir secondFiles
    touch testFileDir
    # create several directories (one inside the other)
    mk -p secondFiles/textFiles/otherFiles
  • remove
    1
    2
    3
    4
    5
    6
    # remove file
    rm testFile
    # remove directory
    rmdir secondFiles
    #remove directory tree, recursive, force
    rm -rf textFiles
  • copy 
    1
    2
    # copy to testfile2
    cp testFile testFile2
  • move and rename
    1
    2
    3
    4
    # move file
    mv testfile2 secondFiles/
    # rename file via mv
    mv testFile2 testFile
  • Hard link: links the original data chunk on disk.Hard links cannot traverse filesystems (physical or shared)
    1
    2
    3
    # create a hard link to a file
    ln ../dir2/file2 hardLinkToFile2
    # deleting the original link doesn't lose the data
  • Soft (symbolic) link: links the original link 
    1
    2
    3
    4
    # create a soft link to a file
    ln -s ../dir2/file2 softLinkToFile2
    # on 'ls' it is represented with an arrow ->
    # deleting the original link loses the data, as a direct access

List, Set, and change standard file permissions

  • Get the list

    1
    2
    # list files
    ls -la

  • Output

    1
    2
    3
    4
    5
    total 8
    drwxr-xr-x  2 user1  user    4096  Feb 28 14:26  .
    -rw-r--r--  1 user1  dbadmin  0     Feb 28 14:26  user1andgroup
    -rw-r--r--  1 user1  user1    0     Feb 28 14:26  user1only
    -rw-r--r--  1 user1  user1    0     Feb 28 14:26  user1yone

    Sets meaning:

    1. directory (‘d’ or ‘-‘’)
    2. ownerPermission-groupPermission-otherPeoplePermission (‘r’, ‘w’, ‘x’ or ‘-‘)
    3. Owner

  • Change permissions

    1
    2
    3
    4
    5
    6
    7
    8
    # grant permissions for user chad, to grant
    chmod u+rwx user1only
    # take away group permissions
    chmod g-rwx user1only
    # take away other users permissions
    chmod o-rwx user1only
    # grant permissions to everyone else
    chmod a+rwx user1only

Permissions via octal numbers

  • 4: read
  • 2: write
  • 1: exec
    You may do all 3 commands in a single one with the octal numbers,
    so to grant all permissions for user, group and other would be 4+2+1=7 -> chmod 777 user1Only
  • Change ownership of files 
    1
    2
    3
    4
    5
    # find files from lostUser
    sudo find / -user lostUser
    # supose the results says they are on '/opt/myapp'
    # change ownership to cloud_user from files on a certain folder
    sudo chown -R cloud_user:cloud_user /opt/myapp

Read and use system documentation

  • Manual
    1
    2
    3
    4
    # manual + command we want to know about, q to exit
    man ls
    # update the man db
    mandb
  • apropos
    1
    2
    # commands related to what you asked about
    apropos ps
  • info
    1
    2
    # similar to man
    info ipc
  • locate
    1
    2
    # its db is updated by a cronjob once a day
    info systemctl.conf
  • the documentation folder: user/share/doc

Manage access to the Root account

  • Get superuser permission
    1
    sudo ls
  • Grant sudo permission
    1
    2
    3
    4
    sudo visudo
    # use either vim or nano
    # find part to grant and add
    # chad ALL=(ALL)  ALL
  • Be added to the WHEEL (CentOS) or SUDO (Ubuntu) group
    1
    2
    3
    4
    5
    6
    7
    8
    # show groups for user1
    groups user1
    # add user1 to group WHEEL
    # -a is important: it adds, if not, it will switch group
    usermod -a -G wheel user1
    # if it fails, do "sudo bang-bang"
    sudo!!
    sudo usermod -a -G wheel user1

Securely copy a file to another server

1
2
# scp file_name userId@destinyServer:destinyFolder
scp myapi.tar.gz cloud_user@SERVER_TWO_PUBLIC_IP:/home/cloud_user/myapi.tar.gz

In

Advantages over bash

  • Libraries: system operations, reading files, listing directories, writing for loops, checking for exit codes…
  • Autocomplete with IDEs
  • Robust testing suite
  • iPython console
  • Python / Miniconda is available on most systems
  • Robust error checking with try and catch blocks
  • Python libraries deal with OS peculiarities under the hood, so it may run everywhere

Installation

  • Check if you have iPython installed

    1
    2
    which python
    which ipython

  • Install Miniconda with Python 3 and update

    1
    2
    3
    4
    5
    bash Miniconda3-latest-Linux-x86_64.sh
    conda update conda
    conda config --add channels conda-forge
    conda update -y --all
    conda install -y ipython

❗ Known issues installing

  • Avoid using the global area: use environments
  • Clean up everything in case of wrong installation 
    1
    2
    conda clean --all
    conda update -y --all

  • Create Environments with Conda

    1
    2
    # Different projects should have their own isolated software environments.
    conda create -n my-project ipython package1 package2 package2

Python Libraries for System Administration

Handy Packages

  • os package. You can use it to list directories, check if files exist, check if symlinks exist, make directories, run system commands, get and set environmental variables, and more. It’s great!
  • subprocess module
  • shutil
  • pprint
  • pytest

Running code

  • Basic notions

    • Run script

      1
      python name-of-script.py

    • Help

      1
      help(os)

    • Import packages

      1
      2
      3
      4
      import os
      import subprocess
      import shutil
      from pprint import pprint

  • Common file and directory operations

    • Working directory

      1
      2
      3
      4
      # Get your current working directly
      # This returns a string
      my_cwd = os.getcwd()
      print(my_cwd)

    • List of directory contents 

      1
      2
      3
      4
      5
      # List the contents of a directory
      # This returns a list
      dir_list = os.listdir()
      for item in dir_list:
        print(item)

    • Absolute path name of file

      1
      2
      # Get the Absolute Path name of a file (file + current working dir)
      os.path.abspath('some-file')

    • File basename

      1
      2
      #Get the basename - returns file
      os.path.basename('/path/to/file')

    • Split directory path

      1
      2
      3
      # Split a directory path - platform independent
      os.path.split(os.getcwd())
      # Out[17]: ('/Users', 'thali')

    • Check path existence

      1
      2
      3
      4
      # Check if a path exists
      os.path.exists('/path/on/filesystem')
      # Check if a path is a symlink
      os.path.islink()

  • Move files and directories

    • Copy directory

      1
      2
      3
      # Copy a directory
      # cp -rf
      shutil.copytree('src', 'dest')

    • Copy file

      1
      2
      3
      # Copy a file
      # cp -rf
      shutil.copyfile('file1', 'file2')

    • Move directory

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
        # Move a directory
        # mv
        shutil.move('src', 'dest')
        ````

      >❗ Not everything is going to be available through python libraries, such as installing system libraries
          ```python
          # Run an arbitrary system command
          command = "echo 'hello'"
          result = subprocess.run(command.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
          #Print the stdout and stderr
          print(result.stdout)
          print(result.stderr)

  • Write to files

    • Create a new file

      1
      2
      3
      4
      5
      # Write to a file (and create it if it doesn't exist)
      # echo "hello" > hello.txt
      f= open("hello.txt","w+")
      f.write("hello!")
      f.close()

    • Append to file

      1
      2
      3
      4
      5
      # Append to a file
      # echo "hello" >> hello.txt
      f = open("hello.txt", "a+")
      f.write("hello again!")
      f.close()

Tests

Tests mostly work by using a function called assert, which is essentially saying “make sure this is true and if not die loudly”.
Put this function in a file called test_my_code.py and run as pytest test_my_code.py.

1
2
3
4
5
def test_system_command():
    """Test the exit code of a system command"""
    command = "echo 'hello'"
    result = subprocess.run(command.split(' '), stdout=subprocess.PIPE)
    assert result.returncode == 0
0%