Learning Power Tools for Linux/UNIX/Mac

Course Handout: (last update 19 October 2011)


These notes may be found at http://caligari.dartmouth.edu/~rc/classes/unix2. The online version has many links to additional information and may be more up to date than the printed notes

Learning Power Tools for Linux/UNIX/Mac

This course will teach participants some of the powerful tools available to Linux/UNIX/Mac users. Users should have taken or be familiar with all the content of the Introduction to Linux/UNIX class. Topics covered will include shell shortcuts, common tools and the UNIX/Linux file system.

Course Goals

Example commands are shown like this
Many commands are shown with links to the man pages (ls)
Output from commands is shown like this
Optional items are shown in brackets, [ like this ]

Some descriptions in these notes have more detail available, and are denoted like this:

More details of this item would appear here. The printed notes include all of the additional information




Those who do not understand Unix are condemned to reinvent it, poorly.

Usenet signature, 1987
-Henry Spencer

Best viewed with ANY browser Powered by Vi

Table of Contents

1.Dennis M. Richie: 1941-2011
2.UNIX File System
3.The User Environment
4.Logging In
5.UNIX Shells
6.Shells cont.
7.Customizing Login Files
8.Filename Wildcards
9.Redirection and Pipes
10.File Permissions
11.Job Control
12.File Utilities
13.Other Useful Commands
14.Software Development
15.Make
16.AFS filesystem
17.Conclusion
18.References

(1)

Dennis M. Richie

1941-2011

Dennis M. Richie, father of C and Unix

(2)

The UNIX File System

(3)

The User Environment

(4)

Logging in and Initialization Files

(5)

UNIX Shells

Interpreting the command line

(where do the spaces go ?)

These comments apply to all the common shells. Exact syntax of internal shell commands differ between shells. Command lines given to a shell for processing (typed at prompt, or out of a shell script file) are interpreted as follows (somewhat simplified): (more on this in the shell programming class)
  1. Separate words by whitespace (space, tab). Quotes override this.
  2. Expand shell aliases (csh/tcsh/ksh only)
  3. Expand shell and environment variables ($VAR, e.g. $HOME, $PRINTER
  4. Expand filename wildcards *?[] into a sorted list of matching filenames (pathnames).
  5. Look for I/O redirection and open the named files for input/output ( >> outfile ). Remove these items - the program does not see this information
  6. Run any commands in "`" and substitute the output. (echo `date`)
  7. Evaluate shell internal commands (e.g. echo, cd)
  8. If not an internal command, the first word on the line is assumed to be an external command, and is search for by examining the $PATH variable left to right. The first match is what gets executed (which ls)
  9. Anything else left (typically option flags and filenames) are passed to the command as arguments.

    Most commands take optional flags introduced by a "-". How the arguments are interpreted is up to each individual program, but most programs accept filename arguments in a predictable way, or attempt to do something natural with standard input if no filename arguments are present.
    Example:
    ls
    ls *
    setenv LSFLAGS -lF
    ls $LSFLAGS

    (6)

    How shells help

    (7)

    Customizing Login Files

    (8)

    Filename Wildcards

    (9)

    File Redirection and Pipes

    (10)

    File Permissions

    (11)

    Job/Process Control

    ^Z
    Suspend current job
    bg
    Put suspended job in background
    fg
    Make background job run in the shell
    kill #
    Kills a job with given process id, (ps -elf | grep unix10 | more)
    renice
    Lets a job run at a lower priority level
    at time
    Runs a job at a given time, UNIX also has "cron" jobs which run on a regular schedule

    (12)

    File Utilities

    Listing, creating, deleting
    ls
    list file names (Ex.)
    rm
    remove files (Ex.)
    mkdir
    make a directory (Ex.)
    rmdir
    remove a directory (if empty) (Ex.)
    pwd
    print working directory
    chmod
    change mode (attributes and permissions)
    chgrp
    change group
    ln
    create links
    find
    search directories for files matching criteria
    quota
    check disk quota
    cp
    copy files and directories (Ex.)
    mv
    move files and directories (Ex.)
    Display, Viewing
    more,less,pg
    view a view one screen at a time, interactive browsing (Ex.)
    view
    browse a file (vi in readonly mode)
    cat
    send a file verbatim to standard output, concatenate files (Ex.)
    head
    show the start of a file (Ex.)
    tail (Ex.)
    show the end of a file
    Filters
    grep
    search files for patterns (Global Regular Expression and Print) (Ex.)
    sed
    stream editor
    awk
    extract patterns, perform processing - simple scripting language
    tr
    translate characters, simple editing (Ex.)
    sort
    sort alaphabetically or numerically (Ex.)
    uniq
    remove duplicates (Ex.)
    cut
    extract fields (columns)
    paste
    join multiple files by columns
    diff
    compare files, get differences (Ex.)
    compress,uncompress,gzip,gunzip
    File compression (Ex.)
    uuencode,uudecode
    encodes binary files so you can send via mail
    strings
    search arbitrary files for printable strings of characters (Ex.)
    od
    octal dump; show exactly what is in a file (debugging) (Ex.)

    (13)

    Other Useful Commands

    Process control

    xload
    Shows machine load (# of ready-to-run processes)
    ps
    list processes
    top
    show top processes
    kill
    send a signal to a process
    nice
    start a process at low priority

    People

    who
    show who is logged in
    finger
    show information about a user

    Mail

    mail, Mail
    basic Unix mailers
    pine
    screen oriented menu-driven mail interface
    thunderbird
    graphical IMAP client from Mozilla Foundation

    Printing

    lpr
    send a file to a printer
    lpq
    check spool queue status
    lprm
    remove a job from the print queue
    pr
    format for printing (text)
    enscript
    format text and images for postscript printing

    Network

    firefox
    graphical web browser from Mozilla Foundation
    sftp,ftp
    transfer files. sftp is encrypted, built on ssh. ftp is plain text and no longer recommended
    ssh,telnet,rlogin
    remote login. ssh is encrypted. telnet is plain text
    talk
    interactive chat
    irc
    internet relay chat -- multiuser talk

    Text processing

    nedit,emacs
    Text editors using X windows
    vi,emacs,joe,pico
    editors using text-only
    nroff
    text formatter
    TeX,LaTeX
    typesetting
    aspell
    spell checker

    Image processing and manipulation, Graphics

    xv
    X image processing
    GIMP
    X image processing
    Photoshop
    X image processing
    pbmplus package
    conversion utilties
    gnuplot
    simple graphics, X or many other (Tek)

    (14)

    Software Development tools on Unix

    Editors

    vi
    The standard text-mode, full-screen editor
    emacs
    Text mode and X windows, very configurable, power user. GNU
    vim, joe, pico
    Alternate editors, text mode. Pico is very simple. Vim is vi compatible but has many extensions
    nedit
    An X-windows only editor. Powerful, and possibly easier for users accustomed to Mac/Windows.

    Compiled languages

    cc, c++, javac
    Found on almost all Unix systems.
    f77 FORTRAN 77,f90 FORTRAN 90, BASIC, Cobol, Pascal
    Available for most versions of Unix.

    Scripting languages

    perl, python, tcl/tk

    Debuggers

    gdb, adb

    Configuration management

    make, pmake, gmake, configure, autoconf

    Revision Control

    rcs
    Revision Control System
    CVS
    Code Version System
    svn
    Subversion
    git
    Distributed version control for large projects

    (15)

    The Make Utility

    make
    A tool for managing collections of files with dependancies (typically program source code).

    Make gets its knowledge of how to build your program from a file called makefile (or Makefile). Make is traditionally used for preprocessing, compiling and linking of source code to build programs, but can also be used for other purposes. It streamlines the process of compiling complex programs by automatically determining which source files of a program need to be recompiled and/or linked. Only the files which are out of date with respect to their dependants are rebuilt.

    A makefile consists of rules with the following format:

    target: dependencies ...
         commands
         ...
    
    By examining the timestamps on the target file and the dependencies, make determines if the target is out of date, in which case the commands (rule) for this target are executed. Dependency files can themselves be targets, and so a dependency tree can be constructed. Default rules are supplied for commonly used operations. A null dependancy set means the target is always out of date, and the rule is always executed.

    A Simple Example of a Makefile

    
    myappl: srcfile1.o srcfile2.o srcfile3.o
        cc -o myapp1 srcfile1.o srcfile2.o srcfile3.o -lm
    srcfile1.o: scrfile1.c myinclude.h
        cc -c srcfile1.c 
    srcfile2.o: scrfile2.c myinclude.h
        cc -c srcfile2.c 
    srcfile3.o: scrfile3.c myinclude.h
        cc -c srcfile3.c 
    clean:
        rm -f myapp1 srcfile1.o srcfile2.o srcfile3.o
    

    Note: You need to put a tab character at the beginning of each command line.

    It is usually easiest to start with an existing makefile and edit it for a new application. Using the name Makefile is preferred - it appears near the top of the directory listing.

    Using Macros (Variables) in Makefiles

    Macros make it easier to define commands and sets of dependancies, and simplify your makefile

    Here is an example macro defintion:

    OBJS=srcfile1.o srcfile2.o srcfile3.o

    To use this macro type $(OBJS)

    Here is a new version of the makefile

    OBJS=srcfile1.o srcfile2.o srcfile3.o
    
    myappl: $(OBJS)
        cc -o myapp1 $(OBJS) -lm
    srcfile1.o: scrfile1.c myinclude.h
        cc -c srcfile1.c 
    srcfile2.o: scrfile2.c myinclude.h
        cc -c srcfile2.c 
    srcfile3.o: scrfile3.c myinclude.h
        cc -c srcfile3.c 
    clean:
        rm -f myapp1 $(OBJS)
    

    Predefined macros (make -p)

    $(CC) - C compiler command (cc)
    $(FC) - Fortran compiler command (f77)
    $(CFLAGS) - C compiler flags
    $(FFLAGS) - Fortran compiler flags

    You can add macros to the command line : make myapp1 CC=gcc

    Suffixes

    Make has many default rules for creating one type of file from another. There are additional built-in macros to make this easier. ($< stands for the file that triggered the rule, i.e. the .c or .f file). The mechanism uses the filename suffix convention used by most compilers.
    For example, to compile a C source code file prog.c and create the object code file prog.o, we need to construct the command:
    cc -c prog.c
    The built-in rule which does this is:

    .SUFFIXES: .c
    .c.o:
        $(CC) $(CFLAGS) -c $<
    .f.o:
        $(FC) (FFLAGS) -c $<
    

    You can add your own default rules to process files in any way that uses the suffix convention, simplifying the explicit commands in the Makefile. Use the command make -p to see a list of all predefined suffixes and macros.

    (16)

    The AFS filesystem

    (17)

    The end

    (18)

    References, Resources, Man pages etc.

    The following is a sampling of the many available books on the subject.
    The links are to publishers web sites, or Amazon.com. Some of the best UNIX books are published/written by O'Reilly & Associates


    Learning Power Tools for Linux/UNIX/Mac: Course Handout
    (last update   19 October 2011)  ©Dartmouth College     http://caligari.dartmouth.edu/~rc/classes/unix2