Shell Programming

Understanding Bash: Elements of Programming

Vladimir Likic — Fri, 28 Sep 2018 14:22:26 +0000

Ever wondered why programming in Bash is so difficult? Bash employs the same constructs as traditional programming languages; however, under the hood, the logic is rather different.

The Bourne-Again SHell (Bash) was developed by the Free Software Foundation (FSF) under the GNU Project, which gives it a somewhat special reputation within the Open Source community. Today, Bash is the default user shell on most Linux installations. Although Bash is just one of several well known UNIX shells, its wide distribution with Linux makes it an important tool to know.

The main purpose of a UNIX shell is to allow users to interact effectively with the system through the command line. A common shell action is to invoke an executable, which in turn causes the kernel to create a new running process. Shells have mechanisms to send the output of one program as input into another and facilities to interact with the filesystem. For example, a user can traverse the filesystem or direct the output of a program to a file.

Although Bash is primarily a command interpreter, it's also a programming language. Bash supports variables, functions and has control flow constructs, such as conditional statements and loops. However, all of this comes with some unusual quirks. This is because Bash attempts to fulfill two roles at the same time: to be a command interpreter and a programming language—and there is tension between the two.

All UNIX shells, including Bash, are primarily command interpreters. This trait has a deep history, stretching all the way to the very first shell and the first UNIX system. Over time, UNIX shells acquired the programming capabilities by evolution, and this has led to some unusual solutions for the programming environment. As many people come to Bash already having some background in traditional programming languages, the unusual perspective that Bash takes with programming constructs is a source of much confusion, as evidenced by many questions posted on Bash forums.

In this article, I discuss how programming constructs in Bash differ from traditional programming languages. For a true understanding of Bash, it's useful to understand how UNIX shells evolved, so I first review the relevant history, and then introduce several Bash features. The majority of this article shows how the unusual aspects of Bash programming originate from the need to blend the command interpreter function seamlessly with the capabilities of a programming language.

Go to Full Article

Cribbage: Sorting Your Hand

Dave Taylor — Mon, 24 Jun 2013 17:57:45 +0000

by Dave Taylor

We've been working on writing code for the game Cribbage, and last time, I created the code needed to pick a random subset of six cards out of a "deck" and display them in an attractive format—like this:


$ sh cribbage.sh
Card 0: 7C
Card 1: 5H
Card 2: 9H
Card 3: 10S
Card 4: 5D
Card 5: AS

The primary task on the agenda this article is to sort the cards after they've been dealt. This means we're going to have to sort the cards by rank while ignoring the suit, then slot them back into the "hand" array. Is there an easy way to do that? Actually, we'll use the sort function.

We can prototype this by using the command line to see what result we get:


$ sh cribbage.sh  | sort -n
Card 0: 4S
Card 1: 7C
Card 2: 9S
Card 3: JC
Card 4: 7H
Card 5: 8C

What the heck? Oh! You can see the problem, right? By telling sort to order things numerically, it properly ignores "Card" but then sees the ordinal value of the card and sorts based on that, rather than on the actual card value itself.

Even if we fix this, however, we still have the problem that face cards will sort before numeric value cards, which isn't what we want. In fact, we want aces to sort as lower than 2s, while jacks, queens and kings sort as higher than 10s.

If you wanted to have aces "high", the easiest way to do that would be to change the display routine, of course: 1 = a deuce, 2 = a three, 12 = king and 13 = ace. Poof. Everything sorts ace-high. That's just not how Cribbage scores them.

To accomplish Cribbage-rank sorting, we'll need to change the output to push out two values: the rank and the total card value. It's going to look ugly, but it's just an interim result.

Here's how I tweak the code to display these values:


showcard()
{
  # given a card value of 0..51 show the suit and rank
  suit=$(( $1 / 13 ))
  rank=$(( ( $1 % 13 ) + 1 ))
  case $rank in
    1)  orank="A" ;;
    11) orank="J" ;;
    12) orank="Q" ;;
    13) orank="K" ;;
     *) orank=$rank ;;
  esac
  showcardvalue=$orank${suits[$suit]}
}

If you compare it to the version we built last month, the main difference is that instead of calculating the rank of the card and then overwriting it with "A", "J", "Q" or "K" as appropriate, we're using a new variable, orank, to store the corrected value. Why? Because now in the main section of the script we also can access the $rank of the card as desired:


showcard ${hand[$card]}
echo "$rank ${hand[$card]}"

For each card chosen, the script has an interim output of rank followed by the numeric value of the card, with no fancy display (even though we're still tapping the showcard function for simplicity). The result:

Go to Full Article

Counting Cards: Cribbage

Dave Taylor — Tue, 28 May 2013 17:43:24 +0000

by Dave Taylor

I've spent the past few months reviewing shell scripting basics, so I think it's time to get back into an interesting project. It's always a good challenge to capture game logic in a shell script, particularly because we're often pushing the envelope with the capabilities of the Bash shell.

For this new project, let's model how a deck of cards works in a script, developing specific functions as we proceed. The game that we'll start with is a two- or three-player card game called Cribbage. The basic functions we'll create also will be easily extended to simple Poker variants and other multi-card evaluation problems.

If you aren't familiar with Cribbage, you've got time to learn more about the game, because I won't actually get to any game-specific elements until next month. Need a good place to learn? Try this: http://www.bicyclecards.com/card-games/rule/cribbage.

The first and most obvious challenge with any card game is modeling the deck of cards. It's not just the deck, however, it's the challenge of shuffling too. Do you need to go through the deck multiple times to randomize the results? Fortunately, that isn't necessary, because you can create a deck—as an array of integer values—in sequential order and randomly pick cards from the deck instead of worrying about shuffling the deck and picking them in sequential order.

This is really all about arrays, and in a shell script, arrays are easy to work with: simply specify the needed index in the array, and it'll be allocated so that it's a valid slot. For example, I simply could use deck[52]=1, and the deck array will have slots 0..52 created (though all the other elements will have undefined values).

Creating the ordered deck of cards, therefore, is really easy:


for i in {0..51}
do
  deck[$i]=$i
done

Since we're going to use the value -1 to indicate that the card has been pulled out of the deck, this would work just as well if everything were set to any value other than -1, but I like the symmetry of deck[$i]=$i.

Notice also the advanced for loop we're employing. Early versions of Bash can't work with the {x..y} notation, so if that fails, we'll need to increment the variable by hand. It's not a big hassle, but hopefully this'll work fine.

To pick a card, let's tap into the magic $RANDOM variable, a variable that has a different value each time you reference it—darn handy, really.

So, picking a card randomly from the deck is as easy as:

Go to Full Article

Working with Stdin and Stdout

Dave Taylor — Mon, 29 Apr 2013 17:33:58 +0000

by Dave Taylor

Previously, I erroneously titled my column as "SIGALRM Timers and Stdin Analysis". It turned out that by the time I'd finished writing it, I had spent a lot of time talking about SIGALRM and how to set up timers to avoid scripts that hang forever, but I never actually got to the topic of stdin analysis. Oops.

So this time, let's start with that topic. The behavior to emulate here is something a lot of utilities do without you paying much attention: they behave differently if their input or output is a pipe or file than they do when it's stdin (the keyboard) or stdout (the screen). Try ls versus ls|cat to see what I mean.

The test command has a helpful flag in this regard: -t. From the man page:


True if the file whose file descriptor number is
file_descriptor is open and is associated with a terminal.

Worth knowing is that file descriptor #0 is stdin; #1 is stdout, and #2 is stderr (pronounced "standard in", "standard out" and "standard error", respectively). That's why using >& to redirect by file descriptors works with 2>&1 to cause error messages to go to stdout just like regular output messages.

Back to the topic though—in practice, the -t test can be used like this:


#!/bin/sh
if [ -t 0 ]; then
  echo script running interactively
else
  echo stdin coming from a pipe or file
fi

It's easy to test:


$ sh inter.sh
script running interactively
$ sh inter.sh < inter.sh
stdin coming from a pipe or file
$ cat inter.sh | sh inter.sh
stdin coming from a pipe or file

Perfect. Now, what about identifying if the output is an interactive terminal, file or pipe? It turns out that you can use the same basic test, just replace the file ID 0 with #1:


if [ -t 1 ] ; then
  echo output going to the screen
else
  echo output redirected to a file or pipe
fi

The results:


$ sh inter.sh
script running interactively
output going to the screen
$ sh inter.sh | cat
script running interactively
output redirected to a file or pipe
$ sh inter.sh > output.txt
$ cat output.txt
script running interactively
output redirected to a file or pipe

Pretty cool, actually.

Let's back up a bit and have another look at file redirection before leaving this topic, however.

I already talked about the common trick of 2>&1 to redirect stderr to stdout—something that's very helpful on the command line. You also can redirect specific lines of output in a shell script to stderr, so your error messages are sent to the screen even if stdout is being sent to a pipe or file:

Go to Full Article

SIGALRM Timers and Stdin Analysis

Dave Taylor — Wed, 20 Mar 2013 18:19:31 +0000

by Dave Taylor

It's not hard to create functions to ensure that your script doesn't run forever. But what if you want portions to be timed while others can take as long as they need? Not so fast, Dave explains in his latest Work the Shell.

In an earlier article, I started building out a skeleton script that would have the basic functions needed for any decent shell script you might want to create. I started with command-line argument processing with getopts, then explored syslog and status logging as scripts. Finally, I ended that column by talking about how to capture signals like Ctrl-C and invoke functions that can clean up temp files and so on before actually giving up control of your shell script.

This time, I want to explore a different facet of signal management in a shell script: having built-in timers that let you specify an allowable quantum of time for a specific function or command to complete with explicit consequences if it hangs.

When does a command hang? Often when you're tapping into a network resource. For example, you might have a script that looks up definitions by handing a query to Google via curl. If everything's running fine, it'll complete in a second or two, and you're on your way.

But if the network's off-line or Google's having a problem or any of the million other reasons that a network query can fail, what happens to your script? Does it just hang forever, relying on the curl program to have its own timeout feature? That's not good.

Alarm Timers

One of the most common alarm timer approaches is to give the entire script a specific amount of time within which it has to finish by spawning a subshell that waits that quantum, then kills its parent. Yeah, kinda Oedipal, but at least we're not poking any eyes out in this script!

The additional lines end up looking like this:


(
sleep 600           # if 10 minutes pass
kill -TERM $$       # send it a SIGTERM signal
)&

There's no "trap" involved—easy enough. Notice especially that the closing parenthesis has a trailing ampersand to ensure that the subshell is pushed into the background and runs without blocking the parent script from proceeding.

A smarter, cleaner way to do this would be for the timer child subshell to send the appropriate SIGALRM signal to the parent—a small tweak:


(
sleep 600            # if 10 minutes pass
kill -ALRM $$        # send it a SIGALRM signal
)&

If you do that, however, what do you need in the parent script to capture the SIGALRM? Let's add that, and let's set up a few functions along the way to continue the theme of useful generic additions to your scripts:

Go to Full Article

Python Scripts as a Replacement for Bash Utility Scripts

Richard Delaney — Wed, 16 Jan 2013 17:22:20 +0000

by Richard Delaney

For Linux users, the command line is a celebrated part of our entire experience. Unlike other popular operating systems, where the command line is a scary proposition for all but the most experienced veterans, in the Linux community, command-line use is encouraged. Often the command line can provide a more elegant and efficient solution when compared to doing a similar task with a graphical user interface.

As the Linux community has grown up with a dependence on the command line, UNIX shells, such as bash and zsh, have grown into extremely formidable tools that complement the UNIX shell experience. With bash and other similar shells, a number of powerful features are available, such as piping, filename wild-carding and the ability to read commands from a file called a script.

Let's look at a real-world example to demonstrate the power of the command line. Every time users log in to a service, their user names are logged to a text file. For this example, let's find out how many unique users use the service.

The series of commands in the following example show the power of more complex utilities by chaining together smaller building blocks:


$ cat names.log | sort | uniq | wc -l

The pipe symbol (|) is used to pass the standard output of one command into the standard input of the next command. In the example here, the output of cat names.txt is passed into the sort command. The output of the sort command is each line of the file rearranged in alphabetical order. This subsequently is piped into the uniq command, which removes any duplicate names. Finally, the output of uniq is passed to the wc command. wc is a counting command, and with the -l flag set, it returns the number of lines. This allows you to chain a number of commands together.

However, sometimes what is needed can become quite complex, and chaining commands together can become unwieldy. In that case, shell scripts are the answer. A shell script is a list of commands that are read by the shell and executed in order. Shell scripts also support some programming language fundamentals, such as variables, flow control and data structures. Shell scripts can be very useful for batch jobs that will be run often and repeatedly. Unfortunately, shell scripts come with some disadvantages:

Shell scripts easily can become overly complicated and unreadable to a developer wanting to improve or maintain them.
Often the syntax and interpreter for these shell scripts can be awkward and unintuitive. The more awkward the syntax, the less readable it is for the developer who must work with these scripts.
The code is generally unusable in other scripts. Code reuse among scripts tends to be difficult, and scripts tend to be very specific to a certain problem.

Go to Full Article