Shell Scripting

Bash Shell Games: Let's Play Go Fish!

Dave Taylor — Tue, 30 Jul 2019 12:00:00 +0000

How to begin developing a computer version of the popular card game.

Between the previous 163 columns I've written here in Linux Journal and the dozens of games I programmed and explored during the creation of my Wicked Cool Shell Scripts book, I've written a lot of Bash shell games. The challenge is to find one that's simple enough where a shell script will work, but isn't so simple that it ends up being only a half-dozen lines.

Magic 8-Ball is a perfect example. It turns out that the entire "predict the future" gizmo was really just a 20-sided die floating in dark purple fluid. So an array of 20 possible values and a random number selector and boom—you've got a magic 8-ball script:


#!/bin/sh

# magic 8 ball. Yup. Pick a random number, output message

# messages harvested from the Wikipedia entry

answers=("It is certain." "It is decidedly so."
  "Without a doubt." "Yes - definitely."
  "You may rely on it." "As I see it, yes." "Most likely."
  "Outlook good." "Yes." "Signs point to yes."
  "Reply hazy, try again." "Ask again later."
  "Better not tell you now." "Cannot predict now."
  "Concentrate and ask again." "Don't count on it."
  "My reply is no." "My sources say no."
  "Outlook not so good." "Very doubtful.")

echo "Oh! Magic 8 Ball, Please Tell Me True..." ; echo ""
/bin/echo -n "What is your question? "
read question

answer=$(( $RANDOM % 20 ))

echo ""
echo "I have looked into the future and I say: "
echo "     ${answers[$answer]}" ; echo ""

exit 0

Let's do a quick run to see if I'm the most popular LJ writer:


$ sh magic8.sh
Oh! Magic 8 Ball, Please Tell Me True...

What is your question? Am I the most popular LJ writer?

I have looked into the future and I say:
     My reply is no.

Ouch, that's harsh. I write the darn divination program, and it just drops a brick on my foot. Yeesh.

More seriously, Magic 8 Ball is too simple to make an interesting shell script. By contrast, Call of Duty is way too complex, even if I did a version with text output instead of gorgeously rendered 3D graphics.

Card Game Function Library

That's why card games prove to be good as programming challenges or exercises: the core mechanism of a 52-card random deck is pretty straightforward, so it's all about the actual cardplay.

Not only that, but as I've written before about card games as shell scripts, I already have a handy set of functions to create, shuffle and display cards out of a deck. If you want to rummage in the archives, I've tackled Acey-Deucey, Baccarat and some bits and pieces of Cribbage.

In order to jump right into the new game that I'm going to describe how to build, Go Fish!, let's steal the following functions from my earlier scripts:

Go to Full Article

Finishing Up the Bash Mail Merge Script

Dave Taylor — Thu, 04 Jul 2019 12:00:00 +0000

by Dave Taylor

Finally, I'm going to finish the mail merge script, just in time for Replicant Day.

Remember the mail merge script I started writing a while back? Yeah, that was quite some time ago. I got sidetracked with the Linux Journal Anniversary special issue (see my article "Back in the Day: UNIX, Minix and Linux"), and then I spun off on a completely different tangent for my last article ("Breaking Up Apache Log Files for Analysis"). I blame it on...

SQUIRREL!

Oh, sorry, back to topic here. I was developing a shell script that would let you specify a text document with embedded field names that could be substituted iteratively across a file containing lots of field values.

Each field was denoted by #fieldname#, and I identified two categories of fieldnames: fixed and dynamic. A fixed value might be #name#, which would come directly out of the data file, while a dynamic value could be #date#, which would be the current date.

More interesting, I also proposed calculated values, specifically #suggested#, which would be a value calculated based on #donation#, and #date#, which would be replaced by the current date. The super-fancy version would have a simple language where you could define the relationship between variables, but let's get real. Mail merge. It's just mail merge.

Reading and Assigning Values

It turns out that the additions needed for this script aren't too difficult. The basic data file has comma-separated field names, then subsequent lines have the values associated with those fields.

Here's that core code:


if [ $lines -eq 1 ] ; then # field names
# grab variable names
declare -a varname=($f1 $f2 $f3 $f4 $f5 $f6 $f7)
else # process fields

# grab values for this line (can contain spaces)
declare -a value=("$f1" "$f2" "$f3" "$f4" "$f5" "$f6" "$f7")

The declare function turns out to be ideal for this, allowing you to create an array varname based on the contents of the first line, then keep replacing the values of the array value, so that varname[1] = value[1], and so on.

To add the additional variables #date# and #suggested#, you simply can append them to the varname and value arrays. The first one is easy, but it did highlight a weakness in the original code that I had to fix by adding quotes as shown:

Go to Full Article

Breaking Up Apache Log Files for Analysis

Dave Taylor — Mon, 27 May 2019 11:00:00 +0000

by Dave Taylor

Dave tackles analysis of the ugly Apache web server log.

I know, in my last article I promised I'd jump back into the mail merge program I started building a while back. Since I'm having some hiccups with my AskDaveTaylor.com web server, however, I'm going to claim editorial privilege and bump that yet again.

What I need to do is be able to process Apache log files and isolate specific problems and glitches that are being encountered—a perfect use for a shell script. In fact, I have a script of this nature that offers basic analytics in my book Wicked Cool Shell Scripts from O'Reilly, but this is a bit more specific.

Oh Those Ugly Log Files

To start, let's take a glance at a few lines out of the latest log file for the site:


$ head sslaccesslog_askdavetaylor.com_3_8_2019
18.144.59.52 - - [08/Mar/2019:06:10:09 -0600] "GET /wp-content/
↪themes/jumpstart/framework/assets/js/nivo.min.js?ver=3.2
 ↪HTTP/1.1" 200 3074
"https://www.askdavetaylor.com/how-to-play-dvd-free-windows-
↪10-win10/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 ↪AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
 ↪64.0.3282.140 Safari/537.36 Edge/18.17763 X-Middleton/1"
 ↪52.53.151.37 - - [08/Mar/2019:06:10:09 -0600] "GET
 ↪/wp-includes/js/jquery/jquery.js?ver=1.12.4 HTTP/1.1"
 ↪200 33766 "https://www.askdavetaylor.com/how-to-play
↪-dvd-free-windows-10-win10/" "Mozilla/5.0 (Windows NT
 ↪10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
 ↪Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
 ↪X-Middleton/1" 18.144.59.52 - - [08/Mar/2019:06:10:09
 ↪-0600] "GET /wp-content/plugins/google-analytics-for-
↪wordpress/assets/js/frontend.min.js?ver=7.4.2 HTTP/1.1"
 ↪200 2544 "https://www.askdavetaylor.com/how-to-play
↪-dvd-free-windows-10-win10/"
 ↪"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 ↪AppleWebKit/537.36 (KHTML, like Gecko)
 ↪Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
 ↪X-Middleton/1"

It's big and ugly, right? Okay, then let's just isolate a single entry to see how it's structured:


18.144.59.52 - - [08/Mar/2019:06:10:09 -0600] "GET
 ↪/wp-content/themes/jumpstart/framework/assets/js/
↪nivo.min.js?ver=3.2 HTTP/1.1" 200 3074
"https://www.askdavetaylor.com/how-to-play-dvd-free-windows-
↪10-win10/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140
 ↪Safari/537.36 Edge/18.17763 X-Middleton/1"

That's still obfuscated enough to kick off a migraine!

Fortunately, the Apache website has a somewhat clearer explanation of what's known as the custom log file format that's in use on my server. Of course, it's described in a way that only a programmer could love:

Go to Full Article

What The @#$%&! (Heck) is this #! (Hash-Bang) Thingy In My Bash Script

Mitch Frazier — Fri, 10 May 2019 11:30:00 +0000

by Mitch Frazier

You've seen it a million times—the hash-bang (#!) line at the top of a script—whether it be Bash, Python, Perl or some other scripting language. And, I'm sure you know what its purpose is: it specifies the script interpreter that's used to execute the script. But, do you know how it actually works? Your initial thought might be that your shell (bash) reads that line and then executes the specified interpreter, but that's not at all how it works. How it actually works is the main focus of this post, but I also want to introduce how you can create your own version of "hash-bang" if you're so inclined.

Go to Full Article

Fun with Mail Merge and Cool Bash Arrays

Dave Taylor — Mon, 25 Mar 2019 11:30:00 +0000

by Dave Taylor

Creating a sed-based file substitution tool.

A few weeks ago, I was digging through my spam folder and found an email message that started out like this:


Dear #name#
Congratulations on winning the $15.7 million lottery payout!
To learn how to claim your winnings, please...

Obviously, it was a scam (does anyone actually fall for these?), but what captured my attention was the #name# sequence. Clearly that was a fail on the part of the sender who presumably didn't know how to use AnnoyingSpamTool 1.3 or whatever the heck he or she was using.

The more general notation for bulk email and file transformations is pretty interesting, however. There are plenty of legitimate reasons to use this sort of substitution, ranging from email newsletters (like the one I send every week from AskDaveTaylor.com—check it out!) to stockholder announcements and much more.

With that as the inspiration, let's build a tool that offers just this capability.

The simple version will be a 1:1 substitution, so #name# becomes, say, "Rick Deckard", while #first# might be "Rick" and #last# might be "Deckard". Let's build on that, but let's start small.

Simple Word Substitution in Linux

There are plenty of ways to tackle the word substitution from the command line, ranging from Perl to awk, but here I'm using the original UNIX command sed (stream editor) designed for exactly this purpose. General notation for a substitution is s/old/new/, and if you tack on a g at the end, it matches every occurrence on a line, not only the first, so the full command is s/old/new/g.

Before going further, here's a simple document that has necessary substitutions embedded:


$ cat convertme.txt
#date#

Dear #name#, I wanted to start by again thanking you for your
generous donation of #amount# in #month#. We couldn't do our
work without support from humans like you, #first#.

This year we're looking at some unexpected expenses,
particularly in Sector 5, which encompasses #state#, as you
know. I'm hoping you can start the year with an additional
contribution? Even #suggested# would be tremendously helpful.

Thanks for your ongoing support. With regards,

Rick Deckard
Society for the Prevention of Cruelty to Replicants

Scan through it, and you'll see there's a lot of substitutions to do: #date#, #name#, #amount#, #month#, #first#, #state# and #suggested#. It turns out that #date# will be replaced with the current date, and #suggested# is one that'll be calculated as the letter is processed, but that's for a bit later, so stay tuned for that.

Go to Full Article

Converting Decimals to Roman Numerals with Bash

Dave Taylor — Mon, 18 Feb 2019 12:30:00 +0000

by Dave Taylor

Decimals to Roman numerals—here we hit all the limitations of Bash shell scripting.

My last few articles have given me a chance to relive my undergraduate computer science degree and code a Roman numeral to decimal converter. It's quite handy when you're watching old movies (when was MCMLVII anyway?), and the basic coding algorithm was reasonably straightforward. (See Dave's "Roman Numerals and Bash" and "More Roman Numerals and Bash".)

The trick with Roman numerals, however, is that it's what's known as a subtractive notation. In other words, it's not a position → value or even symbol → value notation, but a sort of hybrid. MM = 2000, and C = 100, but MMC and MCM are quite different: the former is 2100, and the latter is 1000 + (–100 + 1000) = 1900.

This means that the conversion isn't quite as simple as a mapping table, which makes it a good homework assignment for young comp-sci students!

Let's Write Some Code

In the Roman numeral to decimal conversion, a lot of the key work was done by this simple function:


mapit() {
   case $1 in
     I|i) value=1 ;;
     V|v) value=5 ;;
     X|x) value=10 ;;
     L|l) value=50 ;;
     C|c) value=100 ;;
     D|d) value=500 ;;
     M|m) value=1000 ;;
      * ) echo "Error: Value $1 unknown" >&2 ; exit 2 ;;
   esac
}

You'll need this function to proceed, but as a cascading set of conditional statements. Indeed, in its simple form, you could code a decimal to Roman numeral converter like this:


while [ $decvalue -gt 0 ] ; do

  if [ $decvalue -gt 1000 ] ; then
    romanvalue="$romanvalue M"
    decvalue=$(( $decvalue - 1000 ))
  elif [ $decvalue -gt 500 ] ; then
    romanvalue="$romanvalue D"
    decvalue=$(( $decvalue - 500 ))
  elif [ $decvalue -gt 100 ] ; then
    romanvalue="$romanvalue C"
    decvalue=$(( $decvalue - 100 ))
  elif [ $decvalue -gt 50 ] ; then
    romanvalue="$romanvalue L"
    decvalue=$(( $decvalue - 50 ))
  elif [ $decvalue -gt 10 ] ; then
    romanvalue="$romanvalue X"
    decvalue=$(( $decvalue - 10 ))
  elif [ $decvalue -gt 5 ] ; then
    romanvalue="$romanvalue V"
    decvalue=$(( $decvalue - 5 ))
  elif [ $decvalue -ge 1 ] ; then
    romanvalue="$romanvalue I"
    decvalue=$(( $decvalue - 1 ))
  fi

done

This actually works, though the results are, um, a bit clunky:


$ sh 2roman.sh 25
converts to roman numeral  X X I I I I I

Or, more overwhelming:

Go to Full Article

Writing Secure Shell Scripts

Dave Taylor — Tue, 05 Feb 2019 12:30:00 +0000

by Dave Taylor

Don't expose your system with sloppy scripts!

Although a Linux desktop or server is less susceptible to viruses and malware than a typical Windows device, there isn't a device on the internet that isn't eventually attacked. The culprit might be the stereotypical nerd in a bedroom testing his or her hacker chops (think Matthew Broderick in War Games or Angelina Jolie in Hackers). Then again, it might be an organized military, criminal, terrorist or other funded entity creating massive botnets or stealing millions of credit cards via a dozen redirected attack vectors.

In any case, modern systems face threats that were unimaginable in the early days of UNIX development and even in the first few years of Linux as a hobbyist reimplementation of UNIX. Ah, back in the day, the great worry was about copyrighted code, and so useful tools constantly were being re-implemented from scratch to get away from the AT&T Bell Labs licenses and so forth.

I have personal experience with this too. I rewrote the Hunt the Wumpus game wumpus from scratch for BSD 4.2 when the Berkeley crowd was trying to get away from AT&T UNIX legal hassles. I know, that's not the greatest claim to fame, but I also managed to cobble together a few other utilities in my time too.

Evolution worked backward with the internet, however. In real life, the lawless Wild West was gradually tamed, and law-abiding citizens replaced the outlaws and thugs of the 1850s and the Gold Rush. Online, it seems that there are more, smarter and better organized digital outlaws than ever.

Which is why one of the most important steps in learning how to write shell scripts is to learn how to ensure that your scripts are secure—even if it's just your own home computer and an old PC you've converted into a Linux-based media server with Plex or similar.

Let's have a look at some of the basics.

Know the Utilities You Invoke

Here's a classic trojan horse attack: an attacker drops a script called ls into /tmp, and it simply checks to see the userid that invoked it, then hands off its entire argument sequence to the real /bin/ls. If it recognizes userid = root, it makes a copy of /bin/sh into /tmp with an innocuous name, then changes its permission to setuid root.

This is super easy to write. Here's a version off the top of my head:


#!/bin/sh

if [ "$USER" = "root" ] ; then
  /bin/cp /bin/sh /tmp/.secretshell
  /bin/chown root /tmp/.secretshell
  /bin/chmod 4666 root /tmp/.secretshell
fi

exec /bin/ls $*

I hope you understand what just happened. This simple little script has created a shell that always grants its user root access to the Linux system. Yikes. Fancier versions would remove themselves once the root shell has been created, leaving no trace of how this transpired.

Go to Full Article

More Roman Numerals and Bash

Dave Taylor — Thu, 27 Dec 2018 12:30:00 +0000

by Dave Taylor

When in Rome: finishing the Roman numeral converter script.

In my last article, I started digging in to a classic computer science puzzle: converting Roman numerals to Arabic numerals. First off, it more accurately should be called Hindu-Arabic, and it's worth mentioning that it's believed to have been invented somewhere between the first and fourth century—a counting system based on 0..9 values.

The script I ended up with last time offered the basics of parsing a specified Roman numeral and converted each value into its decimal equivalent with this simple function:


mapit() {
   case $1 in
     I|i) value=1 ;;
     V|v) value=5 ;;
     X|x) value=10 ;;
     L|l) value=50 ;;
     C|c) value=100 ;;
     D|d) value=500 ;;
     M|m) value=1000 ;;
      * ) echo "Error: Value $1 unknown" >&2 ; exit 2 ;;
   esac
}

Then I demonstrated a slick way to use the underutilized seq command to parse a string character by character, but the sad news is that you won't be able to use it for the final Roman numeral to Arabic numeral converter. Why? Because depending on the situation, the script sometimes will need to jump two ahead, and not just go left to right linearly, one character at a time.

Instead, you can build the main loop as a while loop:


while [ $index -lt $length ] ; do

    our code

    index=$(( $index + 1 ))
done

There are two basic cases to think about in terms of solving this algorithmic puzzle: the subsequent value is greater than the current value, or it isn't—for example, IX versus II. The first is 9 (literally 1 subtracted from 10), and the second is 2. That's no surprise; you'll need to know both the current and next values within the script.

Sharp readers already will recognize that the last character in a sequence is a special case, because there won't be a next value available. I'm going to ignore the special case to start with, and I'll address it later in the code development. Stay tuned, sharpies!

Because Bash shell scripts don't have elegant in-line functions, the code to get the current and next values won't be value=mapit(romanchar), but it'll be a smidge clumsy with its use of the global variable value:


mapit ${romanvalue:index-1:1}
currentval=$value

mapit ${romanvalue:index:1}
nextval=$value

It's key to realize that in the situation where the next value isn't greater than the current value (for example, MC), you can't automatically conclude that the next value isn't going to be part of a complex two-value sequence anyway. Like this: MCM. You can't just say M=1000 and C=500, so let's just convert it to 1500 and process the second M when we get to it. MCM=1900, not 2500!

The basic logic turns out to be pretty straightforward:

Go to Full Article

Roman Numerals and Bash

Dave Taylor — Mon, 26 Nov 2018 13:00:00 +0000

by Dave Taylor

Fun with retro-coding a Roman numeral converter—I head back to my college years and solve me homework anew!

I earned a bachelor's degree in computer science back in the dawn of computing. Well, maybe it wasn't quite that long ago, but we did talk about Ada and FORTRAN in class. As a UCSD alumnus, however, it's no surprise that UCSD Pascal was the programming language of choice. Don't worry; no punch cards and no paper tape were involved in my educational endeavors.

As with modern computer science study, we spent a lot of time coding algorithms and solving problems and puzzles. I'm a board-gamer, so I was quite happy to try to solve the "dining philosophers problem", the "four color problem" or the "traveling salesman problem". You might well have tried to solve the same darn problems.

One coding problem that has stuck with me is a Roman numeral conversion program. As part of my first programming class, I recall it being a pretty tricky problem, but we didn't have the internet or GitHub to scrounge around for smart solutions or inspiration.

So, in the spirit of retro-coding, let's build a script that can convert Roman numerals into regular decimal equivalent values.

Roman Numerals

I know, you're saying "um, remind me, what are Roman numerals?", even though you see them all the time in movies and books. You just ignore the MCMLXIII that shows up as a copyright notice. What's funny is that the general industry consensus is that studios use those Roman numerals instead of the more understandable "Copyright 1963" to obfuscate the age of the film (by the way, MCMLXIII = 1963).

It turns out that Roman numerals are interesting because they are essentially grouped into logical segments. At its most basic, each letter has a specific decimal value, so let's start there (see Table 1 for the values).

Table 1. Roman Numerals and Their Values

Symbol	I	V	X	L	C	D	M
Value	1	5	10	50	100	500	1000

If you wanted to write the value 60 as Roman numerals, that's easy: LX. Reverse the two values, however, and it's a completely different value: XL = 40. Why? Because when a lower value symbol appears before a higher value symbol, it's considered a reduction of that value. The fancy name for this is subtractive notation.

In other words, LX = 50 + 10, but XL = L – X = 50 – 10.

Now you can see how the earlier value breaks into clusters of values based on whether a subsequent value is higher or lower than the current value. Here's the logic:

MCMLXIII = M + CM + L + X + III = 1000 + 900 + 50 + 10 + 3

Go to Full Article

Normalizing Filenames and Data with Bash

Dave Taylor — Tue, 30 Oct 2018 13:40:02 +0000

by Dave Taylor

URLify: convert letter sequences into safe URLs with hex equivalents.

This is my 155th column. That means I've been writing for Linux Journal for:


$ echo "155/12" | bc
12

No, wait, that's not right. Let's try that again:


$ echo "scale=2;155/12" | bc
12.91

Yeah, that many years. Almost 13 years of writing about shell scripts and lightweight programming within the Linux environment. I've covered a lot of ground, but I want to go back to something that's fairly basic and talk about filenames and the web.

It used to be that if you had filenames that had spaces in them, bad things would happen: "my mom's cookies.html" was a recipe for disaster, not good cookies—um, and not those sorts of web cookies either!

As the web evolved, however, encoding of special characters became the norm, and every Web browser had to be able to manage it, for better or worse. So spaces became either "+" or %20 sequences, and everything else that wasn't a regular alphanumeric character was replaced by its hex ASCII equivalent.

In other words, "my mom's cookies.html" turned into "my+mom%27s+cookies.html" or "my%20mom%27s%20cookies.html". Many symbols took on a second life too, so "&" and "=" and "?" all got their own meanings, which meant that they needed to be protected if they were part of an original filename too. And what about if you had a "%" in your original filename? Ah yes, the recursive nature of encoding things....

So purely as an exercise in scripting, let's write a script that converts any string you hand it into a "web-safe" sequence. Before starting, however, pull out a piece of paper and jot down how you'd solve it.

Normalizing Filenames for the Web

My strategy is going to be easy: pull the string apart into individual characters, analyze each character to identify if it's an alphanumeric, and if it's not, convert it into its hexadecimal ASCII equivalent, prefacing it with a "%" as needed.

There are a number of ways to break a string into its individual letters, but let's use Bash string variable manipulations, recalling that ${#var} returns the number of characters in variable $var, and that ${var:x:1} will return just the letter in $var at position x. Quick now, does indexing start at zero or one?

Here's my initial loop to break $original into its component letters:

Go to Full Article