Scripting

Creating an Application-Based Terminal Session

Andy Carlson — Thu, 12 Oct 2017 15:11:28 +0000

One of my first exposures to computers in a work environment was using a Wyse terminal to access a console-based application for data entry. It wasn't until a while later that I learned about terminals in UNIX and how they work. Once I understood it, I wanted to create my own self-contained application that was tied to a telnet or SSH session.

The method I've found to accomplish this involves the use of the trap command and some (seemingly) intimate loops. The example code will be inserted into the .bashrc file of the user to whom you wish to give application access. The example code was used for spam message entry into the sa-learn application for Spam Assassin:


instanceid="$RANDOM"
clean_session() {
    [[ -f /tmp/spam.message-$instanceid ]] && rm /tmp/spam.message-$instanceid
    exit
}


trap clean_session SIGINT

while true; do
    clear
    printf "******** Spam Message Entry Menu ********\n"
    printf "Enter the body of spam message, then press Ctrl-D\n"
    printf "To exit at any time, press Ctrl-C\n\n"
    cat > /tmp/spam.message-$instanceid
    clear
    printf "Processing....\n\n"
    /opt/bin/sa-learn --spam /tmp/spam.message-$instanceid
    printf "\n\n"
    rm /tmp/spam.message-$instanceid
    read -s -n 1 -p "Press any key to continue...."
done
exit

Let's examine the logic and functionality. The first line establishes a unique session ID for application access. If you have multiple users logging on with the same user name this will help keep data straight between sessions. The clean_session function needs to be declared at the beginning of the script for use with the following trap statement. The trap statement will run the clean_session function whenever it detects that the user entered Ctrl-C (interrupt signal or SIGINT).

For the purpose of establishing a the data-entry application, you'll use an infinite while loop. Once data is entered the first time and the end of the loop is reached, the loop just starts over and is ready for more data entry. This is where the clean_session function comes into play. If users want to exit, as per the instructional printouts, they type Ctrl-C. Normally, this would drop them into a shell, but since you want to isolate their usage to this application, you'll invoke the clean_session function, which will clean up their temporary file (if it exists) and log them off. The body of the while loop just accepts input (the cat statement), feeds the data into sa-learn and cleans up the temporary file.

For this example, you will need to modify the sudoers file to allow your user root access to sa-learn; the actual location of sa-learn might vary with your installation.

Go to Full Article

Non-Linux FOSS: Scripts in Your Menu Bar!

Shawn Powers — Wed, 23 Nov 2016 17:19:15 +0000

by Shawn Powers

There are hundreds of applications for OS X that place information in the menu bar. Usually, I can find one that almost does what I want, but not quite. Thankfully I found BitBar, which is an open-source project that allows you to write scripts and have their output refreshed and put on the menu bar.

I personally use it to fetch Bitcoin and Ethereum prices, but because you're limited only by what you can get Bash to output, it's extremely flexible. Even the method by which you change update frequency is elegant. By adding a refresh rate to the name of your script, the program refreshes only as often as you desire. You can see an example of my Bitcoin price-fetching script in the screenshot (Figure 1). Also notice the file format for my additional scripts (Figure 2).

Figure 1. Bitcoin Price-Fetching Script

Figure 2. File Format for Additional Scripts

You can download the binary or the source code here. There is also a huge library of user-contributed scripts so you don't have to start from scratch. My Bitcoin script is actually from this repository, and I based my other scripts off that. Whether you want to pull text from an API like me or possibly grep the temperature from a weather page, BitBar is simple and elegant at the same time.

Go to Full Article

Bash Shell Script: Building Your March Madness Bracket

Jim Hall — Tue, 15 Mar 2016 18:30:00 +0000

by Jim Hall

I must admit that I don't really follow basketball. But, I do like to engage with folks at work, and every spring I've always felt a little left out when my work colleagues fill out their NCAA March Madness basketball brackets. If your office is like mine, it seems everyone gets very excited to build their brackets and follow the basketball games and play in an office pool.

I wanted to play too, but I just didn't know enough about the teams to make an informed decision to fill out my March Madness bracket. So a few years ago, I found another way. I wrote a program to build my basketball bracket for me.

Computational models that predict the outcomes of sports games are not new. A little Googling will uncover methods such as the Pythagorean Expectation Model or other algorithms that build on previous team performance to predict future game outcomes. But that requires some research into sports statistics and following each team throughout the season. That's too much work for me, so I used a different method that should be familiar to many of my fellow nerds: the “Dungeons and Dragons Model”.

That's right! You can apply a simple D16 method to build a March Madness basketball bracket. How scientific is this? Probably not very, but it's enough to give me a stake in following March Madness basketball, but not enough that I feel particularly saddened if my bracket doesn't perform well. That's good enough for me.

Let's build a shell script to build an NCAA March Madness basketball bracket. I'll create the script under these assumptions:

The NCAA March Madness basketball brackets are seeded with the NCAA's ranking of 64 college basketball teams, divided into four regions, and ranked #1 through #16.
The NCAA March Madness basketball brackets are always initialized with the same contests: #1 vs #16, #8 vs #9, #5 vs #12, and so on.
A #1 ranked team should perform better than a #16 team, but a #8 team should perform about the same as a #9 team.

With these assumptions, let's examine the D16 method. In a tabletop role-playing game, you might throw a 1D16 to determine the outcome of an encounter. You would compare the value of the 1D16 to a player's statistic, such as Dexterity or Strength. This kind of throw assumes a “probability” or “likelihood” of an outcome based on the relative strength of the player.

Similarly, you can compare the outcome of a 1D16 to a team's NCAA ranking to determine the outcome of a team's performance. A #1 team should be a strong team, so let's assume the #1 team has 15 out of 16 “chances” to win, and one out of sixteen “chances” to lose. Without any other inputs, the #1 team would win if the 1D16 value is two or greater, and the #1 team would lose if the 1D16 value is one.

Go to Full Article

System Status as SMS Text Messages

Dave Taylor — Tue, 29 Sep 2015 20:41:49 +0000

by Dave Taylor

If you're paying really close attention, you'll remember that in my last article, I was exploring the rudiments of a script that would accept a list of words as input and create a word search grid, suitable for printing. It turns out that's crazy hard to do as a shell script—it just doesn't have the muscle to implement any sort of functional algorithm in an elegant fashion. So, I'm going to bail on it, at least until I can find someone else's open-source code I can explore for inspiration.

Or, of course, if you're motivated and have some time to experiment, go back to my April 2015 column, read through it, then try your own hand at implementing something.

Before I get letters about the oddity of being the shell script programming columnist who is bailing on a script, I will point out that there's a lot to learn from this experience actually. Most specifically, although it's nice to imagine that the Linux environment is completely egalitarian, and that every script, every language and every program is as powerful and well designed as every other, it's clear that's not the case.

Take Perl versus Awk, for example. Awk is powerful and I use it frequently, but although there are major software programs written in Perl, you'd be hard-pressed to find any significant software, functions, applications or utilities programmed directly in Awk. The same goes for C++ versus PHP, for example, or any modern structured language versus, well, the Bourne Again Shell. There, I said it. Shell script programming can take you only so far, and then you realize that you've hit the edges of the environment and its capabilities, and it's time to jump to another language.

Indeed, when I wrote my popular book Wicked Cool Shell Scripts, there was a tiny C program that snuck in by necessity: it was a few lines of C to do a certain date calculation that would have been dozens, if not hundreds, of lines of shell script.

Having said that, I will rush back to defend the shell as a powerful, lightweight programming and prototyping environment perfect for a variety of tasks because of its super-easy access to the power and capabilities of the entire Linux environment and, by extension, the entire Internet.

What's your take? You read this column, so it's reasonable for me to conclude that you are interested in learning more about programming within the Linux shell environment. How often do you bump into the bleeding edge of your shell and realize you have to flip into Perl, Ruby, C, Cobol (just kidding!) or another more sophisticated development environment to solve the problem properly?

Go to Full Article

Where's That Pesky Hidden Word?

Dave Taylor — Fri, 28 Aug 2015 18:17:07 +0000

by Dave Taylor

I've been promising my 11-year-old for a long time now that I'd write a program that would let you build custom word searches based on a list of words given by the user. I wrote one years and years ago in C, but since I can't find that code any more and wanted to tackle another interesting project for this column, that's what I'm going to look at herein.

There aren't any well-established algorithms around word searches, so it's also a good opportunity to come up with something from scratch, which means that as with the most interesting projects, we'll actually need to think about the program before we start coding. It's good discipline!

I also should admit that I've loved solving word search puzzles since I was a young pup, so there's an element of enjoyment for me in this project too, not just the pleasure of doing something for my daughter!

In case you've never seen a word search, it's typically a grid of letters, and within that is a set of listed words that can be found running horizontally, vertically or diagonally, spelled forward or backward. Figure 1 shows a tiny example.

Figure 1. Word Search Example

Looking Figure 1, how many words can you find? I can find CAT, DOG, GOD (which is, of course, DOG backward), TIC, COP and, more nerdy, ROM and ARG too. And this brings up one interesting characteristic of word searches: there always are incidental words that can appear, so one important task for the resultant program is to ensure that no obscenities appear.

Upon first glance at a word search, it seems like the way to do it is to populate the grid randomly, then flip letters to make the words fit. But, it seems to me that a better strategy is essentially to make a crossword puzzle, then fill in the empty holes with random letters.

So that's going to be our first strategy for building the word search. Each word also will be randomly oriented horizontal, vertical or diagonal, as well as forward or backward. For now, let's just worry about forward or backward, meaning the initial word orientation code will look like this:


orient()
{
   # randomly pick an orientation and 
   # shift the word to match
   local direction;
   word=$1   # to make things neat and tidy
  if [ $(( RANDOM % 2 )) -eq 1 ] ; then
     # we need to reverse the value of $word
     word="$(echo $word | rev )"
   fi
}

Arrays are created by initializing them in the Bash shell, in a format that looks like this:


arrayname=( value1, value2, ... valueN )

And since this is going to be a lot about arrays, let's start by loading up the wordlist as an array, orienting words randomly as we proceed.

Go to Full Article

Days Between Dates?

Dave Taylor — Thu, 20 Nov 2014 18:58:58 +0000

by Dave Taylor

Alert readers will know that I'm working on a major revision to my popular Wicked Cool Shell Scripts book to come out later this year. Although most of the scripts in this now ten-year-old book still are current and valuable, a few definitely are obsolete or have been supplanted by new technology or utilities. No worries—that's why I'm doing the update.

One script I'll be adding is a complicated one that I'm going to develop here in my Linux Journal column: daysago. The script will take a specified date in the past and tell you how many days have elapsed between that date and the current day and time.

You might be thinking that's fairly complicated, and it is, but not in the way you might be thinking. The actual calculation is really easy because of how Linux systems store and manipulate dates. The challenge is in parsing the input.

The first part of the book includes a library of useful scripting utilities, however, and one just so happens to be what we want—no coincidence that!

Valid Date?

The easiest way to deal with something as complicated as a date is to force the work onto the user. There are a couple different strategies for that, but let's be lazy for now and prompt the user for the month, then day, then year, requiring numeric values. Then, we'll need to check whether it's valid.

Validating a user-specified date is pretty straightforward until we get to the issue of leap years. We're used to thinking that every four years is a leap year, but the formula is quite a bit more complicated than that, and it can be summarized with this set of rules:

Years divisible by four are leap years, unless...
The year also is divisible by 100, except if...
The year is divisible by 400, in which case it is.

Is that complicated enough? Of course, if we're just looking at leap years in the last few decades, it's not a very big deal, but it's inevitable that someone will try something like Feb 29, 1776, in which case we need to know whether it's valid.

Or, we can be lazy.

Since I like the lazy solution to things (remember, I'm not writing production code here, I'm demonstrating concepts), let's cheat by using the Linux cal command. Because it lets us specify month/year, we can hand off the question of whether there's a February 29 in the year 1776 by just asking for a display of 2/1776:


$ cal 2 1776
   February 1776
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29

It looks like 1776 was indeed a leap year. No wonder they had time to draft the Constitution before summer came along and made Philly too darn hot for anyone to work!

To turn this command into a script, a simple grep and test for nonzero results does the trick:

Go to Full Article

Considering Legacy UNIX/Linux Issues

Dave Taylor — Thu, 28 Aug 2014 17:34:54 +0000

by Dave Taylor

Gah, so frustrating! Ten years ago I wrote a rather popular book called Wicked Cool Shell Scripts, and I'm working on a new edition—a Tenth Anniversary release. There are lots of new scripts, entirely new chapters and updates to the older stuff. Fortunately, Bash hasn't evolved that much in the last decade, so just about everything still works fine (although there are some scripts I'm now realizing can't handle spaces in filenames—something I talked about years ago in this very column).

But, there were problems when I pushed out the following script to my Google Plus followers (find me on G+ at http://profiles.google.com/d1taylor) and asked those that had access to a Linux or UNIX system to give it a quick run:


#!/bin/sh
# how many commands: count how many executable commands
#   are in your current PATH.

myPATH="$(echo $PATH | sed -e 's/ /~~/g' -e 's/:/ /g')"
count=0 ; nonex=0

for dirname in $myPATH ;  do
  directory="$(echo $dirname | sed 's/~~/ /g')"
  if [ -d "$directory" ] ; then
    for command in $(ls "$directory") ; do 
      if [ -x "$directory/$command" ] ; then
        count="$(( $count + 1 ))"
      else
        nonex="$(( $nonex + 1 ))"
      fi
    done
  fi
done
echo "$count commands, and $nonex entries that weren't \
   marked executable"
exit 0

It's simple enough really—I'm using sed to split the $PATH value into space-separated values, then the for loop to step through them one by one, counting how many entries are marked as executable (the -x test).

Of course, you have to take into account that there might be spaces in directory names within the PATH (like /User Applications/bin), so I also convert spaces to ~~ and then later in the for loop convert them back at the last possible moment. But that's not rocket science, just basic scripting.

Why test to see if the directory in the PATH is an actual directory (the -d test), you may ask? Because when people can add their own directories to the system PATH, it can get messy, and it's entirely possible that there is an entry that's not a valid directory. So that's just error management really. Perhaps an else echo "Error: Entry $directory isn't a directory?" would be a good addition.

In any case, I posted this script, and people ran it on various systems, reporting answers ranging from 1,100 to more than 3,000 executable commands in their PATH (Ubuntu 13.10). More than 3,000 commands? Sheesh! Except then there was my friend Chris who said:

Sun OS 5.8 The line "for command..." gives me this error "syntax error at line 10: '$' unexpected".

That's this line in the script:

Go to Full Article

Debugging Web Sites

Dave Taylor — Wed, 30 Apr 2014 17:13:30 +0000

by Dave Taylor

I know, I'm in the middle of a series of columns about how to work with ImageMagick on the command line, but when other things arise, well, I imagine that a lot of you are somehow involved in the management of servers or systems, so you all understand firefighting.

Of course, this means you all also understand the negative feedback loop that is an intrinsic part of system administration and IT management. I mean, people don't call you and the CEO doesn't send a memo saying, "system worked all day, printer even printed. Thanks!"

Nope, it's when things go wrong that you hear about them, and that propensity to ignore the good and have to deal with the bad when it crops up is not only a characteristic of being in corporate IT, it's just as true if you're running your own system—which is how it jumped out of the pond and bit me this month.

It all started ten years ago with my Ask Dave Taylor site. You've probably bumped into it, as it's been around for more than a decade and served helpful tutorial information for tens of millions of visitors in that period.

Ten years ago, the choice of Movable Type as my blogging platform made total sense and was a smart alternative to the raw, unfinished WordPress platform with its never-ending parade of hacks and problems. As every corporate IT person knows, however, sometimes you get locked in to the wrong platform and are then stuck, with the work required to migrate becoming greater and greater each month nothing happens.

For the site's tenth anniversary, therefore, it was time. I had to bite the bullet and migrate all 3,800 articles and 56,000 comments from Movable Type to WordPress, because yes, WordPress won and is clearly the industry standard for content management systems today.

The task was daunting, not just because of the size of the import (it required the consulting team rewriting the standard import tool to work with that many articles and comments), but because the naming scheme changed. On Movable Type, I'd always had it set to convert the article's name into a URL like this:

Name: Getting Started with Pinterest

URL: /getting_started_with_pinterest.html

That was easy and straightforward, but on WordPress, URLs have dashes, not underscores, and, more important, they don't end with .html because they're generated dynamically as needed. This means the default URL for the new WordPress site would look like this:

New URL: /getting-started-with-pinterest/

URLs can be mapped upon import so that the default dashes become underscores, but it was the suffix that posed a problem, and post-import there were 3,800 URLs that were broken because every single link to xx_xx.html failed.

Go to Full Article

Web Administration Scripts—Redux

Dave Taylor — Wed, 26 Feb 2014 21:20:11 +0000

by Dave Taylor

It's been months, and I'm still dealing with a DDOS (distributed denial of service) attack on my server—an attack that I can see is coming from China, but there's not really much I can do about it other than try to tweak firewall settings and so on.

In my last article, I started describing analyzing Apache log files and system settings to try to create some scripts that can watch for DDOS attacks and flag them before that inevitable e-mail from a customer about the site being down or incredibly slow.

Honestly, it's pretty disheartening dealing with these anonymous attacks, but that's another column. If you've had experience with this sort of long-term problem, I'd be interested in hearing about how you finally resolved it.

The first script I worked on simply tracks how many processes are running on the server. It's just a small step from monitoring the output of the uptime program, which, of course, could be done as easily as:


load="$(uptime | cut -d\  -f14 | cut -d. -f1)"
if [ $load -gt 4 ] ; then
  echo "Uptime greater than threshold. What's going on?"
  uptime
fi

That's marginally interesting, but let's go back to analyzing the number of processes running, as that's a better indication that there might be a DDOS happening. Last time, I used a loop of ps ; sleep and kept track of the min/max values. After running for a while, the script's output was something like this:


Current ps count=90: min=76, max=150, tally=70 and average=107

Interpret this as after running for 90 cycles, the minimum number of Apache (httpd) processes running at any given time was 76, max was 150 and average was 107.

Once you've run this script for a few hours, you'll have a pretty good idea of typical traffic load on your server, which is critical for you to be able to detect out-of-pattern variations. If my average Apache process count is 107 and the server has 917 processes, there's a problem. On the other hand, if the average load is 325 processes, 600 isn't too far out of band and could be representative of a spike in traffic rather than the beginnings of an actual attack.

I wrapped up last month's article by showing a script suitable for running from cron that would look for abnormal traffic spikes. But it had some problems, as I highlighted at the end of the column, my nerd version of a cliffhanger—no murder unsolved, no car in the driveway, no police sirens, just some code that needs improvement. Hey, work with me here!

Go to Full Article

Cribbage: Sorting Your Hand

Dave Taylor — Mon, 24 Jun 2013 17:57:45 +0000

by Dave Taylor

We've been working on writing code for the game Cribbage, and last time, I created the code needed to pick a random subset of six cards out of a "deck" and display them in an attractive format—like this:


$ sh cribbage.sh
Card 0: 7C
Card 1: 5H
Card 2: 9H
Card 3: 10S
Card 4: 5D
Card 5: AS

The primary task on the agenda this article is to sort the cards after they've been dealt. This means we're going to have to sort the cards by rank while ignoring the suit, then slot them back into the "hand" array. Is there an easy way to do that? Actually, we'll use the sort function.

We can prototype this by using the command line to see what result we get:


$ sh cribbage.sh  | sort -n
Card 0: 4S
Card 1: 7C
Card 2: 9S
Card 3: JC
Card 4: 7H
Card 5: 8C

What the heck? Oh! You can see the problem, right? By telling sort to order things numerically, it properly ignores "Card" but then sees the ordinal value of the card and sorts based on that, rather than on the actual card value itself.

Even if we fix this, however, we still have the problem that face cards will sort before numeric value cards, which isn't what we want. In fact, we want aces to sort as lower than 2s, while jacks, queens and kings sort as higher than 10s.

If you wanted to have aces "high", the easiest way to do that would be to change the display routine, of course: 1 = a deuce, 2 = a three, 12 = king and 13 = ace. Poof. Everything sorts ace-high. That's just not how Cribbage scores them.

To accomplish Cribbage-rank sorting, we'll need to change the output to push out two values: the rank and the total card value. It's going to look ugly, but it's just an interim result.

Here's how I tweak the code to display these values:


showcard()
{
  # given a card value of 0..51 show the suit and rank
  suit=$(( $1 / 13 ))
  rank=$(( ( $1 % 13 ) + 1 ))
  case $rank in
    1)  orank="A" ;;
    11) orank="J" ;;
    12) orank="Q" ;;
    13) orank="K" ;;
     *) orank=$rank ;;
  esac
  showcardvalue=$orank${suits[$suit]}
}

If you compare it to the version we built last month, the main difference is that instead of calculating the rank of the card and then overwriting it with "A", "J", "Q" or "K" as appropriate, we're using a new variable, orank, to store the corrected value. Why? Because now in the main section of the script we also can access the $rank of the card as desired:


showcard ${hand[$card]}
echo "$rank ${hand[$card]}"

For each card chosen, the script has an interim output of rank followed by the numeric value of the card, with no fancy display (even though we're still tapping the showcard function for simplicity). The result:

Go to Full Article