Bash

Writing GitHub Web Hooks with Bash

Andy Carlson — Wed, 07 Aug 2019 16:30:00 +0000

Bring your GitHub repository to the next level of functionality.

For the past year since Microsoft has acquired GitHub, I've been hosting my Git repositories on a private server. Although I relished the opportunity and challenge of setting it all up, and the end product works well for my needs, doing this was not without its sacrifices. GitHub offers a clean interface for configuring many Git features that otherwise would require more time and effort than simply clicking a button. One of the features made easier to implement by GitHub that I was most fond of was web hooks. A web hook is executed when a specific event occurs within the GitHub application. Upon execution, data is sent via an HTTP POST to a specified URL.

This article walks through how to set up a custom web hook, including configuring a web server, processing the POST data from GitHub and creating a few basic web hooks using Bash.

Preparing Apache

For the purpose of this project, let's use the Apache web server to host the web hook scripts. The module that Apache uses to run server-side shell scripts is mod_cgi, which is available on major Linux distributions.

Once the module is enabled, it's time to configure the directory permissions and virtual host within Apache. Use the /opt/hooks directory to host the web hooks, and give ownership of this directory to the user that runs Apache. To determine the user running an Apache instance, run the following command (provided Apache is currently running):


ps -e -o %U%c| grep 'apache2\|httpd'

These commands will return a two-column output containing the name of the user running Apache and the name of the Apache binary (typically either httpd or apache2). Grant directory permission with the following chown command (where USER is the name of the user shown in the previous ps command):


chown -R USER /opt/hooks

Within this directory, two sub-directories will be created: html and cgi-bin. The html folder will be used as a web root for the virtual host, and cgi-bin will contain all shell scripts for the virtual host. Be aware that as new sub-directories and files are created under /opt/hooks, you may need to rerun the above chown to verify proper access to files and sub-directories.

Here's the configuration for the virtual host within Apache:

Go to Full Article

The Bash Trap Command

Mitch Frazier — Wed, 07 Aug 2019 15:15:00 +0000

by Mitch Frazier

If you've written any amount of bash code, you've likely come across the trap command. Trap allows you to catch signals and execute code when they occur. Signals are asynchronous notifications that are sent to your script when certain events occur. Most of these notifications are for events that you hope never happen, such as an invalid memory access or a bad system call. However, there are one or two events that you might reasonably want to deal with. There are also "user" events available that are never generated by the system that you can generate to signal your script. Bash also provides a psuedo-signal called "EXIT", which is executed when your script exits; this can be used to make sure that your script executes some cleanup on exit.

Go to Full Article

Bash Shell Games: Let's Play Go Fish!

Dave Taylor — Tue, 30 Jul 2019 12:00:00 +0000

by Dave Taylor

How to begin developing a computer version of the popular card game.

Between the previous 163 columns I've written here in Linux Journal and the dozens of games I programmed and explored during the creation of my Wicked Cool Shell Scripts book, I've written a lot of Bash shell games. The challenge is to find one that's simple enough where a shell script will work, but isn't so simple that it ends up being only a half-dozen lines.

Magic 8-Ball is a perfect example. It turns out that the entire "predict the future" gizmo was really just a 20-sided die floating in dark purple fluid. So an array of 20 possible values and a random number selector and boom—you've got a magic 8-ball script:


#!/bin/sh

# magic 8 ball. Yup. Pick a random number, output message

# messages harvested from the Wikipedia entry

answers=("It is certain." "It is decidedly so."
  "Without a doubt." "Yes - definitely."
  "You may rely on it." "As I see it, yes." "Most likely."
  "Outlook good." "Yes." "Signs point to yes."
  "Reply hazy, try again." "Ask again later."
  "Better not tell you now." "Cannot predict now."
  "Concentrate and ask again." "Don't count on it."
  "My reply is no." "My sources say no."
  "Outlook not so good." "Very doubtful.")

echo "Oh! Magic 8 Ball, Please Tell Me True..." ; echo ""
/bin/echo -n "What is your question? "
read question

answer=$(( $RANDOM % 20 ))

echo ""
echo "I have looked into the future and I say: "
echo "     ${answers[$answer]}" ; echo ""

exit 0

Let's do a quick run to see if I'm the most popular LJ writer:


$ sh magic8.sh
Oh! Magic 8 Ball, Please Tell Me True...

What is your question? Am I the most popular LJ writer?

I have looked into the future and I say:
     My reply is no.

Ouch, that's harsh. I write the darn divination program, and it just drops a brick on my foot. Yeesh.

More seriously, Magic 8 Ball is too simple to make an interesting shell script. By contrast, Call of Duty is way too complex, even if I did a version with text output instead of gorgeously rendered 3D graphics.

Card Game Function Library

That's why card games prove to be good as programming challenges or exercises: the core mechanism of a 52-card random deck is pretty straightforward, so it's all about the actual cardplay.

Not only that, but as I've written before about card games as shell scripts, I already have a handy set of functions to create, shuffle and display cards out of a deck. If you want to rummage in the archives, I've tackled Acey-Deucey, Baccarat and some bits and pieces of Cribbage.

In order to jump right into the new game that I'm going to describe how to build, Go Fish!, let's steal the following functions from my earlier scripts:

Go to Full Article

Job Control: the Bash Feature You Only Think You Don't Need

Mitch Frazier — Thu, 25 Jul 2019 11:30:00 +0000

by Mitch Frazier

There are basically three types of people in the world: those who know little or nothing about bash job control, those who know enough to believe that it's nothing that they would ever use, and those who can just skim the rest of this post. Now, don't get me wrong, I'm not saying that bash's job control is going to change your world, but there are a couple simple everyday scenarios where job control can be useful, and often, it even can eliminate an "oh crap" moment.

Go to Full Article

Finishing Up the Bash Mail Merge Script

Dave Taylor — Thu, 04 Jul 2019 12:00:00 +0000

by Dave Taylor

Finally, I'm going to finish the mail merge script, just in time for Replicant Day.

Remember the mail merge script I started writing a while back? Yeah, that was quite some time ago. I got sidetracked with the Linux Journal Anniversary special issue (see my article "Back in the Day: UNIX, Minix and Linux"), and then I spun off on a completely different tangent for my last article ("Breaking Up Apache Log Files for Analysis"). I blame it on...

SQUIRREL!

Oh, sorry, back to topic here. I was developing a shell script that would let you specify a text document with embedded field names that could be substituted iteratively across a file containing lots of field values.

Each field was denoted by #fieldname#, and I identified two categories of fieldnames: fixed and dynamic. A fixed value might be #name#, which would come directly out of the data file, while a dynamic value could be #date#, which would be the current date.

More interesting, I also proposed calculated values, specifically #suggested#, which would be a value calculated based on #donation#, and #date#, which would be replaced by the current date. The super-fancy version would have a simple language where you could define the relationship between variables, but let's get real. Mail merge. It's just mail merge.

Reading and Assigning Values

It turns out that the additions needed for this script aren't too difficult. The basic data file has comma-separated field names, then subsequent lines have the values associated with those fields.

Here's that core code:


if [ $lines -eq 1 ] ; then # field names
# grab variable names
declare -a varname=($f1 $f2 $f3 $f4 $f5 $f6 $f7)
else # process fields

# grab values for this line (can contain spaces)
declare -a value=("$f1" "$f2" "$f3" "$f4" "$f5" "$f6" "$f7")

The declare function turns out to be ideal for this, allowing you to create an array varname based on the contents of the first line, then keep replacing the values of the array value, so that varname[1] = value[1], and so on.

To add the additional variables #date# and #suggested#, you simply can append them to the varname and value arrays. The first one is easy, but it did highlight a weakness in the original code that I had to fix by adding quotes as shown:

Go to Full Article

What The @#$%&! (Heck) is this #! (Hash-Bang) Thingy In My Bash Script

Mitch Frazier — Fri, 10 May 2019 11:30:00 +0000

by Mitch Frazier

You've seen it a million times—the hash-bang (#!) line at the top of a script—whether it be Bash, Python, Perl or some other scripting language. And, I'm sure you know what its purpose is: it specifies the script interpreter that's used to execute the script. But, do you know how it actually works? Your initial thought might be that your shell (bash) reads that line and then executes the specified interpreter, but that's not at all how it works. How it actually works is the main focus of this post, but I also want to introduce how you can create your own version of "hash-bang" if you're so inclined.

Go to Full Article

Pattern Matching In Bash

Mitch Frazier — Mon, 15 Apr 2019 11:30:00 +0000

by Mitch Frazier

Wildcards have been around forever. Some even claim they appear in the hieroglyphics of the ancient Egyptians. Wildcards allow you to specify succinctly a pattern that matches a set of filenames (for example, *.pdf to get a list of all the PDF files). Wildcards are also often referred to as glob patterns (or when using them, as "globbing"). But glob patterns have uses beyond just generating a list of useful filenames. The bash man page refers to glob patterns simply as "Pattern Matching".

Go to Full Article

Fun with Mail Merge and Cool Bash Arrays

Dave Taylor — Mon, 25 Mar 2019 11:30:00 +0000

by Dave Taylor

Creating a sed-based file substitution tool.

A few weeks ago, I was digging through my spam folder and found an email message that started out like this:


Dear #name#
Congratulations on winning the $15.7 million lottery payout!
To learn how to claim your winnings, please...

Obviously, it was a scam (does anyone actually fall for these?), but what captured my attention was the #name# sequence. Clearly that was a fail on the part of the sender who presumably didn't know how to use AnnoyingSpamTool 1.3 or whatever the heck he or she was using.

The more general notation for bulk email and file transformations is pretty interesting, however. There are plenty of legitimate reasons to use this sort of substitution, ranging from email newsletters (like the one I send every week from AskDaveTaylor.com—check it out!) to stockholder announcements and much more.

With that as the inspiration, let's build a tool that offers just this capability.

The simple version will be a 1:1 substitution, so #name# becomes, say, "Rick Deckard", while #first# might be "Rick" and #last# might be "Deckard". Let's build on that, but let's start small.

Simple Word Substitution in Linux

There are plenty of ways to tackle the word substitution from the command line, ranging from Perl to awk, but here I'm using the original UNIX command sed (stream editor) designed for exactly this purpose. General notation for a substitution is s/old/new/, and if you tack on a g at the end, it matches every occurrence on a line, not only the first, so the full command is s/old/new/g.

Before going further, here's a simple document that has necessary substitutions embedded:


$ cat convertme.txt
#date#

Dear #name#, I wanted to start by again thanking you for your
generous donation of #amount# in #month#. We couldn't do our
work without support from humans like you, #first#.

This year we're looking at some unexpected expenses,
particularly in Sector 5, which encompasses #state#, as you
know. I'm hoping you can start the year with an additional
contribution? Even #suggested# would be tremendously helpful.

Thanks for your ongoing support. With regards,

Rick Deckard
Society for the Prevention of Cruelty to Replicants

Scan through it, and you'll see there's a lot of substitutions to do: #date#, #name#, #amount#, #month#, #first#, #state# and #suggested#. It turns out that #date# will be replaced with the current date, and #suggested# is one that'll be calculated as the letter is processed, but that's for a bit later, so stay tuned for that.

Go to Full Article

Cat-Proofing Your Screen Locker with Bash

Mitch Frazier — Wed, 20 Feb 2019 13:00:00 +0000

by Mitch Frazier

I have a computer in my bedroom. I also have cats. Unfortunately, cats and screen lockers don't mix well, particularly at night. To be accurate, it's more a problem with the display power management than the actual screen locking. Here's the way it works: I run a script to "shut the lights off at night" (that is, lock the screen and force the display to power down), and that works great, until one of the cats jumps on the desk and causes the mouse to move and turn the display back on. And the cats don't even have to touch the mouse; the slight movement of the desk is enough to cause the mouse to react. Recently, I'd had enough of it and figured there had to be a way to disable the mouse and "refactor" the script.

Go to Full Article

Converting Decimals to Roman Numerals with Bash

Dave Taylor — Mon, 18 Feb 2019 12:30:00 +0000

by Dave Taylor

Decimals to Roman numerals—here we hit all the limitations of Bash shell scripting.

My last few articles have given me a chance to relive my undergraduate computer science degree and code a Roman numeral to decimal converter. It's quite handy when you're watching old movies (when was MCMLVII anyway?), and the basic coding algorithm was reasonably straightforward. (See Dave's "Roman Numerals and Bash" and "More Roman Numerals and Bash".)

The trick with Roman numerals, however, is that it's what's known as a subtractive notation. In other words, it's not a position → value or even symbol → value notation, but a sort of hybrid. MM = 2000, and C = 100, but MMC and MCM are quite different: the former is 2100, and the latter is 1000 + (–100 + 1000) = 1900.

This means that the conversion isn't quite as simple as a mapping table, which makes it a good homework assignment for young comp-sci students!

Let's Write Some Code

In the Roman numeral to decimal conversion, a lot of the key work was done by this simple function:


mapit() {
   case $1 in
     I|i) value=1 ;;
     V|v) value=5 ;;
     X|x) value=10 ;;
     L|l) value=50 ;;
     C|c) value=100 ;;
     D|d) value=500 ;;
     M|m) value=1000 ;;
      * ) echo "Error: Value $1 unknown" >&2 ; exit 2 ;;
   esac
}

You'll need this function to proceed, but as a cascading set of conditional statements. Indeed, in its simple form, you could code a decimal to Roman numeral converter like this:


while [ $decvalue -gt 0 ] ; do

  if [ $decvalue -gt 1000 ] ; then
    romanvalue="$romanvalue M"
    decvalue=$(( $decvalue - 1000 ))
  elif [ $decvalue -gt 500 ] ; then
    romanvalue="$romanvalue D"
    decvalue=$(( $decvalue - 500 ))
  elif [ $decvalue -gt 100 ] ; then
    romanvalue="$romanvalue C"
    decvalue=$(( $decvalue - 100 ))
  elif [ $decvalue -gt 50 ] ; then
    romanvalue="$romanvalue L"
    decvalue=$(( $decvalue - 50 ))
  elif [ $decvalue -gt 10 ] ; then
    romanvalue="$romanvalue X"
    decvalue=$(( $decvalue - 10 ))
  elif [ $decvalue -gt 5 ] ; then
    romanvalue="$romanvalue V"
    decvalue=$(( $decvalue - 5 ))
  elif [ $decvalue -ge 1 ] ; then
    romanvalue="$romanvalue I"
    decvalue=$(( $decvalue - 1 ))
  fi

done

This actually works, though the results are, um, a bit clunky:


$ sh 2roman.sh 25
converts to roman numeral  X X I I I I I

Or, more overwhelming:

Go to Full Article