HOW-TOs

Oops! Debugging Kernel Panics

Petros Koutoupis — Wed, 07 Aug 2019 23:30:00 +0000

A look into what causes kernel panics and some utilities to help gain more information.

Working in a Linux environment, how often have you seen a kernel panic? When it happens, your system is left in a crippled state until you reboot it completely. And, even after you get your system back into a functional state, you're still left with the question: why? You may have no idea what happened or why it happened. Those questions can be answered though, and the following guide will help you root out the cause of some of the conditions that led to the original crash.

Figure 1. A Typical Kernel Panic

Let's start by looking at a set of utilities known as kexec and kdump. kexec allows you to boot into another kernel from an existing (and running) kernel, and kdump is a kexec-based crash-dumping mechanism for Linux.

Installing the Required Packages

First and foremost, your kernel should have the following components statically built in to its image:


CONFIG_RELOCATABLE=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_DEBUG_INFO=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_PROC_VMCORE=y

You can find this in /boot/config-`uname -r`.

Make sure that your operating system is up to date with the latest-and-greatest package versions:


$ sudo apt update && sudo apt upgrade

Install the following packages (I'm currently using Debian, but the same should and will apply to Ubuntu):


$ sudo apt install gcc make binutils linux-headers-`uname -r`
 ↪kdump-tools crash `uname -r`-dbg

Note: Package names may vary across distributions.

During the installation, you will be prompted with questions to enable kexec to handle reboots (answer whatever you'd like, but I answered "no"; see Figure 2).

Figure 2. kexec Configuration Menu

And to enable kdump to run and load at system boot, answer "yes" (Figure 3).

Figure 3. kdump Configuration Menu

Configuring kdump

Open the /etc/default/kdump-tools file, and at the very top, you should see the following:

Go to Full Article

Digging Through the DevOps Arsenal: Introducing Ansible

Petros Koutoupis — Wed, 07 Aug 2019 18:15:00 +0000

by Petros Koutoupis

If you need to deploy hundreds of server or client nodes in parallel, maybe on-premises or in the cloud, and you need to configure each and every single one of them, what do you do? How do you do it? Where do you even begin? Many configuration management frameworks exist to address most, if not all, of these questions and concerns. Ansible is one such framework.

You may have heard of Ansible already, but for those who haven't or don't know what it is, Ansible is a configuration management and provisioning tool. (I'll get to exactly what that means shortly.) It's very similar to other tools, such as Puppet, Chef and Salt.

Why use Ansible? Well, because it's simple to master. I don't mean that the others are not simple, but Ansible makes it easy for individuals to pick up quickly. That's because Ansible uses YAML as its base to provision, configure and deploy. And because of this approach, tasks are executed in a specific order. During execution, if you trip over a syntax error, it will fail once you hit it, potentially making it easier to debug.

Now, what's YAML? YAML (or YAML Ain't Markup Language) is a human-readable data-serialization language mostly used to capture configuration files. You know how JSON is easier to implement and use over XML? Well, YAML takes a more simplistic approach than JSON. Here's an example of a typical YAML structure containing a list:


data:
    - var1:
        a: 1
        b: 2
    - var2:
        a: 1
        b: 2
        c: 3

Now, let's swing back to Ansible. Ansible is an open-source automation platform freely available for Linux, macOS and BSD. Again, it's very simple to set up and use, without compromising any power. Ansible is designed to aid you in configuration management, application deployment and the automation of assorted tasks. It works great in the realm of IT orchestration, in which you need to run specific tasks in sequence and create a chain of events that must happen on multiple and different servers or devices.

Here's a good example: say you have a group of web servers behind a load balancer. You need to upgrade those web servers, but you also need to ensure that all but one server remains online for the upgrade process. Ansible can handle such a complex task.

Ansible uses SSH to manage remote systems across the network, and those systems are required to have a local installation of not only SSH but also Python. That means you don't have to install and configure a client-server environment for Ansible.

Install Ansible

Although you can build the package from source (either from the public Git repository or from a tarball), most modern Linux distributions will have binary packages available in their local package repositories. You need to have Ansible installed on at least one machine (your control node). Remember, all that's required on the remote machines are SSH and Python.

Go to Full Article

Build a Versatile OpenStack Lab with Kolla

John S. Tonello — Wed, 07 Aug 2019 17:30:00 +0000

by John S. Tonello

Hone your OpenStack skills with a full deployment in a single virtual machine.

It's hard to go anywhere these days without hearing something about the urgent need to deploy on-premises cloud environments that are agile, flexible and don't cost an arm and a leg to build and maintain, but getting your hands on a real OpenStack cluster—the de facto standard—can be downright impossible.

Enter Kolla-Ansible, an official OpenStack project that allows you to deploy a complete cluster successfully—including Keystone, Cinder, Neutron, Nova, Heat and Horizon—in Docker containers on a single, beefy virtual machine. It's actually just one of an emerging group of official OpenStack projects that containerize the OpenStack control plane so users can deploy complete systems in containers and Kubernetes.

To date, for those who don't happen to have a bunch of extra servers loaded with RAM and CPU cores handy, DevStack has served as the go-to OpenStack lab environment, but it comes with some limitations. Key among those is your inability to reboot a DevStack system effectively. In fact, rebooting generally bricks your instances and renders the rest of the stack largely unusable. DevStack also limits your ability to experiment beyond core OpenStack modules, where Kolla lets you build systems that can mimic full production capabilities, make changes and pick up where you left off after a shutdown.

In this article, I explain how to deploy Kolla, starting from the initial configuration of your laptop or workstation, to configuration of your cluster, to putting your OpenStack cluster into service.

Why OpenStack?

As organizations of all shapes and sizes look to speed development and deployment of mission-critical applications, many turn to public clouds like Amazon Web Services (AWS), Microsoft Azure, Google Compute Engine, RackSpace and many others. All make it easy to build the systems you and your organization need quickly. Still, these public cloud services come at a price—sometimes a steep price you only learn about at the end of a billing cycle. Anyone in your organization with a credit card can spin up servers, even ones containing proprietary data and inadequate security safeguards.

OpenStack, a community-driven open-source project with thousands of developers worldwide, offers a robust, enterprise-worthy alternative. It gives you the flexibility of public clouds in your own data center. In many ways, it's also easier to use than public clouds, particularly when OpenStack administrators properly set up networks, carve out storage and compute resources, and provide self-service capabilities to users. It also has tons of add-on capabilities to suit almost any use case you can imagine. No wonder 75% of private clouds are built using OpenStack.

Go to Full Article

Writing GitHub Web Hooks with Bash

Andy Carlson — Wed, 07 Aug 2019 16:30:00 +0000

by Andy Carlson

Bring your GitHub repository to the next level of functionality.

For the past year since Microsoft has acquired GitHub, I've been hosting my Git repositories on a private server. Although I relished the opportunity and challenge of setting it all up, and the end product works well for my needs, doing this was not without its sacrifices. GitHub offers a clean interface for configuring many Git features that otherwise would require more time and effort than simply clicking a button. One of the features made easier to implement by GitHub that I was most fond of was web hooks. A web hook is executed when a specific event occurs within the GitHub application. Upon execution, data is sent via an HTTP POST to a specified URL.

This article walks through how to set up a custom web hook, including configuring a web server, processing the POST data from GitHub and creating a few basic web hooks using Bash.

Preparing Apache

For the purpose of this project, let's use the Apache web server to host the web hook scripts. The module that Apache uses to run server-side shell scripts is mod_cgi, which is available on major Linux distributions.

Once the module is enabled, it's time to configure the directory permissions and virtual host within Apache. Use the /opt/hooks directory to host the web hooks, and give ownership of this directory to the user that runs Apache. To determine the user running an Apache instance, run the following command (provided Apache is currently running):


ps -e -o %U%c| grep 'apache2\|httpd'

These commands will return a two-column output containing the name of the user running Apache and the name of the Apache binary (typically either httpd or apache2). Grant directory permission with the following chown command (where USER is the name of the user shown in the previous ps command):


chown -R USER /opt/hooks

Within this directory, two sub-directories will be created: html and cgi-bin. The html folder will be used as a web root for the virtual host, and cgi-bin will contain all shell scripts for the virtual host. Be aware that as new sub-directories and files are created under /opt/hooks, you may need to rerun the above chown to verify proper access to files and sub-directories.

Here's the configuration for the virtual host within Apache:

Go to Full Article

Words, Words, Words--Introducing OpenSearchServer

Marcel Gagné — Wed, 07 Aug 2019 13:46:43 +0000

by Marcel Gagné

How to create your own search engine combined with a crawler that will index all sorts of documents.

In William Shakespeare's Hamlet, one of my favorite plays, Prince Hamlet is approached by Polonius, chief counselor to Claudius, King of Denmark, who happens to be Hamlet's stepfather, and uncle, and the new husband of his mother, Queen Gertrude, whose recently deceased last husband was the previous King of Denmark. That would be Hamlet's biological father for those who might be having trouble following along. He was King Hamlet. Polonius, I probably should mention, is also the father of Hamlet's sweetheart, Ophelia. Despite this hilarious sounding setup, Hamlet is most definitely not a comedy. (Note: if you need a refresher, you can read Hamlet here.)

For reasons I won't go into here, Hamlet is doing a great job of trying to convince people that he's completely lost it and is pretending to be reading a book when Polonius approaches and asks, "What do you read, my lord?"

Hamlet replies by saying, "'Words, words, words." In other words, ahem, nothing of any importance, you annoying little man.

Shakespeare wrote a lot of words. In fact, writers, businesses and organizations of any size tend to amass a lot of words in the form of countless documents, many of which seem to contain a great deal of importance at the time they are written and subsequently stored on some lonely corporate server. There, locked in their digital prisons, these many texts await the day when somebody will seek out their wisdom. Trouble is, there are so many of them, in many different formats, often with titles that tell you nothing about the content inside. What you need is a search engine.

Google is a pretty awesome search engine, but it's not for everybody, especially if the documents in question aren't meant for consumption by the public at large. For those times, you need your own search engine, combined with a crawler that will index all sorts of documents, from OpenDocument format, to old Microsoft Docs, to PDFs and even plain text. That's where OpenSearchServer comes into play. OpenSearchServer is, as the name implies, an open-source project designed to perform the function of crawling through and indexing large collections of documents, such as you would find on a website.

I'm going to show you how to go about getting this documentation site set up from scratch so that you can see all the steps. You may, of course, already have a web server up and running, and that's fine. I've gone ahead and spun up a Linode server running Ubuntu 18.04 LTS. This is a great way to get a server up and running quickly without spending a lot of money if you don't want to, and if you've never done this, it's also kind of fun.

Go to Full Article

Bash Shell Games: Let's Play Go Fish!

Dave Taylor — Tue, 30 Jul 2019 12:00:00 +0000

by Dave Taylor

How to begin developing a computer version of the popular card game.

Between the previous 163 columns I've written here in Linux Journal and the dozens of games I programmed and explored during the creation of my Wicked Cool Shell Scripts book, I've written a lot of Bash shell games. The challenge is to find one that's simple enough where a shell script will work, but isn't so simple that it ends up being only a half-dozen lines.

Magic 8-Ball is a perfect example. It turns out that the entire "predict the future" gizmo was really just a 20-sided die floating in dark purple fluid. So an array of 20 possible values and a random number selector and boom—you've got a magic 8-ball script:


#!/bin/sh

# magic 8 ball. Yup. Pick a random number, output message

# messages harvested from the Wikipedia entry

answers=("It is certain." "It is decidedly so."
  "Without a doubt." "Yes - definitely."
  "You may rely on it." "As I see it, yes." "Most likely."
  "Outlook good." "Yes." "Signs point to yes."
  "Reply hazy, try again." "Ask again later."
  "Better not tell you now." "Cannot predict now."
  "Concentrate and ask again." "Don't count on it."
  "My reply is no." "My sources say no."
  "Outlook not so good." "Very doubtful.")

echo "Oh! Magic 8 Ball, Please Tell Me True..." ; echo ""
/bin/echo -n "What is your question? "
read question

answer=$(( $RANDOM % 20 ))

echo ""
echo "I have looked into the future and I say: "
echo "     ${answers[$answer]}" ; echo ""

exit 0

Let's do a quick run to see if I'm the most popular LJ writer:


$ sh magic8.sh
Oh! Magic 8 Ball, Please Tell Me True...

What is your question? Am I the most popular LJ writer?

I have looked into the future and I say:
     My reply is no.

Ouch, that's harsh. I write the darn divination program, and it just drops a brick on my foot. Yeesh.

More seriously, Magic 8 Ball is too simple to make an interesting shell script. By contrast, Call of Duty is way too complex, even if I did a version with text output instead of gorgeously rendered 3D graphics.

Card Game Function Library

That's why card games prove to be good as programming challenges or exercises: the core mechanism of a 52-card random deck is pretty straightforward, so it's all about the actual cardplay.

Not only that, but as I've written before about card games as shell scripts, I already have a handy set of functions to create, shuffle and display cards out of a deck. If you want to rummage in the archives, I've tackled Acey-Deucey, Baccarat and some bits and pieces of Cribbage.

In order to jump right into the new game that I'm going to describe how to build, Go Fish!, let's steal the following functions from my earlier scripts:

Go to Full Article

What Does It Take to Make a Kernel?

Petros Koutoupis — Tue, 23 Jul 2019 12:00:00 +0000

by Petros Koutoupis

The kernel this. The kernel that. People often refer to one operating system's kernel or another without truly knowing what it does or how it works or what it takes to make one. What does it take to write a custom (and non-Linux) kernel?

So, what am I going to do here? In June 2018, I wrote a guide to build a complete Linux distribution from source packages, and in January 2019, I expanded on that guide by adding more packages to the original guide. Now it's time to dive deeper into the custom operating system topic. This article describes how to write your very own kernel from scratch and then boot up into it. Sounds pretty straightforward, right? Now, don't get too excited here. This kernel won't do much of anything. It'll print a few messages onto the screen and then halt the CPU. Sure, you can build on top of it and create something more, but that is not the purpose of this article. My main goal is to provide you, the reader, with a deep understanding of how a kernel is written.

Once upon a time, in an era long ago, embedded Linux was not really a thing. I know that sounds a bit crazy, but it's true! If you worked with a microcontroller, you were given (from the vendor) a specification, a design sheet, a manual of all its registers and nothing more. Translation: you had to write your own operating system (kernel included) from scratch. Although this guide assumes the standard generic 32-bit x86 architecture, a lot of it reflects what had to be done back in the day.

The exercises below require that you install a few packages in your preferred Linux distribution. For instance, on an Ubuntu machine, you will need the following:

binutils
gcc
grub-common
make
nasm
xorriso

An Extreme Crash Course into the Assembly Language

Note: I'm going to simplify things by pretending to work with a not-so-complex 8-bit microprocessor. This doesn't reflect the modern (and possibly past) designs of any commercial processor.

Go to Full Article

Oracle Linux on Btrfs for the Raspberry Pi

Charles Fisher — Mon, 22 Jul 2019 11:30:00 +0000

by Charles Fisher

Enterprise comes to the micro server.

Oracle Linux 7 has been released for the Raspberry Pi 3. The release packages Btrfs as the root filesystem on the UEK-branded Linux 4.14 Long Term Support (LTS) kernel. A bootable disk image with a minimal install is provided along with a standard ISO installer.

CentOS appears to support only the "Mustang" Applied Micro X-Gene for AArch64, and it provides the older AArch32 environment for all models of the Raspberry Pi. Oracle Linux is a compelling option among RPM distributions in supporting AArch64 for the Pi Model 3.

This is not to say that Oracle AArch64 Linux is without flaw, as Oracle warns that this is "a preview release and for development purposes only; Oracle suggests these not be used in production." The non-functional WiFi device is missing firmware and documentation, which Oracle admits was overlooked. No X11 graphics are included in the image, although you can install them. The eponymous database client (and server) are absent. Oracle has provided a previous example of orphaned software with its Linux for SPARC project, which was abandoned after two minor releases. There's no guarantee that this ARM version will not suffer the same fate, although Oracle has responded that "our eventual target is server class platforms". One possible hardware target is the Fujitsu A64FX, a new server processor that bundles 48 addressable AArch64 cores and 32GB of RAM on one die, asserted to be the "fastest server processor" that exists.

AArch64 on the Pi

You'll need a Raspberry Pi Model 3 to run Oracle Linux. The 3B+ is the best available device, and you should choose that over the predecessor Model 3B and all other previous models. Both Model 3 boards retain the (constraining) 1GB of RAM—a SODIMM socket would be far more practical. The newer board has a CPU that is 200MHz faster and a Gigabit-compatible Ethernet port (that is limited to 300Mbit due to the USB2 linkage that connects it). A Model A also exists, but it lacks many of the ports on the 3B. More important, the Model 3 platform introduces a 64-bit CPU.

Go to Full Article

Arduino from the Command Line: Break Free from the GUI with Git and Vim!

Matthew Hoskins — Tue, 16 Jul 2019 11:30:00 +0000

by Matthew Hoskins

Love Arduino but hate the GUI? Try arduino-cli.

In this article, I explore a new tool released by the Arduino team that can free you from the existing Java-based Arduino graphical user interface. This allows developers to use their preferred tools and workflow. And perhaps more important, it'll enable easier and deeper innovation into the Arduino toolchain itself.

The Good-Old Days

When I started building hobby electronics projects with microprocessors in the 1990s, the process entailed a discrete processor, RAM, ROM and masses of glue logic chips connected together using a point-to-point or "wire wrapping" technique. (Look it up kids!) Programs were stored on glass-windowed EPROM chips that needed to be erased under UV light. All the tools were expensive and difficult to use, and development cycles were very slow. Figures 1–3 show some examples of my mid-1990s microprocessor projects with discrete CPU, RAM and ROM. Note: no Flash, no I/O, no DACs, no ADCs, no timers—all that means more chips!

Figure 1. Example Mid-1990s Microprocessor

Figure 2. Example Mid-1990s Microprocessor

Figure 3. Example Mid-1990s Microprocessor

It all changed in 2003 with Arduino.

The word "Arduino" often invokes a wide range of opinions and sometimes emotion. For many, it represents a very low bar to entry into the world of microcontrollers. This world before 2003 often required costly, obscure and closed-source development tools. Arduino has been a great equalizer, blowing the doors off the walled garden. Arduino now represents a huge ecosystem of hardware that speaks a (mostly) common language and eases transition from one hardware platform to another. Today, if you are a company that sells microcontrollers, it's in your best interest to get your dev boards working with Arduino. It offers a low-friction path to getting your products into lots of hands quickly.

It's also important to note that Arduino's simplicity does not inhibit digging deep into the microcontroller. Nothing stops you from directly twiddling registers and using advanced features. It does, however, decrease your portability between boards.

Go to Full Article

An AI Wizard of Words

Marcel Gagné — Mon, 15 Jul 2019 11:00:00 +0000

by Marcel Gagné

A look at using OpenAI's Generative Pretrained Transformer 2 (GPT-2) to generate text.

It's probably fair to say that there's more than one person out there who is worried about some version of artificial intelligence, or AI, possibly in a robot body of some kind, taking people's jobs. Anything that is repetitive or easily described is considered fair game for a robot, so driving a car or working in a factory is fair game.

Until recently, we could tell ourselves that people like yours truly—the writers and those who create things using some form of creativity—were more or less immune to the march of the machines. Then came GPT-2, which stands for Generative Pretrained Transformer 2. I think you'll agree, that isn't the sexiest name imaginable for a civilization-ending text bot. And since it's version 2, I imagine that like Star Trek's M-5 computer, perhaps GPT-1 wasn't entirely successful. That would be the original series episode titled, "The Ultimate Computer", if you want to check it out.

So what does the name "GPT-2" stand for? Well, "generative" means pretty much what it sounds like. The program generates text based on a predictive model, much like your phone suggests the next word as you type. The "pretrained" part is also quite obvious in that the model released by OpenAI has been built and fine-tuned for a specific purpose. The last word, "Transformer", refers to the "transformer architecture", which is a neural network design architecture suited for understanding language. If you want to dig deeper into that last one, I've included a link from a Google AI blog that compares it to other machine learning architecture (see Resources).

On February 14, 2019, Valentine's Day, OpenAI released GPT-2 with a warning:

Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper.

I've included a link to the blog in the Resources section at the end of this article. It's worth reading partly because it demonstrates a sample of what this software is capable of using the full model (see Figure 1 for a sample). We already have a problem with human-generated fake news; imagine a tireless machine capable of churning out vast quantities of news and posting it all over the internet, and you start to get a feel for the dangers. For that reason, OpenAI released a much smaller model to demonstrate its capabilities and to engage researchers and developers.

Go to Full Article