This article is sponsored by Technologic Systems.

Here are some tips compiled from our seasoned engineers on what they wish they'd known about embedded Linux back when they were "newbs". Newcomers and seasoned veterans alike should get some good nuggets of information and possibly a fun perspective looking back at our own humble beginnings. We'll try not to overwhelm you as we make our way through the list. We're not here to rewrite the books, but we do want to provide a personal perspective. If you're in the camp of people who've been using desktop Linux, just be aware that embedded Linux is a different animal, especially when it comes to space constraints, different CPU architecture (ARM), resilience to sudden power outages and inability to install any mainline Linux kernel or distribution you please. Or, maybe you're in the microprocessor camp moving toward a more generalized and capable embedded Linux system. Either way, we'll assume you have at least some knowledge of Linux as we walk through this guide.

Getting Around

One of the first things you've probably already come to notice in an embedded Linux product is the lack of a nice desktop environment. No GUI here, my friend! Here, the command line is king. Just as you're probably very proficient with a mouse, you'll come to develop muscle memory to make you as proficient if not more using a keyboard. Right now, you're probably feeling more like this finger-typer here, and that's okay! We're here to tell you things are going to get better.

Serial Console (RS-232)

One of the first things you'll use is a serial console. Yes, "serial console" as in the archaic Microsoft Hyperterminal program to connect to your board to get a command prompt, or shell in Linux-speak. The serial console is the most trusted method of communicating with the board since it's not prone to network failure as is SSH and Telnet, plus you'll be able to see both startup and shutdown messages. For connecting via serial console, take a look at minicom (Linux/Mac/Cygwin) or PuTTY (Windows). A serial console must be configured to talk at the same rate of speed as the board you're connecting to. It seems the most common is 115200 Bps with no parity bit and no hardware or software flow control. This is what minicom setup (minicom -s) looks like on a Mac:

You'll need some parts for this: a null modem cable and a serial port on your computer. If you don't have one, you can get a USB to Serial converter.

Pro tip: resize (if available) is amazing for serial consoles that don't auto-grab the terminal size.

Understanding UARTS like RS-232, RS-485

It's advisable to brush up on your UART communication skills, since connecting to RS-232 serial console is much the same as connecting to a standard TTL-level UART or RS-485. The following are some resources that may help your understanding:

• Universal asynchronous receiver/transmitter

• Serial Communication

• Serial RS232 connections in Python

Secure Shell

A heavily accessed tool in the toolbelt is secure shell, and you'll want to be sure to become efficient with it. It comes down to two main commands that use the same underlying technology: SSH and SCP.

SSH

SSH allows you to connect to your board's shell prompt using the network. It uses port 22 by default, and you can set up port forwarding in your router to allow for remote access to the board. Typical usage is very simple:

1 ssh @
2 ssh root@192.168.1.50

SCP

SCP can be thought of as "Secure Copy". This allows you to transfer files to and from your board. If you can ssh to your board, you can copy files. Typical usage:

scp  @:
scp foobar.txt root@192.168.1.50:/root/

Screen and Tmux

Instead of having multiple windows open to the same board, try using screen or tmux. Each of these programs can provide multiple shells for a single connection with the added bonus of keeping your shells active and open even when your connection is closed. Screen usually comes preloaded, is smaller than tmux and is usually sufficient. Tmux is more powerful and has its own advantages. Since screen is so readily available, it's nice to dedicate muscle memory to it.

Editing Files

Since you'll be spending a lot of time in the command line, you'll want to try to become proficient at using a non-GUI editor. There are two main contenders here: vim and Emacs. Do a Google search for vim vs. Emacs and you'll quickly find there are two separate camps, each very passionate about their preferred editor. The engineers here at Technologic Systems are mostly in the vim camp, but encourage you to decide for yourself which is right for you.

Vi and Vim

Most embedded Linux boards will come preloaded with at least vi, the original vim. Vim is simply "vi improved". A vim install takes a bit more storage, but in this author's opinion, it's totally worth every extra byte.

[Note: Derek Wyatt has a fantastic and entertaining set of videos for learning Vim, from basics to advanced usage.]

Emacs

Emacs has a more traditional word editor feel than vim, which many beginners find comforting. If it's not installed by default, it easily can be as it's in most package management systems like apt-get and yum.

There's a guided tour of Emacs at GNU.org if you're interested in learning more.

Disk and File Corruption

You've seen this message before. Admit it, you've unplugged a disk without ejecting it first because you think that message is for chumps, and you've never had any problems with data on your USB thumbdrive. In the embedded Linux world, this is an absolute must, as disks are prone to data and filesystem corruption when the carpet is pulled from underneath. This includes sudden power outages. You can attempt to use fsck to repair the disk, which works for a while, but you'll inevitably find yourself in a situation where it cannot repair the disk any longer. We've written a whitepaper on filesystem corruption and ways to prevent it from happening.

A utility to get in the habit of using is sync, which commits buffer cache to disk. You can read more about it in the sync man page.

NFS

Standing for Network File System, NFS is a great tool to use for accessing files between two computers on a network. The definition is simple, but the implications are many. For example, you can set up (or export) an NFS share containing a tarball of a Linux distribution and set up your board to load it. You can also set up an NFS share to edit your project files in an IDE on your desktop machine and then compile it on the embedded board without the need for manual copy/paste. The Ubuntu community has a really nice article on this: "Setting Up NFS How To".

Programming

Whoa. What an overwhelming topic. It's an exciting one, and certainly important, but there are so many different languages, platforms, memory addresses, processors, styles, etc., it'd be very difficult not to overwhelm a reader in this article. Let's just say that in the embedded Linux world, the language of choice is usually C because of low-level hardware optimization. In fact, the Linux kernel is written in C, a decision that Linux creator Linus Torvalds has a very non-apologetic stance on. Because of limited resources, efficient and fast (aka inexpensive) code is sought after. For example, floating point arithmetic is generally frowned upon because it involves a lot of layers, including the kernel, to make floating point happen, which is very expensive and time consuming. If your CPU has a hardware floating point unit, then it's not really a problem.

One pitfall newbies might fall into is NIH (not invented here) syndrome, where they feel they have to re-invent the wheel. In actuality, there are a ton of existing resources and existing libraries (such as key/value storage like Berkeley DB, send/receive data via HTTP or FTP). A new development project ought to be preceded by an extensive search or research phase before jumping in. Some Linux API calls exist that a lot of people don't realize, like sched_setscheduler(), mlock(), mlockall(), etc. Also, having an understanding about /dev/mem makes development in userspace much easier and forgoes the need to develop kernel drivers. Resist the temptation to put more and more of your embedded application into the kernel and stick with the bare minimum and you'll almost always be better off in the long term. Along those lines, don't think of your device as a monolithic application and instead separate things into processes. Not everything needs to be optimized, so don't be afraid to use tmpfs filesystem or pipes to communicate or shell scripts. Really, it comes down to not underestimating the virtue of simplicity. Use layers, but use them strategically, being sure not to add them until necessary, else your system almost always will get slower, have difficulty troubleshooting failure modes and result in higher demands on hardware. Layering is an under-utilized advantage of embedded programmers coming from microcontrollers and over-utilized by developers coming from a PC or sysadmin background.

ctags and cscope

If you're doing anything related to software development, be sure to take the time to learn about ctags and cscope. These give you a method for jumping to and from function and variable declarations across your project files and can work together. Both vim and Emacs work with ctags (etags for Emacs). You can learn more about ctags and cscope by reading our guide, Tag Jumping with ctags and cscope.

Kernel Compiling

Eventually in your embedded Linux travels, you'll come across some instructions that say you need a particular kernel module. When you find out your kernel doesn't have it, it's intimidating to think of what you'll need to do in order to compile it yourself from scratch. But, it's not actually as scary as you might think, and it's a very good way to improve your understanding of Linux. Each single board computer or computer on module will have slightly different instructions and different versions of the kernel, and Technologic Systems' engineers have worked hard to write guides on several of their products. The guides aren't incredibly long, and they go over things like choosing the module to compile via make menuconfig and inserting the module via insmod when you've successfully compiled them. Take a look at the following guides as a starting point, and then reach out to a support channel to find out more specific information about your board:

• TS-7680 kernel compile guide

• 75XX Kernel Compile

• TS-4900 Kernel Compile Guide

NAND Flash Considerations

NAND flash is really fast, but don't write to it a lot as it's not as fantastic as it seems. eMMC flash is more reliable, doubly so when configured in SLC mode.

Checking Space

Disk space is usually constrained in an embedded Linux device. You may have the ability to increase storage using an SD card or USB thumbdrive, but likely, you'll be booting from an onboard flash drive that might have only 4GB or less. Make sure you're aware of file sizes and their units—for example, MB is different than MiB (MiB = Mebibyte = 1024KiB; MB = Megabyte = 1,000KB). Commands like du and df help identify how the storage is being utilized. You also can remove and purge unnecessary locales or language translations to free up space. If you're in a Debian-like environment, be sure to use tools like apt-get clean and autoremove to keep things on the up and up.

Reading/Writing Images

Undoubtedly, you'll need to restore a factory image or load a prepared image onto your embedded Linux device. Usually the preferred tool for generating these images is dd. We have a guide that will walk you through how to do this in various platforms. See How to Write an SD Card Image (Linux, Windows, Mac OSX).

Distributions and Tarballs

Since Linux is a file-based OS, you can create a tarball of your distribution and extract it either to a prepared partition or NFS filesystem and it'll boot. You can create a tarball using a command similar to:

tar -cvpf /root.tar --directory=/ --exclude=proc --exclude=sys
↪--exclude=dev/pts --exclude=backups .

Recommended Reading

Let's switch to a first-person view here for a moment. When I first started in embedded Linux, this book was an invaluable resource for me. I'd recommend this be on the required reading list for newly hired employees or interns. It explains in very simple terms exactly what's happening in an embedded Linux platform, much like the products we sell at Technologic Systems. So, please take a look and read it: Embedded Linux Primer: A Practical Real-World Approach by Christopher Hallinan.

Conclusion

Hopefully you feel like you've come away with some great tips to get you going in the embedded Linux world. There's still so much to learn, so get out there and start exploring!