Databases

Reality 2.0 Episode 24: A Chat About Redis Labs (Podcast Transcript)

Katherine Druckman — Fri, 02 Aug 2019 15:40:07 +0000

Doc Searls and Katherine Druckman talk to Yiftach Shoolman of Redis Labs about Redis, Open Source licenses, company culture and more.

Listen to the podcast here.

Katherine Druckman: Hey, Linux Journal readers, I am Katherine Druckman, joining you again for our awesome, cool podcast. As always, joining us is Doc Searls, our editor-in-chief. Our special guest this time is Yiftach Shoolman of Redis Labs. He is the CTO and co-founder, and he was kind enough to join us. We’ve talked a bit, in preparation for the podcast, about Redis Labs, but I wondered if you could just give us sort of an overview for the tiny fraction of the people listening that don’t know all about Redis Labs and Redis. If you could just give us a little brief intro, that’d be great.

Yiftach Shoolman: Thank you very much for hosting me, first. Redis is an extremely popular in-memory data structure database that’s used by many people as just a caching system, but many of them have shifted from just simple cache to a real database, even in the open source world. Just in terms of numbers, only on Docker Hub, Redis has been launched for almost 1.8 billion times, something like five million every day, so it’s extremely popular. It’s used everywhere. Redis Labs is the company behind the open source. When I say “behind the open source,” we sponsor, I would say, 99% of all the open source activities, if not 100%. We also have enterprise products, which is called Redis Enterprise.

It is available as a cloud service on all the public clouds, as well as a fully-managed Redis cloud service, as well as softwares that you can download and install everywhere. This is our story in general. The way we split between open source and commercial, which is today very tricky, is that we keep the Redis core as open-core BSD, by the way. On top of that, we added what we call enterprise layers that allows Redis to be deployed in an enterprise environment in the most scalable and highly available way. We have all the goodies that you need, including active-active, including data persistence layer, etc., all the boring stuff that the enterprise needs, in addition to that, a lot of security features. In addition to that, we extended Redis with what we call modules. Some of them were initially open source, and then we changed the license. This is probably the reason that you called me.

Katherine Druckman: Right. That was in the news, certainly.

Go to Full Article

FOSS Project Spotlight: Appaserver

Tim Riley — Fri, 14 Dec 2018 13:00:00 +0000

by Tim Riley

An introduction to an application server that allows you to build MySQL user interfaces without programming.

Assume you are tasked to write a browser-based, MySQL user interface for the table called CITY. CITY has two columns. The column names are city_name and state_code—each combined are the primary key.

Your user interface must enable users to execute the four main SQL operations: select, insert, update and delete. The main characteristics for each operation are:

The select operation needs an HTML prompt form to request a query. It also needs a where clause generator to select from CITY. After forking MySQL and retrieving the raw rows, it needs to translate them into an HTML table form.
The HTML table form needs to be editable, and user edits need to be translated into update statements.
Each resulting row following the execution of a query is a candidate for deletion.
The insert operation needs a blank form. It also needs to translate Apache's common gateway interface (CGI) into insert statements.

So, you might create the source file called city.c and type in all the required code. Of course, relational databases have relations. One city has many persons residing in it. Assume the PERSON table has the column names of full_name, street_address, city_name and state_code. full_name and street_address combined are the primary key (Figure 1).

Figure 1. Database Schema of Many Persons Residing in One City

Are you going to create the source file called person.c too? What about customer.c, inventory.c, order.c, ...?

Alternatively, you might create the source files called select.c, insert.c, update.c and delete.c. Then each of these modules would need as input:

A single table name.
The table's additional attributes.
The table's column names and additional attributes.
A recursive list of related tables.
Apache's CGI dictionary output.

The principle behind Appaserver is this multi-module approach. Appaserver stores table names in a table. Each table's column names and relations are also stored in tables. Taking the table-driven concept to the nth degree forms a database of a database. You can glean a detailed understanding of how the Appaserver database is modeled from https://appahost.com/appaserver_database_schema.pdf.

Go to Full Article

PostgreSQL 10: a Great New Version for a Great Database

Reuven M. Lerner — Mon, 05 Mar 2018 16:38:22 +0000

by Reuven M. Lerner

Reuven reviews the latest and most interesting features in PostgreSQL 10.

PostgreSQL has long claimed to be the most advanced open-source relational database. For those of us who have been using it for a significant amount of time, there's no doubt that this is true; PostgreSQL has consistently demonstrated its ability to handle high loads and complex queries while providing a rich set of features and rock-solid stability.

But for all of the amazing functionality that PostgreSQL offers, there have long been gaps and holes. I've been in meetings with consulting clients who currently use Oracle or Microsoft SQL Server and are thinking about using PostgreSQL, who ask me about topics like partitioning or query parallelization. And for years, I've been forced to say to them, "Um, that's true. PostgreSQL's functionality in that area is still fairly weak."

So I was quite excited when PostgreSQL 10.0 was released in October 2017, bringing with it a slew of new features and enhancements. True, some of those features still aren't as complex or sophisticated as you might find in commercial databases. But they do demonstrate that over time, PostgreSQL is offering an amazing amount of functionality for any database, let alone an open-source project. And in almost every case, the current functionality is just the first part of a long-term roadmap that the developers will continue to follow.

In this article, I review some of the newest and most interesting features in PostgreSQL 10—not only what they can do for you now, but what you can expect to see from them in the future as well. If you haven't yet worked with PostgreSQL, I'm guessing you'll be impressed and amazed by what the latest version can do. Remember, all of this comes in an open-source package that is incredibly solid, often requires little or no administration, and which continues to exemplify not only high software quality, but also a high-quality open-source project and community.

PostgreSQL Basics

If you're new to PostgreSQL, here's a quick rundown: PostgreSQL is a client-server relational database with a large number of data types, a strong system for handling transactions, and functions covering a wide variety of tasks (from regular expressions to date calculations to string manipulation to bitwise arithmetic). You can write new functions using a number of plugin languages, most commonly PL/PgSQL, modeled loosely on Oracle's PL/SQL, but you also can use languages like Python, JavaScript, Tcl, Ruby and R. Writing functions in one of these extension languages provides you not only with the plugin language's syntax, but also its libraries, which means that if you use R, for example, you can run statistical analyses inside your database.

Go to Full Article

SQL Server on Linux

John S. Tonello — Thu, 15 Jun 2017 20:54:12 +0000

by John S. Tonello

When Wim Coekaerts, Microsoft's vice president for open source, took the stage at LinuxCon 2016 in Toronto last summer, he came not as an adversary, but as a longtime Linux enthusiast promising to bring the power of Linux to Microsoft and vice versa. With the recent launch of SQL Server for Linux, Coekaerts is clearly having an impact.

PowerShell for Linux and bash for Windows heralded the beginning, but the arrival of SQL Server, one of the most popular relational databases out there, offers Linux shops some real opportunities—and a few conundrums.

Clearly, the opportunity to deploy SQL Server on something other than a Windows Server means you can take advantage of the database's capabilities without having to manage Windows hosts to do it. If you're a mostly-Linux shop (or want to be) and you have customers looking to deploy workloads and applications that require SQL Server, you now have a true Linux solution.

At the same time, if you're big enough to have cabinets full of database server hardware, you probably have databases that serve real-time workloads and databases that underpin your data warehouse. If you're running the latter on beefy hardware necessary to manage the overhead of both Window Server and SQL Server, the advent of SQL Server on Linux might give you an alternative.

For instance, you might shift your lower-resource data warehouses to resource-sipping Linux servers with SQL Server. That could save you on hardware and migration costs, given that there are no structural differences between SQL Server running on Windows or Linux.

If you were contemplating shifting your data warehouse from SQL Server to MariaDB or Oracle to take advantage of Linux hardware savings, you wouldn't have to fret about the conversion costs. Even though you'd still pay for SQL Server licenses, you could save on the costs to convert and migrate to make up the difference.

On the conundrum side, you may ask why you might need Microsoft's offering at all. Afterall, open-source databases like MariaDB (or MySQL) and PostgreSQL are robust, well tested, free and supported by large communities. Why introduce a historically closed-source proprietary tool to your open-source environment? SQL Server 2016 Standard lists for about $3,717 per core, though the Developer and Express versions are free, with Express able to handle up to 10GB for your data-driven applications.

Go to Full Article

Stunnel Security for Oracle

Charles Fisher — Thu, 28 Jul 2016 20:00:00 +0000

by Charles Fisher

Oracle has integrated modern Transport Layer Security (TLS) network encryption into its eponymous database product, and TLS usage no longer requires the Advanced Security option beginning with the 10.2 database release. Legacy configurations lacking TLS exchange encrypted passwords, but the session payload is transmitted in clear text and is intercepted easily by anyone with control over the intermediate network. Databases holding sensitive content should avoid clear-text traffic configurations.

It is possible to use the stunnel utility to wrap the Oracle Transparent Network Substrate (TNS) Listener "invisibly" with TLS encryption as an isolated process, and this configuration appears to be compatible both with Oracle's sqlplus command-line utility and with database links that are used for distributed transactions between multiple database instances. There are several benefits to stunnel over the TNS Listener's native TLS implementation:

The stunnel utility can be far less expensive. Older Oracle database releases required the Advanced Security option to use TLS, which is licensed at $15,000 per CPU according to the latest pricing, but TLS is now included with Standard Edition SE2.
The stunnel utility and the associated dependent libraries (that is, OpenSSL) are patched far more often, and updates can be applied immediately with no database "bounce" if stunnel is used in an "inetd" configuration. Oracle issued eight total patched versions of OpenSSL in 2015 for Oracle Linux 7. Database patches are issued only four times per year at regular quarterly intervals and require instance bounces/outages. An urgent SSL/TLS update will have lengthy delays when implemented as a database patch (due in part to an overabundance of caution by most DBAs), but will be far easier to apply as a simple OS utility patch with no downtime. For this reason, security-sensitive code that may require immediate updates should be kept out of the database server whenever possible. The stunnel utility meets this requirement very well.
The stunnel utility can run as a separate user and group inside a "chroot jail" that has limited visibility to the rest of the system. Oracle's server TLS implementation runs with the full privilege of the TNS Listener. A compromise of the TLS engine can be drastically less dangerous if it is confined within a chroot() jail. Privilege separation and chroot() are well-recognized security techniques, and many security-sensitive installations likely will disable listener TLS for this reason alone.

Let's proceed with adding stunnel TLS services to Oracle.

Go to Full Article

EnterpriseDB's EDB Postgres Advanced Server and EDB Postgres Enterprise Manager

James Gray — Wed, 27 Apr 2016 17:00:00 +0000

by James Gray

The elegance of open source is on full display with new product releases like EnterpriseDB's (EDB's) new PostreSQL-based database solutions. On the heels of the significant PostgreSQL 9.5 update come two EnterpriseDB solutions that take Postgres further, namely EDB Postgres Advanced Server 9.5 and EDB Postgres Enterprise Manager 6.0.

EDB Postgres Advanced Server integrates additional capabilities and security into Postgres that large companies and governments require in order to use it, and this new v9.5 features the following: preconfigured integration with Hadoop and MongoDB, enhanced security with password profiles, expanded compatibility with Oracle to ease and speed migrations, and dramatic performance increases through vertical scaling optimizations.

Meanwhile, EDB Postgres Enterprise Manager, EDB's single console for tuning, monitoring and administering large Postgres deployments, adds enhancements as well in this v6.0 release. These include Nagios support, failover management, a streaming replication wizard, audit log alerts and an improved alert UI.

Go to Full Article

Tune Up Your Databases!

Shawn Powers — Thu, 03 Mar 2016 18:54:57 +0000

by Shawn Powers

My last full-time job was manager of a university's database department. Ironically, I know very, very little about databases themselves. I'm no longer in charge of college databases, but I still do have a handful of MySQL servers that run my various Web applications. Apart from apt-get install, I have no idea how to make databases work. Thankfully, help is available.

MySQLTuner is a Perl script that checks your local (or remote) MySQL server and gives recommendations for improving security and performance. It does not edit files or actually make changes to the server, but it does give a very lengthy list of recommendations. If you (like me) are the sort of person who just tends to copy/paste database setup instructions, running MySQLTuner is a really good idea.

You can download your copy at http://mysqltuner.com. Be sure to read the documentation to get the most use out of the program. And, if you discover security problems like the ones shown in my screenshot? Fix them!

Thanks to its ability to help improve and secure MySQL servers that otherwise might be vulnerable, MySQLTuner gets this month's Editors' Choice award. If you're imperfect like me, download a copy today and fine-tune your databases!

Go to Full Article

Use Your Database!

Reuven Lerner — Thu, 25 Feb 2016 19:30:00 +0000

by Reuven Lerner

I love high-level, dynamically typed languages, such as Python, Ruby and JavaScript. They're easy—and even fun—to use. They let me express myself richly, and they lend themselves to code that easily can be reused and maintained. It's no surprise that interest in such languages is on the rise, especially when creating Web applications.

Now, one of the downsides of these languages is that they tend to execute more slowly than static languages, such as Java, C# and Go. But for a very large number of Web applications, this speed difference doesn't matter, or it is justified by the productivity gain enjoyed by the engineers, or it can be (somewhat) handled by throwing hardware at the problem.

However, the fact that dynamic languages run more slowly than their static counterparts doesn't mean you want to ignore the speed issue completely. Once you know your way around dynamic languages, as well the frameworks built in them, you get a sense of what runs quickly and what doesn't.

In just the past few weeks, however, I've encountered a pattern—or perhaps I should say, an "anti-pattern"—in the code that several of my consulting clients had written. This anti-pattern is well known to experienced developers, but it seems to be less well known than I would have hoped or expected. That anti-pattern, stated simply, is that you should have the database do as much work as possible.

There are several reasons for throwing as much as possible at your database server. First and foremost, your database almost certainly is written in C, so it's likely to execute more quickly than your high-level, dynamic code.

Second, your database has been highly optimized through the years, such that retrieving data from it has been tuned to take into account memory, disk and the frequency of the retrievals.

Third, although network bandwidth is cheap nowadays, it's not infinitely fast. This means that although you could, in theory, write a database query in Ruby that returns a large number of rows and then filters through them using Enumerate#map, if you have the database do some of this for you, it can reduce the amount of data you're retrieving dramatically and, thus, lead to faster application responses and less network usage.

So in this article, I explore this anti-pattern of doing work in an application that probably should be done in the database. You'll see how you can get the same results, but much faster, by applying this rule. There's obviously no one right way to do things, but having the database do as much work as possible is likely to make your applications faster and easier to maintain.

Go to Full Article

Django Models

Reuven Lerner — Tue, 27 Oct 2015 19:50:28 +0000

by Reuven Lerner

In my last article, I continued looking at the Django Web framework, showing how you can create and modify models. As you saw, Django expects you to describe your models using Python code. The model description is then transformed into SQL and compared with any previous version of the model that might have existed. Django then creates a "migration", a file that describes how you can move from one version of the model definition to the next. A migration is a fantastic tool, one that allows developers to move their database forward (and backward) in defined chunks. Migrations make it easier to collaborate with others and upgrade existing applications.

The thing is, migrations have little or nothing to do with the day-to-day application that you want to run. They are useful for the creation and maintenance of your application's models, but in your application, you're going to want to use the models themselves.

So in this article, I look at Django's ORM (object-relational mapper). You'll see how how Django allows you to perform all the traditional CRUD (create-read-update-delete) actions you need and expect within your application, so that you can use a database to power your Web application.

For the purposes of this article, I'll be using the "atfapp" application within the "atfapp" project that I created in last month's article. The model, of an appointment calendar, is defined as follows in atfapp/models.py:


class Appointment(models.Model):
    starts_at = models.DateTimeField()
    ends_at = models.DateTimeField()
    meeting_with = models.TextField()
    notes = models.TextField()
    minutes = models.TextField()
    def __str__(self):
        return "{} - {}: Meeting with {} 
           ↪({})".format(self.starts_at,
                                self.ends_at,
                                self.meeting_with,
                                self.notes)

As you can see, the above model has four fields, indicating when the meeting starts, ends, with whom you are meeting and notes for before the meeting starts. The first two fields are defined to be DateTime fields in Django, which is translated into an SQL TIMESTAMP time in the database.

Creating a New Appointment

The easiest and best way to get your hands dirty with Django models is to use the Django interactive shell—meaning, the Python interactive shell within the Django environment. Within your project, just type:


django-admin shell

and you'll be placed in the interactive Python interpreter—or if you have it installed, in IPython. At this point, you can start to interact with your project and its various applications. In order to work with your Appointment object, you need to import it. Thus, the first thing I do is write:

Go to Full Article

Django Models and Migrations

Reuven Lerner — Thu, 30 Jul 2015 20:10:39 +0000

by Reuven Lerner

In my last two articles, I looked at the Django Web application framework, written in Python. Django's documentation describes it as an MTV framework, in which the acronym stands for model, template and views.

When a request comes in to a Django application, the application's URL patterns determine which view method will be invoked. The view method can then, as I mentioned in previous articles, directly return content to the user or send the contents of a template. The template typically contains not only HTML, but also directives, unique to Django, which allow you to pass along variable values, execute loops and display text conditionally.

You can create lots of interesting Web applications with just views and templates. However, most Web applications also use a database, and in many cases, that means a relational database. Indeed, it's a rare Web application that doesn't use a database of some sort.

For many years, Web applications typically spoke directly with the database, sending SQL via text strings. Thus, you would say something like:


s = "SELECT first_name, last_name FROM Users where id = 1"

You then would send that SQL to the server via a database client library and retrieve the results using that library. Although this approach does allow you to harness the power of SQL directly, it means that your application code now contains text strings with another language. This mix of (for example) Python and SQL can become difficult to maintain and work with. Besides, in Python, you're used to working with objects, attributes and methods. Why can't you access the database that way?

The answer, of course, is that you can't, because relational databases eventually do need to receive SQL in order to function correctly. Thus, many programs use an ORM (object-relational mapper), which translates method calls and object attributes into SQL. There is a well established ORM in the Python world known as SQLAlchemy. However, Django has opted to use its own ORM, with which you define your database tables, as well as insert, update and retrieve information in those tables.

So in this article, I cover how you create models in Django, how you can create and apply migrations based on those model definitions, and how you can interact with your models from within a Django application.

Models

A "model" in the Django world is a Python class that represents a table in the database. If you are creating an appointment calendar, your database likely will have at least two different tables: People and Appointments. To represent these in Django, you create two Python classes: Person and Appointment. Each of these models is defined in the models.py file inside your application.

Go to Full Article