Cloud

The Taloflow Instance Manager (Tim)

Petros Koutoupis — Fri, 08 Feb 2019 13:00:00 +0000

For years, modern workloads have shifted to the cloud, with AWS being the most popular. And although this shift has cut down operating costs significantly, millions, if not billions, of dollars still are wasted to maintain all those virtual instances—even when they are not in use.

To help alleviate both the burden and headache of managing your cloud-hosted virtual machines, Taloflow built the Taloflow Instance Manager (Tim), which can reduce your expenditures by as much as 40%. Tim monitors your AWS resources and suggests automations that effortlessly save you money in real time.

Taloflow is a Vancouver- and California-based startup, offering a Software-as-a-Service (SaaS) platform that seamlessly integrates into your preferred cloud service provider to set up alerts, capture metrics and automate a list of useful actions. The company is focused solely on bringing artificial intelligence (AI) automation and intelligence to cloud services. Currently, Taloflow is an operation of at least eight talented engineers coming from all business backgrounds (from startups to enterprises).

Figure 1. The Taloflow Team

One of the key differences with Tim is that it works in real time. Unlike its competition, which is focused primarily on accountants and finance departments, Tim takes a bottoms-up approach and shifts that focus onto the engineers and operators pulling the levers on these cloud virtual instances. Think of it as bot or tool helping developers manage their resources and monitor their workflows. Tim will provide recommendations to those same engineers on how to optimize the performance, as well as the cost in the cloud.

The current implementation of Tim is available under a freemium model. This is intended to encourage early adoption, and it also allows users to hit the ground running and get started quickly. Depending on usage, number of users and the required performance, a paid tier or Enterprise Model eventually will be offered by March 2019.

Tim's basic model runs on Taloflow's own cloud, and depending on the customer's security preferences, the company will offer and provision private instances for each user (under the Enterprise subscription model). This will look like a Kubernetes image running on-premises at the customer site.

Go to Full Article

Weekend Reading: Cloud

Carlie Fairchild — Sat, 22 Dec 2018 12:30:00 +0000

by Carlie Fairchild

The cloud has become synonymous with all things data storage. It additionally equates to the many web-centric services accessing that same back-end data storage, but the term also has evolved to mean so much more.

This weekend join us as we explore the cloud.

CloudWatch Is of the Devil, but I Must Use It

by Corey Quinn

Let's talk about Amazon CloudWatch.

Everything You Need to Know about the Cloud and Cloud Computing, Part I

by Petros Koutoupis

An in-depth breakdown of the technologies involved in making up the cloud and a survey of cloud-service providers.

Everything You Need to Know about the Cloud and Cloud Computing, Part II: Using the Cloud

by Petros Koutoupis

How to get started with AWS, install Apache, create an EFS volume and much more.

The Agony and the Ecstasy of Cloud Billing

by Corey Quinn

Cloud billing is inherently complex; it's not just you.

Nextcloud 13: How to Get Started and Why You Should

by Marco Fioretti

In its simplest form, the Nextcloud server is "just" a personal, free software alternative to services like Dropbox or iCloud. You can set it up so your files are always accessible via the internet, from wherever you are, and share them with your friends. However, Nextcloud can do so much more.

Simple Cloud Hardening

by Kyle Rankin

Apply a few basic hardening principles to secure your cloud environment.

Vendor Lock-in: Now in the Cloud!

by Kyle Rankin

Go to Full Article

CloudWatch Is of the Devil, but I Must Use It

Corey Quinn — Wed, 31 Oct 2018 11:30:00 +0000

by Corey Quinn

Let's talk about Amazon CloudWatch.

For those fortunate enough to not be stuck in the weeds of Amazon Web Services (AWS), CloudWatch is, and I quote from the official AWS description, "a monitoring and management service built for developers, system operators, site reliability engineers (SRE), and IT managers." This is all well and good, except for the part where there isn't a single named constituency who enjoys working with the product. Allow me to dispense some monitoring heresy.

Better, let me describe this in the context of the 14 Amazon Leadership Principles that reportedly guide every decision Amazon makes. When you take a hard look at CloudWatch's complete failure across all 14 Leadership Principles, you wonder how this product ever made it out the door in its current state.

"Frugality"

I'll start with billing. Normally left for the tail end of articles like this, the CloudWatch billing paradigm is so terrible, I'm leading with it instead. You get billed per metric, per month. You get billed per thousand metrics you request to view via the API. You get billed per dashboard per month. You get billed per alarm per month. You get charged for logs based upon data volume ingested, data volume stored and "vended logs" that get published natively by AWS services on behalf of the customer. And, you get billed per custom event. All of this can be summed up best as "nobody on the planet understands how your CloudWatch metrics and logs get billed", and it leads to scenarios where monitoring vendors can inadvertently cost you thousands of dollars by polling CloudWatch too frequently. When the AWS charges are larger than what you're paying your monitoring vendor, it's not a wonderful feeling.

"Invent and Simplify"

CloudWatch Logs, CloudWatch Events, Custom Metrics, Vended Logs and Custom Dashboards all mean different things internally to CloudWatch from what you'd expect, compared to metrics solutions that actually make some fathomable level of sense. There are, thus, multiple services that do very different things, all operating under the "CloudWatch" moniker. For example, it's not particularly intuitive to most people that scheduling a Lambda function to invoke once an hour requires a custom CloudWatch Event. It feels overly complicated, incredibly confusing, and very quickly, you find yourself in a situation where you're having to build complex relationships to monitor things that are themselves far simpler.

Go to Full Article

Why the Failure to Conquer the Desktop Was Great for GNU/Linux

Glyn Moody — Mon, 06 Aug 2018 11:30:00 +0000

by Glyn Moody

AI: open source's next big win.

Canonical recently launched Ubuntu 18.04 LTS. It's an important release. In part, that's because Canonical will support it for five years, making it one of the relatively rare LTS products in Ubuntu's history. Ubuntu 18.04 also marks a high-profile return to GNOME as the default desktop, after a few years of controversial experimentation with Unity. The result is regarded by many as the best desktop Ubuntu so far (that's my view too, for what it's worth). And yet, the emphasis at launch lay elsewhere. Mark Shuttleworth, CEO of Canonical and founder of Ubuntu, said:

Multi-cloud operations are the new normal. Boot-time and performance-optimised images of Ubuntu 18.04 LTS on every major public cloud make it the fastest and most efficient OS for cloud computing, especially for storage and compute-intensive tasks like machine learning.

The bulk of the official 18.04 LTS announcement is about Ubuntu's cloud computing features. On the main web site, Ubuntu claims to be "The standard OS for cloud computing", citing (slightly old) research that shows "70% of public cloud workloads and 54% of OpenStack clouds" use it. Since Canonical is a privately held company, it doesn't publish a detailed breakdown of its operations, just a basic summary. That means it's hard to tell just how successful the cloud computing strategy is proving. But, the fact that Shuttleworth is now openly talking about an IPO—not something to be undertaken lightly—suggests that there is enough good news to convince investors to throw plenty of money at Canonical when the prospectus spells out how the business is doing.

Go to Full Article

As Cloud Computing Providers Post Record Profits, One Company Wants to Make Them Obsolete

Graham Templeton — Thu, 28 Jun 2018 12:00:00 +0000

by Graham Templeton

Could the world's idle computers make the $200 billion cloud computing industry obsolete?

In the final years of the Obama administration, it seemed that the world was witnessing the emergence of an odd alliance between the largest establishment tech companies and the traditionally anti-establishment community of independent techies. The highest profile case involved Apple and other major corporations siding with civil rights organizations to advocate against weakening device encryption on behalf of law enforcement, and it it led to a hope that these companies were finally seeing the profit potential in having secure, satisfied customers. That hope has now evaporated in the face of continuous betrayals of user trust, and rather than take it lying down, tech users are looking into alternative solutions that put them in control.

The recently passed federal Clarifying Lawful Overseas Use of Data Act, or the CLOUD Act, makes it perfectly clear why this shift away from self-interested stakeholders is so necessary. The new law was motivated by a legal demand for Microsoft to hand user data to US law enforcement, even though that data was stored outside US borders and, thus, in a different legal jurisdiction. At first, it seemed that Microsoft was planning to stand up for privacy and national sovereignty by opposing the demand in court—but, predictably, the corporation rolled over just as soon as it deemed that its own interests were protected.

The goal of the CLOUD Act is for the US to be able to compel any company that does business in the country to provide information to US law enforcement, even if that information is not actually stored within the US. If it's successful, a company's willingness to fight will become essentially irrelevant. It makes all cloud computing and cloud storage companies suspect, simply by virtue of being cloud computing companies.

Groups like the Electronic Frontier Foundation and the American Civil Liberties Union believe it also could allow US law enforcement to seize information in foreign jurisdictions because those regions have more lenient privacy laws than the US itself. The reverse, in which foreign law enforcement agencies operating under more stringent rules than the US, could gain access to protected information stored in the country, is also a major concern.

Go to Full Article

Nextcloud 13: How to Get Started and Why You Should

Marco Fioretti — Mon, 21 May 2018 11:45:00 +0000

by Marco Fioretti

Nextcloud could be the first step toward replacing proprietary services like Dropbox and Skype.

In this article, I first describe what the Nextcloud server is and how to install and set it up on GNU/Linux systems. Then I explain how to configure the optional Nextcloud features, which may be the first steps toward making Nextcloud the shell of a complete replacement for many proprietary platforms existing today, such as Dropbox, Facebook and Skype.

Figure 1. A safe home for all your data that all your devices can reach—that's what Nextcloud wants to be.

Why Nextcloud and Not ownCloud?

Nextcloud, whose version 13 was released in February 2018, was spun off the popular ownCloud project in 2016, out of licensing and other disagreements. See the Resources section for some of the most complete feature-by-feature comparisons between Nextcloud and ownCloud. The most basic capabilities are still almost identical, two years after the fork. Some of the functions described here, however, are easier to integrate in Nextcloud than in its ancestor. In addition, my personal reasons for recommending Nextcloud over ownCloud are the following:

Licensing and pricing policies: all the official components of Nextcloud are both free as in freedom and as in free beer. You pay only for support and update services. That's not the case with ownCloud.
Long-term roadmap: at the moment, ownCloud seems to be more focused on corporate customers and more relevant for investors, while Nextcloud seems to be more focused on extending "direct" user-to-user communication and cooperation features.

Figure 2. The Original Nextcloud/ownCloud Functions: File and Picture Storage, Dropbox-Style

A Word on Security

Several good reasons to choose Nextcloud as the online home for your own files and data are related to security. I don't cover them in detail in this introductory article, but I want to mention at least some of them.

Nextcloud refuses continuous (that is, malicious) attempts to authenticate from any computer, except those whose IP addresses are included in "brute-force IP whitelists". (Of course, the best possible whitelist you can configure is an empty one.)

Go to Full Article

Everything You Need to Know about the Cloud and Cloud Computing, Part II: Using the Cloud

Petros Koutoupis — Tue, 15 May 2018 11:45:00 +0000

by Petros Koutoupis

How to get started with AWS, install Apache, create an EFS volume and much more.

The cloud is here to stay, regardless of how you access data day to day. Whether you are uploading and sharing new photos with friends in your social-media account or updating documents and spreadsheets alongside your peers in your office or school, chances are you're connecting to the cloud in some form or another.

In the first part of this series, I explored what makes up the cloud and how it functions when all of its separate moving pieces come together. In this article, building from Part I's foundations, I cover using the cloud through some actual examples.

Getting Started with AWS

For the purposes of this article, I'm focusing on a few of the top offerings provided by Amazon Web Services (AWS). Please know that I hold no affiliation to or with Amazon, nor am I stating that Amazon offerings exceed those of its competitors.

If you haven't already, be sure to register an account. But before you do, understand that charges may apply. Amazon, may provide a free tier of offerings for a limited time, typically a year, to newly registered users. In most cases, the limitations to these offerings are far less than ideal for modern use cases. It is a pay-as-you go model, and you'll be charged only as long as the instance or service continues to be active.

As soon as you are registered and logged in from within your web browser, you'll be greeted by a fairly straightforward dashboard.

Figure 1. The AWS Main Dashboard of services and resources.

Compute

At first, companies leveraging cloud compute applied a straight copy-and-paste of their very own data centers for deploying standard web/application/database servers. The model was the same. There is nothing wrong with that approach. The transition for most converting from on-premises to the cloud would have been somewhat seamless—at least from the perspective of the user accessing those resources. The only real difference being that it was just in a different data center and without the headache of maintaining the infrastructure supporting it.

In the world of AWS, virtual compute servers are managed under the Elastic Cloud Computing (EC2) stack, from whole virtual instances to containers and more. Let's begin an example EC2 experiment by navigating to the EC2 dashboard.

Figure 2. The Elastic Cloud Computing Dashboard

Go to Full Article

Everything You Need to Know about the Cloud and Cloud Computing, Part I

Petros Koutoupis — Wed, 09 May 2018 13:30:00 +0000

by Petros Koutoupis

An in-depth breakdown of the technologies involved in making up the cloud and a survey of cloud-service providers.

The cloud has become synonymous with all things data storage. It additionally equates to the many web-centric services accessing that same back-end data storage. But the term also has evolved to mean so much more.

Cloud computing provides more simplified access to server, storage, database and application resources, with users provisioning and using the minimum set of requirements they see fit to host their application needs. In the past decade alone, the paradigm shift toward a wider and more accessible network has forced both hardware vendors and service providers to rethink their strategies and cater to a new model of storing information and serving application resources. As time continues to pass, more individuals and businesses are connecting themselves to this greater world of computing.

What Is the Cloud?

Far too often, the idea of the "cloud" is confused with the general internet. Although it's true that various components making up the cloud can be accessible via the internet, they are not one and the same. In its most general terms, cloud computing enables companies, service providers and individuals to provision the appropriate amount of computing resources dynamically (compute nodes, block or object storage and so on) for their needs. These application services are accessed over a network—and not necessarily a public network. Three distinct types of cloud deployments exist: public, private and a hybrid of both.

The public cloud differentiates itself from the private cloud in that the private cloud typically is deployed in the data center and under the proprietary network using its cloud computing technologies—that is, it is developed for and maintained by the organization it serves. Resources for a private cloud deployment are acquired via normal hardware purchasing means and through traditional hardware sales channels. This is not the case for the public cloud. Resources for the public cloud are provisioned dynamically to its user as requested and may be offered under a pay-per-usage model or for free.

Some of the world's leading public cloud offering platforms include:

Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform
IBM Cloud (formerly SoftLayer)

As the name implies, the hybrid model allows for seamless access and transitioning between both public and private deployments, all managed under a single framework.

For those who prefer either to host their workload internally or partially on the public cloud—sometimes motivated by security, data sovereignty or compliance—private and hybrid cloud offerings continue to provide the same amount of service but all within your control.

Using cloud services enables you to achieve the following:

Go to Full Article

The Agony and the Ecstasy of Cloud Billing

Corey Quinn — Tue, 17 Apr 2018 14:40:00 +0000

by Corey Quinn

Cloud billing is inherently complex; it's not just you.

Back in the mists of antiquity when I started reading Linux Journal, figuring out what an infrastructure was going to cost was (although still obnoxious in some ways) straightforward. You'd sign leases with colocation providers, buy hardware that you'd depreciate on a schedule and strike a deal in blood with a bandwidth provider, and you were more or less set until something significant happened to your scale.

In today's brave new cloud world, all of that goes out the window. The public cloud providers give with one hand ("Have a full copy of any environment you want, paid by the hour!"), while taking with the other ("A single Linux instance will cost you $X per hour, $Y per GB transferred per month, and $Z for the attached storage; we simplify this pricing into what we like to call 'We Make It Up As We Go Along'").

In my day job, I'm a consultant who focuses purely on analyzing and reducing the Amazon Web Services (AWS) bill. As a result, I've seen a lot of environments doing different things: cloud-native shops spinning things up without governance, large enterprises transitioning into the public cloud with legacy applications that don't exactly support that model without some serious tweaking, and cloud migration projects that somehow lost their way severely enough that they were declared acceptable as they were, and the "multi-cloud" label was slapped on to them. Throughout all of this, some themes definitely have emerged that I find that people don't intuitively grasp at first. To wit:

It's relatively straightforward to do the basic arithmetic to figure out what a current data center would cost to put into the cloud as is—generally it's a lot! If you do a 1:1 mapping of your existing data center into the cloudy equivalents, it invariably will cost more; that's a given. The real cost savings arise when you start to take advantage of cloud capabilities—your web server farm doesn't need to have 50 instances at all times. If that's your burst load, maybe you can scale that in when traffic is low to five instances or so? Only once you fall into a pattern (and your applications support it!) of paying only for what you need when you need it do the cost savings of cloud become apparent.

Go to Full Article

Simple Cloud Hardening

Kyle Rankin — Tue, 10 Apr 2018 15:30:00 +0000

by Kyle Rankin

Apply a few basic hardening principles to secure your cloud environment.

I've written about simple server-hardening techniques in the past. Those articles were inspired in part by the Linux Hardening in Hostile Networks book I was writing at the time, and the idea was to distill the many different hardening steps you might want to perform on a server into a few simple steps that everyone should do. In this article, I take the same approach only with a specific focus on hardening cloud infrastructure. I'm most familiar with AWS, so my hardening steps are geared toward that platform and use AWS terminology (such as Security Groups and VPC), but as I'm not a fan of vendor lock-in, I try to include steps that are general enough that you should be able to adapt them to other providers.

New Accounts Are (Relatively) Free; Use Them

One of the big advantages with cloud infrastructure is the ability to compartmentalize your infrastructure. If you have a bunch of servers racked in the same rack, it might be difficult, but on cloud infrastructures, you can take advantage of the technology to isolate one customer from another to isolate one of your infrastructure types from the others. Although this doesn't come completely for free (it adds some extra overhead when you set things up), it's worth it for the strong isolation it provides between environments.

One of the first security measures you should put in place is separating each of your environments into its own high-level account. AWS allows you to generate a number of different accounts and connect them to a central billing account. This means you can isolate your development, staging and production environments (plus any others you may create) completely into their own individual accounts that have their own networks, their own credentials and their own roles totally isolated from the others. With each environment separated into its own account, you limit the damage attackers can do if they compromise one infrastructure to just that account. You also make it easier to see how much each environment costs by itself.

In a traditional infrastructure where dev and production are together, it is much easier to create accidental dependencies between those two environments and have a mistake in one affect the other. Splitting environments into separate accounts protects them from each other, and that independence helps you identify any legitimate links that environments need to have with each other. Once you have identified those links, it's much easier to set up firewall rules or other restrictions between those accounts, just like you would if you wanted your infrastructure to talk to a third party.

Go to Full Article