The New Stack Makers

Data lakes are an enticing idea: pour all of a company's information--both structured and unstructured--into a single system, and then everyone can access it and analyze it on demand. It's a great idea, but in practice, actually getting that data back out of the lake and into the hands of the people who need it can be a lot more difficult than it sounds. Kelly Stirman, CMO and vice president of Dremio, has been working with data for decades. In a past life, he worked on strategy at MongoDB, and before that, MarkLogic. In his latest role, he's tasked with helping companies find ways to solve their data lake problems. Chief among those problems, he said, is making that data usable by those who need it.

At ChefConf 2018 in Chicago in late May, we sat down with Chef CTO and co-founder Adam Jacob, and Senior VP of Products and Engineering Corey Scobie, to discuss the cultural currents in the enterprise. Specifically, they focused on the proliferation of DevOps and DevOps tools within the business community. One of the things that is potentially a roadblock for uptake of DevOps in the enterprise has been the internal culture of many organizations. With so many new ideas and tools out there, it can feel like your company doesn't get it at all, and even perhaps fears DevOps. Jacob said that the whole IT organization inside of an enterprise must change to properly embrace DevOps, and he said this must even extend to the opinion IT has of itself. If an organization is doing the wrong thing, or is impeding the uptake of DevOps, said Jacob, administrators and IT workers must push to change those impediments, not resign themselves this being the state of things forever and always. "When you know something's not right in the world, in your company, in your organization, or whatever, you have two choices. You go back to work and you accept that, or you go back to work and you do something about it. One of the themes we've been talking about this week is that IT needs to change their thinking in the enterprise to continue to be successful and relevant, and to actually serve the business," said Corey Scobie, Senior Vice President of Product and Engineering at Chef. Watch on YouTube: https://www.youtube.com/watch?v=5WPx6M0fzO0

Yaron Haviv, CTO and co-founder of Iguazio joins TC Currie for this episode of The New Stack Makers.  Haviv’s deep technological experience in the fields of big data, cloud, storage and networking led to him leading the team creating Iguazio’s new product nuclio. This open-source project, launched in December 2017, is an extremely fast serverless platform with a real-time processing engine.  Extremely fast meaning 100x faster than AWS Lambda. “Everyone knows that the value of data diminishes over time,” said Haviv, “so that means when the data starts flowing into the system, you need to start aggregating contextualizing and acting on it as it flows.” The question was how to process data at extreme speed and at scale, moving from the notion that data is something that you store and query to thinking of data as something you continuously process.

Michael Hausenblas, Red Hat developer advocate for OpenShift, still prefers the term Functions as a Service. In the push to expand the serverless capabilities of Kubernetes, Hausenblas, at KubCon in May, told us that one of the key desires for enterprises is to be able to host Amazon Lambda-like functionality internally as well as in public clouds. He expects as more users adopt serverless functions, that developers will increasingly become comfortable with function accessibility via APIs. Hausenblas then detailed some of the changes this shift would bring to developers and their tools. "On the upside, I see a lot of positive developments so that it's pretty easy to actually write something. You can write it any language, as long as it's in Node.js. From a developer point of view, I don't think this is a big problem. It's more on the operational side. So if you actually think about who is going to be on call when you're operating that at scale , since you don't necessarily have any administrators around any more," said Hausenblas. Watch on YouTube: https://youtu.be/MwX2FPBCR2o

Artificial intelligence may be at the peak of its hype cycle for modern businesses, but for the IT administrator it is still a headache, requiring software and processes that may entirely outside the normal sphere of operations. But help is on the way. At Kubecon + CloudNativeCon 2018 last month, David Aronchick introduced the first working version (0.1) of Kubeflow, software that packages the most popular AI and machine learning software so it can be easily run on the Kubernetes container orchestration engine. "We can't ask people to be experts in everything," Aronchick told the audience at Kubecon, in a keynote introducing the technology.  He noted that today's AI tools, such as Jupyter Notebooks or TensorFlow, have to be managed individually and often at great effort (the "bespoke" model). Kubeflow could rationalize this unruly set of software, but offering a common platform upon which they can all be based, and offered easily to the end-user, the data scientist. Watch on YouTube: https://youtu.be/S7qpvr2bZ2U

In the race to make this weird, wild world of distributed, containerized applications compatible with the virtualized infrastructure upon which most enterprises depend, perhaps no project has made more progress than Kata Containers.  The product of collaboration between the Hyper.sh project and Intel’s Clear Containers, Kata aims to pair individual containers with hypervisors, creating that direct link with the hardware that typifies first-generation virtualization, and isolating host Linux kernels from one another. Google’s recent gVisor project follows a similar path, creating a minimal Linux kernel for the container hosts that reduces the likelihood of exploit. Some folks contend these architectures may render many of the more aggressive security systems being conceived for containerized environments unnecessary or redundant.  But in a conversation for The New Stack Makers, Aqua Security co-founder and CTO Amir Jerbi told us he believes that even the mode of process isolation gVisor and Kata introduce, would carry with it into practice some security challenges.  Try orchestrating a microservices environment with isolated instances in a multi-tenant environment, he suggests, and see what happens.

Back in May, the world's biggest Kubernetes users and contributors gathered in Denmark for KubeCon Europe. One of the topics we were keen to see addressed by this meeting of the minds was the future of Kubernetes, and the CNCF as a whole. To find out just what is being done to prepare the Kubernetes project for the long haul, we sat down with Jaice Singer DuMars, open source software governance program manager at Google, Aparna Sinha, group product manager for Kubernetes at Google, and Dan Kohn, executive director of CNCF. Watch on YouTube: https://youtu.be/sUEsQS9vl4Y

InSpec is a tool for compliance and security testing. It started around operating systems and traditional VMs, and has grown into the security space as a result. Now, it has expanded beyond that. At this year's ChefConf held recently in May, TNS Founder Alex Williams sat down with Dominik Richter, Senior Product Manager at Chef, andAnnie Hedgpeth, Cloud Automation Engineer at 10th Magnitude. On today's episode of The New Stack Makers, the trio dove into the discussion of how InSpec fills a gap in today's DevOps workflow, and the project's goals for the future. Watch on YouTube: https://www.youtube.com/watch?v=PotvNcL6voc

As organizations look to DevOps as a means to achieve digital transformation, they realize they must accelerate the end-to-end software delivery process, and also make it safer.   “It can seem like a boil-the-ocean type of problem. That makes it hard to figure out how you’re going to incrementally derive value from it,” said Andrew Phillips, a product manager in Google Cloud Platform’s DevOps division, in this episode of The New Stack Makers. By separating the developer feedback cycle from the rollout process, organizations can find a manageable starting point. Tools like Spinnaker were built for this purpose, Phillips said, “to provide an abstraction so that development teams can have a simplified experience, while still providing the operations teams with the ability to manage and tweak and define it in exactly the way that makes the most sense for the organization.”

There is no single, best set of tools for continuous integration / continuous development (CI/CD) with Kubernetes — each organization will use the tools that are best suited for its specific use case. For this reason, the Cloud Native Computing Foundation (CNCF) does not advocate for one toolset over another when it comes to building, deploying and managing applications on Kubernetes, says Ihor Dvoretskyi, developer advocate at the CNCF. However, they do analyze what integrations and support are needed in the Kubernetes core and identify CI/CD trends in the process. “The hardest thing [about running on Kubernetes] is gluing all the pieces together. You need to think more holistically about your systems and get a deep understanding of what you're working with,” said Chris Short, a DevOps consultant and CNCF ambassador. “There [must be] a greater understanding of what your business is considering key metrics, or some people call it true north, and bringing that into the operations and development world more closely.” In this podcast, we talk with Dvoretskyi and Short about the trends they’re seeing in DevOps and CI/CD with Kubernetes, the role of the Kubernetes community in improving CI/CD and some of the challenges organizations face as they consider the plethora of tools available today.

At KubeCon + CloudNativeCon in Copenhagen in early May, we caught up with Red Hat's newest CTO, Brandon Philips, to discuss the state of CoreOS and containers as a whole. Philips and his team were acquired by Red Hat back in January, and the pairing has focused a good amount of Linux thought leadership onto the Kubernetes project. In Copenhagen, he and Diane Mueller, director of community development at Red Hat, sat down to talk about what it's been like merging the CoreOS and Red Hat teams. The CoreOS team, and Red Hat overall, has been working closely with the CNCF to expand the capabilities of Kubernetes for enterprise users. One big part of this effort has been the Kubernetes Operators Project, said Philips. Watch on YouTube: https://youtu.be/en4yHnjfi-s

The 20th anniversary of OSCON arrives next month on July 16, 2018, and Portland town can't wait for the geeks to come for a visit. This year continues the O'Reilly effort to move from that long-time practice of language-focus to the story of the past four years thinking more how the developer works every day. On today's episode of The New Stack Makers, TNS Editor-in-Chief Alex Williams sat down with Rachel Roumeliotis, OSCON Chair and Vice President of Content Strategy at O'Reilly Media. Throughout the discussion, the two discussed what guests can expect at the conference, including what tracks are of note, the technologies behind them, and how open source must continue to bring diversity to its workforce.

The buzz at ChefConf, this past week, was all around the Habitat project. This open source project brings a deployment layer to developers, allowing them to package up their applications as just about anything under the sun. Considering that Chef remains a popular path to configuration management and environment configuration, adding this new unified deployment preparation layer brings developers even more exciting ways to automate their continuous integration and deployment processes. "I describe it, personally, as two things. The first is a packaging system. What you do is, you create a plan file for your application, you write the application first, and then that plan file defines in Bash for Linux and Powershell for Windows how exactly that app should be built, installed, and run. The wonderful thing is that gets packaged into a HART artifact, because we heart you, though it actually means Habitat Artifact. That package can be exported to almost anything. It can be exported into a Docker container. That's by far the most popular exporter that people use," said Nell Shamrell-Harrington, senior development engineer at Chef. Watch on YouTube: https://www.youtube.com/watch?v=9yQRixcCmXc

You're likely running some form of server on AWS right now, somewhere in the world. If so, you're also entrusting your systems to the reliability and stability of the folks at Amazon: that they've thought through their devops practices and have your back when it comes to scaling, outages, and backup. And you'd be correct to place your trust there: Amazon's AWS track record only gets better every year as billions of applications successfully complete their business goals within its confines. But just how does Amazon maintain such a spectacularly science-fiction sized hardware and software hosting system? The answer is not only devops, it's also Chef. At Chefconf this past week, we sat down with Jonathan Weiss, senior software development manager and Mark Rambow, software development manager, both from Amazon's AWS team. We traced the lines of control and practice inside their company to find just what it is that allows them to not only operate at scale, but to allow everyone else in the world to operate at scale as well. Watch on YouTube: https://www.youtube.com/watch?v=w4iMgvkxT9g

When it comes to securing an application deployed on Kubernetes, automation through a CI/CD pipeline is key. Using cloud-native security tools that hook right into Jenkins or your favorite CI/CD tool, enterprise security teams can set policies for developers who are building container images. The pipeline enforces those policies through automated vulnerability scanning of each image during the build process. Developers only deploy images that the security team is confident in because they've been scanned. “CI/CD automation is key because of the scale,” said Liz Rice, technology evangelist at Aqua Security, in this podcast with The New Stack. “You couldn't possibly manually check all these different images when you're shipping potentially hundreds or thousands of deploys in a day.” Watch on YouTube: https://youtu.be/MC3x2CV3ozA