In this O'Reilly Radar Podcast: Edd Dumbill on the data lake, and Rajiv Maheswaran on the science of moving dots.
In a recent blog post, Edd Dumbill, VP of strategy at Silicon Valley Data Science, wrote about the phrase “data lake.” Likening it to a dream, he described a data lake as “a place with data-centered architecture, where silos are minimized, and processing happens with little friction in a scalable, distributed environment…Data itself is no longer restrained by initial schema decisions, and can be exploited more freely by the enterprise.” He explained that he called it a “dream” because “we’ve a way to go to make the vision come true” — but noted he’s optimistic the dream can be realized.
Putting ourselves in the shoes of the user is key to building better systems and services.
In this podcast episode, Tim O’Reilly talks about building systems and services for people, keeping a close eye on the end user’s experience to build better, more efficient systems that actually work for the people using them. Highlighting a quote from Jeff Sussna, O’Reilly makes a deeper connection between development and the ultimate purpose for building systems and services — user experience:
“[Jeff Sussna says in his blog post Empathy: The Essence of DevOps]: ‘It’s not about making developers and sysadmins report to the same VP. It’s not about automating all your configuration procedures. It’s not about tipping up a Jenkins server, or running your applications in the cloud, or releasing your code on Github. It’s not even about letting your developers deploy their code to a PaaS. The true essence of DevOps is empathy.’
“Understanding the other people that you work with and how you’re going to work together more effectively. That word ‘empathy’ struck me and it made me connect the world of DevOps with the world of user experience design.”
Tim O'Reilly and Carl Bass discuss the future of making things, and Astro Teller on Google X's approach to solving big problems.
I recently lamented the lag in innovation in relation to the speed of technological advancements — do we really need a connected toaster that will sell itself if neglected? Subsequently, I had a conversation with Josh Clark that made me rethink that position; Clark pointed out that play is an important aspect of innovation, and that such whimsical creations as drum pants could ultimately lead to more profound innovations.
In the first segment of this podcast episode, Tim O’Reilly and Autodesk CEO Carl Bass have a wide-ranging discussion about the future of making things. Bass notes that innovation tends to start by “looking at the rear window”:
“The first naïve response is to take a new technology and do the old thing with it. It takes a while until you can start reimagining things…the first thing that you need is this new tool set in software, hardware, and materials, but the more important thing — and the more difficult thing, obviously — is a new mind-set. How are you going to think about this problem differently? How are you going to reimagine what you can do? That’s the exciting part.”
In separate interviews, authors Hugh Howey and Ramez Naam discuss science fiction and their views of the future.
Science fiction long has fueled the imaginations of scientists and inspired (or foreshadowed) technological advancement. We have only to look back at the works of Isaac Asimov, or even Kurt Vonnegut, and episodes of “Star Trek” or movies like “Minority Report” for science fiction technologies that are (or nearly are) coming into existence today.
In this podcast episode, author, scientist, and futurist Ramez Naam explains to O’Reilly’s Mac Slocum that science fiction had a direct influence on his current interests in human enhancement and telepathy. Naam grew up reading science fiction (“like a lot of geeks,” he says) and once he started reading scientific journals and papers, he started seeing the connections. Naam says, “I found out that a lot of science fiction ideas were becoming actually possible — that scientists were implanting electrodes in the brains of animals and getting them to move robot arms by thought, to help people who were paralyzed.”
Josh Clark and Tim O’Reilly on designing beyond screens, and beyond a single device.
As the Internet is increasingly embedded into our physical world, it’s important to start designing for physical and intentional interactions with interfaces to supplement the passive, data-gathering interactions — designing smart devices that service us in the background, but upon which we also can exert our will.
In this episode, Josh Clark (in an interview) and Tim O’Reilly (in a keynote) both address the importance of designing for contextual awareness and physical interaction. Clark stresses that we’re not facing a challenge of technology, but a challenge of imagination. O’Reilly argues that we’re not paying enough attention to the aspects of people and time in designing the Internet of Things, and that the entire system in which we operate is the user interface — as we design this new world, we must think about user needs first.
Data tools are less important than the way you frame your questions.
Max Shron and Jake Porway spoke with me at Strata a few weeks ago about frameworks for making reasoned arguments with data. Max’s recent O’Reilly book, Thinking with Data, outlines the crucial process of developing good questions and creating a plan to answer them. Jake’s nonprofit, DataKind, connects data scientists with worthy causes where they can apply their skills.
A few of the things we talked about:
- The importance of publishing negative scientific results
- Give Directly, an organization that facilitates donations directly to households in Kenya and Uganda. Give Directly was able to model income using satellite data to distinguish thatched roofs from metal roofs.
- Moritz Stefaner calling for a “macroscope”
- Project Cybersyn, Salvador Allende’s plan for encompassing the entire Chilean economy in a single real-time computer system
- Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed by James C. Scott
After we recorded this podcast episode at Strata Santa Clara, Max presided over a webcast on his book that’s archived here.
An Interview with Neal Ford
I recently interviewed O’Reilly author Neal Ford (Functional Thinking, The Productive Programmer) on the subject of polyglot programming. In 2006, Neal wrote a blog post which resurrected the term, suggesting that as modern applications become more complex, it is important for developers to leverage knowledge of multiple languages and use the right tool for the job. In the interview, we discuss the benefits and challenges of polyglot programming, how it has evolved in recent years, and the impact it’s had on software development.
Some key highlights in our conversation include:
- What is polyglot programming? [Discussed at 0:15]
- What are some of the benefits? [Discussed at 1:39]
- How polyglot programming has affected software development in recent years [Discussed at 4:25]
- Downsides to polyglot programming? What are the trade-offs? [Discussed at 6:22]
- Best practices when starting out in polyglot programming [Discussed at 8:58]
- Resources for keeping up on trends and new technologies [Discussed at 12:48]
A chat about the future of UI/UX design with Alasdair Allan, Josh Marinacci and Tony Santos.
At our OSCON conference this summer, Jon Bruner, Renee DiResta and I sat down with Alasdair Allan, a hardware hacker and O’Reilly author; Josh Marinacci, a researcher with Nokia; and Tony Santos, a user experience designer with Mozilla. Our discussion focused on the future of UI/UX design, from the perils of designing from the top down to declining diversity in washing machines to controlling your car from anywhere in the world.
Here are some highlights from our chat:
- Alasdair’s Ignite talk on the bad design of UX in the Internet of Things: the more widgets and dials and sliders that you add on are delayed design decisions that you’re putting onto the user. (1:55 mark)
- Looking at startups working in the Internet of Things, design seems to be “pretty far down on the general level of importance.” Much of the innovation is happening on Kickstarter and is driven by hardware hackers, many of whom don’t have design experience — and products are often designed as an end to themselves, as opposed to parts of a connected ecosystem. “We’re not building an Internet of Things, we’re building a series of islands…we should be looking at systems.” (3:23)
A conversation with Chris Anderson, Nick Pinkston, and Jie Qi
Manufacturing is hard, but it’s getting easier. In every stage of the manufacturing process–prototyping, small runs, large runs, marketing, fulfillment–cheap tools and service models have become available, dramatically decreasing the amount of capital required to start building something and the expense of revising and improving a product once it’s in production.
In this episode of the Radar podcast, we speak with Chris Anderson, CEO and co-founder of 3D Robotics; Nick Pinkston, a manufacturing expert who’s working to make building things easy for anyone; and Jie Qi, a student at the MIT Media Lab whose recent research has focused on the factories of Shenzhen.
Along the way we talk about the differences between Tesla’s auto plant and its previous incarnation as the NUMMI plant; the differences between on-shoring, re-shoring and near-shoring; and how the innovative energy of Kickstarter and the Maker movement can be brought to underprivileged populations.
Many of these topics will come up at Solid, O’Reilly’s new conference about the intersection of software and the physical world. Solid’s call for proposals open through December 9. We’re planning a series of Solid meet-ups, plant tours, and books about the collision of real and virtual; if you’ve got an idea for something the series should explore, please reach out!
Hadoop, Sqoop, and ZooKeeper
Kathleen Ting (@kate_ting), Technical Account Manager at Cloudera, and our own Andy Oram (@praxagora) sat down to discuss how to work with structured and unstructured data as well as how to keep a system up and running that is crunching that data.
Key highlights include:
- Misconfigurations consist of almost half of the support issues that the team at Cloudera is seeing [Discussed at 0:22]
- ZooKeeper, the canary in the Hadoop coal mine [Discussed at 1:10]
- Leaky clients are often a problem ZooKeeper detects [Discussed at 2:10]
- Sqoop is a bulk data transfer tool [Discussed at 2:47]
- Sqoop helps to bring together structured and unstructured data [Discussed at 3:50]
- ZooKeep is not for storage, but coordination, reliability, availability [Discussed at 4:44]
You can view the full interview here: