In this O'Reilly Radar Podcast: Edd Dumbill on the data lake, and Rajiv Maheswaran on the science of moving dots.
In a recent blog post, Edd Dumbill, VP of strategy at Silicon Valley Data Science, wrote about the phrase “data lake.” Likening it to a dream, he described a data lake as “a place with data-centered architecture, where silos are minimized, and processing happens with little friction in a scalable, distributed environment…Data itself is no longer restrained by initial schema decisions, and can be exploited more freely by the enterprise.” He explained that he called it a “dream” because “we’ve a way to go to make the vision come true” — but noted he’s optimistic the dream can be realized.
The core principle in bitcoin is decentralization, and it has important implications for security.
Editor’s note: this is an excerpt from Chapter 10 of our recently released book Mastering Bitcoin, by Andreas Antonopoulos. You can read the full chapter here. Antonopoulos will be speaking at our upcoming event Bitcoin & the Blockchain, January 27, 2015, in San Francisco. Find out more about the event and reserve your spot here.Securing bitcoin is challenging because bitcoin is not an abstract reference to value, like a balance in a bank account. Bitcoin is very much like digital cash or gold. You’ve probably heard the expression “Possession is nine tenths of the law.” Well, in bitcoin, possession is ten tenths of the law. Possession of the keys to unlock the bitcoin, is equivalent to possession of cash or a chunk of precious metal. You can lose it, misplace it, have it stolen, or accidentally give the wrong amount to someone. In every one of those cases, end users would have no recourse, just as if they dropped cash on a public sidewalk.
However, bitcoin has capabilities that cash, gold, and bank accounts do not. A bitcoin wallet, containing your keys, can be backed up like any file. It can be stored in multiple copies, even printed on paper for hardcopy backup. You can’t “backup” cash, gold, or bank accounts. Bitcoin is different enough from anything that has come before that we need to think about bitcoin security in a novel way too.
The core principle in bitcoin is decentralization and it has important implications for security. A centralized model, such as a traditional bank or payment network, depends on access control and vetting to keep bad actors out of the system. By comparison, a decentralized system like bitcoin pushes the responsibility and control to the end users. Because security of the network is based on Proof-Of-Work, not access control, the network can be open and no encryption is required for bitcoin traffic. Read more…
A look at the social and moral implications of living in a deeply connected, analyzed, and informed world.
We’ll now look at both the light and the shadows of this new dawn, the social and moral implications of living in a deeply connected, analyzed, and informed world. This is both the promise and the peril of big data in an age of widespread sensors, fast networks, and distributed computing.
Solving the big problemsThe planet’s systems are under strain from a burgeoning population. Scientists warn of rising tides, droughts, ocean acidity, and accelerating extinction. Medication-resistant diseases, outbreaks fueled by globalization, and myriad other semi-apocalyptic Horsemen ride across the horizon.
Can data fix these problems? Can we extend agriculture with data? Find new cures? Track the spread of disease? Understand weather and marine patterns? General Electric’s Bill Ruh says that while the company will continue to innovate in materials sciences, the place where it will see real gains is in analytics.
It’s often been said that there’s nothing new about big data. The “iron triangle” of Volume, Velocity, and Variety that Doug Laney coined in 2001 has been a constraint on all data since the first database. Basically, you could have any two you want fairly affordably. Consider:
- A coin-sorting machine sorts a large volume of coins rapidly, but assumes a small variety of coins. It wouldn’t work well if there were hundreds of coin types.
- A public library, organized by the Dewey Decimal System, has a wide variety of books and topics, and a large volume of those books — but stacking and retrieving the books happens at a slow velocity.
What’s new about big data is that the cost of getting all three Vs has become so cheap it’s almost not worth billing for. A Google search happens with great alacrity, combs the sum of online knowledge, and retrieves a huge variety of content types. Read more…
The blockchain is like layers in a geological formation — the deeper you go, the more stability you gain.
Editor’s note: this is an excerpt from Chapter 7 of our recently released book Mastering Bitcoin, by Andreas Antonopoulos. You can read the full chapter here. Antonopoulos will be speaking at our upcoming event Bitcoin & the Blockchain, January 27, 2015, in San Francisco. Find out more about the event and reserve your spot here.The blockchain data structure is an ordered back-linked list of blocks of transactions. The blockchain can be stored as a flat file, or in a simple database. The bitcoin core client stores the blockchain metadata using Google’s LevelDB database. Blocks are linked “back,” each referring to the previous block in the chain. The blockchain is often visualized as a vertical stack, with blocks layered on top of each other and the first block serving as the foundation of the stack. The visualization of blocks stacked on top of each other results in the use of terms like “height” to refer to the distance from the first block, and “top” or “tip” to refer to the most recently added block.
Each block within the blockchain is identified by a hash, generated using the SHA256 cryptographic hash algorithm on the header of the block. Each block also references a previous block, known as the parent block, through the “previous block hash” field in the block header. In other words, each block contains the hash of its parent inside its own header. The sequence of hashes linking each block to its parent creates a chain going back all the way to the first block ever created, known as the genesis block. Read more…
As we increasingly depend on connected devices, primary concerns will narrow to safety, reliability, and survivability.
Editor’s note: this interview with GE’s Bill Ruh is an excerpt from our recent report, When Hardware Meets Software, by Mike Barlow. The report looks into the new hardware movement, telling its story through the people who are building it. For more stories on the evolving relationship between software and hardware, download the free report.More than one observer has noted that while it’s relatively easy for consumers to communicate directly with their smart devices, it’s still quite difficult for smart devices to communicate directly, or even indirectly, with each other. Bill Ruh, a vice president and corporate officer at GE, drives the company’s efforts to construct an industrial Internet that will enable devices large and small to chat freely amongst themselves, automatically and autonomously. From his perspective, the industrial Internet is a benign platform for helping the world become a quieter, calmer, and less dangerous place.
“In the past, hardware existed without software. You think about the founding of GE and the invention of the light bulb — you turned it on and you turned it off. Zero lines of code. Today, we have street lighting systems with mesh networks and 20 million lines of code,” says Ruh. “Machines used to be completely mechanical. Today, they are part digital. Software is part of the hardware. That opens up huge possibilities.”
A hundred years ago, street lighting was an on-or-off affair. In the future, when a crime is committed at night, a police officer might be able to raise the intensity of the nearby street lights by tapping a smart phone app. This would create near-daylight conditions around a crime scene, and hopefully make it harder for the perpetrators to escape unseen. “Our machines are becoming much more intelligent. With software embedded in them, they’re becoming brilliant,” says Ruh. Read more…
The evolving marketplace is making new data applications and interactions possible.
Here’s a look at some options in the evolving, maturing marketplace of big data components that are making the new applications and interactions we’ve been looking at possible.
First used in social network analysis, graph theory is finding more and more homes in research and business. Machine learning systems can scale up fast with tools like Parameter Server, and the TitanDB project means developers have a robust set of tools to use.
Are graphs poised to take their place alongside relational database management systems (RDBMS), object storage, and other fundamental data building blocks? What are the new applications for such tools?
Inside the black box of algorithms: whither regulation?It’s possible for a machine to create an algorithm no human can understand. Evolutionary approaches to algorithmic optimization can result in inscrutable, yet demonstrably better, computational solutions.
If you’re a regulated bank, you need to share your algorithms with regulators. But if you’re a private trader, you’re under no such constraints. And having to explain your algorithms limits how you can generate them.
As more and more of our lives are governed by code that decides what’s best for us, replacing laws, actuarial tables, personal trainers, and personal shoppers, oversight means opening up the black box of algorithms so they can be regulated.
Years ago, Orbitz was shown to be charging web visitors who owned Apple devices more money than those visiting via other platforms, such as the PC. Only that’s not the whole story: Orbitz’s machine learning algorithms, which optimized revenue per customer, learned that the visitor’s browser was a predictor of their willingness to pay more. Read more…
In this O'Reilly Radar Podcast: David Rose on fairy tale inspiration, and Simon King on designing for future context.
In this podcast episode, David Rose, an instructor at MIT’s Media Lab and CEO at Ditto Labs, sits down with Mary Treseler, O’Reilly’s director of strategic content for our design space. In the interview, Rose defines his mission: “to make technology more elegant, more embedded, and hopefully, more humane.” Technology itself isn’t what drives Rose — he’s looking for inspiration in places that have captured and fueled our imaginations for centuries:
“I’m trying to be very, sort of, fairy-tale driven rather than tech driven. In the book [Enchanted Objects], I go back to some of the patterns that are revealed through Hans Christian Andersen or the Brothers Grimm or other pop culture, like spy culture or Harry Potter or Frodo, and I try to think about what those technologies are or how those services are transferable from one person to another.
“Super powers like Superman’s ability to fly don’t count because he can’t give that to anyone else, but if it’s boots that allow you to walk many miles that you wouldn’t otherwise be able to walk or a purse that replenishes or a magic carpet that could transport anybody, those qualify because those are objects that can be used by many people. I have gone back, studied these crystal balls and other objects of enchantment and magic, and think about how those could be used as a way to inspire the inventors of The Internet of Things today.”
Behind the scenes, there's a lot more to bitcoin and blockchain than first meets the eye.
Editor’s note: this is an excerpt from Chapter 1 of our recently released book Mastering Bitcoin, by Andreas Antonopoulos. You can read the full chapter here. Antonopoulos will be speaking at our upcoming event Bitcoin & the Blockchain, January 27, 2015, in San Francisco. Find out more about the event and reserve your spot here.Bitcoin is a collection of concepts and technologies that form the basis of a digital money ecosystem. Units of currency called bitcoins are used to store and transmit value among participants in the bitcoin network. Bitcoin users communicate with each other using the bitcoin protocol, primarily via the Internet; although, other transport networks can also be used. The bitcoin protocol stack, available as open source software, can be run on a wide range of computing devices, including laptops and smartphones, making the technology easily accessible.
Users can transfer bitcoin over the network to do just about anything that can be done with conventional currencies, such as buy and sell goods, send money to people or organizations, or extend credit. Bitcoin technology includes features that are based on encryption and digital signatures to ensure the security of the bitcoin network. Bitcoins can be purchased, sold, and exchanged for other currencies at specialized currency exchanges. Bitcoin, in a sense, is the perfect form of money for the Internet because it is fast, secure, and borderless. Read more…
A look at a few ways humans mesh with the rest of our data systems.
Here’s a look at a few of the ways that humans — still the ultimate data processors — mesh with the rest of our data systems: how computational power can best produce true cognitive augmentation.
Deciding betterOver the past decade, we fitted roughly a quarter of our species with sensors. We instrumented our businesses, from the smallest market to the biggest factory. We began to consume that data, slowly at first. Then, as we were able to connect data sets to one another, the applications snowballed. Now that both the front office and the back office are plugged into everything, business cares. A lot.
While early adopters focused on sales, marketing, and online activity, today, data gathering and analysis is ubiquitous. Governments, activists, mining giants, local businesses, transportation, and virtually every other industry lives by data. If an organization isn’t harnessing the data exhaust it produces, it’ll soon be eclipsed by more analytical, introspective competitors that learn and adapt faster.
Whether we’re talking about a single human made more productive by a smartphone-turned-prosthetic-brain, or a global organization gaining the ability to make more informed decisions more quickly, ultimately, Strata + Hadoop World has become about deciding better.
What does it take to make better decisions? How will we balance machine optimization with human inspiration, sometimes making the best of the current game and other times changing the rules? Will machines that make recommendations about the future based on the past reduce risk, raise barriers to innovation, or make us vulnerable to improbable Black Swans because they mistakenly conclude that tomorrow is like yesterday, only more so? Read more…
Joi Ito on the evolution of manufacturing.
Editor’s note: this interview with Joichi Ito is an excerpt from our recent report, When Hardware Meets Software, by Mike Barlow. The report looks into the new hardware movement, telling its story through the people who are building it. For more stories on the evolving relationship between software and hardware, download the free report.Joichi Ito is the director of the MIT Media Lab. Ito, who is also co-chair of the O’Reilly Solid Conference, recalls sending a group of MIT students to Shenzhen so they could see for themselves how manufacturing is evolving. “Once they got their heads around the processes in a deep way, they understood the huge differences between prototyping and manufacturing. Design for prototyping and design for manufacturing are fundamentally different,” says Ito. The problem in today’s world, according to Ito, is that “we have abstracted industrial design to the point where we think that we can just throw designs over a wall” and somehow they will magically reappear as finished products.
The trip to Shenzhen helped the students understand the manufacturing process from start to finish. “In Shenzhen, they have a $12 phone. It’s amazing. It has no screws holding it together. It’s clearly designed to be as cheap as possible. It’s also clearly designed by someone who really understands manufacturing and understands what consumers want.”
Ito also sees a significant difference between what’s happening on the factory floors in Shenzhen and the maker movement. “We’re not talking about low-volume, DIY manufacturing,” he says. Instead, Ito’s students are working through the problems and challenges of a real, live paradigm shift — the kind of gut-wrenching upheaval described in Thomas S. Kuhn’s seminal book, The Structure of Scientific Revolutions. From Kuhn’s point of view, a paradigm shift isn’t a cause for celebration or blithe headlines — it’s a sharp and unexpected blow that topples old theories, wrecks careers, and sweeps aside entire fields of knowledge. Read more…