- How to Easily Resize and Cache Images for the Mobile Web (Pete Warden) — I set up a server running the excellent ImageProxy open-source project, and then I placed a Cloudfront CDN in front of it to cache the results. (a how-to covering the tricksy bits)
- Google’s Position on Net Neutrality Changes? (Wired) — At issue is Google Fiber’s Terms of Service, which contains a broad prohibition against customers attaching “servers” to its ultrafast 1 Gbps network in Kansas City. Google wants to ban the use of servers because it plans to offer a business class offering in the future. [...] In its response [to a complaint], Google defended its sweeping ban by citing the very ISPs it opposed through the years-long fight for rules that require broadband providers to treat all packets equally.
- The Future of Programming (Bret Victor) — gorgeous slides, fascinating talk, and this advice from Alan Kay: I think the trick with knowledge is to “acquire it, and forget all except the perfume” — because it is noisy and sometimes drowns out one’s own “brain voices”. The perfume part is important because it will help find the knowledge again to help get to the destinations the inner urges pick.
- psd.rb — Ruby code for reading PSD files (MIT licensed).
ENTRIES TAGGED "Google"
Mobile Image Cache, Google on Net Neutrality, Future of Programming, and PSD Files in Ruby
Retreading old topics can be a powerful source of epiphany, sometimes more so than simple extra-box thinking. I was a computer science student, of course I knew statistics. But my recent years as a NoSQL (or better stated: distributed systems) junkie have irreparably colored my worldview, filtering every metaphor with a tinge of information management.
Lounging on a half-world plane ride has its benefits, namely, the opportunity to read. Most of my Delta flight from Tel Aviv back home to Portland lacked both wifi and (in my case) a workable laptop power source. So instead, I devoured Nate Silver’s book, The Signal and the Noise. When Nate reintroduced me to the concept of statistical overfit, and relatedly underfit, I could not help but consider these cases in light of the modern problem of distributed data management, namely, operators (you may call these operators DBAs, but please, not to their faces).
When collecting information, be it for a psychological profile of chimp mating rituals, or plotting datapoints in search of the Higgs Boson, the ultimate goal is to find some sort of usable signal, some trend in the data. Not every point is useful, and in fact, any individual could be downright abnormal. This is why we need several points to spot a trend. The world rarely gives us anything clearer than a jumble of anecdotes. But plotted together, occasionally a pattern emerges. This pattern, if repeatable and useful for prediction, becomes a working theory. This is science, and is generally considered a good method for making decisions.
On the other hand, when lacking experience, we tend to over value the experience of others when we assume they have more. This works in straightforward cases, like learning to cook a burger (watch someone make one, copy their process). This isn’t so useful as similarities diverge. Watching someone make a cake won’t tell you much about the process of crafting a burger. Folks like to call this cargo cult behavior.
How Fit are You, Bro?
You need to extract useful information from experience (which I’ll use the math-y sounding word datapoints). Having a collection of datapoints to choose from is useful, but that’s only one part of the process of decision-making. I’m not speaking of a necessarily formal process here, but in the case of database operators, merely a collection of experience. Reality tends to be fairly biased toward facts (despite the desire of many people for this to not be the case). Given enough experience, especially if that experience is factual, we tend to make better and better decisions more inline with reality. That’s pretty much the essence of prediction. Our mushy human brains are more-or-less good at that, at least, better than other animals. It’s why we have computers and Everybody Loves Raymond, and my cat pees in a box.
Imagine you have a sufficient amount of relevant datapoints that you can plot on a chart. Assuming the axes have any relation to each other, and the data is sound, a trend may emerge, such as a line, or some other bounding shape. A signal is relevant data that corresponds to the rules we discover by best fit. Noise is everything else. It’s somewhat circular sounding logic, and it’s really hard to know what is really a signal. This is why science is hard, and so is choosing a proper database. We’re always checking our assumptions, and one solid counter signal can really be disastrous for a model. We may have been wrong all along, missing only enough data. As Einstein famously said in response to the book 100 Authors Against Einstein: “If I were wrong, then one would have been enough!”
Database operators (and programmers forced to play this role) must make predictions all the time, against a seemingly endless series of questions. How much data can I handle? What kind of latency can I expect? How many servers will I need, and how much work to manage them?
So, like all decision making processes, we refer to experience. The problem is, as our industry demands increasing scale, very few people actually have much experience managing giant scale systems. We tend to draw our assumptions from our limited, or biased smaller scale experience, and extrapolate outward. The theories we then tend to concoct are not the optimal fit that we desire, but instead tend to be overfit.
Overfit is when we have a limited amount of data, and overstate its general implications. If we imagine a plot of likely failure scenarios against a limited number of servers, we may be tempted to believe our biggest odds of failure are insufficient RAM, or disk failure. After all, my network has never given me problems, but I sure have lost a hard drive or two. We take these assumptions, which are only somewhat relevant to the realities of scalable systems and divine some rules for ourselves that entirely miss the point.
In a real distributed system, network issues tend to consume most of our interest. Single-server consistency is a solved problem, and most (worthwhile) distributed databases have some sense of built in redundancy (usually replication, the root of all distributed evil).
The App Store model has increased the uncertainty of the software release process
The recent unavailability of the Apple Developer’s Portal just underscores how increasingly dependent developers have become on third parties during the software lifecycle. For those who are not following the fun and games, the developer.apple.com sites, which include much of the functionality needed to develop Mac and iOS applications, has been unavailable for more than a week as of this writing. Although iTunes Connect, the portal used to actually deploy apps to the App Stores, has remained available, the remainder of the site territory has been off-limits. This is all thanks to a security intrusion (evidently by an over-zealous researcher.)
The App Store model has fundamentally changed how software is distributed, mostly for the better (IMHO), but it has also removed some of the control of the release process from the hands of the developers and companies they work for. As I have spelled out previously in my book on iOS enterprise development, the fact that Apple has the final say on if and when software goes into the store has required more conservative release timelines. If you want to release on the first of September, you need to count back at least two weeks for “gold master”, because you need to upload the app, potentially go through a round of rejection from Apple, and then upload a fixed version.
Android apps don’t suffer from this lag, because most of the Android stores don’t do any significant checking of the applications uploaded to them. The Devil’s Deal that Apple developers have made with Apple is that in return for the longer wait time to get apps in the store (and having to follow Apple’s rules), they get a de facto seal of approval from Apple. In other words, it is assumed that apps in the iTunes store are more stringently policed and less likely to crash or do harm (deliberately or else-wise.)
The current downtime has brought that deal into question, however. Suddenly, developers who need new provisioning certificates, passbook certificates, or push notification certificates find themselves with nowhere to go. Even if iTunes Connect is available, it doesn’t do you any good if you can’t get a distribution certificate to sign your app for the store. I’m sure that there are developers at this moment who have had their finely tuned release strategies thrown into disarray by the in-availability of the developer portal.
Being essentially at the mercy of Apple’s whims (or Google’s, for that matter) can’t be a pleasant sensation for a company or individual trying to get a new piece of software out the door. The question that the developer community will have to answer is if the benefits of the App Store model make it worth the hassles, in the long run.
Rules of the Internet, Bigness of the Data, Wifi ADCs, and Google Flirts with Client-Side Encryption
- Ten Rules of the Internet (Anil Dash) — they’re all candidates for becoming “Dash’s Law”. I like this one the most: When a company or industry is facing changes to its business due to technology, it will argue against the need for change based on the moral importance of its work, rather than trying to understand the social underpinnings.
- Data Storage by Vertical (Quartz) — The US alone is home to 898 exabytes (1 EB = 1 billion gigabytes)—nearly a third of the global total. By contrast, Western Europe has 19% and China has 13%. Legally, much of that data itself is property of the consumers or companies who generate it, and licensed to companies that are responsible for it. And in the US—a digital universe of 898 exabytes (1 EB = 1 billion gigabytes)—companies have some kind of liability or responsibility for 77% of all that data.
- x-OSC — a wireless I/O board that provides just about any software with access to 32 high-performance analogue/digital channels via OSC messages over WiFi. There is no user programmable firmware and no software or drivers to install making x-OSC immediately compatible with any WiFi-enabled platform. All internal settings can be adjusted using any web browser.
- Google Experimenting with Encrypting Google Drive (CNet) — If that’s the case, a government agency serving a search warrant or subpoena on Google would be unable to obtain the unencrypted plain text of customer files. But the government might be able to convince a judge to grant a wiretap order, forcing Google to intercept and divulge the user’s login information the next time the user types it in. Advertising depends on the service provider being able to read your data. Either your Drive’s contents aren’t valuable to Google advertising, or it won’t be a host-resistant encryption process.
Microvideos for MIcrohelp, Organic Search, Probabilistic Programming, and Cluster Management
- How to Make Help Microvideos For Your Site (Alex Holovaty) — Instead of one monolithic video, we decided to make dozens of tiny, five-second videos separately demonstrating features.
- How Google is Killing Organic Search — 13% of the real estate is organic results in a search for “auto mechanic”, 7% for “italian restaurant”, 0% if searching on an iPhone where organic results are four page scrolls away. SEO Book did an extensive analysis of just how important the top left of the page, previously occupied by organic results actually is to visitors. That portion of the page is now all Google. (via Alex Dong)
- Church — probabilistic programming language from MIT, with tutorials. (via Edd Dumbill)
- mesos — a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark (a new framework for low-latency interactive and iterative jobs), and other applications. Mesos is open source in the Apache Incubator. (via Ben Lorica)
OSCON 2013 Speaker Series
Note: Amy Unruh, Google Cloud Platform Developer Relations, is just one of the many fantastic speakers we have at OSCON this year. If you are interested in attending to check out Amy’s talk or the many other cool sessions, click over to the OSCON website where you can use the discount code OS13PROG to get 20% off your registration fee.
At this year’s Google I/O, we launched the PHP runtime for Google App Engine, part of the Google Cloud Platform. App Engine is a service that lets you build web apps using the same scalable infrastructure that powers many of Google’s own applications. With App Engine, there are no servers to maintain; you just upload your application, and it’s ready to go.
App Engine’s services support and simplify many aspects of app development. One of those services is Task Queues, which lets you easily add asynchronous background processing to your PHP app, and allows you to simultaneously make your applications more responsive, more reliable, and more scalable.
The App Engine Task Queue service allows your application to define tasks, add them to a queue, and then use the queue to process them asynchronously, in the background. App Engine automatically scales processing capacity to match your queue configuration and processing volume. You define a Task by specifying the application-specific URL of a handler for the task, along with (optionally) parameters or a payload for the task, and other settings, then add it to a Task Queue.
Thread Problems, Better Image Search, Open Standards, and GitHub Maps
- Multithreading is Hard — The compiler and the processor both conspire to defeat your threads by moving your code around! Be warned and wary! You will have to do battle with both. Sample code and explanation of WTF the eieio barrier is (hint: nothing to do with Old McDonald’s server farm). (via Erik Michaels-Ober)
- Improving Photo Search (Google Research) — volume of training images, number of CPU cores, and Freebase entities. (via Alex Dong)
- Is Google Dumping Open Standards for Open Wallets? (Matt Asay) — it’s easier to ship than standardise, to innovate than integrate, but the ux of a citizen in the real world is pants. Like blog posts? Log into Facebook to read your friends! (or Google+) Chat is great, but you’d better have one client per corporation your friends hang out on. Nobody woke up this morning asking for features to make web pages only work on one browser. The user experience of isolationism is ugly.
- GitHub Renders GeoJSON — Under the hood we use Leaflet.js to render the geoJSON data, and overlay it on a custom version of MapBox’s street view baselayer — simplified so that your data can really shine. Best of all, the base map uses OpenStreetMap data, so if you find an area to improve, edit away.
Could technology be bringing people closer together?
I had quite an experience at Maker Faire this weekend. So instead of a follow up on Google I/O today I’m going talk about how wearables, specifically Google Glass, seem to be bringing people closer together rather than farther apart. So, more on Google I/O later in the week.
A Tale of Two Events
I first broke out my Google Glass at Google I/O where Glass Explorers and Googlers filled the Moscone West sporting the device. Glass Explorers are those that pre-ordered the I/O last year and winners of the #IfIHadGlass contest. The mood towards Glass at I/O was, generally, split into the have’s and have not’s. Those with them proudly showed them off while others fell into the following camps: carefully measured excitement, cool intrigue, and those who were over it. I think for the most part the subdued reaction was a reflection of attendees wanting to be able to get into the action immediately. It was a shame that Glass wasn’t available for purchase to those at I/O this year.
In stark contrast to that reaction was the response I received from attendees of this past weekend’s Maker Faire. My first inkling of what was ahead were the whispers. I would hear excitedly, “Is that the Google Glass?” which made me smile. However, when I met up with my 11:30 a.m. appointment at his booth and started talking about and sharing the Glass with him and his colleagues a mob quickly formed. Frankly, I got scared for a moment as a mass of people forced inward towards me, and then thought what if someone just takes off with these? But, no one did. These mini-mobs happened to me twice, both times in the Electronics area (not surprisingly). The outcome of these Glass flash mobs, however, was quite simply lovely. Individuals were polite, asked me questions, wanted to take pictures of themselves with it and that was it. Throughout the day people would comment on them, stop me to talk, but it was always a pleasure with people smiling ear to ear when I had them play with the device.
What will wearables really mean to society?
The quick answer for now—who knows? I have to say I was a bit overwhelmed by all of this social engagement. I had anticipated some notice, but this? Now, granted, the attendees of a Maker Faire might skew towards being interested in new gadgets and devices but my experience was unexpected—and wonderful. I talked to more random, happy people at this event than I have in a long while. It has given me a new perspective on recent issues that have come up regarding the Glass, such as invasion of privacy and the idea that we are disconnecting with the world more and more via personal devices, when in fact I was finding just the opposite. Maybe in time everyone will have a Glass or have seen one and it won’t be a big deal. But for now, it is generating interaction and discussion about technology with young and old alike.
Oh, and here you can see what it is like to be attacked by a T-Rex from my POV via the Glass, scary stuff. Click here to see the T-Rex Attack.
This will be the first post in a series on my journey through the world with Glass.
Two views on new Google Maps; a look at predictive, intelligent apps; and Aaron Swartz's and Kevin Poulsen's anonymous inbox launches.
Google aims for a new level of map customization
Google introduced a new version of Google maps at Google I/O this week that learns from each use to customize itself to individual users, adapting based on user clicks and searches. A post on the Google blog outlines the updates, which include recommendations for places you might enjoy (based upon your map activity), ratings and reviews, integrated Google Earth, and tours generated from user photos, to name a few.
I just read a Forbes article about Glass, talking about the split between those who are “sure that it is the future of technology, and others who think society will push back against the technology.”
I don’t see this as a dichotomy (and, to be fair, I’m not sure that the author does either). I expect to see both, and I’d like to think a bit more about what these two apparently opposing sides mean.
Push back is inevitable. I hope there’s a significant push back, and that it has some results. Not because I’m a Glass naysayer, but because we, as technology users, are abused so often, and push back so weakly, that it’s not funny. Facebook does something outrageous; a few technorati whine; they add option 1023 to their current highly intertwined 1022 privacy options that have been designed so they can’t be understood or used effectively; and sooner or later, it all dies down. A hundred fifty users have left Facebook, and half a million more have joined. When Apple puts another brick in their walled garden, a few dozen users (myself included) bitch and moan, but does anyone leave? Personally, I’m tired of getting warnings whenever I install software that doesn’t come from the Apple Store (I’ve used the Store exactly twice), and I absolutely expect that a not-too-distant version of OS X won’t allow me to install software from “untrusted” sources, including software I’ve written. Will there be push back? Probably. Will it be effective? I don’t know; if things go as they are now, I doubt it.
There will be push back against Glass; and that’s a good thing. I think Google, of all the companies out there, is most likely to listen and respond positively. I say that partly because of efforts like the Data Liberation Front, and partly because Eric Schmidt has acknowledged that he finds many aspects of Glass creepy. But going beyond Glass: As a community of users, we need to empower ourselves to push back. We need to be able to push back effectively against Google, but more so against Apple, Facebook, and many other abusers of our data, rather than passively accept the latest intrusion as an inevitability. If Glass does nothing more than teach users that they can push back, and teach large corporations how to respond constructively, it will have accomplished much.
Is Glass the future? Yes; at least, something like Glass is part of the future. As a species, we’re not very good at putting our inventions back into the box. About three years ago, there was a big uptick in interest in augmented reality. You probably remember: Wikitude, Layar, and the rest. You installed those apps on your phone. They’re still there. You never use them (at least, I don’t). The problem with consumer-grade AR up until now has been that it was sort of awkward walking around looking at things through your phone’s screen. (Commercial AR–heads-up displays and the like–is a completely different ball game.) Glass is the first attempt at broadly useful platform for consumer AR; it’s a game changer.
Could Glass fail? Sure; I know more failed startups than I can count where the engineers did something really cool, and when they released it, the public said “what is that, and why do you think we’d want it?” Google certainly isn’t immune from that disease, which is endemic to an engineering-driven culture; just think back to Wave. I won’t deny that Google might shelve Glass if they consider unproductive, as they’ve shelved many popular applications. But I believe that Google is playing long-ball here, and thinking far beyond 2014 or 2015. In a conversation about Bitcoin last week, I said that I doubt it will be around in 20 years. But I’m certain we will have some kind of distributed digital currency, and that currency will probably look a lot like Bitcoin. Glass is the same. I have no doubt that something like Glass is part of our future. It’s a first, tentative, and very necessary step into a new generation of user interfaces, a new way of interacting with computing systems and integrating them into our world. We probably won’t wear devices around on our glasses; it may well be surgically implanted. But the future doesn’t happen if you only talk about hypothetical possibilities. Building the future requires concrete innovation, building inconvenient and “creepy” devices that nevertheless point to the next step. And it requires people pushing back against that innovation, to help developers figure out what they really need to build.
Glass will be part of our future, though probably not in its current form. And push back from users will play an essential role in defining the form it will eventually take.