"Java" entries

Scaling People, Process, and Technology with Python

OSCON 2013 Speaker Series

NOTE: If you are interested in attending OSCON to check out Dave’s talk or the many other cool sessions, click over to the OSCON website where you can use the discount code OS13PROG to get 20% off your registration fee.

Since 2009, I’ve been leading the optimization team at AppNexus, a real-time advertising exchange. On this exchange, advertisers participate in real-time auctions to bid on individual ad impressions. The highest bid wins the auction, and that advertiser gets to show an ad. This allows advertisers to carefully target where they advertise—maximizing the effectiveness of their advertising budget—and lets websites maximize their ad revenue.

We do these auctions often (~50 billion a day) and fast (<100 milliseconds). Not surprisingly, this creates a lot of technical challenges. One of those challenges is how to automatically maximize the value advertisers get for their marketing budgets—systematically driving consumer engagement through ad placements on particular websites, times of day, etc.—and we call this process “optimization.” The volume of data is large, and the algorithms and strategies aren’t trivial.

In order to win clients and build our business to the scale we have today, it was crucial that we build a world-class optimization system. But when I started, we didn’t have a scalable tech stack to process the terabytes of data flowing through our systems every day, and we didn't have the team to do any of the required data modeling.

People

So, we needed to hire great people fast. However, there aren’t many veterans in the advertising optimization space, and because of that, we couldn’t afford to narrow our search to only experts in Java or R or Matlab. In order to give us the largest talent pool possible to recruit from, we had to choose a tech stack that is both powerful and accessible to people with diverse experience and backgrounds. So we chose Python.

Python is easy to learn. We found that people coding in R, Matlab, Java, PHP, and even those who have never programmed before could quickly learn and get up to speed with Python. This opened us up to hiring a tremendous pool of talent who we could train in Python once they joined AppNexus. To top it off, there’s a great community for hiring engineers and the PyData community is full of programmers who specialize in modeling and automation.

Additionally, Python has great libraries for data modeling. It offers great analytical tools for analysts and quants and when combined, Pandas, IPython, and Matplotlib give you a lot of the functionality of Matlab or R. This made it easy to hire and onboard our quants and analysts who were familiar with those technologies. Even better, analysts and quants can share their analysis through the browser with IPython.

Process

Now that we had all of these wonderful employees, we needed a way to cut down the time to get them ramped up and pushing code to production.

First, we wanted to get our analysts and quants looking at and modeling data as soon as possible. We didn’t want them worrying about writing database connector code, or figuring out how to turn a cursor into a data frame. To tackle this, we built a project called Link.

Imagine you have a MySQL database. You don’t want to hardcode all of your connection information because you want to have a different config for different users, or for different environments. Link allows you to define your “environment” in a JSON config file, and then reference it in code as if it is a Python object.

Now, with only three lines of code you have a database connection and a data frame straight from your mysql database. This same methodology works for Vertica, Netezza, Postgres, Sqlite, etc. New “wrappers” can be added to accommodate new technologies, allowing team members to focus on modeling the data, not how to connect to all these weird data sources.

By having the flexibility to easily connect to new data sources and APIs, our quants were able to adapt to the evolving architectures around us, and stay focused on modeling data and creating algorithms.

Second, we wanted to minimize the amount of work it took to take an algorithm from research/prototype phase to full production scale. Luckily, with everyone working in Python, our quants, analysts, and engineers are using the same language and data processing libraries. There was no need to re-implement an R script in Java to get it out across the platform.
Read more…

Comment

Google Glass: What Developers Need to Know about This New Platform

Creating Glassware today and what's in store for tomorrow

You’ve likely already seen pictures of people using Google Glass, if not had an actual in-the-wild spotting as well. After getting a quick demo myself, I spoke with Maximiliano Firtman about his talk at Fluent conference that covers what developers need to start doing and thinking about when it comes to developing apps for this new environment.

Key highlights include:

  • The current version supports cloud-based web applications that can be built in any language using the Mirror API. [Discussed at 0:30]
  • A forthcoming SDK will support native app development, essentially Android apps written in Java. [Discussed at 2:20]
  • The only truly augmented reality type application currently available is Google Maps. [Discussed at 3:30]
  • Developers need to think outside the technical details as well, and spend time considering how people will be interacting with Google Glass—it’s a uniquely new paradigm with unique use cases. [Discussed at 4:14]
  • While the beta (Explorer) program is currently closed, Max expects to see more devices available and “on the street” within the next year. [Discussed at 6:10]

You can view the full interview here:

Read more…

Comment

JavaScript: Not as Expected

A good match for the similarly unexpected Web?

JavaScript’s ever-growing importance still takes people by surprise. Every time I post about things JavaScript makes possible, I get pushback from people who refuse to be impressed by JavaScript. Why? Because it isn’t what they wanted.

In the course of a week, I get to hear from different quarters about how JavaScript is half Lisp, and terrible either because it dares to be half-Lisp or because it only manages to be half Lisp. Similarly, as functional programming has become more visible, I’ve heard more from people who think JavaScript programming is too functional or not functional enough. People disappointed in JavaScript because it doesn’t have strong typing are a constant, as are people who find prototypal inheritance perverse. JavaScript syntax—I’m sure someone must like it.

It’s tempting to tell the story of JavaScript as a series of historical accidents. Brendan Eich put together LiveScript, applying a variety of techniques to meet a particular set of needs quickly. Since then, we’ve been dealing with JavaScript’s shift from a simple object manipulation language to a much broader and more powerful toolkit, unable to change approach because of the unique dynamics of the browser world.

Read more…

Comment: 1

Why we need Go

Rob Pike on how Go fits into today's computing environment

Go programming languageThe Go programming language was created by Rob Pike, Ken Thompson, and Robert Griesemer. Pike (@rob_pike) recently told me that Go was born while they were waiting a long while for some code to compile — too long.

C++ and Java have long been the go-to languages for big server or system programs, but they were created almost 30 and 20 years ago, respectively. They don’t address very well the issues programmers see today like use of concurrency and incorporating big data and they’re not optimal for the current programming environment.

One main reason that Go will succeed is how it deals with concurrency. It outpaces Java and C++ as well as Python, Ruby, and all the other scripting languages. It simply provides a better model, with Java a close second, that is able to work within the computing environment into which it was born.

During a recent interview, Pike elaborated on the need for Go and where it fits in today’s programming landscape. Highlights from our discussion include: Read more…

Comments: 4
Four short links: 7 September 2012

Four short links: 7 September 2012

Collections, Games, Accessibility, and Science

  1. GS-Collections (GitHub) — Goldman Sachs open-sourced (Apache-licensed) their Java collection library, full of lambda goodness. No report on whether it requires a 750G bailout.
  2. Learning ZIL — old manual for the interactive fiction programming language that Zork and other Infocom games were written in. Virtual machines on a Z80 processor? They were hardcore before your time.
  3. NZ Government Web Toolkit — information and guides on accessibility standards.
  4. Workshop on Research and Resource Commons in Scientific Research: Final ReportThis diverse group discussed the current state of policy and technology as it relates to a scientific research commons, and identified key opportunities and challenges, as well as next steps, for the scientific community in general and Creative Commons in particular. Wilbanks describes as, “Sort of a wrapup after seven years of SC.” (via John Wilbanks)
Comment: 1

Developer Week in Review: Oracle's big bet fails to pay off

Google dodges a bullet, a new Perl in town, and GCC loses an OS.

Oracle fails to convince a jury that Google owes them big bucks, the annual refresh of Perl has arrived, and FreeBSD says goodbye to an increasingly restrictive GCC license.

Comment

Developer Week in Review: Oracle’s big bet fails to pay off

Google dodges a bullet, a new Perl in town, and GCC loses an OS.

Oracle fails to convince a jury that Google owes them big bucks, the annual refresh of Perl has arrived, and FreeBSD says goodbye to an increasingly restrictive GCC license.

Comment
Four short links: 15 May 2012

Four short links: 15 May 2012

Mobile Money, Actors in java, Actors in python, and a Decision-Making Tool

  1. Mobile Money (The Economist) — Many people know that “mobile money”—financial transactions on mobile phones—has taken off in Africa. How far it has gone, though, still comes as a bit of a shock. Three-quarters of the countries that use mobile money most frequently are in Africa, and mobile banking in some of them has reached extraordinary levels.
  2. Akka — Apache-licensed Java high-performance concurrency library built around the concept of “actors“. (via Hacker News)
  3. Pykka — actors in Python. (via Hacker News)
  4. Loom.io Project — help crowdfund a collaborative decision-making tool. They’re using it as they build the tool, and it’s the implementation of a process they use in real life. I know many organisations who need a free open-source web application that helps groups make better decisions together. You should probably read more about the interesting company Enspiral which is behind loom.io.
Comment

Developer Week in Review: Java on trial

The trial of the century continues, cat feeders and coding, and PHP sites at risk.

Google and Oracle continue to duke it out in court, with more than just Android at risk. One developer uses cat feeders as a way to look at good software, and the PHP developers take a second try at fixing a critical bug.

Comment

Editorial Radar: Functional languages

The benefits of functional languages and functional language techniques.

O'Reilly editors Mike Loukides and Mike Hendrickson discuss the advantages of functional programming languages and how functional language techniques can be deployed with almost any language.

Comment: 1