ENTRIES TAGGED "algorithms"

Simpler workflow tools enable the rapid deployment of models

The importance of data science tools that let organizations easily combine, deploy, and maintain algorithms

Data science often depends on data pipelines, that involve acquiring, transforming, and loading data. (If you’re fortunate most of the data you need is already in usable form.) Data needs to be assembled and wrangled, before it can be visualized and analyzed. Many companies have data engineers (adept at using workflow tools like Azkaban and Oozie), who manage1 pipelines for data scientists and analysts.

A workflow tool for data analysts: Chronos from airbnb
A raw bash scheduler written in Scala, Chronos is flexible, fault-tolerant2, and distributed (it’s built on top of Mesos). What’s most interesting is that it makes the creation and maintenance of complex workflows more accessible: at least within airbnb, it’s heavily used by analysts.

Job orchestration and scheduling tools contain features that data scientists would appreciate. They make it easy for users to express dependencies (start a job upon the completion of another job), and retries (particularly in cloud computing settings, jobs can fail for a variety of reasons). Chronos comes with a web UI designed to let business analysts3 define, execute, and monitor workflows: a zoomable DAG highlights failed jobs and displays stats that can be used to identify bottlenecks. Chronos lets you include asynchronous jobs – a nice feature for data science pipelines that involve long-running calculations. It also lets you easily define repeating jobs over a finite time interval, something that comes in handy for short-lived4 experiments (e.g. A/B tests or multi-armed bandits).

Read more…

Comment
Four short links: 8 March 2013

Four short links: 8 March 2013

Comparing Algorithms, Programming & Visual Arts, Data Brokers, and Your Brain on Ebooks

  1. mlcompa free website for objectively comparing machine learning programs across various datasets for multiple problem domains.
  2. Printing Code: Programming and the Visual Arts (Vimeo) — Rune Madsen’s talk from Heroku’s Waza. (via Andrew Odewahn)
  3. What Data Brokers Know About You (ProPublica) — excellent run-down on the compilers of big data about us. Where are they getting all this info? The stores where you shop sell it to them.
  4. Subjective Impressions Do Not Mirror Online Reading Effort: Concurrent EEG-Eyetracking Evidence from the Reading of Books and Digital Media (PLOSone) — Comprehension accuracy did not differ across the three media for either group and EEG and eye fixations were the same. Yet readers stated they preferred paper. That preference, the authors conclude, isn’t because it’s less readable. From this perspective, the subjective ratings of our participants (and those in previous studies) may be viewed as attitudes within a period of cultural change.
Comment
Four short links: 11 January 2013

Four short links: 11 January 2013

Comms 101, RoboTurking, Geek Tourism, and Implementing Papers

  1. How to Redesign Your App Without Pissing Everybody Off (Anil Dash) — the basic straightforward stuff that gets your users on-side. Anil’s making a career out of being an adult.
  2. Clockwork Raven (Twitter) — open source project to send data analysis tasks to Mechanical Turkers.
  3. Updates from the Tour in China (Bunnie Huang) — my dream geek tourism trip: going around Chinese factories and bazaars with MIT geeks.
  4. How to Implement an Algorithm from a Scientific PaperI have implemented many complex algorithms from books and scientific publications, and this article sums up what I have learned while searching, reading, coding and debugging. (via Siah)
Comment
Four short links: 17 October 2012

Four short links: 17 October 2012

Invisible Data Economy, Hacked Value, Open Algorithms Textbook, and Mobile Testing

  1. Beyond Goods and Services: The Unmeasured Rise of the Data-Driven Economy — excellent points about data as neither good nor service, and how data use goes unmeasured by economists and thus doesn’t influence policy. According to statistics from the Bureau of Economic Analysis, real consumption of ‘internet access’ has been falling since the second quarter of 2011. In other words, according to official U.S. government figures, consumer access to the Internet—including mobile—has been a drag on economic growth for the past year and a half. (via Mike Loukides)
  2. How Crooks Turn Even Crappy Hacked PCs Into Money (Brian Krebs) — show to your corporate IT overlords, or your parents, to explain why you want them to get rid of the Windows XP machines. (via BoingBoing)
  3. Open Data Structures — an open content textbook (Java and C++ editions; CC-BY licensed) on data structures. (via Hacker News)
  4. Mobiforge — test what gets sent back to mobile browsers. This site sends the HTTP headers that a mobile browser would. cf yesterday’s Responsivator. (via Ronan Cremin)
Comment
Four short links: 16 October 2012

Four short links: 16 October 2012

News App, Data Wrangler, Responsive Previews, and Accountable Algorithms

  1. cir.ca — news app for iPhone, which lets you track updates and further news on a given story. (via Andy Baio)
  2. DataWrangler (Stanford) — an interactive tool for data cleaning and transformation. Spend less time formatting and more time analyzing your data. From the Stanford Visualization Group.
  3. Responsivator — see how websites look at different screen sizes.
  4. Accountable Algorithms (Ed Felten) — When we talk about making an algorithmic public process open, we mean two separate things. First, we want transparency: the public knows what the algorithm is. Second, we want the execution of the algorithm to be accountable: the public can check to make sure that the algorithm was executed correctly in a particular case. Transparency is addressed by traditional open government principles; but accountability is different.
Comment: 1
Four short links: 8 October 2012

Four short links: 8 October 2012

Drone Conflict, 3D Scanning Booths, Bitcoin Consensus, and Moar Coders

  1. Beware the Drones (Washington Times) — the temptation to send difficult to detect, unmanned aircraft into foreign airspace with perceived impunity means policymakers will naturally incline towards aggressive use of drones and hyperactive interventionism, leading us to a future that is ultimately plagued by more, not less warfare and conflict. This. Also, what I haven’t seen commented on with the Israeli air force shooting down a (presumably Hezbollah) drone: low cost of drones vs high cost of maintaining an air force to intercept, means this is asymmetric unmanned warfare.
  2. Scanbooth (github) — a collection of software for running a 3D scanning booth. Greg Borenstein said to me, “we need tools to scan and modify before 3D printing can take off.” (via Jeremy Herrman)
  3. Bitcoin’s Value is Decentralization (Paul Bohm) — Bitcoin isn’t just a currency but an elegant universal solution to the Byzantine Generals’ Problem, one of the core problems of reaching consensus in Distributed Systems. Until recently it was thought to not be practically solvable at all, much less on a global scale. Irrespective of its currency aspects, many experts believe Bitcoin is brilliant in that it technically made possible what was previously thought impossible. (via Mike Loukides)
  4. Blue Collar Coder (Anil Dash) — I am proud of, and impressed by, Craigslist’s ability to serve hundreds of millions of users with a few dozen employees. But I want the next Craigslist to optimize for providing dozens of jobs in each of the towns it serves, and I want educators in those cities to prepare young people to step into those jobs. Time for a Massively Multiplayer Online Economy, as opposed to today’s fun economic games of Shave The Have-Nots and Race To The Oligarchy.
Comment

Wall Street’s robots are not out to get you

Putting high-frequency trading into perspective.

ABOVE by Lyfetime, on FlickrTechnology is critical to today’s financial markets. It’s also surprisingly controversial. In most industries, increasing technological involvement is progress, not a problem. And yet, people who believe that computers should drive cars suddenly become Luddites when they talk about computers in trading.

There’s widespread public sentiment that technology in finance just screws the “little guy.” Some of that sentiment is due to concern about a few extremely high-profile errors. A lot of it is rooted in generalized mistrust of the entire financial industry. Part of the problem is that media coverage on the issue is depressingly simplistic. Hyperbolic articles about the “rogue robots of Wall Street” insinuate that high-frequency trading (HFT) is evil without saying much else. Very few of those articles explain that HFT is a catchall term that describes a host of different strategies, some of which are extremely beneficial to the public market.

I spent about six years as a trader, using automated systems to make markets and execute arbitrage strategies. From 2004-2011, as our algorithms and technology became more sophisticated, it was increasingly rare for a trader to have to enter a manual order. Even in 2004, “manual” meant instructing an assistant to type the order into a terminal; it was still routed to the exchange by a computer. Automating orders reduced the frequency of human “fat finger” errors. It meant that we could adjust our bids and offers in a stock immediately if the broader market moved, which enabled us to post tighter markets. It allowed us to manage risk more efficiently. More subtly, algorithms also reduced the impact of human biases — especially useful when liquidating a position that had turned out badly. Technology made trading firms like us more profitable, but it also benefited the people on the other sides of those trades. They got tighter spreads and deeper liquidity.

Read more…

Comments: 28
Can Future Advisor be the self-driving car for financial advice?

Can Future Advisor be the self-driving car for financial advice?

A startup mashes personal and government data with algorithms to provide automated advice.

Given the turmoil in financial markets and uncertainty abroad, good financial advice has never been more valuable. Startup Future Advisor looks to democratize personalized financial advice using the Internet, data and algorithms.

Comments: 3
Four short links: 21 May 2012

Four short links: 21 May 2012

Objectivist C, Robotrading, Meme Culture, and Mobile-controlled Peripherals

  1. Objectivist C — very clever. In Objectivist-C, each program is free to acquire as many resources as it can, without interference from the operating system. (via Tim O’Reilly)
  2. Zynga and Facebook Stock Oddities (The Atlantic) — signs of robotrading, a reminder that we’re surrounded by algorithms and only notice them when they go awry.
  3. The Final ROFLcon and Mobile’s Impact on Internet Culture (Andy Baio) — These days, memes spread faster and wider than ever, with social networks acting as the fuel for mass distribution. But it’s possible we may see less mutation and remixing in the near future. As Internet usage shifts from desktops and laptops to mobile devices and tablets, the ability to mutate memes in a meaningful way becomes harder.
  4. Oh Mi Bod — I was impressed to learn that one can buy vibrators that can be controlled from an iPhone. Insert iBone joke here. (via Cary Gibson)
Comment
Four short links: 2 May 2012

Four short links: 2 May 2012

Elective Dickery, Probabilistic Data Analysis, Data Cleaning, and SSL Security

  1. Punting on SxSW (Brad Feld) — I came across this old post and thought: if you can make money by being a dick, or make money by being a caring family person, why would you choose to be a dick? As far as I can tell, being a dick is optional. Brogrammers, take note. Be more like Brad Feld, who prioritises his family and acts accordingly.
  2. Probabilistic Structures for Data Mining — readable introduction to useful algorithms and datastructures showing their performance, reliability, and resources trade-off. (via Hacker News)
  3. Dataset — a Javascript library for transforming, querying, manipulating data from different sources.
  4. Many HTTPS Servers are Insecure — 75% still vulnerable to the BEAST attack.
Comment