"data" entries

Four short links: 23 January 2015

Four short links: 23 January 2015

Investment Themes, Python Web Mining, Code Review, and Sexist Brilliance

  1. 16 Andreessen-Horowitz Investment Areas — I’m struck by how they’re connected: there’s a cluster around cloud development, there are two maybe three on sensors …
  2. Patterna web mining module for the Python programming language. It has tools for data mining (Google, Twitter and Wikipedia API, a web crawler, a HTML DOM parser), natural language processing (part-of-speech taggers, n-gram search, sentiment analysis, WordNet), machine learning (vector space model, clustering, SVM), network analysis and <canvas> visualization.
  3. Code Review — FogCreek’s code review checklist.
  4. Expectations of Brilliance Underlie Gender Distributions Across Academic Disciplines (Science) — Surveys revealed that some fields are believed to require attributes such as brilliance and genius, whereas other fields are believed to require more empathy or hard work. In fields where people thought that raw talent was required, academic departments had lower percentages of women. (via WaPo)
Four short links: 7 January 2015

Four short links: 7 January 2015

Program Synthesis, Data Culture, Metrics, and Information Biology

  1. Program Synthesis ExplainedThe promise of program synthesis is that programmers can stop telling computers how to do things, and focus instead on telling them what they want to do. Inductive program synthesis tackles this problem with fairly vague specifications and, although many of the algorithms seem intractable, in practice they work remarkably well.
  2. Creating a Data-Driven Culture — new (free!) ebook from Hilary Mason and DJ Patil. The editor of that team is the luckiest human being alive.
  3. Ev Williams on Metrics — a master-class in how to think about and measure what matters. If what you care about — or are trying to report on — is impact on the world, it all gets very slippery. You’re not measuring a rectangle, you’re measuring a multi-dimensional space. You have to accept that things are very imperfectly measured and just try to learn as much as you can from multiple metrics and anecdotes.
  4. Nature, the IT Wizard (Nautilus) — a fun walk through the connections between information theory, computation, and biology.
Four short links: 5 January 2015

Four short links: 5 January 2015

3D Art Reuse, Faceted Data Browser, Robotics Roundup, and Social Signal Interpretation

  1. Lincoln Gallery Reuse — UK gallery placed 3D models of their works online and are sharing what people did with them. Some beautiful art in here! (via BoingBoing)
  2. Kesihif — open source browser for faceted data.
  3. 2014 Robotics IPOs, Acquisitions, and Failures (RoboHub) — good roundup of what happened in 2014.
  4. SSIan open source platform for social signal interpretation.
Four short links: 18 December 2014

Four short links: 18 December 2014

Manufacturer Rootkits, Dangerous Dongle, Physical Visualisation, and Cryptoed Comms

  1. Popular Chinese Android Smartphone Backdoored By ManufacturerCoolpad is the third largest smartphone builder in China, and ranks sixth worldwide with 3.7 percent global market share. It trails only Lenovo and Xiaomi in China and is the leader of China’s 4G market with 16 percent market share. Coolpad outsells Samsung and Apple in China, and has said it plans to expand globally with a goal of 60 million phones worldwide. For now, its high-end Halo Dazen phones are the only ones containing the backdoor, Palo Alto said. Backdoor enabled installation of other apps, dial numbers, send messages, and report back to the mothership. The manufacturer even ran the command-and-control nodes for the malware.
  2. USB Driveby — dongle that plugs into USB, and tries to root the box. Specifically, when you normally plug in a mouse or keyboard into a machine, no authorization is required to begin using them. The devices can simply begin typing and clicking. We exploit this fact by sending arbitrary keystrokes meant to launch specific applications (via Spotlight/Alfred/Quicksilver), permanently evade a local firewall (Little Snitch), install a reverse shell in crontab, and even modify DNS settings without any additional permissions.
  3. Physical Data Visualisationsa chronological list of physical visualizations and related artifacts. (via Flowing Data)
  4. Dissentan anonymous communication substrate intended primarily for applications built on a broadcast communication model: for example, bulletin boards, wikis, auctions, or voting. Users of an online group obtain cryptographic guarantees of sender and receiver anonymity, message integrity, disruption resistance, proportionality, and location hiding. And a pony.
Four short links: 11 December 2014

Four short links: 11 December 2014

Crowdsourcing Framework, Data Team Culture, Everybody Scrolls, and Honeypot Data

  1. Hive — open source crowdsourcing framework from NYT Labs.
  2. Prezi Data Team Culture — good docs on logging, metrics, etc. The vision is a great place to start.
  3. Scroll Behaviour Across the Web (Chartbeat) — nobody reads above the fold, they immediately scroll.
  4. threat_research (github) — shared raw data and stats from honeypots.
Four short links: 9 December 2014

Four short links: 9 December 2014

Tab Tool, Ad Manifesto, Cultural Heritage, and Software Sustainability

  1. tab — command-line tool for doing heavy lifting with tab-separated files.
  2. Acceptable Ads — manifesto from the makers of AdBlock Plus. (via Monday Note)
  3. Cultural Heritage of Humanity (Matt Webb) — Matt points to UNESCO’s Representative List of the Intangible Cultural Heritage of Humanity and comments: When the aliens land and set up shop and they’re like, “Guys, so what have you got?” And we’re all… “Uh, lasers? We’ll trade you lasers for a starship drive.” And the aliens will be: “Nope, what else?” Then we’ll say: “Tsiattista poetic duelling. Turkish coffee. Jazz.” Bingo. Kudos to UNESCO for prepping our inventory ahead of time.
  4. Apache (and Other Foundations) Considered Useful (Chris Aniszczyk) — have over a decade of experience being built for the sole purpose of allowing independent open source communities to flourish with fair governance models […] This is important because the incentives between individuals small companies, large companies, heavily funded companies and even academics are different and need to be accounted for in a fair open source governance structure. Sustainability of software commons is an unsolved problem, but foundations make it tractable.

Signals from Strata + Hadoop World in Barcelona 2014

From the Internet of Things to data-driven fashion, here are key insights from Strata + Hadoop World in Barcelona 2014.

Experts from across the big data world came together for Strata + Hadoop World in Barcelona 2014. We’ve gathered insights from the event below.

#IoTH: The Internet of Things and Humans

“If we could start over with these capabilities we have now, how would we do it differently?” Tim O’Reilly continues to explore data and the Internet of Things through the lens of human empowerment and the ability to “use technology to give people superpowers.”

Read more…

Four short links: 14 November 2014

Four short links: 14 November 2014

Completing Maps, ChatOps, Career Design, and Data Privacy

  1. Missing Maps Fill In the Blanks (New Scientist) — OpenStreetMap project to crowdmap slums around the world.
  2. Chatops — devops deployment chatter with Hubot.
  3. Alternatives to Tech Career Ladders — Spotify trying to figure out how to keep engineers challenged as they become more senior.
  4. Mozilla’s Data Privacy Principles — well-articulated and useful: without pre-defined principles, it’s so easy to accidentally collect or poorly protect data.
Four short links: 11 November 2014

Four short links: 11 November 2014

High-Volume Logs, Regulated Broadband, Oculus Web, and Personal Data Vacuums

  1. Infrastructure for Data Streams — describing the high-volume log data use case for Apache Kafka, and how it plays out in storage and infrastructure.
  2. Obama: Treat Broadband and Mobile as Utility (Ars Technica) — In short, Obama is siding with consumer advocates who have lobbied for months in favor of reclassification while the telecommunications industry lobbied against it.
  3. MozVR — a website, and the tools that made it, designed to be seen through the Oculus Rift.
  4. All Cameras are Police Cameras (James Bridle) — how the slippery slope is ridden: When the Wall was initially constructed, the public were informed that this [automatic license plate recognition] data would only be held, and regularly purged, by Transport for London, who oversee traffic matters in the city. However, within less than five years, the Home Secretary gave the Metropolitan Police full access to this system, which allowed them to take a complete copy of the data produced by the system. This permission to access the data was granted to the Police on the sole condition that they only used it when National Security was under threat. But since the data was now in their possession, the Police reclassified it as “Crime” data and now use it for general policing matters, despite the wording of the original permission. As this data is not considered to be “personal data” within the definition of the law, the Police are under no obligation to destroy it, and may retain their ongoing record of all vehicle movements within the city for as long as they desire.
Four short links: 6 November 2014

Four short links: 6 November 2014

Javascript Testing, Dark Data, Webapp Design, and Design Trumps Data

  1. Karma — kick-ass open source Javascript test environment.
  2. The Dark Market for Personal Data (NYTimes) — can buy lists of victims of sexual assault, of impulse buyers, of people with sexually transmitted disease, etc. The cost of a false-positive when those lists are used for marketing is less than the cost of false-positive when banks use the lists to decide whether you’re a credit risk. The lists fall between the cracks in privacy legislation; essentially, the compilation and use of lists of people are unregulated territory.
  3. 7 Principles of Rich Web Applications — “rich web applications” sounds like 2007 wants its ideas back, but the content is modern and useful. Predict behaviour for negative latency.
  4. Collaborative Filtering at LinkedIn (PDF) — This paper presents LinkedIn’s horizontal collaborative filtering infrastructure, known as browsemaps. Great lessons learned, including context and presentation of browsemaps or any recommendation is paramount for a truly relevant user experience. That is, design and presentation represents the largest ROI, with data engineering being a second, and algorithms last. (via Greg Linden)