- Why Narrative and Structure are Important (Ed Yong) — Ed looks at how Atul Gawande’s piece on death and dying, which is 12,000 words long, is an easy and fascinating read despite the length.
- Understanding Science (Berkeley) — simple teaching materials to help students understand the process of science. (via BoingBoing comments)
- Sax: Symbolic Aggregate approXimation — SAX is the first symbolic representation for time series that allows for dimensionality reduction and indexing with a lower-bounding distance measure. In classic data mining tasks such as clustering, classification, index, etc., SAX is as good as well-known representations such as Discrete Wavelet Transform (DWT) and Discrete Fourier Transform (DFT), while requiring less storage space. In addition, the representation allows researchers to avail of the wealth of data structures and algorithms in bioinformatics or text mining, and also provides solutions to many challenges associated with current data mining tasks. One example is motif discovery, a problem which we recently defined for time series data. There is great potential for extending and applying the discrete representation on a wide class of data mining tasks. Source code has “non-commercial” license. (via rdamodharan on Delicious)
- Open Source OSCON (RedMonk) — The business of selling open source software, remember, is dwarfed by the business of using open source software to produce and sell other services. And yet historically, most of the focus on open source software has accrued to those who sold it. Today, attention and traction is shifting to those who are not in the business of selling software, but rather share their assets via a variety of open source mechanisms. (via Simon Phipps)
ENTRIES TAGGED "statistics"
Narrative and Structure, Teaching Science, Time-Series Statistics, and Who Benefits from Open Source
Statistical Jeopardy Wins, Mobile Taxonomy, Geodata Mystery, and Machine Learning Blog
- What is IBM’s Watson? (NY Times) — IBM joining the big data machine learning race, and hatching a Blue Gene system that can answer Jeopardy questions. Does good, not great, and is getting better.
- Google Lays Out its Mobile Strategy (InformationWeek) — notable to me for Rechis said that Google breaks down mobile users into three behavior groups: A. “Repetitive now” B. “Bored now” C. “Urgent now”, a useful way to look at it. (via Tim)
- BP GIS and the Mysteriously Vanishing Letter — intrigue in the geodata world. This post makes it sound as though cleanup data is going into a box behind BP’s firewall, and the folks who said “um, the government should be the depot, because it needs to know it has a guaranteed-untampered and guaranteed-able-to-access copy of this data” were fired. For more info, including on the data that is available, see the geowanking thread.
- Streamhacker — a blog talking about text mining and other good things, with nltk code you can run. (via heraldxchaos on Delicious)
Google Docs APIs, Wikileaks Founder Profile, DNA Hacking, and Abusing the Numbers
- Appscale — open source implementation of Google App engine’s APIs built on top of Amazon’s APIs, from UCSB. You can deploy on Amazon or on any Amazon API-compliant cloud such as Eucalyptus.
- Information Pioneers — the Chartered Institute for IT has a pile of video clips about famous IT pioneers (Lovelace, Turing, Lamarr, Berners-Lee, etc.).
- This Week in Law — podcast from Denise Howell, covering IT law and policy. E.g., this week’s episode covers “Google Books, Elena Kagen, owning virtual land, double-dipping game developers, Facebook tips, forced follow bug and fragile egos, embedding tweets, Star Trek Universe liability, and more.”
- <a href="
GMail CRM, Django Best Practices, Stats-Think, and WoW Number Crunching
- Rapportive — a simple social CRM built into Gmail. They replace the ads in Gmail with photos, bio, and info from social media sites. (via ReadWrite Web)
- Best Practices in Web Development with Django and Python — great set of recommendations. (via Jon Udell‘s article on checklists)
- Think Like a Statistician Without The Math (Flowing Data) — Finally, and this is the most important thing I’ve learned, always ask why. When you see a blip in a graph, you should wonder why it’s there. If you find some correlation, you should think about whether or not it makes any sense. If it does make sense, then cool, but if not, dig deeper. Numbers are great, but you have to remember that when humans are involved, errors are always a possibility. This is basically how to be a scientist: know the big picture, study the details to find deviations, and always ask “why”.
- WoW Armory Data Mining — a blog devoted to data mining on the info from the Wow Amory, which has a lot of data taken from the servers. It’s baseball statistics for World of Warcraft. Fascinating! (via Chris Lewis)
Data Adjustments, Grasping Telcos, Open Data Panacea Denied, Newspaper Software
- How to Seasonally Adjust Data — Most statisticians, economists and government agencies that report data use a method called the X12 procedure to adjust data for seasonal patterns. The X12 procedure and its predecessor X11, which is still widely used, were developed by the U.S. Census Bureau. When applied to a data series, the X12 process first estimates effects that occur in the same month every year with similar magnitude and direction. These estimates are the “seasonal” components of the data series. (via bengebre on Delicious)
- Vodafone Chief: Mobile Groups Should Be Able to Bypass Google (Guardian) — Vodafone and other telcos want to charge both ends, to charge not just the person with a monthly mobile data subscription but also the companies with whom that person communicates. It’s double-dipping and offensively short-sighted. Vodafone apparently wants to stripmine all the value their product creates. This is not shearing the sheep, this is a recipe for lamb in mint sauce.
- Open Data is Not A Panacea, But It Is A Start — The reality is that releasing the data is a small step in a long walk that will take many years to see any significant value. Sure there will be quick wins along the way – picking on MP’s expenses is easy. But to build something sustainable, some series of things that serve millions of people directly, will not happen overnight. And the reality, as Tom Loosemore pointed out at the London Data Store launch, it won’t be a sole developer who ultimately brings it to fruition. (via sebchan on Twitter)
- Our GeoDjango EC2 Image for News Apps — Chicago Tribune releasing an Amazon EC2 image of the base toolchain they use. Very good to see participation and contribution from organisations historically seen as pure consumers of technology. All business are becoming technology-driven businesses, realising the old mindset of “leave the tech to those who do it best” isn’t compatible with being a leader in your industry.
Desirable Devices, iPhone Piracy Numbers, Internet Trend Numbers, Value of Privacy
- New Device Desirable, Old Device Undesirable — “I’m going to take my new device wherever I go,” said Larson, holding the expensive item directly in the eyeline of several reporters. “That way no one on the street, inside the elevator, or at my place of business will ever mistake me for the sort of individual who does not own the new device.” Added Larson, “The new device brings me satisfaction.” (via liza on Twitter)
- iPhone Piracy — over 70% of submitted game scores for this game were from pirated copies. Having seen our data and the fact that not a single pirate bought Tap-Fu after playing it, these arguments all sound a bit delusional to me. It seems like an attempt at trying to be legitimate while hiding the real reason. They should just change their page to say “We pirate because we can”. That seems to be a much more honest statement based on the data we’ve seen. (via timoreilly on Twitter)
- World Internet Project — global research into Internet adoption and trends. Found via the New Zealand partner who published their dataset in the New Zealand Social Science Datasets repository.
- The Eternal Value of Privacy (Bruce Schneier) — powerful notes about the right to privacy. Privacy protects us from abuses by those in power, even if we’re doing nothing wrong at the time of surveillance. [...] Privacy is a basic human need. [...] For if we are observed in all matters, we are constantly under threat of correction, judgment, criticism, even plagiarism of our own uniqueness. We become children, fettered under watchful eyes, constantly fearful that — either now or in the uncertain future — patterns we leave behind will be brought back to implicate us, by whatever authority has now become focused upon our once-private and innocent acts. We lose our individuality, because everything we do is observable and recordable.
Global Broadband, A/B Testing Stats, Streaming with SSDs, Online Videos Sell
- OECD Broadband Portal — global data on broadband penetration and pricing available from June 2009.
- Easy Statistics for A/B Testing — it really is easy. And it mentions hamsters. This is worth reading. (via Hacker News)
- last.fm’s SSD Streaming Infrastructure — Each single SSD can support around 7000 concurrent listeners, and the serving capacity of the machine topped out at around 30,000 concurrent connections in it’s tested configuration. Lots of hardware and OS configuration geeking here, it’s great. (via Hacker News)
- Videos Sell More Product — Zappos sells 6-30% more merchandise when accompanied by video demos. By the end of next year, Zappos will have ten full working video studios, with the goal of producing around 50,000 product videos by 2010, up from the 8,000 videos they have on the site today (via johnclegg on Twitter)
Heat Maps in R, EC2 Blackhat Tricks, Snickersome Unicode, and Decoding Statistics
- Heat Maps in R — We used financial data here because it’s easier to access than the airline data, but it’s actually a pretty interesting way of looking at a financial time series. Weekend and holiday effects are a bit more obvious, and it’s a bit like being able to see the daily, weekly, monthly and yearly closes all at once (by scanning your eye over the calendar in different directions). Includes source code. (via migurski on Delicious)
- BlackHat and EC2 — Theft of resources is the red-headed step-child of attack classes and doesn’t get much attention, but on cloud platforms where resources are shared amongst many users these attacks can have a very real impact. With this in mind, we wanted to show how EC2 was vulnerable to a number of resource theft attacks and the videos below demonstrate three separate attacks against EC2 that permit an attacker to boot up massive numbers of machines, steal computing time/bandwidth from other users and steal paid-for AMIs. (via straup on Delicious)
- Funny Characters in Unicode — I never get tired of the wacky stuff in Unicode. I love the thought of a Unicode committee somewhere arguing passionately about the number of buttons on the snowman …. (via Hacker News)
- Statistics to English Translation — The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate. This matters. Bandying around numbers with misleading labels, or misinterpreting numbers that have a precise and defined meaning, does not further understanding. (Said 78.4% of statisticians, with a 20% confidence factor probability of false positives)