Data Science

The AI Delusion
by Gary Smith
Oxford University Press, 256 pp., USD $27.95.

“I for one welcome our new computer overlords” – Ken Jennings, Jeopardy champion

            In The AI Delusion, economist Gary Smith provides a warning for mankind. However, it is not a warning about machines, it is about ourselves and our tendency to trust machines to make decisions for us. Artificial Intelligence is fantastic for limited and focused tasks but is not close to actual general intelligence. Professor Smith points out that machines, for which all patterns in data appear equally meaningful, have none of the real-world understanding required to filter out nonsense. Even worse is the fact that many of the new algorithms hide their details so we have no way of determining if the output is reasonable. Even human beings, when not engaging their critical thinking skills, mistakenly draw conclusions from meaningless patterns. If we blindly trust conclusions from machines, we are falling for the AI delusion and will certainly suffer because of it.

The Real Danger of Artificial Intelligence

Speculators about the future of artificial intelligence (AI) tend to fall into one of two camps. The first group believes that, when hardware reaches the same level of complexity and processing speed as a human brain, machines will quickly surpass human-level intelligence and lead us into a new age of scientific discovery and inventions. As part of his final answer of the man vs. machine match against IBM’s Watson, former Jeopardy! champion Ken Jennings seemed to indicate that he was in this first camp by welcoming our computer overlords. The impressive AI system, which beat him by answering natural language questions and appeared to understand and solve riddles, made fully intelligent machines seem to be right around the corner.[1]

The second camp dreads an AI revolution. Having grown up on sci-fi movies, like the Matrix and the Terminator, they worry that superior intelligence will lead machines to decide the fate of mankind, their only potential threat, in a microsecond. Alternatively, and more realistically, they see a risk that AI machines may simply not value or consider human life at all and unintentionally extinguish us in their single-minded pursuit of programmed tasks. Machines may find a creative solution that people did not anticipate and endanger us all.

Gary Smith convincingly presents his belief that neither of these views is correct. If achieving true AI is like landing on the moon, all of the impressive recent advances are more like tree-planting than rocket-building. New advancements are akin to adding branches to the tree, and getting us higher off the ground, but not on the path towards the moon.

Humanity has turned away from the exceedingly difficult task of trying to mimic the way the brain works and towards the easier applications (such as spell-checkers and search engines) that leverage what computers do well. These new applications are useful and profitable but, if the goal is for machines to be capable of understanding the world, we need to start over with a new approach to AI. Machines gaining human-like intelligence is not something around the corner unless we start building rockets. 

The AI Delusion warns us that the real danger of AI is not that computers are smarter than we are but that we think computers are smarter than we are. If people stop thinking critically and let machines make important decisions for them, like determining jail sentences or hiring job candidates, any one of us may soon become a victim of an arbitrary and unjustifiable conclusion. It is not that computers are not incredibly useful; they allow us to do in minutes what might take a lifetime without them. The point is that, while current AI is artificial, it is not intelligent.

The Illusion of Intelligence

Over the years I have learned a tremendous amount from Gary Smith’s books and his way of thinking. It seems like a strange compliment but he is deeply familiar with randomness. He knows how random variables cluster, how long streaks can be expected to continue, and what random walks look like. He can examine a seemingly interesting statistical fluke in the data and conclude “you would find that same pattern with random numbers!” and then prove it by running a simulation. He uses this tactic often in his books and it is extremely effective. How can you claim that a pattern is meaningful when he just created it out of thin air?

The AI Delusion begins with a painful example for the political left of the United States. Smith points a finger at the over-reliance on automated number-crunching for the epic failure of Hillary Clinton’s presidential campaign in 2016. Clinton had a secret weapon: a predictive modeling system. Based on historical data, the system recommended campaigning in Arizona in an attempt for a blowout victory while ignoring states that Democrats won in prior years. The signs were there that the plan needed adjusting: her narrow victory over Bernie Sanders, the enthusiastic crowds turning out for Trump, and the discontent of blue-collar voters who could no longer be taken for granted. However, since her computer system did not measure those things, they were considered unimportant. Clinton should have heeded the advice of sociologist William Bruce Cameron: “not everything that can be counted counts, and not everything that counts can be counted.” Blindly trusting machines to have the answers can have real consequences. When it comes to making predictions about the real world, machines have blind spots, and we need to watch for them.

In contrast, machines are spectacular at playing games; they can beat the best humans at practically every game there is. Games like chess were traditionally considered proxies for intelligence, so if computers can crush us, does that mean that they are intelligent? As Smith reviews various games, he shows that the perception that machines are smart is an illusion. Software developers take advantage of mind-boggling processing speed and storage capabilities to create programs that appear smart. They focus on a narrow task, in a purified environment of digital information, and accomplish it in a way that humans never would. Smith points out the truth behind the old joke that a computer can make a perfect chess move while it is in a room that is on fire; machines do not think, they just follow instructions. The fact that they’re good at some things does not mean they will be good at everything.

In the early days of AI, Douglas Hofstadter, the author of the incredibly ambitious book Gödel, Escher, Bach: An Eternal Golden Braid, tackled the seemingly impossible task of replicating the way a human mind works. He later expressed disappointment as he saw the development of AI take a detour and reach for the tops of trees rather than the moon:

To me, as a fledgling [artificial intelligence] person, it was self-evident that I did not want to get involved in that trickery. It was obvious: I don’t want to be involved in passing off some fancy program’s behavior for intelligence when I know that it has nothing to do with intelligence.

A New Test for AI

The traditional test for machine intelligence is the Turing Test. It essentially asks the question: “Can a computer program fool a human questioner into thinking it is a human?” Depending on the sophistication of the questioner, the freedom to ask anything at all can pose quite a challenge for a machine. For example, most programs would be stumped by the question “Would flugly make a good name for a perfume?” The problem with this test is that it is largely a game of deception. Pre-determined responses and tactics, such as intentionally making mistakes, can fool people without representing any useful advance in intelligence. You may stump Siri with the ‘flugly’ question today, but tomorrow the comedy writers at Apple may have a witty response ready: “Sure, flidiots would love it.” This would count as the trickery Hofstadler referred to. With enough training, a program will pass the test but it would not be due to anything resembling human intelligence; it would be the result of a database of responses and a clever programmer who anticipated the questions.

Consider Scrabble legend Nigel Richards. In May 2015, Richards, who does not speak French, memorized 386,000 French words. Nine weeks later he won the first of his two French-language Scrabble World Championships. This can provide insight into how computers do similarly amazing things without actually understanding anything. Another analogy is the thought experiment from John Searle in which someone in a locked room receives and passes back messages under the door in Chinese. The person in the room does not know any Chinese; she is just following computer code that was created to pass the Turing Test in Chinese. If we accept that the person in the room following the code does not understand the questions, how can we claim that a computer running the code does?

A tougher test to evaluate machine intelligence is the Winograd Schema Challenge. Consider what the word ‘it’ refers to in the following sentences:

I can’t cut that tree down with that axe; it is too thick

I can’t cut that tree down with that axe; it is too small.

A human can easily determine that, in the first sentence, ‘it’ refers to the tree while, in the second, ‘it’ is the axe. Computers fail these types of tasks consistently because, like Nigel Richards, they do not know what words mean. They don’t know what a tree is, what an axe is, or what it means to cut something down. Oren Etzioni, a professor of computer science, asks “how can computers take over the world, if they don’t know what ‘it’ refers to in a sentence?”

One of my favorite surprises from the book is the introduction of a new test (called the Smith Test of course) for machine intelligence:

Collect 100 sets of data; for example, data on U.S. stock prices, unemployment, interest

rates, rice prices, sales of blue paint in New Zealand, and temperatures in Curtin,

Australia. Allow the computer to analyze the data in any way it wants, and then report the statistical relationships that it thinks might be useful for making predictions. The

computer passes the Smith test if a human panel concurs that the relationships selected by the computer make sense.

This test highlights the two major problems with unleashing sophisticated statistical algorithms on data. One problem is that computers do not know what they have found; they do not know anything about the real world. The other problem is that it is easy, even with random data, to find associations. That means that, when given a lot of data, what computers find will almost certainly be meaningless. Without including a critical thinker in the loop, modern knowledge discovery tools may be nothing more than noise discovery tools.

It is hard to imagine how a machine could use trickery to fake its way through a test like this. Countless examples in the book show that even humans who are not properly armed with a sense of skepticism can believe that senseless correlations have meaning:

• Students who choose a second major have better grades on average. Does this mean a struggling student should add a second major?
• Men who are married live longer than men who are divorced or single. Can men extend their lifespans by tying the knot?
• Emergency room visits on holidays are more likely to end badly. Should you postpone emergency visits until the holidays are over?
• Freeways with higher speed limits have fewer traffic fatalities. Should we raise speed limits?
• Family tension is strongly correlated with hours spent watching television. Will everyone get along better if we ditch the TV?
• People who take driver-training courses have more accidents than people who do not. Are those courses making people more reckless?
• Students who take Latin courses score higher on verbal ability. Should everyone take Latin?

Many people incorrectly assume causal relationships in questions like these and unthinking machines would certainly do so as well. Confounding variables only become clear when a skeptical mind is put to use. Only after thinking carefully about what the data is telling us, and considering alternate reasons why there might be an association, can we come to reasonable conclusions.

Gary Smith’s specialty is teaching his readers how to spot nonsense. I’m reminded of a memorable speech from the movie My Cousin Vinny[2]:

Vinny: The D.A.’s got to build a case. Building a case is like building a house. Each piece of evidence is just another building block. He wants to make a brick bunker of a building. He wants to use serious, solid-looking bricks, like, like these, right? [puts his hand on the wall]
Bill: Right.
Vinny: Let me show you something.
[He holds up a playing card, with the face toward Billy]
Vinny: He’s going to show you the bricks. He’ll show you they got straight sides. He’ll show you how they got the right shape. He’ll show them to you in a very special way, so that they appear to have everything a brick should have. But there’s one thing he’s not gonna show you. [He turns the card, so that its edge is toward Billy]
Vinny: When you look at the bricks from the right angle, they’re as thin as this playing card. His whole case is an illusion, a magic trick…Nobody – I mean nobody – pulls the wool over the eyes of a Gambini.

Professor Smith endeavors to make Gambinis out of us all. After reading his books, you are taught to look at claims from the right angle and see for yourself if they are paper thin. In the case of The AI Delusion, the appearance of machine intelligence is the magic trick that is exposed. True AI would be a critical thinker with the capability to separate the meaningful from the spurious, the sensible from the senseless, and causation from correlation.

Data-Mining for Nonsense

The mindless ransacking of data and looking for patterns and correlations, which is what AI does best, is at the heart of the replication crisis in science. Finding an association in a large dataset just means that you looked, nothing more. Professor Smith writes about a conversation he had with a social psychologist at Sci Foo 2015, an annual gathering of scientists and writers at Googleplex. She expressed admiration for Daryl Bem, a social psychologist, who openly endorsed blindly exploring data to find interesting patterns. Bem is known, not surprisingly, for outlandish claims that have been refuted by other researchers. She also praised Diederik Stapel who has even admitted that he made up data. Smith changed the subject. The following day a prominent social psychologist said that his field is the poster-child for irreproducible research and that his default assumption is that every new study is false. That sounds like a good bet. Unfortunately, adding more data and high-tech software that specializes in discovering patterns will make the problem worse, not better.

To support the idea that computer-driven analysis is trusted more than human-driven analysis, Smith recounts a story about an economist in 1981 who was being paid by the Reagan administration to develop a computer simulation that predicted that tax revenue would increase if tax rates were reduced. He was unsuccessful no matter how much the computer tortured the data. He approached Professor Smith for help and was not happy when Smith advised him to simply accept that reducing tax rates would reduce tax revenue (which is, in fact, what happened). The effort to find a way to get a computer program to provide the prediction is telling; even back in the 80s people considered computers to be authoritative. If the machine says it, it must be true.

Modern day computers can torture data like never before. A Dartmouth graduate student named Craig Bennett used an MRI machine to search for brain activity in a salmon as it was shown pictures and asked questions. The sophisticated statistical software identified some areas of activity! Did I mention that the fish was dead? Craig grabbed it from a local market. There were so many areas (voxels) being examined by the machine that it would inevitably find some false positives. This was the point of the study; people should be skeptical of findings that come from a search through piles of data. Craig published his research and won the Ig Nobel Prize, which is awarded each year to “honor achievements that first make people laugh, and then make them think.” The lesson for the readers of AI Delusion is that anyone can read the paper and chuckle at the absurdity of the idea that the brain of a dead fish would respond to photographs but the most powerful and complex neural net in the world, given the same data, would not question it.

One of the biggest surprises in the book was the effective criticism of popular statistical procedures including stepwise regression, ridge regression, neural networks, and principal components analysis. Anyone under the illusion that these procedures will protect them against the downsides of data-mining is disabused of that notion. Professor Smith knows their histories and technical details intimately. Ridge regression, in particular, takes a beating as a “discredited” approach. Smith delivers the checkmate, in true Smithian style, by sending four equivalent representations of Milton Friedman’s model of consumer spending to a ridge regression specialist to analyze:

I did not tell him that the data were for equivalent equations. The flimsy foundation of ridge regression was confirmed in my mind by the fact that he did not ask me anything about the data he was analyzing. They were just numbers to be manipulated. He was just like a computer. Numbers are numbers. Who knows or cares what they represent? He estimated the models and returned four contradictory sets of ridge estimates.

Smith played a similar prank on a technical stock analyst. He sent fictional daily stock prices based on student coin flips to the analyst to see if it would be a good time to invest. The analyst never asked what companies the price history was from but became very excited about the opportunity to invest in a few of them. When Smith informed him that they were only coin flips, he was disappointed. He was not disappointed that his approach found false opportunities in noise but that he could not bet on his predictions. He was such a firm believer in his technical analysis that he actually believed he could predict future coin flips.

Automated stock-trading systems, similar to AI, are not concerned with real world companies; the buy and sell decisions are based entirely on transitory patterns in the price and the algorithms are tuned to the primarily meaningless noise of historical data. I wondered why, if stock trading systems are garbage, investment companies spend billions of dollars on trading centers as close to markets as possible. Smith explains this as well: they want to exploit tiny price discrepancies thousands of times per second or to front-run orders from investors and effectively pick-pocket them. This single-minded pursuit of a narrow goal without concern for the greater good is unfortunately also a feature of AI. The mindless world of high-frequency trading, both when it is profitable (exploiting others) and when it is not (making baseless predictions based on spurious patterns), serves as an apt warning about the future that awaits other industries if they automate their decision-making.

Gary Smith draws a clear distinction between post-hoc justification for patterns found rummaging through data and the formation of reasonable hypotheses that are then validated or refuted based on the evidence. The former is unreliable and potentially dangerous while the latter was the basis of the scientific revolution. AI is built, unfortunately, to maximize rummaging and minimize critical thinking. The good news is that this blind spot ensures that AI will not be replacing scientists in the workforce anytime soon.

There Are No Shortcuts

If you have read other books from Gary Smith, you know to expect many easy-to-follow examples that demonstrate his ideas. Physicist Richard Feynman once said “If you cannot explain something in simple terms, you don’t understand it.” Smith has many years of teaching experience and has developed a rare talent for boiling ideas down to their essence and communicating them in a way that anyone can understand.

Many of the concepts seem obvious after you have understood them. However, do not be fooled into believing they are self-evident. An abundance of costly failures have resulted from people who carelessly disregarded them. Consider the following pithy observations…

We think that patterns are unusual and therefore meaningful.

Patterns are inevitable in Big Data and therefore meaningless.

The bigger the data the more likely it is that a discovered pattern is meaningless.

You see at once the danger that Big Data presents for data-miners. No amount of statistical sophistication can separate out the spurious relationships from the meaningful ones. Even testing predictive models on fresh data just moves the problem of finding false associations one level further away. The scientific way is theory first and data later.

Even neural networks, the shining star of cutting edge AI, are susceptible to being fooled by meaningless patterns. The hidden layers within them make the problem even worse as they hide the features they rely on inside of a black box that is practically impossible to scrutinize. They remind me of the witty response from a family cook responding to a question from a child about dinner choices: “You have two choices: take it or leave it.”

The risk that data used to train a neural nets is biased in some unknown way is a common problem. Even the most sophisticated model in the world could latch on some feature, like the type of frame around a picture it is meant to categorize, and become completely lost when new pictures are presented to it that have different frames. Neural nets can also fall victim to adversarial attacks designed to derail them by obscuring small details that no thinking entity would consider important. The programmers may never figure out what went wrong and it is due to the hidden layers.

A paper was published a couple days ago in which researchers acknowledged that the current approaches to AI have failed to come close to human cognition. Authors from DeepMind, as well as Google Brain, MIT, and the University of Edinburgh write that “many defining characteristics of human intelligence, which developed under much different pressures, remain out of reach for current approaches.”[3] They conclude that “a vast gap between human and machine intelligence remains, especially with respect to efficient, generalizable learning.”

The more we understand about how Artificial Intelligence currently works, the more we realize that ‘intelligence’ is a misnomer. Software developers and data scientists have freed themselves from the original goal of AI and have created impressive software capable of extracting data with lightning speed, combing through it and identifying patterns, and accomplishing tasks we never thought possible. In The AI Delusion, Gary Smith has revealed the mindless nature of these approaches and made the case that they will not be able to distinguish meaningful from meaningless any better than they can identify what ‘it’ refers to in a tricky sentence. Machines cannot think in any meaningful sense so we should certainly not let them think for us.

---

[1] Guizzo, Erico. “IBM’s Watson Jeopardy Computer Shuts Down Humans in Final Game.” IEEE Spectrum: Technology, Engineering, and Science News. February 17, 2011. Accessed November 05, 2018. https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/ibm-watson-jeopardy-computer-shuts-down-humans.

[2] My Cousin Vinny. Directed by Jonathan Lynn. Produced by Dale Launer. By Dale Launer. Performed by Joe Pesci and Fred Gwynne

[3] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. “Relational inductive biases, deep learning, and graph networks.” arXiv preprint arXiv:1806.01261, 2018.

There have been conflicting reports about the US crime rate in the media.  On one hand, President Obama said last year that “there’s been an incredible drop in violent crime” and “our crime rate today is substantially lower than it was five years ago, 10 years ago, 20 years ago, 30 years ago.”  On the other hand, the then presidential candidate Donald Trump stated that “violent crime has increased in cities across America.  The New York Times described ‘the startling rise in murders,’ in our major cities.”  So which is it?

It is true that the New York Times reported that “data … based on reports from more than 60 cities,  showed notable increases in murders in about two dozen cities in the first three months of the year compared to last year and about a 9 percent increase nationwide.”  However, the long-term trend is undeniable.  The murder rate is practically at an all-time low.

Crime Trends Chart

As the final project for my Data Visualization and Communication class in the Berkeley MIDS program, my team came up with some useful charts, fueled by data from the Chicago Police Department’s CLEAR system, where users can see for themselves what the trend is for a wide variety of crimes.  We categorized the types of crimes according to what we felt would be in the public interest and adjusted for the recent decline in population in Chicago by calculating the crime rate per 100,000 residents per year.  This also provides context so people can understand what the average risk of victimization is per year for any particular crime.

In addition to addressing the misleading statements made by politicians, we also wanted to provide a counterpoint to news broadcasting in this country, which has a financial incentive to focus on the sensational crimes that bring in viewers.  News watchers receive a steady diet of stories about homicide and home invasions without proper context.  Our chart also aims to dispel the perception that those are common types of crime and place them in perspective.  The news is entertaining but not very informative, when it comes to communicating the likelihood of various crimes taking place.

Interactive Crime Maps

While the Crime Trends chart are of interest to the general public seeking information about crime trends, our interactive crime maps are of particular use to someone, such as a home-buyer, interested in crime rates in specific community areas within Chicago.

Users can also select and isolate statistics for communities if they want to quantify the safety of nearby areas or determine the relative safety of traveling through a particular series of communities.  By toggling the selection of areas they wish to compare, they will be able to determine which places are best to avoid.

Enjoy!

–> Interact with our awesome charts here <–

I just completed Berkeley’s enjoyable DATASCI W201 class “Research Design and Applications for Data Analysis” and figured I might as well share a few of my essays since I crafted them with such tender-loving care.  Enjoy!

 

Week 5 – Up Is Up by Jay Cordes – Tuesday, 07 June 2016, 09:22 AM

At my last job, there were many examples where decision makers took questionable facts for granted. “Up is up” was even a favorite saying of one of the executives, and I would often have back and forth exchanges with him in meetings when I thought he was being duped by data.

In one case, practically everyone was fooled by a case of regression towards the mean. My business unit designed and conducted experiments to determine which web page designs maximized ad revenue for our customers, who collect domain names. Invariably, some of their domains would perform worse than others, and it was decided that we should try to optimize those “underperformers” by hand (typically, we’d use a generic design across the board). Being scientifically rigorous and only working on a random half was considered overkill.

My friend Brad was the one tasked with actually making the modifications and was given the list of domain names each week that could use help. Every single time he hand-optimized a set of domains, the revenue on those sites would significantly increase the following day (around 30-40% if I recall). He was praised by customers and managers alike who started to wonder why he’s not hand-optimizing all of the domains on our system. Brad was savvy enough to be skeptical of results and smiled knowingly when I pointed out that there’s practically no way for him to fail since the worst domains would be expected to improve anyway just by chance.

Well, one time he forgot to work on the set of domains given to him. The next day, their total revenue skyrocketed by about 40% and the customer wrote “whatever you did worked!” Since the increase was at least as much as we’d seen with his hand-optimizing, it was now realized that he could actually have been harming revenue. If we hadn’t assumed that things could only get better and used a randomized control (selected from the domains he was told to optimize), we would’ve clearly seen whether or not his work was having a benefit. The moral to the story is: up is definitely not always up.

 

Week 7 – Phones Don’t Cause Cancer, The World is P-Hacking by Jay Cordes – Tuesday, 21 June 2016, 06:46 PM

The article, “Study finds cell phone frequency linked to cancer,” (http://www.mysuncoast.com/health/news/study-finds-cell-phone-frequency-linked-to-cancer/article_26411b44-290a-11e6-bffc-af73cd8bda5f.html) reports that a recent study shows that cell phone radiation increases brain tumors in rats. It also provides safety tips, such as keeping phones away from your ear.

The problem with the article is subtle: the failure to recognize the likelihood of spurious results due to repeated testing. “P-hacking” refers to when researchers hide or ignore their negative results. It’s similar to editing a video of yourself shooting baskets and only showing a streak of 10 consecutive makes. This study wasn’t p-hacked, but the result should still be regarded with skepticism due to the number of other studies that did not find similar evidence. In other words, global p-hacking is occurring when articles focus on this experiment in isolation.

Another way to think of p-hacking is in regards to “regression toward the mean”, which occurs because (1) luck almost always plays some role in results, and (2) it is more likely to be present at the extremes. A small amount of luck will lead to future results that are slightly less extreme. However, if the results are due entirely to luck, they will evaporate completely upon re-testing. With many failed attempts comes a higher likelihood that luck played a big role in the successes.

The article doesn’t recognize that results from other studies should weaken the confidence you should have in this one. It also mentions that the radiation is non-ionizing, but doesn’t note that only ionizing radiation (high frequency electromagnetic radiation including ultraviolet, x-ray, and gamma rays) has been shown to cause mutations. From a purely physical standpoint, cancer-causing cell phone radiation is very unlikely. Also, no mention is made of the fact that cancer rates haven’t skyrocketed in response to the widespread cell phone usage around the world.

The only criticism I have for the experiment itself is the fact that the results were broken down by male/female. What explanation could there be for why females were not affected by radiation? Would the result have been significant if the sexes were combined? A couple “misses” do seem to have been edited out.

I chose this blind spot to discuss because I believe it scores a perfect 10 in terms of its pervasiveness and negative impact on interpreting results. At my last job, we ran a randomized controlled test on web pages of a few different colors and found no significant difference in revenue. My boss then recommended that we look at the results by country, which made me wince and explain why that’s a data foul. An analyst did look at the results and found that England significantly preferred teal during the experiment. Due to my objections, we thankfully did not roll out the teal web page in England, but kept the experiment running. Over the next few weeks, England appeared to significantly hate teal. It pays to be skeptical of results drawn from one of many experiments.

 

Week 10 – Jay Cordes: Visualizations Can Affect Your Intelligence by Jay Cordes – Monday, 11 July 2016, 11:33 PM

The worst visualization I’ve ever seen actually makes you dumber for having seen it. It’s this one from Fox News evidently attempting to show that global warming is not a thing…

Figure 1 – Wow, this is nonsense.

It’s not bad in the sense of Edward Tufte’s ”chartjunk” (there’s no graphical decoration at all). It’s also not violating the idea of data-ink maximization (there’s actually not enough ink used, since there’s no legend explaining what the colors represent). It’s bad because:

1. It’s flat out nonsense. It turns out that the map on the left shows whether temperatures were higher or lower than normal the prior March, while the map it’s being compared to is the absolute minimum temperature on a particular day.
2. Even if the chart were actually comparing what it pretends to be, it would still be meaningless. Can a one month exception cast doubt on a decades long trend?
3. If the colors on the map on the left map actually meant what they appear to mean, then we are supposed to believe that Minnesota hit 100 degrees in March of 2012? When it did set the record for the hottest average March temperature in history, but it was only 48.3 degrees.

On the other hand, one of the best charts I’ve seen is this one from the Economist that we used in our presentation on Brexit.

Figure 2 – Now we’re learning something

There’s a density of information that is presented very simply and efficiently: Which European countries believe they’d be better off outside the European Union? Is it more that they think their economics and commerce would improve or their migration controls? What are their population sizes? Why did England vote to leave? Unlike the Fox News chart, the data is presented truthfully and clearly. Answers to all of these questions can be quickly drawn from it. I also like the subtle details, such as making it clear what the colors of the circles represent by simply coloring the font of the axis labels. The grid lines are white on a light gray background to avoid cluttering up the chart. Also, the population sizes are rounded to one of three values to simplify comparison. This chart definitely follows the idea of data ink maximization.

Sources: http://mediamatters.org/blog/2013/03/25/another-misleading-fox-news-graphic-temperature/193247 http://www.economist.com/news/briefing/21701545-britains-decision-leave-european-union-will-cause-soul-searching-across-continentand http://files.dnr.state.mn.us/natural_resources/climate/twin_cities/ccmarch1981_2010.html http://shoebat.com/wp-content/uploads/2016/06/20160702_FBC814.png http://m5.paperblog.com/i/48/483755/every-day-is-april-fools-at-fox-news-L-byleIS.jpeg

These days, if you listen to presidential candidates of various hand-sizes or watch a lot of news, you probably think that a crime wave is overtaking the United States.  However, if you happen to be a data science student who’s looking at crime data and working on homework assignments like these, you probably know better.

Here’s some recent data about crime in Chicago…

Pretty much every type of crime is dropping year after year and it’s been happening for quite awhile. If you insist on worrying about stuff, you should know that theft, battery, and criminal damage are the most common types of crime and make up about half of all arrests.  Also, in case you’re curious, those cycles in the figure above are due to the fact that more crime occurs when it’s hot.

It’s incredibly consistent: as the temperature goes up, so does the crime rate.  So, if you’re really worried about crime, stay in your safe room during hot days.  Actually, it turns out that safe rooms aren’t that necessary either.  Home invasions, while they make good news stories, actually made up 0.085% of all crime in Chicago.  That’s not 8.5%, that’s 8.5 one hundredths of a percent.  Similarly, homicide, while making up about a quarter of the crime stories on the news actually represent less than 0.2% of all crime.  There should probably be some kind of a warning label on newscasts: “We’ve chosen to show you tons of murders only because it’s exciting and brings in viewers.  We are in no way implying that the Zombie Apocalypse is upon us.”

So, rest easy kids!  Put down the guns, because you’ll probably just hurt yourselves.  And whatever you do, don’t travel into the past, because that’s where the real danger lies.

While I couldn’t have been more excited to start the MIDS program (Master of Information and Data Science) at UC Berkeley, the very first class just happened to conflict with the opportunity to meet this Guy…

I guess that proves that even more than a “data wonk,” I am at heart a “science freak.”  We ended up even getting to walk with Mr. Guy on the way to go see his buddy Neil deGrasse Tyson give an entertaining lecture about Sci-Fi movies and their scientific accuracy and/or lack of it, so it was a full-on geek-out that night.  But I digress!

Prior to the first live classes, we had access to a killer application that provided access to tons of pre-recorded lectures.  I have it on my desktop, my iPhone, and my iPad…

You can stream the lectures or download them for offline viewing later and can adjust the speed of the video to really plow through them in record time.  If you go fast enough, you feel like you’re Neo downloading knowledge directly into your brain.  Before long, your screen has check marks by everything and then you look up and say to yourself “I know Python.”  You know you’re addicted to the app when you grab your iPad and are actually not sure whether you’d rather check to see what your classmates and professor are posting on the wall or play Plants vs. Zombies.  Admit it, just by reading this and you’re already starting thinking about studying for those GREs and going back to school!

Finally, it was time for live classes to begin…

Not your gramma’s graduate school!  It was a lively and interesting discussion and nobody sang the Brady Bunch theme song.  There also wasn’t any singing when the 2nd homework assignment was handed out and was surprisingly challenging: writing python files from within a bash script and getting up to speed on a code versioning app called git.  Being pretty new to git, I forgot to add a comment once at the end of a “git commit” command and was thrown into a nightmare of a text editor called vim that I COULD NOT ESCAPE FROM without doing web research.  Turns out that the command I was looking for was “:wq” to close the window.  As a former software developer, stuff like that drives me crazy.  Even if a programmer was sick enough to think that that was a good command for something as common as closing a window, did he/she not have a manager to say “Okay, you know that someone other than you might end up using this program, right?”

A friend of mine backed up my initial impressions of git’s user-friendliness: “Yes, git is overwhelmingly cryptic.  A basic workflow can be described in a more straightforward way, but God forbid anything go wrong.  I’ve spent many half-days trying to fix a mistake in git.  It’s kind of amazing to me that it has caught on as much as it has.”  He also supplied me with this helpful advice…

I ended up getting through the assignment without too much trouble, but evidently others went a bit astray, because the following day, the professor posted a message on the wall: “…Please don’t force push to the github playground … if you do that, that means you’re going to overwrite someone else’s changes.  If you do that, we can’t keep track of who all made changes.  That means we’ll also be modifying the assignment – we just want you to have one change in the github playground.”

After reading that, I naturally had to post the following to the wall…

Good times!

With the recent passing of Oliver Sacks, I was reminded of the moving essay he wrote after learning he only had a few months to live.  I’m very grateful that he spent one of his last precious days on this planet sharing his reflections and thoughts about what he finds worthwhile.  I’m always interested in how thoughtful people spend their time, since I tend to think of us all as being short on time and never truly aware of its value until it’s almost gone.

So, let’s discuss it now.  In this age of cell-phones, we’ve become adept at never “wasting” time, but not as good at “spending” time.  As a believer in the wisdom of the masses, I think one of the best ways to find out what I should be doing with my free time is to compile the ideas from essays, such as the gem Dr. Sacks and others have left us with.  After all, “ask the audience” beats “phone a friend” hands down.

Supposing that all of the top responses are worth doing and that I should spend my free time in proportion to the number of votes received, here are the ideas…

 

How To Spend Your Time	Votes	%
Connect with Loved Ones.  Deepen friendships / make a difference in lives around you.	14	22%
Have Fun.  “Squeeze as much happiness” as you can.  Laugh.  Be entertained by cats.	7	11%
Be In The Moment.  Take dog for a walk, enjoy a sunset, travel, eat favorite food.	7	11%
Learn and Discover.  Develop abilities or character / new levels of understanding and insight	6	9%
Do Something Useful.  Your profession / achieve things that contribute to society	5	8%
Do Something Creative.  Art / writing.	5	8%
Be Kind.  Be gentle to each other.	4	6%
Relieve Suffering.  Help people / share knowledge.	4	6%
Read.  Connect with people who came before us.	3	5%
Enjoy Music.	3	5%
Enjoy a Hobby.  Gardening.	3	5%
Give your kids Opportunity, Skills, and Empathy to enjoy their lives.  Show them wonder.	3	5%

 

Sounds like a great day to me!  Well, except for the gardening (someday, I expect that a “gardening gene” will activate and I’ll suddenly have an irresistible urge to pull weeds until my back hurts).  The clear winner was connecting with loved ones.  I’ve always been one to repeat the mantra “when you’re on your deathbed, you’re not going to wish you spent more time at work.”  One of my proudest moments was when, instead of asking for a raise, I took the bold step of asking my manager to cut my workweek and pay by 20% so I could spend more time with my family.  Her initial response was an incredulous “you can’t do that, you’re a manager!” but later had a change of heart and said she admired the move.  I definitely noticed the increase in my time and quality of life much more than I missed the extra pay.  If you can afford to do this, why wait for a health scare to send you home to the people who matter most in your life?

In second place, there was a tie between “having fun” and “being in the moment”, if those are even different things.  Since reading these essays, I’ve been more on the lookout for opportunities to find memorable moments in life to savor and it’s already paying off.  My most recent precious moment was when my family and I were staying at a hotel on the beach and my 12-year-old daughter wanted to go out to see the ocean at 10 pm!  I wasn’t even sure if we were allowed to go out there, but armed with the wisdom of the masses telling me that unique experiences like that are important, we went out into the darkness.  Nobody else was around and it was so dark and clear that, as invisible waves crashed on the shore, we could actually see the Milky Way stretching across the sky for the first time.  She later said that she’ll remember that experience forever.

In our daily rush to check off our to-do lists, moments like those are the ones that most of us let slip by because we “don’t have time”, while the few who truly know the value of time understand that what we actually don’t have time for are missed opportunities.