Comparing the opinions of economic experts and the general public

Last week on Marginal Revolution, there was a link to a wonderful paper comparing the policy opinions of economic experts to those of the general public. The paper, by Paola Sapienza and Luigi Zingales, found some pretty significant discrepancies between the two groups. The authors attributed this difference to the degree of trust each group put in the implicit assumptions embedded into the economists’ answers. It’s an excellent and fascinating read. However, like pretty much all academic economics papers, it displays its data in cumbersome text tables rather than figures. In this blog post, I’ve created some figures from the data that I think are easier to read than tables.

For each policy question, the paper provides two numbers. One is the percentage of economic experts who agree with the policy position. The second is the percentage of general public respondents who agree with the policy position. This sounds like a good opportunity for a scatter plot:

There’s a light negative correlation (r = -0.47) between how much economic experts and the general public agree with these positions. As shown in the bottom right, the vast majority of economists prefer a carbon tax to car emissions standards, whereas the vast majority of the general public prefers the reverse. As shown in the top left, the vast majority of the general public believes that requiring the government to “Buy American” is an effective way to improve manufacturing employment, whereas the majority of economists do not. For the exact wording of the questions, see the Appendix to the original paper.

Another way to visualize the same data is with a slopegraph. Below, the left column shows all the policy positions ranked by agreement from economic experts. The right column shows the same policy positions ranked by agreement from the general public. This type of plot vividly shows how unanimous the experts are on a few beliefs: It’s very hard to predict the stock market, we’re on the left side of the Laffer Curve, and the US economy is fiscally unsustainable without healthcare cuts or taxes hikes.

Slopegraphs are useful in this context because they create less text overlap than scatter plots. With the scatter plot above, I had to use interactive mouseover events to selectively show the text for individual data points. This wouldn’t be possible in a publication, since most economics journals still require static PDFs.

Even with a slopegraph, there is still some risk of overlapping text. To keep this from happening, I added some light repulsion to the data points.

For those interested, the experts in this dataset come from the Economic Expert Panel, a diverse set of economists comprising Democrats, Republicans, and Independents. This panel is the same panel that generates the data used in my Which Famous Economist website.

[Python code for the slope graph. Scatter plot is in page source.]


Four pitfalls of hill climbing

One of the great developments in product design has been the adoption of A/B testing. Instead of just guessing what is best for your customers, you can offer a product variant to a subset of customers and measure how well it works. While undeniably useful, A/B testing is sometimes said to encourage too much “hill climbing”, an incremental and short-sighted style of product development that emphasizes easy and immediate wins.

Discussion around hill climbing can sometimes get a bit vague, so I thought I would make some animations that describe four distinct pitfalls that can emerge from an overreliance on hill climbing.

1. Local maxima

If you climb hills incrementally, you may end up in a local maximum and miss out on an opportunity to land on a global maximum with much bigger reward. Concerns about local maxima are often wrapped up in critiques of incrementalism.

Local maxima and global maxima can be illustrated with hill diagrams like the one above. The horizontal axis represents product space collapsed into a single dimension. In reality, of course, there are many dimensions that a product could explore.

2. Emergent maxima

If you run short A/B tests, or A/B tests that do not fully capture network effects, you might not realize that a change that initially seems bad may be good in the long run. This idea, which is distinct from concerns about incrementalism, can be described with a dynamic reward function animation. As before, the horizontal axis is product space. Each video frame represents a time step, and the vertical axis represents immediate, measurable reward.

When a product changes, the intial effect is negative. But eventually, customers begin to enjoy the new version, as shown by changes in the reward function. By waiting at a position that initially seemed negative, you are able to discover an emergent mountain, and receive greater reward than you would have from short-term optimization.

3. Novelty effects

Short-term optimization can be bad, not only because it prevents discovery of emergent mountains, but also because some hills can be transient. One way a hill can disappear is through novelty effects, where a shiny new feature can be engaging to customers in the short term, but uninteresting or even negative in the long term.

4. Loss of differentiation

Another way a hill can disappear is through loss of differentiation from more dominant competitors. Your product may occupy a market niche. If you try to copy your competitor, you may initially see some benefits. But at some point, your customers may leave because not much separates you from your more dominant competitor. Differentiation matters in some dimensions more than others.

You can think of an industry as a dynamic ecosystem where each company has its own reward function. When one company moves, it changes its own reward function as well as the reward functions of other companies. If this sounds like biology, you’re not mistaken. The dynamics here are similar to evolutionary fitness landscapes.

While all of the criticisms of hill climbing have obvious validity, I think it is easy for people to overreact to them. Here are some caveats in defense of hill climbing:

  • The plots above probably exaggerate the magnitude and frequency with which reward functions change.
  • There is huge uncertainty and disagreement about what future landscapes will look like. In most cases, it’s better to explore regions that increase (rather than decrease) reward, making sure to run long term experiments when needed.
  • The space is high dimensional. Even if your product is at a local maximum in one dimension, there are many other dimensions to explore and measure.
  • We may overestimate the causal relationship between bold product moves and company success. Investors often observe that companies who don’t make bold changes are doomed to fail. While I don’t doubt that there is some causation here, I think there is also some reverse causation. Bold changes require lots of resources. Maybe it’s mostly the success-bound companies who have enough resources to afford the bold changes.

Special thanks to Marika Inhoff, John McDonnell and Alex Rosner for comments on a draft of this post.


How to make polished Jupyter presentations with optional code visibility

Jupyter notebooks are great because they allow you to easily present interactive figures. In addition, these notebooks include the figures and code in a single file, making it easy for others to reproduce your results. Sometimes though, you may want to present a cleaner report to an audience who may not care about the code. This blog post shows how to make code visibility optional, and how to remove various Jupyter elements to get a clean presentation.

On the top is a typical Jupyter presentation with code and some extra elements. Below that is a more polished version that removes some of the extra elements and makes code visibility optional with a button.

To make the code optionally visible, available at the click of a button, include a raw cell at the beginning of your notebook containing the JavaScript and HTML below. This code sample is inspired by a Stack Overflow post, but makes a few improvements such as using a raw cell so that the button position stays fixed, changing the button text depending on state, and displaying gradual transitions so the user understands what is happening.

  function code_toggle() {
    if (code_shown){
      $('#toggleButton').val('Show Code')
    } else {
      $('#toggleButton').val('Hide Code')
    code_shown = !code_shown

  $( document ).ready(function(){
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

It’s pretty straightforward to remove the extra elements like the header, footer, and prompt numbers. That being said, you may want to still include some attribution to the Jupyter project and to your free hosting service. To do all of this, just include a raw cell at the end of your notebook with some more JavaScript.


<footer id="attribution" style="float:right; color:#999; background:#fff;">
Created with Jupyter, delivered by Fastly, rendered by Rackspace.

One shortcoming with what we have so far is that users may still see some code or other unwanted elements while the page is loading. This can be especially problematic if you have a long presentation with many plots. To avoid this problem, add a raw cell at the very top of your notebook containing a preloader. This example preloader includes an animation that signals to users that the page is still loading. It heavily inspired by this preloader created by @mimoYmima.

  jQuery(document).ready(function($) {



<style type="text/css">
  div#preloader { position: fixed;
      left: 0;
      top: 0;
      z-index: 999;
      width: 100%;
      height: 100%;
      overflow: visible;
      background: #fff url('') no-repeat center center;


<div id="preloader"></div>

To work with these notebooks, you can clone my GitHub repository. While the notebooks render correctly on nbviewer (unpolished, polished), they do not render correctly on the GitHub viewer.


New Blog Address

Welcome to the new location for The File Drawer! This blog is now hosted on Github Pages and powered by Jekyll. My old blog at will be shutting down soon.

I actually really liked WordPress, but I wanted to have a little bit more control over what I can put in my posts. In particular, I wanted to be able to insert my own JavaScript animations, for example in this post on the recent presidential debates.

It was pretty fun to migrate everything over from WordPress – which is not to say there weren’t some hiccups along the way – but I was able to do so by following some nice instructions in this post by Joshua Lande.

For those of you using RSS, the feed for should be searchable in your RSS readers, but please let me know if it’s not.


10 classic dialogues you can find on the internet

Some videos on the internet are so good that I’ve watched them twice. Below is a list of 10 of my favorite interviews and dialogues. Obviously this isn’t an endorsement of all the positions taken. I just think they are very well done and fun to watch. The last four are best watched on 1.4x speed.

  1. 1971 Michael Parkinson interviews Muhammad Ali.

  2. 1974 Michael Parkinson interviews Muhammad Ali again, with better sound.

  3. 1997 Steve Jobs interacting with the audience when announcing the Microsoft deal.

  4. 1997 Steve Jobs interacting with the audience at WWDC.

  5. 2006 Stephen Colbert interviews Eleanor Holmes Norton.

  6. 2009 Robert Wright and Joel Achenbach

  7. 2009 Tyler Cowen and Peter Singer

  8. 2010 Robert Wright and Mickey Kaus

  9. 2011 Robert Wright and Mickey Kaus

  10. 2012 Glenn Loury and Ann Althouse

If I had to recommend just one, it would be Cowen/Singer.