Protobi

← All blog posts View post

<p>How many ideas might you expect to find in customers' responses to open-ended survey question?
Here's an interesting empirical analysis of text verbatim coding from a recent survey,
looking at actual data compared to expected values under Zipf's Law and Heap's Law.</p>

<p>The survey question was "Why did you choose the product you selected?".  Respondents provided free-text responses.
200 responses were coded in Protobi using the new <a href="http://protobi.com/post/text-verbatims">verbatim coding widget </a>by a professional analyst.</p>

Four responses were  blank and  excluded from this analysis.  Codes are sanitized for display.

<div class="chart-title">Frequency distribution of individual codes</div>
The first graph shows the frequency distribution.  There appears to be a "long-tail" distribution where there are a few
high-frequency codes, and many lower-frequency codes.
<div class=figure id="code-frequency">Loading...</div>
<img src="/uploads/images/image-2015-7-28-verbatims-empiric-2024-08-30-03-12-16.png" alt="" style="max-width: 400px"/>

<div class="chart-title">Frequency of individual codes vs Rank (Zipf's Law)</div>
<p>According to <a href="https://en.wikipedia.org/wiki/Zipf%27s_law">Zipf's Law</a> we'd expect the frequency of each code to be inversely proportional
to its rank:</p>

<div class=katex data-display=true> z(r) = z_{max} \cdot r ^ {-\alpha}</div>

<p>Zipf's law can be derived from the power-law probability distribution, which describes many long-tail phenomena.</p>

<p>Here the blue line represents actual frequencies.
 The green line is the expected value per theory, with a slope <span class=katex>\alpha = 1.0</span>, and intercept equal to the frequency of the most common response.
 The grey line is the least squares estimate, with a slope <span class=katex>\alpha = 1.001</span>.</p>

<p>Simply put, a slope of -1 means that we'd expect the 2nd most common code to appear about 1/2 as often as the 1st,
 the 3rd most common code to appear about 1/3 as often, the 4th most common code to appear 1/4 as often, etc.</P>

<p>Per the graph below, Zipf's Law appears to match extremely well (except at the tails which is often the case in practice):</p>
<img class=figure id="codes-zipf" src="/uploads/images/image-2015-7-28-verbatims-empiric-2024-08-30-03-13-12.png" alt="" style="max-width: 400px"/>

<div class="chart-title">Unique codes encountered vs responses seen (Heap's Law)</div>
<p>A practical question is "How many respondents do we need to discover all the codes that there are to be found (above some minimum prevalence)?".
Or conversely, with our planned sample size, how many  ideas will likely be out there left undiscovered?
This is a variation of the <a href="https://en.wikipedia.org/wiki/Coupon_collector%27s_problem">Coupon Collector's Problem</a> <a href="#1">[2]</a>
(or Baseball Card Collector's problem?).</p>

<p>The number of codes we'd expect to encounter at any given sample size we might expect to be described
by <a href="https://en.wikipedia.org/wiki/Heaps%27_law">Heap's Law</a>:

<div class=katex data-display=true>N(t) \approx k \cdot t ^ \beta </div>

<div class=katex data-display=true>W(n) \approx k \cdot  ^ \beta </div>

<p>where <span class=katex>N(t)</span> is the number of distinct codes we would expect to find in <span class=katex>t</span> responses,
and <span class=katex>k</span> and <span class=katex>\beta</span> are estimated empirically.

<p>It turns out that Heaps's Law is a consequence of Zipf's Law above <a href="#1">[1]</a>.  And in the special case where <span class=katex>\alpha</span>=1
in the Zipf distribution, then there is an exact formula for Heap's Law     based on the
<a href="https://cs.uwaterloo.ca/research/tr/1993/03/W.pdf">Lambert W function</a> <a href="#3">[3]</a>:</p>

<p>(Amazingly, this dataset yields <span class=katex>\alpha</span>=1.001 and Lambert W appeared in another completely unrelated analysis we recently did to <a href="/post/optimal-price-in-discrete-choice-models">calculate optimal price in a discrete choice model</a>).</p>

<p>In the graph below, the blue line shows the actual number of unique codes encountered in the first N responses.
The grey line represents the least-squares estimate, yielding an estimated exponent of <span class=katex>\beta</span> = 0.460,
well in the range of 0.4 to 0.6 based on other analyses of English-language text.  Again we see that the data matches Heap's Law very well.</p>

<div class=figure id="codes-heap">Loading...</div>

<div class="chart-title">So what...</div>
<p>We're actively exploring ways to make text open end responses a rich source of insight for market researchers.
And make text analysis fun and easy.  This is an early step, looking at the data with thought-leading clients.  </p>

<p>A practical outcome
may be simple diagnostic metrics that help identify if the data is undercoded (i.e. there may be ideas yet to be discerned)
or overcoded (i.e. we might be making more distinctions than the sample size might support).</p>

<h1>Research questions</h1>
<p>Looking at the text responses and the process of coding them raises a number of interesting questions.
At a high level:</p>
<ul>
<li>When do "interesting" ideas appear?  The most common answers are presumably kind of known. The really rare responses may not be relevant.</li>
<li>What makes a response "interesting" to an end client?  Which ideas does the client consider to be the "pearls"? </li>
<li>Do end clients and product/marketing managers code differently than analysts? Do they make different distinctions? </li>
</ul>
Other questions are more technical:
<ul>
<li>Do most verbatim questions follow these curves?  Do they fall in a close or wide range?</li>
<li>How often do responses include multiple ideas that match several codes?</li>
<li>Is it common for codes to coalesce and split as analysis proceeds?</li>
<li>Can the computer learn from the initial codes and provide good auto-guesses as coding proceeds?</li>
</ul>

<p>If you have text survey data and are interested in mining it further, contact us at <a href="mailto:support@protobi.com">support@protobi.com</a>.</p>

<h3>References</h3>
<li id="1">[1] Lu¨ L, Zhang Z-K, Zhou T (2010) <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2996287/pdf/pone.0014139.pdf">Zipf’s Law Leads to Heaps’ Law: Analyzing Their Relation in Finite-Size Systems.</a>
PLoS ONE 5(12): e14139. doi:10.1371/journal.pone.0014139</li>
<li id="2">[2] Marco Ferrante, Monica Saltalamacchia (2014)
<a href="http://www.mat.uab.cat/matmat/PDFv2014/v2014n02.pdf">The Coupon Collector’s Problem</a>
   MATerials MATemàtics, Volum 2014, treball no. 2, 35 pp. ISSN: 1887-1097
   Publicació electrònica de divulgació del Departament de Matemàtiques de la Universitat Autònoma de Barcelona</li>
<li id="3">[3] R.M. Corless, G.H. Gonnet, D.E.G. Hare, D.M. Jeffrey and D. E. Knuth, <a href="https://cs.uwaterloo.ca/research/tr/1993/03/W.pdf">"On Lambert's W Function"</a>, Technical Report CS-93-03,
         University of Waterloo, January 1993.</li>

<style>
    .figure {
      margin-bottom: 40px;
    }
    .chart-title {
      font-weight: bold;
      color: #444;
      font-size: 18px;
      margin-top: 20px;
    }
    .axis path,
    .axis line {
      fill: none;
      stroke: black;
      shape-rendering: crispEdges;
    }

.hover {
      stroke: #fc0;
      stroke-width: 4px;
    }

.line-random {
      stroke: #aaa;
    }

.line-theory {
      stroke: #9b6;
      stroke-width: 4px;
    }

.dot-actual {
      fill: #39C;
      stroke: #26a;
      r: 5px;
      stroke-width: 1px;
    }

.line-actual {
      fill: none;
      stroke: #39c;
      stroke-width: 4px;
    }
    .line-estimate {
      fill: none;
      stroke: #ccc;
      stroke-width: 4px;
    }

.axis text {
      font-family: sans-serif;
      font-size: 14px;
    }

.tw-bs .label {
      font-family: sans-serif;
      font-size: 14px;
      font-weight: normal;
    }

.bar {
      fill: #39c;
    }
  </style>

// draw streamlines
var line_graph = function (el, attributes, series) {
  console.log("Rendering streamlines");
  var self = this;
  this.$el = $(el);
  this.el = this.$el[0]
  this.attributes = attributes;
  this.$el.html("<h1>Hi!</h1>");

var svg = this.svg = d3.select(this.el).html('').append('svg')
          .style('height', this.attributes.height)
          .style('width', this.attributes.width);

xScale = (this.attributes.xAxisLog ? d3.scale.log() : d3.scale.linear())
          .domain(this.attributes.xAxisDomain)
          .range([this.attributes.margin.left, this.attributes.width - this.attributes.margin.right]);

yScale = (this.attributes.yAxisLog ? d3.scale.log() : d3.scale.linear())
          .domain(this.attributes.yAxisDomain)
          .range([this.attributes.height - this.attributes.margin.bottom, this.attributes.margin.top]);

xAxis = d3.svg.axis()
          .scale(xScale)
          .orient('bottom')
          .ticks(5)
          .tickFormat(d3.format(this.attributes.xAxisFmt))
  ;

yAxis = d3.svg.axis()
          .scale(yScale)
          .ticks(5)
          .orient('left')
          .tickFormat(d3.format(this.attributes.yAxisFmt))
  ;

svg.append('g')
          .attr('class', 'axis')
          .attr('transform', "translate(0, " + (this.attributes.height - this.attributes.margin.bottom) + ")")
          .call(xAxis);

svg.append('g')
          .attr('class', 'axis')

.attr('transform', "translate(" + this.attributes.margin.left + ", 0)")
          .call(yAxis);

var gradient = svg.append("svg:defs")
          .append("svg:linearGradient")
          .attr("id", "gradient")
          .attr("x1", "0%")
          .attr("y1", "100%")
          .attr("x2", "100%")
          .attr("y2", "100%")

gradient.append("svg:stop")
          .attr("offset", "0%")
          .attr("stop-color", "#36f")
          .attr("stop-opacity", 0.2);

gradient.append("svg:stop")
          .attr("offset", "50%")
          .attr("stop-color", "#36f")
          .attr("stop-opacity", 1);

gradient.append("svg:stop")
          .attr("offset", "100%")
          .attr("stop-color", "#fff")
          .attr("stop-opacity", 0.1);

svg.append("text")
          .attr("class", "x label")
          .attr("text-anchor", "end")
          .attr("x", this.attributes.width / 2)
          .attr("y", this.attributes.height - 6)
          .text(this.attributes.xAxisTitle);

svg.append("text")
          .attr("class", "y label")
          .attr("transform", "rotate(-90)")
          .attr("x", 0 - (this.attributes.height / 2))
          .attr("dy", "1em")
          .style("text-anchor", "middle")
          .text(this.attributes.yAxisTitle);

series.forEach(function (seri) {

var line = d3.svg.line()
            .x(function (point, idx) {
              return xScale(point.x);
            })
            .y(function (point, idx) {
              return yScale(point.y);
            });

if (seri.line) {
      var line = svg.append("path")

.attr("d", line(seri.data))
              .attr("stroke", "blue")
              .attr("stroke-width", 1)
              .attr("fill", "none")
              .attr("class", seri.line.class)
              .on("mouseover", function (d) {
                d3.select(this).classed("hover", true);
              })
              .on("mouseout", function (d) {
                d3.select(this).classed("hover", false);
              })

}
mage
    if (seri.dot) {
      svg.selectAll("g.dot")
              .data(seri.data)
              .enter()
              .append("g")
              .append("circle")
              .attr("class", seri.dot.class)
              .attr("r", 4)
              .attr("cx", function (d, i) {
                return xScale(d.x);
              })
              .attr("cy", function (d, i) {
                return yScale(d.y);
              })

}
  });
};

// draw streamlines
var bar_graph = function (el, attributes, series) {

var self = this;
  this.$el = $(el);
  this.el = this.$el[0]
  this.attributes = attributes;
  this.$el.html("<h1>Hi!</h1>");

var svg = this.svg = d3.select(this.el).html('').append('svg')
          .style('height', this.attributes.height)
          .style('width', this.attributes.width);

//  xScale = d3.scale.linear()
//          .domain(this.attributes.xAxisDomain)
//          .range([this.attributes.margin.left, this.attributes.width - this.attributes.margin.right]);

var width = this.attributes.width - this.attributes.margin.left - this.attributes.margin.right;
  var height = this.attributes.height - this.attributes.margin.top - this.attributes.margin.bottom;

var xScale = d3.scale.ordinal()
          .domain(series.map(function (d) {
            return d.name;
          }))
          .rangeRoundBands([this.attributes.margin.left, this.attributes.width - this.attributes.margin.right], .1)

yScale = d3.scale.linear()
          .domain(this.attributes.yAxisDomain)
          .range([this.attributes.height - this.attributes.margin.bottom, this.attributes.margin.top]);

xAxis = d3.svg.axis()
          .scale(xScale)
          .orient('bottom')
          .ticks(5)
  ;

yAxis = d3.svg.axis()
          .scale(yScale)
          .ticks(5)
          .orient('left')
          .tickFormat(d3.format(this.attributes.yAxisFmt))
  ;

svg.append('g')
          .attr('class', 'axis')
          .attr('transform', "translate(0, " + (this.attributes.height - this.attributes.margin.bottom) + ")")
          .call(xAxis)
          .selectAll("text")
          .style("text-anchor", "end")
          .attr("dx", "-.8em")
          .attr("dy", ".15em")
          .attr("transform", "rotate(-90)");

svg.append('g')
          .attr('class', 'axis')
          .attr('transform', "translate(" + this.attributes.margin.left + ", 0)")
          .call(yAxis);

gradient.append("svg:stop")
          .attr("offset", "0%")
          .attr("stop-color", "#36f")
          .attr("stop-opacity", 0.2);

gradient.append("svg:stop")
          .attr("offset", "50%")
          .attr("stop-color", "#36f")
          .attr("stop-opacity", 1);

gradient.append("svg:stop")
          .attr("offset", "100%")
          .attr("stop-color", "#fff")
          .attr("stop-opacity", 0.1);

svg.selectAll(".bar")
          .data(series)
          .enter().append("rect")
          .attr("class", "bar")
          .attr("x", function (d) {
            return xScale(d.name);
          })
          .attr("y", function (d) {
            return yScale(d.value);
          })
          .attr("height", function (d) {
            return height - yScale(d.value) + self.attributes.margin.top;
          })
          .attr("width", xScale.rangeBand());
};

setTimeout(function () {

var recoded = coded.filter(function (val) {
    return val.code != ''
  })
  coded = recoded;
  var series = [];
/*

for (var i = 0; i < 100; i++) {

var shuffled = shuffle(coded)

*/
series.push({
    name: "Heap's Law",
    dot: null,
    line: {
      class: "line-estimate"
    },
    data: coded.map(function (row, idx) {
      return {
        x: idx+1,
        y: 1.6963 * Math.pow(idx, 0.5)
      }
    })
  });

/*
  series.push({
      name: "Heap's Law",
      dot: null,
      line: {
        class: "line-theory"
      },
      data: coded.map(function (row, idx) {
        var b = (idx + 1);
        return {
          x: b,
          y: (b ) / (gsl_sf_lambert_W0(b))
        }
      })
    });

*/
/*
  var counter = new Counter();
  coded.forEach(function (row) {
    counter.add(row.code);
  });
  console.log(counter);
  series.push({
    name: "Actual # codes",
    dot: {
      class: "dot-actual"
    },
    line: {
      class: "line-actual"
    },
    data: counter.counts.map(function (val, idx) {
      return {
        x: idx,
        y: val
      }
    })
  });
*/

$("#codes-vs-index").html("Boo");
  // Render once with expected profit
  line_graph("#codes-heap", {
    width: 600,
    height: 400,

xAxisTitle: 'Respondent #',
    yAxisTitle: 'Cumulative # Unique Codes',
    xAxisDomain: [0, 200],
    yAxisDomain: [0, 20],
    yAxisFmt: '.0f',
    xAxisFmt: '.0f',

margin: {
      top: 20,
      left: 60,
      bottom: 50,
      right: 40
    }
  }, series);

xAxisTitle: "Coded value",//'Respondent #',
    yAxisTitle: '# of responses',
    yAxisDomain: [0, 50],
    yAxisFmt: '.0f',
    bar: {
      class: "bar-frequency"
    },
//    xAxisFmt: '.0f',

margin: {
      top: 20,
      left: 60,
      bottom: 150,
      right: 40
    }
  }, bardata);

var freqdata = bardata.map(function (entry, idx) {
    return {
      x: idx +1,
      y: entry.value
    }
  });

var freqseries = [];
  freqseries.push({
        name: "Actual",
        data: freqdata.map(function(entry, idx) {
          return {
            x: idx+1,
            y: 57.267 * Math.pow(idx + 1, -1.001)
          }
        }),
        line: { class: "line-estimate"}
      })

freqseries.push({
    name: "Theory",
    data: freqdata,
    line: { class: "line-actual"},
    dot: {class: "dot-actual"}
  })

freqseries.push({
      name: "Actual",
      data: freqdata.map(function(entry, idx) {
        return {
          x: idx+1,
          y: 45  / (idx+1)
        }
      }),
      line: { class: "line-theory"}
    })

// Render once with expected profit
  line_graph("#codes-zipf", {
    width: 600,
    height: 460,

xAxisTitle: 'Code response rank ',
    yAxisTitle: '# of responses',
    xAxisDomain: [1, 20],
    yAxisDomain: [1, 50],
    yAxisFmt: '.0f',
    xAxisFmt: '.0f',
    xAxisLog: true,
    yAxisLog: true,

margin: {
      top: 20,
      left: 60,
      bottom: 50,
      right: 40
    }
  }, freqseries);

return this;
}, 10);

function shuffle(original) {

var array = original.map(function (val) {
    return val;
  })
  var currentIndex = array.length, temporaryValue, randomIndex;

// While there remain elements to shuffle...
  while (0 !== currentIndex) {

// Pick a remaining element...
    randomIndex = Math.floor(Math.random() * currentIndex);
    currentIndex -= 1;

// And swap it with the current element.
    temporaryValue = array[currentIndex];
    array[currentIndex] = array[randomIndex];
    array[randomIndex] = temporaryValue;
  }

return array;
}

var coded = [
{"code":"Cost"},
{"code":""},
{"code":""},
{"code":"Smart phone"},
{"code":"Portable"},
{"code":"Cost"},
{"code":"Keep existing monitor"},
{"code":"Easy"},
{"code":"Keep existing monitor"},
{"code":"No landline"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Internet ready"},
{"code":"Cost"},
{"code":"Already have it"},
{"code":"Internet ready"},
{"code":"Easy"},
{"code":"No landline"},
{"code":"Cost"},
{"code":"Portable"},
{"code":"No landline"},
{"code":"Portable"},
{"code":"Cost"},
{"code":"Keep existing monitor"},
{"code":"Keep existing monitor"},
{"code":"Portable"},
{"code":"No need to change"},
{"code":"Cost"},
{"code":"No need to change"},
{"code":"Internet ready"},
{"code":"No need to change"},
{"code":"No need to change"},
{"code":"No landline"},
{"code":"Cost"},
{"code":"Portable"},
{"code":"Reliable"},
{"code":"Easy"},
{"code":"Wireless"},
{"code":"Easy"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Internet ready"},
{"code":"Easy"},
{"code":""},
{"code":"Easy"},
{"code":"Keep existing monitor"},
{"code":"Already have it"},
{"code":"Portable"},
{"code":"Easy"},
{"code":"Seemed best"},
{"code":"Cost"},
{"code":"Internet ready"},
{"code":"Easy"},
{"code":"Cost"},
{"code":"Familiar"},
{"code":"Portable"},
{"code":"Portable"},
{"code":""},
{"code":"No need to change"},
{"code":"No need to change"},
{"code":"Cost"},
{"code":"Portable"},
{"code":"Internet ready"},
{"code":"Cost"},
{"code":"No need to change"},
{"code":"Familiar"},
{"code":"Familiar"},
{"code":"Smart phone"},
{"code":"Cost"},
{"code":"Portable"},
{"code":"Easy"},
{"code":"Familiar"},
{"code":"No need to change"},
{"code":"Cost"},
{"code":"Easy"},
{"code":"Familiar"},
{"code":"Portable"},
{"code":"Easy"},
{"code":"Internet ready"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Smart phone"},
{"code":"Easy"},
{"code":"Familiar"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Cost"},
{"code":"No need to change"},
{"code":"Portable"},
{"code":"Unsure"},
{"code":"Wireless"},
{"code":""},
{"code":"No need to change"},
{"code":"Smart phone"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Wireless"},
{"code":"Have equipment"},
{"code":"No need to change"},
{"code":"Appealing"},
{"code":"Easy"},
{"code":"Seemed best"},
{"code":"Seemed best"},
{"code":"Appealing"},
{"code":"Cost"},
{"code":"Appealing"},
{"code":"Portable"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Cost"},
{"code":"Appealing"},
{"code":"Easy"},
{"code":"Appealing"},
{"code":"Portable"},
{"code":"Appealing"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Unsure"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Reliable"},
{"code":"Wireless"},
{"code":"Easy"},
{"code":"Appealing"},
{"code":"Portable"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Internet ready"},
{"code":"Easy"},
{"code":"Familiar"},
{"code":"Have equipment"},
{"code":"Wireless"},
{"code":""},
{"code":"Already have it"},
{"code":"Cost"},
{"code":"Familiar"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Portable"},
{"code":"Cost"},
{"code":"Have equipment"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Keep existing monitor"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Cost"},
{"code":"Internet ready"},
{"code":"Portable"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Appealing"},
{"code":"Internet ready"},
{"code":"Easy"},
{"code":"Easy"},
{"code":"Cost"},
{"code":"Cost"},
{"code":"Internet ready"},
{"code":"Easy"},
{"code":"Have equipment"},
{"code":"Appealing"},
{"code":"Wireless"},
{"code":"Easy"},
{"code":"Unsure"},
{"code":"Already have it"},
{"code":"Easy"},
{"code":"Smart phone"},
{"code":"Cost"},
{"code":""},
{"code":"Wireless"},
{"code":"Easy"},
{"code":"No landline"},
{"code":"No need to change"},
{"code":"Familiar"},
{"code":"Seemed best"},
{"code":"Appealing"},
{"code":"Cost"},
{"code":"Paying is better"},
{"code":"No need to change"},
{"code":"Cost"},
{"code":"Appealing"},
{"code":"Paying is better"},
{"code":"Seemed best"},
{"code":"No need to change"},
{"code":"No landline"},
{"code":"Paying is better"},
{"code":"Seemed best"},
{"code":"Paying is better"},
{"code":"Paying is better"},
{"code":"Internet ready"},
{"code":"Already have it"}
];

</script>

Date

Status

Slug edit

Thumbnail

Categories Manage