Fix tsmatchsel() to account properly for null rows.

ts_typanalyze.c computes MCE statistics as fractions of the non-null rows,
which seems fairly reasonable, and anyway changing it in released versions
wouldn't be a good idea.  But then ts_selfuncs.c has to account for that.
Failure to do so results in overestimates in columns with a significant
fraction of null documents.  Back-patch to 8.4 where this stuff was
introduced.

Jesper Krogh

diff --git a/src/backend/tsearch/ts_selfuncs.c b/src/backend/tsearch/ts_selfuncs.c
index 1f0a42d9b1216b3f779d7e2edfd14fb8e4d4a457..709d48c61783bd3b11767b5e1dab95e4a1a9974e 100644 (file)
--- a/src/backend/tsearch/ts_selfuncs.c
+++ b/src/backend/tsearch/ts_selfuncs.c
@@ -188,11 +188,17 @@ tsquerysel(VariableStatData *vardata, Datum constval)
            /* No most-common-elements info, so do without */
            selec = tsquery_opr_selec_no_stats(query);
        }
+
+       /*
+        * MCE stats count only non-null rows, so adjust for null rows.
+        */
+       selec *= (1.0 - stats->stanullfrac);
    }
    else
    {
        /* No stats at all, so do without */
        selec = tsquery_opr_selec_no_stats(query);
+       /* we assume no nulls here, so no stanullfrac correction */
    }
 
    return selec;

diff --git a/src/include/catalog/pg_statistic.h b/src/include/catalog/pg_statistic.h
index 0e831ef29829f7e552bb94b2015498aab0c42ff3..edccf254b1b4e2dfef9c09b156597541b374e44f 100644 (file)
--- a/src/include/catalog/pg_statistic.h
+++ b/src/include/catalog/pg_statistic.h
@@ -244,6 +244,8 @@ typedef FormData_pg_statistic *Form_pg_statistic;
  * type with identifiable elements (for instance, tsvector).  staop contains
  * the equality operator appropriate to the element type.  stavalues contains
  * the most common element values, and stanumbers their frequencies.  Unlike
+ * MCV slots, frequencies are measured as the fraction of non-null rows the
+ * element value appears in, not the frequency of all rows.  Also unlike
  * MCV slots, the values are sorted into order (to support binary search
  * for a particular value).  Since this puts the minimum and maximum
  * frequencies at unpredictable spots in stanumbers, there are two extra