Posts tonen met het label statistics. Alle posts tonen
Posts tonen met het label statistics. Alle posts tonen

vrijdag 12 mei 2017

Simulation panel-invariant variable: OLS versus fixed effects

Below is a simulation of the consequence of controlling for fixed effects. In some cases this is desirable, in others not.

In the wage equation below, the wage depends on gender (b = 2), effort (b = 10), and ability (b = 3). There is no unobserved heterogeneity. A straightforward OLS estimation will return the correct b's.

However, as gender and ability are panel-invariant, the fixed effect regression does not yield any effect -- although the effect of effort is unbiased.

On the other hand, say that ability and effort are correlated, and ability is not observed, the estimated beta if effort in OLS would be biased, but not in a fixed effects estimation. The latter, however, would not allow estimating the beta for gender.

Good riddance.



clear
gene byte gender = .
gene float effort = .
gene float wage = .
gene float ability = .
gene long id = .

forvalues i = 1/1000 {
local a = runiform()
local g = runiform() > .5

set obs `=`i'*5'

replace ability = `a' if missing(id)
replace gender = `g' if missing(id)
replace id = `i' if missing(id)
}
replace effort = runiform()
replace wage = 2*gender + 10*effort + 3*ability + .5*runiform()


regress wage gender effort ability
areg wage gender effort ability, abs(id)


/*
. regress wage gender effort ability

      Source |       SS           df       MS      Number of obs   =     5,000
-------------+----------------------------------   F(3, 4996)      >  99999.00
       Model |  50282.7806         3  16760.9269   Prob > F        =    0.0000
    Residual |  104.339671     4,996  .020884642   R-squared       =    0.9979
-------------+----------------------------------   Adj R-squared   =    0.9979
       Total |  50387.1202     4,999  10.0794399   Root MSE        =    .14452

------------------------------------------------------------------------------
        wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      gender |   1.999089   .0040916   488.58   0.000     1.991067     2.00711
      effort |   10.00101   .0070513  1418.33   0.000     9.987187    10.01483
     ability |   3.004343   .0071066   422.75   0.000     2.990411    3.018275
       _cons |   .2498592   .0058902    42.42   0.000     .2383118    .2614065
------------------------------------------------------------------------------

. 
end of do-file

. do "/var/folders/p2/2v2ckxtd2794655ypfbmfg9w0000gn/T//SD31177.000000"

. areg wage gender effort ability, abs(id)
note: gender omitted because of collinearity
note: ability omitted because of collinearity

Linear regression, absorbing indicators         Number of obs     =      5,000
                                                F(   1,   3999)   = 1602779.17
                                                Prob > F          =     0.0000
                                                R-squared         =     0.9983
                                                Adj R-squared     =     0.9979
                                                Root MSE          =     0.1450

------------------------------------------------------------------------------
        wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      gender |          0  (omitted)
      effort |   10.00054   .0078993  1266.01   0.000     9.985056    10.01603
     ability |          0  (omitted)
       _cons |   2.810143   .0044419   632.65   0.000     2.801435    2.818852
-------------+----------------------------------------------------------------
          id |      F(999, 3999) =    401.808   0.000        (1000 categories)
*/

maandag 6 februari 2017

Making sense of measuring effect size (eta-squared)

Introduction: it's impossible

We like to know whether effects are significant, or more significant than others, but generally avoid the issue of whether effects are substantial. On rare occasions, when an OLS regression suffices, a statistician may want to report R-squared values. However, do not expect much enthusiasm, as R-squared can only increase when adding additional variables even if they do not make sense. Adjusted R-squared, AIC and BIC measures correct for this. Also, R-squared has to be computed differently depending on whether or not there is a constant in the model (see Hayashi, 2000, p. 20). Worse, if your model is not OLS, you will need to define some pseudo-R-squared.

Needless to say, if you want an R-squared value for any subset of explanatory factors, statisticians will hate you. This is not possible, because of collinearity, they will say. You can stepwise add or delete variables and check the change in R-squared, but this will include or remove variance that is due to other variables. Sadly!

Objection: it may be possible

The people at UCLA, in fact Philip B. Ender, thought differently. They have a Stata package called -regeffectsize- for grabs which partitions the variance explained by the separate explanatory factors, giving a good indication of how substantial effects are. The help-file is not very instructive, so I deciphered the .ado myself, and here's the report. I'm sorry this is not (yet) in LaTeX, but speed over quality for now.

The structure of the program is as follows:
  • From the regression, they take:
    • TSS, total sum of squares in Y, which is RSS+MSS
    • RSS: residual sum of squares
    • MSS: the model sum of squares
    • DFR: residual degrees of freedom, which is n-k (n: sample size, k: number of variables in the model)
    • If you divide RSS by DFR you get the mean summed error MSE
  • From the F-test (simply -test-), they take:
    • The F-value
Obviously, eta-squared as we know it is ETA2 = MSS/TSS and it is generally decomposed in a within and between effect. Typically we get an ETA2 for the second level of a multi-level model, to indicate how much variation exists between versus within the second level units. This is also one formulation of the intraclass correlation coefficient ICC.

So how do they get to the (semi)partial eta-squared formula? There are three formulations, which are more or less the same and build up. The basis is the effect sum of squares, which is ESS = F*MSE.

Semipartial eta-squared

ETA2 = ESS/TSS = F*MSE/TSS

A glimpse of genius, if you ask me, to use a distribution unit (F) into the equation for the partitioning of variance. You might think that if F is really key, then our partitioning will be little more than a comparison of significance levels. The higher ESS, the more important the explanatory variable – for this it needs to be significant. In Hayashi, 2000, p. 53, I found some intuition that may be helpful. The F-test is a monotone transformation of the likelihood ratio.

Recall that MSE = RSS/DFR = RSS/(n-k) and add that F = (n-k)*(RSSr/RSSu-1) when there is only one test variable (#r = 1) to compare with the null. It is easy to see that n-k drops from the RHS and we are left with:

ETA2 = [ RSS*(RSSr-RSSu)/RSSu ] / TSS

RSSu are the unrestricted errors when beta is as calculated. Hence RSSu = RSS (the model residual sum of squares). RSSr are the restricted errors under the null (i.e. beta = 0). We can rewrite the equation to:

ETA2 = (RSSr-RSS)/TSS

Say there is only one variable in the model, then RSSr = TSS, and ETA2 = R2 = MSS/TSS. In general, the better the prediction, the lower RSS will be and the higher ETA2. Note that by definition RSS > RSSr, unless there is nothing explained, and then ETA2 = 0. What matters is the difference if we 'exclude' a variable: then RSSr rises proportionally to the importance of that variable.

Is this a good metric? I would tend to think so. The advantage of this approach in my view is that the beta's of other variables are unbiased by the omission of one variable, so MSS is unchanged and RSSr will be correct. There is also no difference in DFR in the models to compare, as it would be with stepwise deletion. To me, this looks useful as an approach to explain how much of total variance is explained by one explanatory variable, although I may overlook something. In that case, let me know.

Oh, I don't know why it would be semi-partial – the .ado uses the macro `spe2', so it's just a guess.

Percentage eta-squared

%ETA2 = 100*ETA2/R2

I think that is obvious enough. The output says 'change eta-squared', but there is no change involved.

Partial eta-squared

Here we have a little change, to not include the variance explained by other variables in the denominator. This will always be larger than the semipartial eta-squared, and will be exponentially higher when ESS approaches MSS. It could be a solution to avoid the issue of having much shared variance in MSS which is not in ESS, I'm not sure.

Part.ETA2 = ESS / (ESS+RSS)

References







dinsdag 3 februari 2015

Maximum Likelihood - with examples of regressions of continuous and discrete dependent variables

The idea of Maximum Likelihood is very simple, yet powerful. A regression returns a residual for every case, telling us how far the estimation is off. In Ordinary Least Squares, we minimize the residual. In Maximum Likelihood, we ... maximize the likelihood of the prediction. It sounds like it is the same thing, and in some cases it is the same thing. But whereas OLS mathematically computes the parameters of the model, ML uses a maximization algorithm (generally Newton-Raphson) that checks multiple possibilities. These are called iterations, and when the model works well, they will 'converge'. If there is no maximum (i.e. the maximizand is non concave), or several maxima, you are out of luck.

Continuous dependent variables

Take a simple regression like y = Xb, and suppose - as in simplified OLS - that the error term is normally distributed. Then e = y - Xb.

Now let f(e) = (2.pi.sigma^2)^-(1/2) . exp(- (e^2)/(2.sigma^2)

In this case f is a normal density function or normal distribution. If the error is 0, the density is highest (around .44), so a good estimation will need to have small errors.

The likelihood function we want to maximize is the joint probability distribution of the sample. This means we compute the density of the error for every case and multiply all of them. We may just as well take the log and sum the logs of the error density. When this sum of logs is maximizes, we have the same set of coefficients b as in OLS. Note that this will be the true b only if the error term is indeed normally distributed.

Discrete dependent variables

When the outcome variable y_obs is binary (0,1), such a linear model will not work. Instead we estimate a function that lies behind the observed outcome. If that function results in a number greater than 0, we will observe a success (1) and if not, a failure (0). Hence again, we have:

y = xb

Yet in this case we cannot simply compute a residual. In the probit case, we take the cumulative probability of xb. It is obvious that this depends on a scaling parameter sigma.

The likelihood function is then P(xb) if y_obs = 1, and 1-P(xb) if y_obs = 1. Another way to express this is:

L_i = P(xb)^y_obs * (1-P(xb))^(1-y_obs).

Maximizing the product of all L_i or the sum of the logs yields good estimates of b.

Note: an alternative to probit is the logit or logistic regression. Then the function P is not the normal cumulative distribution function, but instead it is the inverse logit:

P(xb) = exp(xb) / (1+ exp(xb) )

The advantage here is that no scaling sigma has to be estimated. It used to be popular in the early days of sociology, before econometrics became dominant in the social sciences.

Links

The last link has a nice picture comparing the cumulative distribution function and the inverse logit (or logistic) function. The latter has 'fatter' tails, even though the difference is negligible in practice.








donderdag 20 maart 2014

Type I and Type II errors

Type I: Ho true -- accept Ha / reject Ho -- try to limit this through minimizing the standard errors on your estimates, estimates should be efficient

Type II: Ha true -- reject Ha / accept Ho -- harder to know, estimates should be more consistent

http://en.wikipedia.org/wiki/Type_I_and_type_II_errors


donderdag 29 juli 2010

What stats package to use?

Introduction

Boys like their toys, and this is not different with statistical packages. It's a perpetual and heated debate and when you've landed at some point and think your workflow is good, technology passes you and sets you back. 

Here's an old discussion that I first consulted, but below I make my own considerations. In short: I'd use Python for big data, Stata for analysis, and R if I have to (e.g. for some graphs). Everything else I would ditch.

Stata

Stata is my program of choice. It is quite expensive, but mind you that for a couple of hundred euros, depending on the flavour, you'll not only get an easy and robust statistical software package, but also fast support, a good community, useful user commands, and a great documentation source. Fun fact: all documentation is read by the wife of the founder, who's not a statistician but perhaps even smarter. If she doesn't understand what the statisticians are saying, it goes back to the drawing board. 

The bad things: forget about ever copy-pasting anything. You'll also need to have a lot of memory on your computer, as Stata loads the whole file and just one at a time (although you can 'preserve' a file temporarily to work on something else in between).

Python

Python is the next language I will learn. I have used chunks without understanding what I was doing, but I like the sound of the language, and it's the logical step-up after Stata, it seems. Many people are using it and so will I.

R

I don't like R. There is a thorough discussion here, circling around leaving Stata for R, but ending up in concluding what I conclude about: R is a mixture of a coding language like Python and a statistical language like Stata, but because it is open source the support is unsure, the community tends to be geeky and unfriendly, the documentation is poor, and the language consistency - even if the structure is good because it is a programming language - is bad. Some commands have their own inner programming language and that is plain bad. 

The good things: it is free and R Studio is a great user interface. It has good graphic capabilities, and 

Mplus

I don't know Mplus. Colleagues use it when there are issues with missing values, and the programmers are said to be the best statisticians in the world. So it must be good, but I don't use it.

SAS

This is old software. It is too complicated, and while it can do a lot through obscure options, it is not flexible enough to do what you want.

SPSS

This is bad software. It is a scandal that some universities still teach this.

Some R resources

Apparently the single best manual for R: https://r4ds.had.co.nz.