Multipart Formula

[matthieugomez]

Several arguments of a function fit typically refer to dataframe variables which are not regressors. Some examples: variables used to compute standard errors, weight variables, variables used to specify rows on which to estimate the model, high dimensional fixed effects, mixed models, etc.

It would be nice to think about the best syntax to refer to these variable. I have thought about three potential syntaxes:

1. Define a macro for each argument that needs to capture variable names

   fit(df, @formula(y ~ x1), @weight(x2), @vcov(cluster(x3+x4)), @where(x5 >= 0), maxiter = 100)

2. Define a model macro that accepts multiple arguments, i.e. @model(expr, args...).

   fit(df, @model(y ~ x1, weight = x2, vcov = cluster(x3+x4), where = (x5 >= 0)), maxiter = 100)

3. Define a fit macro that accepts multiple arguments (syntax closest to reg in Stata and to @with in DataFramesMeta.jl), i.e. @fit(expr1, expr2, args...)

   @fit df y ~ x1 weight = x2 vcov = cluster(x3+x4) where = (x5 >= 0) maxiter = 100
   # or (either would work)
   @fit(df, y ~ x1, weight = x2, vcov = cluster(x3+x4), where = (x5 >= 0), maxiter = 100)

An additional benefit is that agreeing on a syntax would help to standardize the names of commonly used arguments like "weights" "vcov" "where" across different packages that do statistical estimations. Enforcing these keyword arguments across different statistical estimations, like in Stata, could do a lot to improve the user experience.

[ararslan]

I don't understand the use case for capturing variable names anywhere but in the actual construction of the model terms. Also, macros don't support keyword arguments.

[matthieugomez]

I have three difference use cases in FixedEffectModels, you can simply read the Readme. Or look at all the R packages that rely on multipart formula
Macros can work with keyword arguments as they are just particular expressions. As in@test syntax

using Base.test
@test 1 ≈ 0.99 atol = 1e-2

[nalimilan]

I always thought passing symbols for variable names would be enough. That doesn't work for where, but I'm not sure I like the idea of supporting where in model fitting functions, as it sounds simpler to perform this step separately.

[matthieugomez]

Maybe I will be clearer if I delve into FixedEffectModels. To estimate these kinds of models, an estimation command requires the user to specify (i) a formula (ii) a set of high dimensional fixed effects (iii) a set of variables to cluster on.
A syntax that use symbol for everything except the formula, as proposed by @nalimilan, would look like that:

fit(@formula(y~x1+x2), fe = (:x3, :x4), vcov = cluster(:x5, :x6))

The syntax looks clumsy. Moreover, this syntax does not allow to use expressions such as x3&x4 in fe or vcov. That is, the syntax allows to regress on x3&x4, but does not allow to cluster on x3&x4.

To avoid these problems, package developers in R have turned to the Extended Model Formula R package. For instance, to estimate models with high dimentional fixed effects, the R package lfe uses the syntax:

felm(y~x1+x2 | x3&x4 + x4 | x6 + x7)

Although this syntax improves on the first one, I don't like it either because it is not clear what each part of the formula refers to. This is why I am pushing for a keyword syntax, something similar to

@formula(y~x1+x2, fe = x3&x4 + x4, vcov = cluster(x6 + x7))

While these examples focus on models with high dimentional fixed effects — the ones I am familiar with — these syntax issues are more common. A lot of R packages rely on the Extended Model Formulas R Package

[nalimilan]

I guess what we could do is that a general macro like @model or @fit would transform every expression starting with ~ into a formula, so that you don't need to repeat @formula. But automatically transforming fe = x3 + x4 into a formula sounds difficult/impossible to me: it doesn't generalize, and yet that macro would have to be defined only once in StatsModels.

So the solutions I can imagine are like:

@fit(y ~ x1+x2, fe = (:x3, :x4), vcov = cluster(:x5, :x6))
@fit(y ~ x1+x2, fe = ~ x3 + x4, vcov = cluster(~ x5 + x6))

[ararslan]

We should just get better about allowing function calls in formulas. (Last time I checked, we either didn't support that at all or only kind of did.) Then you can do what R does for example with offset in Poisson regression: @formula(y ~ a + b + offset(c)) In your case it'd be @formula(y ~ x1 + x2 + fe(x3, x4) + cluster(x6, x7)).

[matthieugomez]

@ararslan I don't think this overloading would work. This overloading would completely abuse mathematical notations : clustered variables refer to methods to compute standard errors, not variables to regress on. Moreover, what if there are some functions defined fe or cluster in the namespace?
Also, note that it is already possible to have function calls in formula in R, yet it did not prevent people to require multipart formulas.

[nalimilan]

So, would one of my examples suit your needs?

[ararslan]

fit wouldn't even need to be a macro for the former example.

fit(@formula(y ~ x1 + x2), fe=(:x3, :x4), vcov=cluster(:x5, :x6), ...)

You just need a Symbol varargs method for cluster.

[matthieugomez]

Yes, that would work. But why would you need to only capture the arguments prefixed by ~? The macro could also capture everything, i.e. something like

macro fit(x, args...)
    Expr(call, :fit, :(@formula($(esc(x)))), (Base.Meta.quot(args[i]) for i in 1:length(args))...)
end 

Each package would then define the fit function.

[nalimilan]

But not all arguments must be formulas. And in some cases you explicitly need to refer to local objects rather than to data frame columns.

[andreasnoack]

What about just using several @formula arguments?

[matthieugomez]

@nalimilan I agree, but this problem is common to every dplyr-like data manipulations (like DataFramesMeta).

@andreasnoack Something like this would work indeed.

fit(df, @formula(y ~ x1), fe = @formula(x3&x4 + x4), weight = @formula(x5), vcov = cluster(@formula(x3+x4))

It is quite verbose. I think I prefer Syntax 1

fit(df, @formula(y ~ x1), @fe(x3&x4 + x4)), @weight(x5), @vcov(cluster(x3+x4)))

but it requires to pollute the user space with macro definitions for each keyword argument.

[matthieugomez]

Maybe we can come back to this issue once there is a commonly accepted way to handle expressions that may or may not use variable names (i.e. something akin to a dplyr-like package). I just wanted to have this issue somewhere — it would be great to nail the syntax of estimation commands.

[nalimilan]

I don't think the two issues are completely related (and that's good news since it makes it easier to fix the present one). Query macros need to deal with identifiers which most frequently refer to data frame columns, with a way to escape this occasionally (e.g. using $x, which was evoked at some point). fit/@fit is quite different, since most arguments do not refer to columns: data frame object, distribution family, number of iterations, etc. Arguments which have to refer to columns are more the exception than the rule. What we need is a way to make them not too painful to use, but not the default either. I'd say my proposals are reasonable compromises, though there are other solutions.