`metadata` method

[Tokazama]

It is often the case that one wants to attach metadata of some sort to an array/graph/etc. How do people feel about adding something basic like metadata(x) = nothing that can then be extended by other packages?

[bkamins]

@pdeffebach had some good design ideas about it in DataFrames.jl in the past.
Now, finally after 0.21.0 release, we are planning to add this functionality to DataFrames.jl.

As this is raised on a higher level let me give the API I envision for DataFrames.jl for now:

• metadata(::DataFrame) returns a Union{Nothing, Dict{Symbol,Any}} that - if filled - gives a DataFrame-level metadata (this can be arbitrary metadata). The restriction would be that symbols starting with DF_ in their name would be reserved for internal use of DataFrames.jl (as a convention)
• metadata(obj::OTHER_TYPES_DEFINED_BY_DATAFRAMES) = metadata(parent(obj))
• metadata(::DataFrame, ::ColumnIndex) returns a String (by default nothing) - which would indicate just a verbose name of the column, with default being just column name
• metadata(obj::OTHER_TYPES_DEFINED_BY_DATAFRAMES, ::ColumnIndex) similar to the above if the column is present in the other type.

If we agree to this design then I can implement it. The key challenge is rules of propagation of metadata, but this is not DataAPI.jl related thing so I leave this discussion for later.

CC @pdeffebach, @nalimilan

[nalimilan]

See JuliaData/DataFrames.jl#1458 for the last attempt at implementing this in DataFrames. Two points:

• I think we need something more general than just having a custom label/verbose name for columns. For example it could be useful to store units, informations about measurement, etc. Label can just be a standard field among others.
• We also need an API to set metadata and to retrieve the list of fields that have been set.

In general a choice has to be made between having 1) a single function in the API which would return a metadata dict which would have to implement specific methods (getindex, setindex! and keys notably); or 2) several functions in the API that would allow doing these operations directly. See the table in my first comment at JuliaData/DataFrames.jl#1458. I think returning an object is simpler since it allows reusing the standard dict API.

[Tokazama]

Metadata could technically be stored at any level of something like a table. For example, each column could be a MetadataArray (i.e. from MetadataArrays.jl) and the table itself could have metadata. I worry that if we started trying to design this around column based indexing it would needlessly complicate and potentially limit its wider usability. Even the definition of what "metadata" is to different people is likely to vary so I'm not sure we should even guarantee it returns a certain type.

[bkamins]

Initially I wanted to write that metadata(::DataFrame, ::ColumnIndex) could also return a Dict{Symbol, Any} - for me it would be OK. In this case there should also be some namespace of resrved key names for internal use.

So personally I would prefer the "single function that returns a metadata dict" approach and later the user can just work on the Dict.

Ah - and now I see we could support metadata(::DataFrame, ::ColumnIndex) that would return a NamedTuple of dictionaries associated with columns.

I agree with @Tokazama that different people will want different things from metadata therefore I believe the API we provide should be maximally simple and flexible.
Therefore I would prefer to think that metadata is just a Dict and there is one global dict for a data frame as a whole and then each column can have column specific dicts. Then the rest - how to work with it - would be delegated to a decision of the user.

[Tokazama]

Allowing the user to decide what to do with whatever metadata return also provides the freedom to further specialize on this later. For example you could always do something like colmeta(df, col) = metadata(df)[col] and then you wouldn't have to worry about reserving key names.

Would a simple PR to DataAPI.jl on this be a good next step right now?

[bkamins]

One will probably need to reserve key names anyway. In particular I do not think that metadata(df)[col] to return a metadata for column col is a good API (if we allowed this then there would be no way to specify global metadata for a table as a whole).

I think this is such a major thing that we should wait for other JuliaData members to comment before moving forward.

[nalimilan]

Maybe we can say that metadata(tbl) and metadata(tbl, col) have to return objects implementing the AbstractDictAPI, giving respectively the table-wise and column-wise metadata? That should be flexible enough for all implementations.

(In practice for DataFrames we would probably store column metadata internally as vectors with one entry per column as JuliaData/DataFrames.jl#1458 does but that can be exposed to users via a lazy AbstractDict object per column. What could be useful to provide in addition is a way to access these vectors for convenience/efficiency.)

[bkamins]

have to return objects implementing the AbstractDictAPI

Agreed

In practice for DataFrames we would probably store column metadata internally as vectors with one entry per column

We can discuss what is best in the PR for DataFrames.jl when it is done (essentially we have two options: dict of vectors or vector of dicts).

[bkamins]

As have been thinking about this issue and #1458 I came to the conclusion that we should go back to the fundamentals. And the core issue is:

It is often the case that one wants to attach metadata of some sort to an array/graph/etc.

What I mean that while we seem all agree that adding metadata to tables is needed, actually I would discuss first what kind of metadata we really think people would store in practice. This is a relevant questions as I think we should not create a functionality that later would be very rarely used. Conversely - if we know exactly what we actually want to use we can design API that supports the required use-cases cleanly.

My two concerns are:

• persistence; most storage formats will not allow to save and load this metadata; which means that, at least in my understanding, the use cases, where people will use metadata will be situations of non-persistent metadata (i.e. something you attach to your table temporarily for programming convenience)
• performance; we do not want to kill the performance of basic operations on tables, because the "table processing engine" would constantly check if metadata needs updating, or if the cos of updating the metadata would be large, or if the memory footprint of allowing to store metadata would be non-negligible (ideally if there is no metadata then the performance should not be affected)

So now let me go down to the starting question - what metadata we see that would be actually used (this is not a comprehensive list - please comment what you think would be really used - not just potentially used):

• metadata for handling how data frame is shown (things like overriding show defaults, maybe custom decimal delimiter, maybe - if in the future we add integration with PrettyTables.jl, some settings of all the options that package provides)
• custom column labels (possibly shown when printing a data frame) - this is tempting, but has a problem with "persistence" - i.e. most formats will not store this information, so my question is - how in practice we see that people will use this functionality?
• custom row lables labels - the same situation
• setting flags that some columns should be treated in a special way, e.g. for geospatial or time series analysis of data frames - this is tempting, but myself I am not 100% convinced it is a good idea, as it will be hard to ensure the metadata is consistent with a parent table (e.g. GeoPandas uses this strategy, and it has a problem with ensuring this integrity)

[Tokazama]

In addition to what you've mentioned here are some types of metadata that I think would be useful for me personally to be able to store:

• Source and collection information: I often have many tables that have some acquisition metadata pertinent to them that is not row or column specific but describes important aspects of all the data in one table.
• Column tracking: When performing semi-automated feature creation I like to keep track of certain operations/parameters/weights that resulted in the formation of a column of measures.
• Attaching metadata to a column that changes how it dispatches later on

it will be hard to ensure the metadata is consistent with a parent table (e.g. GeoPandas uses this strategy, and it has a problem with ensuring this integrity)

I think it depends on how much you care to take ownership of handling all metadata. I would prefer handling metadata be given a minimal interface. It could potentially have a methods for things like joins so that something like join_metadata would also join dictionaries but could be taken advantage of for custom metadata types.

I also think that I/O on metadata should be entirely dependent on the package supporting I/O. There aren't many file types equipped to flexibly handle metadata and it seems like the best thing for DataAPI.jl is to just make it simple to extract metadata.

[pdeffebach]

I agree with all that is said here. As the author of one of the previous attempts I think that meta-data is important and people coming from R and Python often don't fully appreciate how useful metadata is for Stata users and how it has hurt the adoption of R in applied economics, especially household surveys.

• custom column labels (possibly shown when printing a data frame) - this is tempting, but has a problem with "persistence" - i.e. most formats will not store this information, so my question is - how in practice we see that people will use this functionality?

My use of metadata in Stata was twofold

1. Pretty printing of columns and keeping track of data. For example, the table below would not have been possible to make programattically without extensive use of column labels. I couldn't imagine trying to write this in R because column metadata in R doesn't persist after joins.

2. Keeping track of the data-cleaning process. In the above table, the variable "Standardized income index" is composed of the 3 variables below it. The note for that variable will tell us as much, and was automatically generated. If you were to type

note list standardized_index

you would see a note that said something along the lines of

A standardized index of 3 variables: net_earnings, consumption, durable_assets

Stata also has metadata about a table, which is often used to denote a source or author. I never used that feature.

With regards to IO, I don't see a huge problem with saving a data frame to two CSVs and providing a convenience method for adding metadata to a DataFrame when the metadata is stored as a Table. Maybe it's a bit heavy handed but it's robust.

[Tokazama]

I think these are all great use cases that I've wanted at some point. As someone who deals with lots of different types of metadata I'd really like to emphasize that less is more as this is implemented. It's easy to get stuck in the weeds on every little implementation detail because you have the combination of situations that arise from row specific, column specific, and general table metadata and all the different types of metadata.

This is loosely the kind structure I'm considering using...

struct Table{T<:AbstractVector,M<:Union{Nothing,AbstractDict{Symbol,Any}}}
    data::Vector{T}
    index::Dict{Symbol,Int}
    meta::M
end

metadata(x::Table) = getfield(x, :meta)

Users don't have to ever worry about metadata unless they decide they want it and developers can create whatever type of fancy metadata that changes dispatch as long as it is a subtype of AbstractDict{Symbol,Any}.

I would think column specific metadata would be easiest if implemented as a column vector with metadata so I could just do metadata(table.column_name). Otherwise concatenating/merging/joining columns becomes the responsibility of DataAPI.jl. Seems more simple for this to be implemented at the level of something like vcat(::MetadataVector, ::MetadataVector).

[nalimilan]

I agree with most of what has been said. Just one point:

I would think column specific metadata would be easiest if implemented as a column vector with metadata so I could just do metadata(table.column_name). Otherwise concatenating/merging/joining columns becomes the responsibility of DataAPI.jl. Seems more simple for this to be implemented at the level of something like vcat(::MetadataVector, ::MetadataVector).

I'm afraid this wouldn't be workable, as it would require users to deal with another new kind of vector just to store metadata. That would force recompiling all functions for that type, and it wouldn't be easy to deal with e.g. CategoricalArray. It would also make things appear more complex for users who load files with metadata (e.g. Stata files with column labels), while one of the strengths of DataFrames is that they just wrap standard arrays.

We can say the table type is responsible for preserving metadata across concatenations/joins. DataAPI itself doesn't have to know anything about that.

[pdeffebach]

With regards to spatial data, which is a natural use case of this, is there anyone in the Julia Data community who has a really detailed knowledge of R's sf package?

It's the best thing ever, being able to use all of dplyr while also maintaining spatial metadata and using spatial joins etc. is incredible.

Perhaps someone who has worked on that project could provide some insights.

[bkamins]

The project is going to be done during JSoC this year. And one of the reasons I am pressing to decide on metadata now is to have a clear guidance how this extra package should integrate with DataFrames.jl.