Hi everyone, I'm Marin Šoštarić, and

today I will present our paper on

benchmarking table extraction. Tables in

PDF contain valuable data that we want

to extract and exploit later. These

tables are visually structured for

humans, but PDF has no machine-readable

structure, which makes table extraction

harder.

We define two subtasks. In TD, we locate

the table bounding box, then in TSR plus

TCR, we extract its structure and

content. Table extraction is the overall

end-to-end task. We ask ourselves, how

do we best evaluate a table extraction

model end-to-end? Prior work tackles the

two subtasks independently. In TD, the

input is a page image, and we use

classical retrieval metrics. In TSR, the

input is a perfectly cropped table.

The two evaluations are done in

isolation with no relationship between

them.

However, table detection feeds the table

structure recognition model, so errors

in the first stage propagate to the

second.

This is why we propose a new end-to-end

metric that accounts for the dependency

between the two tasks.

We benchmark 15 models on four datasets,

introduce end-to-end metrics.

We also study inference cost and model

calibration for subset of models. Two of

the datasets are new, one automatically

generated from our craft sources, one

manually annotated. They also contain

negative samples, which balances

precision to better reflect real-world

performance.

We benchmark 15 models ranging from

heuristic, computer vision, general, and

visual language models to commercial

tools.

Model performance depends strongly on

document layout.

The most expensive models are not

necessarily the best.

Thank you for your attention. Find more

details in the paper.