Taming LibreOffice’s Quadratic Table Layout in an RTF→PDF Pipeline

How we turned a conversion that couldn’t finish in 2 hours into a 4-minute job — with a 40-line preprocessing function, no new infrastructure, and no loss of fidelity.

The symptom

Our document pipeline converts statistical reporting output — RTF listings produced by SAS — into PDF and HTML using headless LibreOffice. It worked fine for years, until one class of files started blowing through a two-hour task timeout:

• File A: 32 MB RTF, ~2,000 pages → converts in 80 seconds ✅
• File B: 31 MB RTF, ~580 pages → still running at 2 hours ❌

Same size. Same producer (SAS). Same converter. Fewer pages in the slow one. What gives?

The diagnosis: it’s not size, it’s shape

Dumping the RTF structure of both files told the story:

File A (fast):  thousands of small tables, one per page, ~50 rows each
File B (slow):  ONE table. 29,222 rows. Zero page breaks. One section.

LibreOffice Writer’s layout engine has a known performance wall: layout cost grows superlinearly (roughly quadratically) with the number of rows in a single table. This is tracked upstream as tdf#162047 — “Table layout is quadratic in complexity”, filed by a LibreOffice developer, still open, with a whole family of related bugs (tdf#84635, tdf#151718). The developer-recommended workaround in the bug discussion is exactly what we ended up building:

split the giant table into smaller tables, and layout “will be fast.”

We verified the quadratic behavior empirically by splitting the same 29k-row table at different sizes and timing the conversion:

Time scales roughly with rows-per-table — what you’d expect if per-table cost is quadratic (total ≈ (R/N) · N² = R·N). It’s not perfectly linear because each conversion also pays a fixed ~60–70 s base cost (LibreOffice startup, parsing 30 MB of RTF, rendering the non-table content) that dominates at small N.

Why the “obvious” fix makes things worse

Our first workaround was the classic one: split the RTF file into chunks, convert each chunk to PDF separately (fanned out to a conversion service), and merge the PDFs. It converts fine — and produces a subtly broken document:

1. Page numbers lie. SAS listings render “Page X of Y” from RTF PAGE/NUMPAGES fields. Each chunk evaluates its own fields, so chunk 2 starts again at “Page 1 of 160”. You end up painting corrected numbers on top of the PDF, while the wrong text stays underneath in the text layer, where every text extractor finds it.
2. Repeating column headers vanish. The header row that repeats on every page is a single row flagged \trhdr at the start of the table. Chunks 2…N are cut mid-table and never receive it — so 70% of your pages have no column headers.
3. You now own a merge step, chunk-boundary cleanup, and a second service.

The insight that fixes all of this: don’t split the document — split the table, inside one document, and convert once.

Anatomy of a SAS-style RTF listing

Three structural facts make the clean fix possible:

{\rtf1\ansi
  {\header ... {Page {\field PAGE} of {\field NUMPAGES}} ...}   ← page numbers live HERE
  \trowd\trhdr ... {column headers}\cell ... {\row}             ← ONE row marked \trhdr
  \trowd ... {data}\cell ... {\row}                             ← 29,221 more rows
  \trowd ... {data}\cell ... {\row}
  ...
}

1. “Page X of Y” is in the page header block, evaluated per page at render time. As long as we keep one document and one conversion, pagination stays native and correct — no stamping.
2. A table in RTF is just an unbroken run of \trowd ... {\row} blocks. There is no <table> element. Any non-table paragraph between two rows ends one table and starts another. That’s the entire splitting mechanism.
3. The repeating header is one \trhdr-flagged row at the top of the table. Copy that row block to the start of each new sub-table, and headers keep repeating everywhere.

The fix, end to end

Step 1 — find the rows and group them into tables

TROWD_RE = re.compile(r"\\trowd")
ROW_END_RE = re.compile(r"\{\\row\}")

def find_row_blocks(text):
    """(start, end) span of each table row: \trowd ... {\row}."""
    blocks, pos = [], 0
    while (m := TROWD_RE.search(text, pos)):
        e = ROW_END_RE.search(text, m.end())
        if not e:
            break
        blocks.append((m.start(), e.end()))
        pos = e.end()
    return blocks

def group_tables(text, blocks):
    """Consecutive rows separated by whitespace only = one table."""
    tables, current = [], []
    for block in blocks:
        if current and text[current[-1][1]:block[0]].strip():
            tables.append(current)      # real content between rows -> new table
            current = []
        current.append(block)
    if current:
        tables.append(current)
    return tables

The whitespace-only rule matters: it means a document with several genuinely separate tables is grouped correctly, and we only ever cut inside a single oversized run of rows.

Step 2 — split, preserving what the output format needs

For PDF, each split re-emits the \trhdr header row and forces a page break, so the duplicated header only ever renders at the top of a fresh page — exactly where headers appear on every other page. Without the page break, you get a header row stranded mid-page at every split point; with it, split pages are visually indistinguishable from natural ones.

# ends the current table; \page pushes the next sub-table (header first)
# to the top of a fresh page. The paragraph is 1pt tall — invisible.
SEPARATOR_PDF = "\n{\\pard\\plain\\fs2\\sl-20\\slmult0\\page\\par}\n"

def split_giant_tables(rtf: bytes, rows_per_table=2000, min_rows=3000,
                       reemit_headers=True) -> bytes:
    """reemit_headers=True for paginated output (PDF); False for HTML,
    which uses the marker-only separator defined in step 3 instead."""
    try:
        text = rtf.decode("latin-1")
        edits, marker = [], 0
        for table in group_tables(text, find_row_blocks(text)):
            if len(table) <= min_rows:        # small tables: never touched
                continue
            n_hdr = 0                          # leading \trhdr header row(s)
            while n_hdr < len(table) and "\\trhdr" in text[slice(*table[n_hdr])]:
                n_hdr += 1
            if reemit_headers:                 # PDF: repeat headers per sub-table
                if n_hdr == 0:
                    continue                   # can't preserve headers -> don't split
                header = text[table[0][0]:table[n_hdr - 1][1]]
                insert = SEPARATOR_PDF + header + "\n"
            data = table[n_hdr:]
            for i in range(rows_per_table, len(data), rows_per_table):
                if not reemit_headers:         # HTML: uniquely numbered marker
                    marker += 1
                    insert = SEPARATOR_HTML.format(n=marker)
                edits.append((data[i - 1][1], insert))

        if not edits:
            return rtf                         # byte-identical no-op
        pieces, prev = [], 0
        for pos, ins in edits:
            pieces += [text[prev:pos], ins]
            prev = pos
        pieces.append(text[prev:])
        return "".join(pieces).encode("latin-1")
    except Exception:
        return rtf                             # fail open: never break a conversion

Guardrails worth copying: a minimum-rows gate so normal documents pass through byte-identical, a no-header → no-split rule (better slow than wrong), and a top-level try/except that returns the original bytes on any surprise. The pathological files are currently failing; the worst acceptable outcome of the preprocessor is “no change.”

Step 3 — HTML needs the opposite trade

HTML has no pages, so header re-emission and page breaks are pointless — worse, duplicated header rows would pollute any text extracted from the HTML downstream. But there’s a subtler problem: splitting produces 15 <table> elements for what is logically one table, and downstream consumers (DOM parsers, data extractors) shouldn’t have to know about your performance workaround.

So for HTML we split with only an invisible separator — no header copies — and tag each cut with an RTF bookmark:

# LibreOffice exports the bookmark as <a name="pa_split_N"> in HTML
SEPARATOR_HTML = (
    "\n{{\\pard\\plain\\fs2\\sl-20\\slmult0"
    "{{\\*\\bkmkstart pa_split_{n}}}{{\\*\\bkmkend pa_split_{n}}}\\par}}\n"
)

This is a str.format template — the doubled {{ }} are escaped literal RTF braces, and {n} is filled per cut with SEPARATOR_HTML.format(n=marker) (see step 2). The numbering must be unique document-wide because RTF bookmark names must be unique; duplicates get renamed or dropped by the converter, which would break the merge step.

…and after conversion, a single regex pass stitches the sub-tables back into one <table> by deleting each marked boundary:

BOUNDARY_RE = re.compile(
    rb"</table>\s*(?:</center>\s*)?"
    rb"<p\b[^>]*><a name=\"pa_split_\d+\"></a>\s*(?:<br/?>\s*)*</p>\s*"
    rb"(?:<center>\s*)?<table\b[^>]*>\s*(?:<col\b[^>]*/?>\s*)*",
    re.IGNORECASE,
)

def merge_split_tables_in_html(html: bytes) -> bytes:
    n_markers = len(re.findall(rb"<a name=\"pa_split_\d+\"></a>", html))
    if n_markers == 0:
        return html
    merged, n_merged = BOUNDARY_RE.subn(b"", html)
    if n_merged != n_markers:      # anything unexpected -> refuse to half-merge
        return html
    return merged

The markers are the load-bearing detail: boundaries between genuinely different tables in the source carry no marker, so they can never be fused. Only the cuts we made are undone. A count check (markers found == boundaries merged) refuses to ship a half-merged document if the converter ever changes its output shape.

Step 4 — wire it into the converter

def convert_rtf(content: bytes, output_format: str) -> bytes:
    if output_format == "pdf":
        content = split_giant_tables(content)               # headers + page breaks
    elif output_format == "html":
        content = split_giant_tables(content, reemit_headers=False)  # markers only

    result = libreoffice_convert(content, output_format)     # one document, one pass

    if output_format == "html":
        result = merge_split_tables_in_html(result)          # one logical table again
    return result

Results

Same 31 MB, 29k-row file, same hardware:

How we validated “zero lines lost”: extract text from both PDFs, strip the page-number strings, and compare line multisets; then re-check any “missing” line with whitespace squashed, because long values wrap mid-word across lines in narrow columns and produce false positives. Trust the wrap-immune diff, not your eyes, on a 580-page document.

Gotchas we hit so you don’t have to

• Page counts are not a property of the RTF. Our file had no explicit page breaks and no page-count metadata — pagination is computed at layout time. The same document produced 573 pages on macOS and 585 in a Linux container with identical content, purely because font substitution fit 51 vs. 50 rows per page. Don’t write tests that assert absolute page counts across environments.
• Split-point visibility depends on arithmetic luck. If rows-per-split is an exact multiple of rows-per-page (2000 ÷ 50, in our container), every forced break lands precisely on a natural page boundary and the splits are literally invisible. When it isn’t a multiple, the page before each split just ends early — cosmetically similar to what keep-together rows already do.
• Old converters may ignore \trhdr entirely. LibreOffice didn’t honor the RTF repeat-header flag from v3.5 until 25.2.2 (tdf#118465). If your container ships a distro-packaged LibreOffice from 2022, repeating headers won’t work no matter what your RTF says. Ship a current build.
• Two-step routes inherit the problem. We tested RTF→DOCX→(fast PDF engine): the DOCX export burned 12+ minutes of CPU before we killed it. The cost is in LibreOffice ingesting the giant table, not in the PDF export — any pipeline that opens the unsplit document pays it.
• Cloud converter APIs have their own walls. The hosted document-conversion services we evaluated either enforce hard server-side timeouts (~45 s) or run table layout in an embedded JS engine with the same big-table pathology. A 600-page single-table document is a stress test most of them fail.

Takeaways

1. When a converter is slow, profile the document’s shape, not just its size. “One 29,000-row table” and “1,500 twenty-row tables” are wildly different workloads to a layout engine.
2. Fix the input, not the pipeline. Preprocessing the RTF (a plain-text format!) beat every alternative we researched — chunk-and-merge services, commercial SDKs, cloud APIs, format detours — on speed, fidelity, and operational complexity.
3. Respect the output contract. “Fast” wasn’t enough: page numbers had to stay native, headers had to repeat, and one logical table had to remain one table for whoever consumes the file next. The format-specific split/merge asymmetry (headers + page breaks for PDF, markers + re-merge for HTML) is what made the fix shippable rather than a demo.
4. Make performance workarounds fail open. Gate on a threshold, refuse to split what you can’t split safely, and return the input untouched on any surprise — a slow conversion is recoverable; a silently corrupted clinical document is not.