Skip to content

Research

I'm an engineer first, but the engineering is aimed at research questions: how transformers represent language, how to serve them efficiently, and how to ground their outputs in verifiable sources.

Currently exploring

Transformer Architectures

Attention variants, positional encodings, normalization schemes, and the design space of encoder and decoder stacks.

Efficient Inference

KV caching, speculative decoding, quantization, and the systems work that makes large models cheap to serve.

Retrieval & Grounding

Dense retrieval, retrieval-augmented generation, and grounding generative models in verifiable sources.

Representation Learning

What embeddings encode, how pretraining objectives shape them, and how to evaluate them honestly.

Generative AI

Decoder-only language modeling, sampling strategies, and evaluation of open-ended generation.

NLP

Tokenization, pretraining pipelines, and task-specific finetuning across the modern NLP stack.

Current reading

  • Attention Is All You Need

    Vaswani et al., 2017 — the reference point for everything in Polaris and Zenith.

  • RoFormer: Enhanced Transformer with Rotary Position Embedding

    Su et al., 2021 — basis for Zenith's positional encoding.

  • Fast Inference from Transformers via Speculative Decoding

    Leviathan et al., 2023 — basis for Zenith's greedy-exact speculative decoding.

  • Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Lewis et al., 2020 — the lineage Meridian's grounded pipeline builds on.

  • Training Compute-Optimal Large Language Models

    Hoffmann et al., 2022 — scaling laws and how to budget compute honestly.

Research roadmap

  1. Now

    Close Meridian's measured domain-transfer gap: SciNLI adaptation for the NLI verifier, whose 78.3% SNLI accuracy does not yet carry over to biomedical prose.

  2. Next

    Scale Meridian past laptop constraints — the ~200K-abstract PubMed corpus and the cited-answer generator trained at scale — plus QLoRA/FSDP in Zenith.

  3. Later

    Toward a broader grounded biomedical platform: richer sources beyond abstracts, stronger calibration, and human-agreement studies for the verifier.

Future publications

Nothing published yet — this section will grow as the research roadmap matures. Engineering write-ups live in the engineering notes in the meantime.

Future talks

None scheduled yet. If you'd like a talk or walkthrough on from-scratch NLP systems — tokenizers to serving — get in touch via the contact page.