How We Got to LLMs: From Word2Vec to the Transformer大型語言模型是怎麼來的：從 Word2Vec 到 Transformer

When ChatGPT appeared, it felt as though AI had suddenly learned to speak. It can feel like a new kind of technology, but LLMs are not new — and they did not begin with ChatGPT. There is no official point at which a language model becomes a large language model. No governing body sets a minimum parameter count, and no single paper marks its birth. If we need a starting point for the modern LLM, 2017 is the clearest answer. That was the year researchers introduced the Transformer — the architecture behind nearly every major LLM today.

BERT and GPT followed, demonstrating what happened when Transformers were pretrained on large amounts of text. GPT-2 was already being described as a “large language model” in 2019; GPT-3 then showed how dramatically new capabilities could emerge as models grew. ChatGPT brought those capabilities to the public—it did not create them (Transformer paper, 2019 usage of “large language model”, GPT-3 paper).

Transformer did not appear from nowhere. It was the result of a chain of breakthroughs stretching back years: word embeddings gave words mathematical meaning, recurrent networks introduced memory, attention helped models focus, and Transformers made the whole process scalable. So the story of LLMs does not begin with a chatbot—or even with something we would call an LLM today. It begins with a more fundamental problem:

How can a machine represent and understand language at all?

This article explores that question in two parts. First, The evolution of language models offers a conceptual walkthrough of four major eras—what each introduced, where it fell short, and how those limitations led to the next breakthrough. Then, Text summarization using four eras of models turns theory into a practical comparison: we give each generation the same summarization task and compare how the models—and their results—have changed.

The evolution of language models

Stage 1: Word Embeddings — Words Become Numbers (2013)

Before 2013, a word was just an index in a lookup table. "Car" and "automobile" were as unrelated to a model as "car" and "trombone"—two arbitrary IDs, no matter how close their meaning.

Word embedding (e.g., Word2Vec) changed the representation. The core idea is that similar things should be close together in vector space; dissimilar things should be far apart. Instead of an ID, embeddings turn every word into a dense vector representation that captures its semantic meaning. We can think of an embedding as a list of a few hundred numbers—learned by analyzing the contexts in which a word appears.

For the first time, "meaning" was something you could measure with a distance function. Distance in that vector space started to encode similarity in meaning: "king" lands much closer to "queen" than to "banana," because king and queen keep turning up in similar contexts and banana never does. The famous example:

Nobody told the model that "king" and "queen" differ by gender the same way "man" and "woman" do. It fell out of the geometry, purely from co-occurrence statistics.

Input: center word (one-hot, dim V)
         ↓
       [W: V×D]             ← input embedding matrix — this is what we keep!
         ↓
   hidden layer (dim D)     ← this word's embedding vector (typically 100-300)
         ↓
       [W′: D×V]            ← output embedding (context) matrix — typically discarded
         ↓
  softmax over vocabulary
         ↓
 predicted context word
 (repeated once per position in the window)

The Wall: The classical word embedding (e.g. Word2Vec) gives every word exactly one vector. "Bank" gets a single point in space, shared between "river bank" and "bank account" — the word means two different things, but the model only has one representation for it. Also, Word2Vec has no opinion on how to combine word vectors into a sentence; it understands words, not order.

Stage 2: Sequence Models — Language Gains Memory (2014–2016)

If word order matters — and "dog bites man" means something very different from "man bites dog" — a model needs to process words in sequence, not as a bag of vectors.

Recurrent Neural Networks (RNNs) do exactly that. They process a sequence by reading one token, updating an internal hidden state (memory), reading the next token, and updating again.

        x_1         x_2         x_3
         ↓           ↓           ↓
... → [RNN]   →  [RNN]   →  [RNN] → ...
       (h_1)       (h_2)       (h_3)

Because the representation of a word now depends on the hidden state accumulated from the words before it, context emerges. "Bank" after "river" and "bank" after "savings" can now result in different final representations, because the model has actually processed what preceded each one.

While elegant, vanilla RNNs suffer from a fatal flaw: it is not capable of handling a long text and it quickly "forgets" distant words. This make it unable to learn long-range dependencies.

Long Short-Term Memory (LSTM) refine this with gates that explicitly decide what to keep, update, or discard from memory — a more disciplined version of the same idea.

The wall: information has to travel through the sequence one step at a time. By the time a long sentence's tenth word arrives, whatever mattered from the first word has usually faded — long-range relationships are hard to preserve, even with an LSTM's gating. And because each step depends on the one before it, an RNN can't be unrolled and processed all at once: training time scales with sequence length and can't be efficiently parallelized across a GPU.

Stage 3: Attention and the Transformer — Every Token Can Look at Every Other Token (2017–2018)

Attention lets a model connect relevant words directly, no matter how far apart they sit in a sentence. A preliminary version of this idea was patched into RNNs back in 2014, allowing a decoder to look back across everything an encoder had seen instead of relying on a single compressed summary vector.

In 2017, a paper with a blunt title took that patch and made it the entire architecture. "Attention Is All You Need" processes every token in parallel, using attention alone to relate tokens to each other—no recurrence required.

RNN: A time-dependent chain
[Token 1] ──> [Token 2] ──> [Token 3] ──> [Token 4]
   ↓             ↓             ↓             ↓
(step 1)      (step 2)      (step 3)      (step 4) 


Transformer: A parallel grid
[Token 1]     [Token 2]     [Token 3]     [Token 4]
   │             │             │             │
   └─────────────┴──────┬──────┴─────────────┘
                        ↓ (Self-Attention)
   ┌─────────────┬──────┴──────┬─────────────┐
   ↓             ↓             ↓             ↓
(step 1)      (step 1)      (step 1)      (step 1)

Under the hood, "related to each other" is driven by a specific mechanism worth naming. Every token produces three learned vectors:

• a Query — what it is looking for,
• a Key — what it has to offer, and
• a Value — what it actually contributes.

Each token's query is compared against every other token's key. Those comparison scores are turned into weights, and the token's new representation becomes a weighted blend of every other token's value.

The part worth sitting with: every token can pull from every other token in the sequence, all at once. It doesn't just look at the one word directly before it, and it isn't limited by whatever an RNN's memory managed to carry forward.

The headline benefit isn't just accuracy, though—it is economics. Every position in the sequence can now be computed simultaneously. Matrix multiplications are exactly what GPUs are designed to do. That is what made training on internet-scale text affordable. The Transformer's real contribution was making scale possible, not making language understanding "solved."

A quick correction: The original 2017 Transformer is not encoder-only, and it isn't decoder-only either. It is an encoder-decoder, built specifically for machine translation. An encoder reads the source sentence; a decoder generates the target sentence, attending back to the encoder as it goes. What came next was researchers taking different halves of that same architecture to build the models we use today:

• Encoder-decoder — the original design. The encoder builds a rich representation of the input; the decoder generates an output while attending to it. This remains the natural fit for tasks with a clear input-to-output shape, like translation (e.g., T5, BART).
• Encoder-only — BERT (2018). Keeps just the encoder half, reading a sequence in both directions at once. It is pretrained by masking random words and predicting them, then fine-tuned per task. Built for understanding text rather than generating it, it dominated NLP benchmarks for two years and, within 18 months, was powering a large share of Google Search.
• Decoder-only — GPT (2018 onward). Keeps just the decoder half, reading strictly left-to-right to predict the next token. Built to generate, not just represent.

The wall: BERT's paradigm requires a labeled dataset and a fresh fine-tuning run for every new task—one model for sentiment analysis, another for entity extraction, another for Q&A. The decoder-only line pointed toward something more general (prompting a single model to do anything), but GPT-1 and GPT-2 were not yet robust enough to prove the idea had no ceiling. Going into 2020, that was still an open question.

(BERT's lineage didn't die, by the way — it lives on as the embedding models behind modern semantic search.)

Stage 4: The Modern LLM Recipe — Decoder-Only Transformers and the Scale Era (2020–Present)

While the public timeline of generative AI seems to start in late 2022, the actual architectural turning point happened in 2020. That was the year the decoder-only architecture won the argument that GPT-1 and GPT-2 couldn't settle—and it won it purely on scale, in a way nobody had fully predicted.

In May 2020, GPT-3 was released at 175 billion parameters (roughly 500x the size of BERT-Large). At that size, it could perform tasks it was never explicitly trained on, specified entirely through a few examples in the prompt. Nobody engineered that behavior in; it emerged. Parameter counts jumped, capability jumped with them in ways that weren't linearly predictable, and "large" stopped being just a decorative adjective. The field started saying LLM.

The result of this pure scaling was a base model with genuine capability but terrible manners—it could generate incredibly realistic text, but it had no real sense of being asked a question. It was a massive autocomplete engine. It took two years of engineering to solve that final piece. Instruction tuning and alignment (using Reinforcement Learning from Human Feedback) closed the gap, transforming the raw GPT-3 architecture into the highly aligned ChatGPT in late 2022.

Strip away the magic, and what was left circa 2020 was a relatively simple recipe—dense, decoder-only Transformers trained on raw internet scrapes. As dense scaling hit hardware and data limits, that recipe had to evolve. Today's state-of-the-art looks vastly different:

• From Dense to Sparse (MoE): We no longer activate every parameter for every word. Trillion-parameter models now use a Mixture of Experts, dynamically routing each token to specialized sub-networks to massively increase capacity without exploding inference costs.
• The Data Wall and Synthetic Data: We have largely exhausted the high-quality public internet. Frontier models are now trained on highly curated synthetic data—often generated and verified by other models—prioritizing dense reasoning, math, and code over raw web scrapes.
• Inference-Time Compute: Autoregressive prediction is no longer just a fast reflex. The modern edge involves training models to "think" (plan, search, and self-correct via hidden reasoning tokens) during inference, trading immediate response time for much deeper logical accuracy.
• Agentic Workflows and Tool Use: We have moved beyond passive chatbots. The LLM is now the central reasoning engine for autonomous agents that can plan multi-step workflows, write and execute their own code, query live databases, call external APIs, and interact with the physical and digital world.
• Native Multimodality: We stopped bolting vision and audio tools onto text models. Modern architectures tokenize text, audio, and video directly into the same latent space from day one.

We have moved from static lookup tables to dynamic sequences, to parallel attention grids, and finally to trillion-parameter reasoning engines. The cleanest way to see exactly how far we have come across this entire evolution is to put one model from every era above through the exact same task.

Text summarization task using four eras of models

The history becomes more concrete when every era faces the same job. For this comparison, I created 2,400 fictional product reviews, each paired with a one-sentence reference summary. The first 2,000 reviews form the training split, 200 are reserved for validation, and 200 form the test split. Every system below was evaluated on the same first 20 held-out test reviews.

This is deliberately a small architectural demonstration, not a leaderboard. The systems differ in size, pretraining, supervision, and whether they extract or generate text. The dataset is also template-generated rather than written by real customers. Those differences make the historical progression visible, but they prevent a clean claim that architecture alone caused any score change.

Here is the first test review:

The WorkNest Chair has been part of my routine for six weeks. It looked much like the photos and felt familiar within a day. Assembly is straightforward with clear instructions. The chair creaks and feels less sturdy than expected. The seat cushion flattens too quickly. The adjustment range is too limited. The drawbacks outweigh the few things it does well.

The reference summary is:

The WorkNest Chair has decent assembly, but weak comfort makes it difficult to recommend.

The task sounds simple: compress seven sentences into one short verdict. But doing that well requires several separate abilities—identifying important sentences, preserving sentiment, combining evidence, and writing something that is both concise and faithful.

1. Word2Vec: represent, average, and copy

Word2Vec is not a summarizer by itself. In this experiment, it provides the representation inside a classical extractive pipeline.

I trained 100-dimensional skip-gram embeddings on the 2,000 training reviews.

word2vec = Word2Vec(
    sentences=corpus,  # Tokenized documents. Each element should look like ["battery", "life", "is", "good"]
    vector_size=100,   # Number of values in each word embedding. Larger vectors can capture more relationships but require more data, memory, and computation.
    window=5,          # Maximum context distance around the target word.
    min_count=3,       # Minimum total frequency required for a word to enter the vocabulary.
    workers=max(1, (os.cpu_count() or 2) - 1), 
    sg=1,              # 1. skip-gram, 0 CBOW
    negative=10,       # Number of negative examples
    epochs=6, 
    seed=SEED,
)

Each sentence becomes the average of its word vectors; averaging all sentence vectors produces a document centroid. The pipeline ranks sentences by cosine similarity to that centroid, adds a small position bonus, and copies at most two high-ranking sentences in their original order.

def summarize_word2vec(document, word_budget=WORD_BUDGET):
    # 1. Break the review document into sentences.
    sentences = split_sentences(document)
    if not sentences:
        return ""

    # 2. Represent every sentence as one vector.
    vectors = [sentence_vector(sentence) for sentence in sentences]

    # 3. Average all sentence vectors to represent the whole document.
    centroid = np.mean(vectors, axis=0)

    # 4. Score sentences by similarity to the document, with a small preference for earlier sentences.
    bias = 0.08
    scores = [cosine(vector, centroid) + bias / (index + 1) for index, vector in enumerate(vectors)]

    # 5. Select the highest-scoring sentences.
    chosen, words = [], 0
    for index in np.argsort(scores)[::-1]:
        sentence_words = len(tokenize(sentences[index]))

        # Skip a sentence if it would exceed the desired length.
        if chosen and words + sentence_words > word_budget:
            continue
        chosen.append(int(index))
        words += sentence_words

        # Stop after reaching the budget or selecting two sentences.
        if words >= word_budget or len(chosen) == 2:
            break

    # 6. Restore the sentences to their original document order.
    return " ".join(sentences[index] for index in sorted(chosen))
    

For the WorkNest review, it returns:

It looked much like the photos and felt familiar within a day. The adjustment range is too limited.

The second sentence is useful, but the first is generic setup rather than the review's verdict. More importantly, the two claims remain disconnected. The pipeline cannot rewrite them as “limited adjustability makes the chair difficult to recommend,” because its output vocabulary consists of complete source sentences.

This is the consequence of both the representation and the decision rule. Averaging word vectors discards order, gives every occurrence of a word the same meaning, and reduces a sentence to one point. Centroid similarity then asks which sentence is most typical of the document—not necessarily which one a reader most needs to know.

Across the 20 reviews, this system achieved a ROUGE-L score of 0.174 and a source four-gram copy rate of 0.916. It was effectively instantaneous at 0.00038 seconds per review on the test machine. That combination—fast, faithful to the source wording, but shallow—is exactly what we should expect from static embeddings plus sentence ranking.

2. A hierarchical BiLSTM: learn which sentences matter

The second system replaces averaged static vectors with a recurrent model. A word-level bidirectional LSTM reads each sentence; a sentence-level bidirectional LSTM then reads the sequence of sentence representations. The result is an importance score for every sentence in the review.

class HierarchicalBiLSTM(nn.Module):
    def __init__(self, vocab_size, embedding_dim=96, hidden_dim=48):
        super().__init__()
        
        # Word embedding layer: (batch, sentences, words, embed_size)
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0) 
        
        # Word-level BiLSTM, this LSTM reads the words inside each sentence.
        # forward state:  48 values + backward state: 48 values => return 96 values (hidden_dim * 2)
        self.word_lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True, bidirectional=True) 

        # Sentence-level BiLSTM, the second BiLSTM reads the sentence representations in document order:
        self.sentence_lstm = nn.LSTM(hidden_dim * 2, hidden_dim, batch_first=True, bidirectional=True)
        self.dropout = nn.Dropout(0.2) # Dropout is disabled during evaluation after.

        # Sentence classifier converts its 96-dimensional contextual representation into one number.
        self.classifier = nn.Linear(hidden_dim * 2, 1)

    def forward(self, documents):
        """
        documents: 3-D tensor. (batch_size, max_sentences, max_words)
        """
        batch, sentence_count, word_count = documents.shape
        
        # Flatten documents into sentences. (16, 30, 36) -> (480, 36)
        flat = documents.reshape(batch * sentence_count, word_count)

        # Embed words and run the word-level LSTM: (480, 36)-> (480, 36, 96)
        word_states, hidden_cell = self.word_lstm(self.embedding(flat))

        # Identify real words. unsqueeze(-1) adds a feature dimension so the mask can be applied across all 96 hidden features.
        word_mask = flat.ne(0).unsqueeze(-1) # (480, 36) → (480, 36, 1)

        # Why fill padding with -1e4? If padding were filled with zero, it might incorrectly win when all real activations for a feature are negative:
        sentence_vectors = word_states.masked_fill(~word_mask, -1e4).max(dim=1).values
        has_words = flat.ne(0).any(dim=1, keepdim=True)
        sentence_vectors = torch.where(has_words, sentence_vectors, torch.zeros_like(sentence_vectors))

        # Reconstruct the document hierarchy: (480, 96) -> (16, 30, 96)
        sentence_vectors = sentence_vectors.reshape(batch, sentence_count, -1)

        # Run the sentence-level BiLSTM
        document_states, _ = self.sentence_lstm(sentence_vectors)

        # Apply dropout and classification: (16, 30, 96)-> (16, 30, 1) -> (16, 30)
        return self.classifier(self.dropout(document_states)).squeeze(-1)

The training data contains reference summaries but no human labels saying which source sentences are important. To create supervision, each training sentence is compared with its review's reference using ROUGE-L. The best-matching sentence becomes a positive ROUGE-derived pseudo-label. At test time, the model sees only the review and predicts sentence importance on its own.

Its summary for the same review is:

The WorkNest Chair has been part of my routine for six weeks. The adjustment range is too limited.

This is still extractive, so it cannot synthesize a new sentence. But it now preserves the product name and selects a concrete weakness. Across the test set, ROUGE-L rises to 0.220, the best result among the three systems before the instruction-tuned LLM.

The improvement comes with two qualifications. First, the model learns the biases of its pseudo-labeling rule: “best lexical match to the reference” is not identical to “most important sentence.” Second, the output still copies the source. Its four-gram copy rate remains 0.836. Recurrence improves the representation of order and context, but it does not by itself make a system abstractive; the output objective still asks this model to select sentences.

The recurrent architecture also retains the limitation described in part one. Word states must be computed in sequence, and information reaches distant positions through repeated updates. The hierarchy shortens that path by modeling words within sentences and then sentences within documents, but it does not remove the sequential bottleneck.

3. An encoder–decoder Transformer: attention plus generation

The first two systems use extractive summarization: they select and copy sentences from the source. Stage 3 moves to abstractive summarization, which generates a new summary and can paraphrase or combine information.

This stage uses DistilBART-CNN, an encoder–decoder Transformer fine-tuned for news summarization. The encoder reads the full review, while the decoder uses cross-attention to consult the source as it generates each output token.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
MODEL_NAME = 'sshleifer/distilbart-cnn-12-6'
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_NAME).to(device).eval()

...

def summarize_transformer(document):
    inputs = tokenizer(
        document, 
        return_tensors='pt', # returns PyTorch tensors. 
        truncation=True, 
        max_length=MAX_INPUT_TOKENS
    ).to(device)
    with torch.inference_mode(): # Disable training-related calculations
        generated = model.generate(
            **inputs, num_beams=4, do_sample=False, min_new_tokens=12, max_new_tokens=80,
            length_penalty=1.0, no_repeat_ngram_size=3, early_stopping=True,
        )
    return tokenizer.decode(generated[0], skip_special_tokens=True).strip()

For the WorkNest review, it produces:

Assembly is straightforward with clear instructions. The drawbacks outweigh the few things it does well. The chair creaks and feels less sturdy than expected.

Although DistilBART is an abstractive summarizer capable of generating new wording, it mostly copies and reorders the source sentences. Its 0.803 four-gram copy rate confirms this behavior across the test set.

Its ROUGE-L score is 0.172, slightly below Word2Vec and the BiLSTM. DistilBART also produces the longest summaries, adding words that do not overlap with the concise references and lowering its ROUGE score.

What the stage does demonstrate cleanly is the removal of the fixed recurrent bottleneck. The encoder maintains a contextual representation for every source position, and each generated token can retrieve different evidence through cross-attention. The model no longer has to compress the entire review into one final recurrent state before it begins writing.

4. An instruction-tuned decoder-only LLM: the task moves into the prompt

The final system is Qwen2.5-1.5B-Instruct, a 1.54-billion-parameter decoder-only Transformer.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_NAME = 'Qwen/Qwen2.5-1.5B-Instruct'
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=dtype).to(device).eval()

...

def summarize_llm(document):
    messages = [
        {'role': 'system', 'content': SYSTEM_PROMPT},
        {'role': 'user', 'content': 'Summarize this user review:\n\n' + document},
    ]
    input_ids = tokenizer.apply_chat_template(
        messages, 
        tokenize=True, 
        add_generation_prompt=True, 
        return_tensors='pt',
        truncation=True, 
        max_length=MAX_INPUT_TOKENS,
    ).to(device)
    with torch.inference_mode():
        generated = model.generate(
            input_ids, 
            max_new_tokens=80, 
            do_sample=False, 
            repetition_penalty=1.05,
            pad_token_id=tokenizer.eos_token_id,
        )
    new_tokens = generated[0, input_ids.shape[-1]:]
    summary = tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
    return summary.removeprefix('Summary:').strip()

It receives no summarization training in this experiment. Instead, its chat prompt specifies the task:

You summarize product reviews. Return exactly one concise factual sentence that captures
the overall sentiment and the most important strength or weakness. Use only the review.

It returns:

The WorkNest Chair, while familiar and easy to assemble, falls short in comfort and adjustability, with issues like frequent creaking, rapid seat cushion flattening, and limited adjustment range, which outweigh its initial positive impression.

This is the first output that behaves like an editor rather than a sentence selector. It names the product, preserves the negative verdict, and combines several pieces of evidence into a grammatical sentence. Across all 20 reviews, it achieved the highest ROUGE-L score, 0.266, while its four-gram copy rate fell to 0.031. In other words, only about 3% of its four-word sequences were copied directly from the source.

The flexibility is not free. Average inference time rose to 0.570 seconds per review, excluding model loading. More importantly, generation introduces a failure mode that strict extraction cannot have: a fluent model can change a fact.

Two test outputs show the risk. One source says the DustPilot 3 “maps rooms and moves around furniture reliably”; Qwen summarizes that it “doesn't map rooms as accurately as some other models.” Another says the MetroPods connection “stays stable”; the generated summary introduces “occasional stability issues.” Both sentences sound plausible, and both reverse or invent evidence. The instruction “use only the review” reduces this risk but does not eliminate it.

Reading the results without turning them into a leaderboard

ROUGE-L measures overlap with a reference, not truth. Compression ratio measures length, not completeness. Copy rate measures reuse, not quality. Latency depends on hardware and decoding settings. The table is useful only when read together with the actual outputs.

Three patterns are more informative than the ranking:

1. Representation and output strategy are separate choices. The BiLSTM understands order but remains extractive because it was trained to score sentences. DistilBART can generate new wording but often copies because that is what its learned distribution favors.
2. Attention changes how evidence is available. The Transformer encoder keeps a contextual state for every source token, and cross-attention lets the decoder retrieve from those states while writing. This is the structural break from recurrence.
3. Instruction tuning changes the interface. In the first three systems, summarization behavior is encoded in ranking logic, pseudo-labels, or task-specific fine-tuning. In the fourth, a substantial part of the specification is expressed in ordinary language. The prompt becomes part of the program.

The progression is therefore not simply “each model scores higher.” It is a shift in where the capability lives. Word2Vec supplies reusable word geometry to an algorithm. The BiLSTM learns contextual sentence importance. The encoder–decoder Transformer learns to read and write with direct access to the source. The instruction-tuned LLM turns summarization into one behavior of a general model—more flexible and more fluent, but also more capable of being confidently wrong.