Choosing the correct activation function in NLP can be confusing because different NLP problems produce very different kinds of outputs.

A sentiment classifier, a spam detector, a named entity recognition model, and a large language model may all process text, but they do not necessarily use the same output activation.

The easiest way to remember the correct choice is to ask:

What exactly is the model predicting?

The core rule is:

One choice from many → Softmax
Independent multiple choices → Sigmoid
Yes/No prediction → Sigmoid
Continuous number → Linear

This guide covers the most common NLP prediction types, including text classification, text generation, next-word prediction, translation, summarization, NER, POS tagging, question answering, and Transformer-based language models.

---

1. Binary Text Classification → Sigmoid

Use sigmoid when the model makes a yes/no or two-class decision.

Examples include:

• Spam vs not spam
• Positive vs negative sentiment
• Sarcastic vs not sarcastic
• Toxic vs non-toxic
• Fake vs real
• Fraudulent vs legitimate

Typical output layer:

tf.keras.layers.Dense(
    1,
    activation="sigmoid"
)

A sigmoid output produces a value between 0 and 1.

For example:

0.08 → negative
0.92 → positive

Typical loss:

loss="binary_crossentropy"

Memory Tip

Two possible answers → one probability → Sigmoid

---

2. Multiclass Text Classification → Softmax

Use softmax when the model must choose exactly one class from several possible classes.

For example, sentiment could be:

negative
neutral
positive

The output layer would be:

tf.keras.layers.Dense(
    3,
    activation="softmax"
)

Example output:

Negative : 0.03
Neutral  : 0.10
Positive : 0.87

These probabilities sum to:

1.00

Typical loss:

loss="sparse_categorical_crossentropy"

or:

loss="categorical_crossentropy"

depending on label encoding.

Common NLP examples

• Sentiment classification
• Intent classification
• News category classification
• Language classification
• Topic classification where only one topic is allowed

Memory Tip

Many classes, choose ONE → Softmax

---

3. Multi-Label Text Classification → Sigmoid

Multiclass and multi-label classification are different.

Suppose an article can contain several topics:

Artificial Intelligence
AWS
Programming
Machine Learning
DevOps

One article could belong to:

AI = Yes
AWS = Yes
Programming = Yes
DevOps = No

Because each label is an independent decision, use:

tf.keras.layers.Dense(
    5,
    activation="sigmoid"
)

Example output:

AI              : 0.96
Machine Learning: 0.91
AWS             : 0.82
Programming     : 0.77
DevOps          : 0.18

Unlike softmax, these probabilities do not need to sum to 1.

Typical loss:

loss="binary_crossentropy"

Memory Tip

Many classes, choose MANY → Sigmoid

---

4. Next-Word Prediction → Softmax

This is one of the most important cases in NLP.

Suppose your vocabulary contains 10,000 words:

vocab_size = 10000

The output layer may be:

tf.keras.layers.Dense(
    vocab_size,
    activation="softmax"
)

The model produces a probability for every possible next word.

For example:

Input:

"I love machine"

Predictions:

learning   : 0.84
vision     : 0.07
software   : 0.03
banana     : 0.0001
...

The model then chooses or samples one word.

Why Softmax?

Because the model is asking:

Which ONE token from my vocabulary should come next?

Memory Tip

Next word = one winner from vocabulary → Softmax

---

5. Next-Character Prediction → Softmax

Character-level generation works exactly the same way.

Suppose the possible characters are:

a-z
A-Z
0-9
spaces
punctuation

If there are 80 characters:

tf.keras.layers.Dense(
    80,
    activation="softmax"
)

For input:

hel

the model might predict:

l : 0.85
p : 0.03
o : 0.02
...

and produce:

hell

Memory Tip

One next character → Softmax

---

6. Text Generation → Repeated Softmax Prediction

Text generation may seem more complicated, but the principle is simple.

A text-generation model repeatedly predicts the next token.

For example:

"I"
 ↓
"love"

"I love"
 ↓
"machine"

"I love machine"
 ↓
"learning"

At every generation step:

Current text
      ↓
Neural network
      ↓
Vocabulary scores
      ↓
Softmax
      ↓
Next token

Therefore:

Text generation is repeated next-token prediction.

A simple LSTM text-generation model may end with:

tf.keras.layers.Dense(
    vocab_size,
    activation="softmax"
)

---

7. LSTM Text Generation → Softmax

A basic LSTM language model might look like:

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(
        vocab_size,
        embedding_dim
    ),

    tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(150)
    ),

    tf.keras.layers.Dense(
        vocab_size,
        activation="softmax"
    )
])

The LSTM learns contextual information.

The final Dense layer asks:

Given this context,
which vocabulary token should come next?

Therefore:

Dense(vocab_size, activation="softmax")

is appropriate.

---

8. Modern Transformer Text Generation → Logits + Softmax

Modern Transformer models often do something slightly different.

Instead of:

Dense(
    vocab_size,
    activation="softmax"
)

they usually output raw values called logits:

Dense(vocab_size)

The model could produce:

Dhaka       14.8
London       3.1
Paris        2.4
Tokyo        1.7

Softmax can then transform these logits into probabilities.

For training, TensorFlow commonly uses:

loss = tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True
)

This means the loss function internally handles the softmax calculation.

Conceptually:

Transformer
     ↓
Vocabulary logits
     ↓
Softmax
     ↓
Token probabilities

Important

These two approaches represent essentially the same prediction problem:

Dense(vocab_size, activation="softmax")

versus:

Dense(vocab_size)

with:

SparseCategoricalCrossentropy(
    from_logits=True
)

The logits approach is common in modern deep learning frameworks.

---

9. How GPT, Qwen and DeepSeek Generate Text

Large language models follow the same fundamental idea.

Suppose a model has:

Vocabulary size = 150,000 tokens

and receives:

"The capital of Bangladesh is"

The network produces a score for every vocabulary token.

For example:

Dhaka       15.9
Chittagong   7.1
London       0.4
Paris       -1.2
...

After softmax:

Dhaka       → very high probability
Chittagong  → much lower probability
London      → very low probability

The decoding algorithm then selects the next token.

The model repeats this process:

Input tokens
    ↓
Transformer
    ↓
Vocabulary logits
    ↓
Softmax probabilities
    ↓
Select next token
    ↓
Append token
    ↓
Repeat

So even sophisticated LLM generation is fundamentally based on:

Repeated next-token prediction over a vocabulary.

---

10. Machine Translation → Softmax

Consider:

English:
"I love machine learning"

French:
"J'aime l'apprentissage automatique"

A sequence-to-sequence translation model generates the target sentence one token at a time.

At every decoder step:

Decoder hidden state
      ↓
Dense(target_vocab_size)
      ↓
Softmax
      ↓
Next French token

Typical output:

tf.keras.layers.Dense(
    target_vocab_size,
    activation="softmax"
)

Memory Tip

Translation = next-token generation in another language

Therefore:

Softmax

---

11. Text Summarization → Softmax

Summarization models also generate sequences.

Input:

Long article

Output:

Short summary

But internally, the summary is generated:

token 1
token 2
token 3
token 4
...

Each token is selected from the vocabulary.

Therefore:

Dense(
    vocab_size,
    activation="softmax"
)

or vocabulary logits followed by softmax.

Memory Tip

Summarization generates text → next-token prediction → Softmax

---

12. Generative Question Answering → Softmax

Suppose the prompt is:

What is the capital of Bangladesh?

A generative model produces:

Dhaka

Internally, it generates one or more tokens.

Therefore:

Question
   ↓
Language model
   ↓
Vocabulary distribution
   ↓
Softmax
   ↓
Generated answer tokens

Memory Tip

Generative QA → Generates tokens → Softmax

---

13. Named Entity Recognition → Softmax Per Token

Named Entity Recognition, or NER, is different from text generation.

Consider:

Barack Obama visited London.

The model might label:

Barack  → PERSON
Obama   → PERSON
visited → OTHER
London  → LOCATION

Each token must receive one class.

If there are nine entity labels:

tf.keras.layers.Dense(
    9,
    activation="softmax"
)

The output might have shape:

(batch_size, sequence_length, number_of_labels)

For example:

(32, 100, 9)

This means every token has its own 9-class probability distribution.

Memory Tip

One label for each token → Softmax per token

---

14. Part-of-Speech Tagging → Softmax Per Token

Consider:

The cat runs quickly

The model predicts:

The     → DET
cat     → NOUN
runs    → VERB
quickly → ADV

Each word gets exactly one grammatical label.

Therefore:

Dense(
    number_of_pos_tags,
    activation="softmax"
)

Memory Tip

One POS class for every token → Softmax

---

15. Sequence Labeling → Softmax

Tasks such as:

• Named Entity Recognition
• POS tagging
• BIO tagging
• Chunking
• Token classification

typically follow:

Tokens
   ↓
Embedding
   ↓
LSTM / BiLSTM / Transformer
   ↓
Dense(number_of_labels)
   ↓
Softmax for each token

The key distinction is:

Text generation:
Predict next TOKEN.

Token classification:
Predict LABEL for every token.

Both commonly involve softmax, but they solve different problems.

---

16. Extractive Question Answering → Position Prediction

Extractive QA does not generate an arbitrary answer.

Suppose:

Context:
"Bangladesh became independent in 1971."

Question:
"When did Bangladesh become independent?"

The model predicts:

Start position → 1971
End position   → 1971

The model therefore produces:

start-position logits
end-position logits

Softmax can be applied across sequence positions.

Conceptually:

Token positions
      ↓
Probability distribution
      ↓
Highest start position
+
Highest end position

Memory Tip

Extractive QA → predict positions, not vocabulary words

---

17. Semantic Similarity → Sigmoid or Cosine Similarity

Suppose:

Sentence A:
"How can I reset my password?"

Sentence B:
"I forgot my password. How do I change it?"

You might predict:

Similarity = 0.94

If your model directly predicts a score from 0 to 1:

Dense(
    1,
    activation="sigmoid"
)

However, embedding models frequently calculate:

Cosine similarity

between sentence embeddings instead.

So semantic similarity does not always require a Dense output activation.

---

18. Text Regression → Linear

Sometimes text is used to predict a continuous numerical value.

Examples:

• Review score
• Readability score
• Estimated price from product description
• Severity score
• Engagement score

Use:

tf.keras.layers.Dense(1)

or explicitly:

tf.keras.layers.Dense(
    1,
    activation="linear"
)

Memory Tip

Text in, number out → Linear

---

19. Star Rating Prediction Can Use Two Different Activations

Suppose you predict Amazon-style ratings:

1 star
2 stars
3 stars
4 stars
5 stars

You can formulate this as classification.

Use:

Dense(
    5,
    activation="softmax"
)

But if you want:

4.37 stars

as a continuous number, you are doing regression:

Dense(1)

This demonstrates an important principle:

Activation depends on how you formulate the target, not merely on the dataset.

---

20. Encoder-Decoder / Seq2Seq Models → Softmax

Classic sequence-to-sequence architecture:

Input sentence
     ↓
Encoder
     ↓
Context representation
     ↓
Decoder
     ↓
Vocabulary prediction
     ↓
Softmax

Common applications include:

• Translation
• Summarization
• Paraphrasing
• Dialogue generation
• Generative question answering

All eventually involve next-token prediction.

---

The Complete NLP Activation Cheat Sheet

NLP Problem	Output	Activation
Spam detection	Yes/No	Sigmoid
Sarcasm detection	Yes/No	Sigmoid
Binary sentiment	Positive/Negative	Sigmoid
Multiclass sentiment	One of N sentiments	Softmax
Intent classification	One intent	Softmax
Single-topic classification	One topic	Softmax
Multi-label topic tagging	Multiple topics	Sigmoid
Next-word prediction	One vocabulary token	Softmax
Next-character prediction	One character	Softmax
Text generation	Repeated next-token prediction	Softmax
LSTM language model	Next token	Softmax
Transformer language model	Vocabulary logits → Softmax	Softmax conceptually
Machine translation	Next target token	Softmax
Summarization	Next summary token	Softmax
Generative QA	Next answer token	Softmax
NER	One label per token	Softmax
POS tagging	One label per token	Softmax
Token classification	One label per token	Softmax
Extractive QA	Start/end position	Softmax over positions
Multi-label toxicity	Several labels	Sigmoid
Semantic similarity 0–1	Score	Sigmoid
Text regression	Continuous number	Linear

---

The Best Memory Formula

Instead of memorizing dozens of NLP tasks, memorize the output relationship:

YES / NO
   ↓
SIGMOID


ONE CLASS FROM MANY
   ↓
SOFTMAX


MANY INDEPENDENT LABELS
   ↓
SIGMOID


ONE NEXT TOKEN FROM VOCABULARY
   ↓
SOFTMAX


GENERATE A SENTENCE
   ↓
REPEAT NEXT-TOKEN SOFTMAX


ONE LABEL FOR EACH TOKEN
   ↓
SOFTMAX PER TOKEN


CONTINUOUS NUMBER
   ↓
LINEAR

An even shorter version is:

Sigmoid = independent yes/no decisions
Softmax = competitors where one wins
Linear = unrestricted numerical prediction

---

Final Rule for NLP Developers

When you are unsure which activation to use, do not start by asking:

“Is this NLP?”

Instead ask:

“What exactly should my output represent?”

If the answer is:

Is it spam?

use:

Dense(1, activation="sigmoid")

If the answer is:

Which sentiment class?

use:

Dense(num_classes, activation="softmax")

If the answer is:

Which topics apply?

use:

Dense(num_topics, activation="sigmoid")

If the answer is:

Which word comes next?

use:

Dense(vocab_size, activation="softmax")

If the answer is:

What number should I predict?

use:

Dense(1)

That one question—“What does my output represent?”—is often enough to choose the correct activation function.