2.8 TOKENS

                                    

     Programming in C    

TOKENS IN C

Tokens are the basic lexical building blocks of a C program.
They are the smallest individual units of a program that are recognized and understood by the compiler.

Characters written in a source program are grouped into tokens according to the rules of the C language. The compiler performs lexical analysis, during which it checks whether the tokens form legal strings as per the syntax rules of C.

C language has five classes of tokens:

1. Identifiers

2. Keywords (Reserved words)

3. Operators

4. Separators

5. Constants

---

Example Showing Tokens

if (x < 5)
    x = x + 2;
else
    x = x + 10;

Tokens Generated

• Keywords : if, else

• Identifiers : x

• Constants : 5, 2, 10

• Operators : <, =, +

• Separators : ;, (, )

---

1 Identifier

An identifier is a sequence of characters created by the programmer to name or identify program elements such as:

• Variables

• Arrays

• Functions

• Labels

Rules for Naming Identifiers

1. The first character must be an alphabet or an underscore (_).

2. Subsequent characters may be alphabets, digits, or underscores.

3. Only the first 31 characters are significant.

4. Keywords cannot be used as identifiers.

5. Identifiers are case-sensitive.

Valid Identifiers

• employee_number

• monthly_pay

• interest_per_annum

• tool_4

Invalid Identifiers

• 230_item (starts with digit)

• #pulse_rate (special symbol)

• total~amount

• /profit margin

---

2 Keywords

Keywords are reserved words that have a special meaning to the compiler.
They cannot be redefined or used as variable names.

Important Characteristics

• Predefined by the C language

• Always written in lowercase

• Used for program structure and control

Keywords in C (C89 – 32 keywords)

Some important ones include:

int, float, char, if, else, while, for, do, return, switch, case, break, continue, struct, union, typedef, static, extern, sizeof, void

Additional Keywords

• C89 added: const, enum, signed, volatile

• C99 added: inline, restrict, _Bool, _Complex, _Imaginary

⚠️ Note: Some compilers provide extended keywords like near, far, asm, which are compiler-specific.

---

3 Constants

A constant is a fixed value written directly in the program.
Its value does not change during program execution.

Types of Constants in C

1. Integer Constants

2. Floating-point Constants

3. Character Constants

4. String Constants

3.1 Integer Constants

• Sequence of digits

• Can be decimal, octal, or hexadecimal

Type	Example
Decimal	92
Octal (starts with 0)	0134
Hexadecimal (0x / 0X)	0x5C

Suffixes:

• U / u → unsigned

• L / l → long

• LL → long long (C99)

Examples:

1984U, 1984L, 12345LL, 123456ULL

3.2 Floating-point Constants

• Have decimal point and optional exponent

• Default type: double

Examples:

0.06
3.14F
2.164E-3
3.141592654L

3.3 Character Constants

• Single character enclosed in single quotes

'A', 'b', '$'

• Stored as numeric ASCII values

Escape Sequences

Used for non-printable characters:

'\n'  // newline
'\t'  // tab
'\b'  // backspace
'\0'  // null character

Hex and Octal representation:

'\010'   // octal
'\xA'    // hexadecimal

⚠️ Portability Tip: Use escape characters like '\b' instead of numeric ASCII codes.

3.4 String Constants

• Sequence of characters enclosed in double quotes

"Welcome"

• Stored as array of characters

• Ends with a null character '\0'

Example:

"HELLO"  // occupies 6 bytes (H E L L O \0)

⚠️ Important Difference

• 'A' → single character

• "A" → string containing A and \0

---

4 Assignment

The assignment operator (=) assigns the value of an expression to a variable.

General Form

variable = expression;

Examples:

i = 6;
i = i + 1;

Important Points

• Left operand must be an lvalue

• Right operand can be any expression

• Assignment itself is an expression

Example:

y = (x = 2 * x);

---

5 Initialization

When a variable is declared without a value, it contains garbage data.

int i;   // tentative declaration

Initialization

Assigning a value at the time of declaration:

int i = 100;

Advantages

• Prevents undefined behavior

• Improves program reliability

---

Conclusion

Tokens form the foundation of C programming.
Understanding identifiers, keywords, constants, operators, separators, assignment, and initialization is essential for:

• Writing correct syntax

• Avoiding compiler errors

• Developing efficient and portable programs

---

📘 Reference

The content is prepared by referring to the standard textbook:
“Computer Fundamentals and Programming in C”
Pradip Dey & Manas Ghosh
Oxford University Press, Second Edition (2018)

---

Aivette-COI
Helping college students learn smartly, revise quickly, and approach exams with confidence.

With love and care 🤍
— Aivette