c++ clang libtooling

Better C++ Syntax Highlighting - Part 1: Introduction

· Updated Jul 14, 2025

I created this blog to have a place to discuss interesting problems I encounter while working on my personal projects. Many of these projects, particularly those focused on computer graphics, are written in C++.

One problem I wanted to tackle was syntax highlighting, as I often use code snippets in my explanations and wanted them to be easily readable. Initially, I integrated PrismJS - a popular library for syntax highlighting in browsers - into my Markdown renderer. However, I quickly discovered that PrismJS struggles with properly highlighting C++ code.

Consider the following example, which showcases a variety of C++20 features I commonly use. Syntax highlighting is handled exclusively by PrismJS:

cpp
Show 261 more lines

Unfortunately, there are several issues with the syntax highlighting.

  • User-defined types: Only declarations of custom types are recognized as classes, with subsequent uses treated as plain tokens. Examples of are the Container concept on line 27 and the Vector3 struct on line 90. This also extends to type aliases, such as Color on line 283, and all standard library types.

  • Enums: As with user-defined types, only declarations of enums are highlighted properly. An example of this is the definition of the Month enum on line 61. Enum values, such as the month names in the Month enum definition, are also highlighted as plain tokens.

  • Class member variables: Class member declarations and references in member function bodies are highlighted as plain tokens. Examples of this can be seen with references to x, y, and z member variables throughout the definition of the Vector3 class.

  • Functions: Preprocessor definitions, constructors, and C++-style casts are incorrectly highlighted as function calls. Examples of this are the use of the ASSERT macro on line 197, the Color constructor on line 284, and uses of C++-style casts static_cast and const_cast on lines 83 and 226 and line 126, respectively.

  • Namespaces: Namespace declarations, as well as namespace qualifiers on types and functions, are highlighted as plain tokens. Examples of this are definitions of the utility namespace on line 17 or the math namespace on line 88, as well as the std qualifier on standard library types. This also extends to using namespace declarations and namespace aliases, such as using namespace std::chrono on line 218.

  • Templates: As with user-defined types, type names in template definitions, specializations, and instantiations are highlighted as plain tokens. This extends to C++20 concepts, such as the Container concept on line 27.

  • Operators: Certain characters are highlighted as operators. For example, lvalue and rvalue references are highlighted as the address-of operator, pointers as the multiplication operator, and template angle brackets as comparison operators.

The list goes on.

PrismJS breaks the source code into tokens based on a set of predefined grammar rules specific to each language. These rules are essentially regular expressions that identify different types of elements in the code, such as keywords, strings, numbers, comments, etc. Once the source code is parsed into tokens, each token is tagged with a set of CSS classes that are then used to apply styling.

Due to the complexity of C++ syntax, however, such an approach is not feasible. It is perfectly valid, for example, for a variable to have the same name as a class (given the class definition is properly scoped):

cpp
namespace detail {
struct MyClass {
...
};
}
int main() {
detail::MyClass MyClass { };
...
}

If we extend PrismJS to highlight all tokens that match class names, we may accidentally end up highlighting more than necessary.

While this example may be contrived, it sheds light on the fundamental issue of syntax highlighting with PrismJS: it is difficult to reason about the structure of the code by only looking at individual tokens. I believe this to be the fundamental reason for the issues pointed out above. Syntax highlighting for C++ simply requires more context. What if we want to extract member variables of a given class? How do we distinguish between local variables and class members? Approaches like using regular expressions or rule-based syntax highlighting quickly grow convoluted, posing a challenge from standpoints in both readability and long-term maintenance.

It makes sense, therefore, that PrismJS skips most this complexity and only annotates tokens it is confidently able to identify.

Abstract Syntax Trees

A more effective approach would be to parse the Abstract Syntax Tree (AST) and add custom annotations to tokens based on the exposed symbols. Abstract syntax trees are data structures that are widely used by compilers to represent the structure of the source code of a program. We can view the AST of a given file with Clang by running the following command:

text
clang -Xclang -ast-dump -fsyntax-only -std=c++20 example.cpp

To better understand the structure of an AST, let’s take a look at (a simplified version of) the one generated for the code snippet at the start of this post:

json
Show 84 more lines

The root of the AST is always the translation unit, which represents the entire compiled C++ file. Symbols from standard library headers are omitted for brevity - including all expands the AST to 561,445 lines, out of which only 1,758 (~0.3%) are relevant to the example. Note that only the top-level nodes are included, with comments indicating which elements these nodes reference in the code. Most child nodes, corresponding to function parameters, call expressions, variable declarations, and other statements within the function body itself, have been omitted for clarity purposes. Don’t worry, we will revisit these later.

Clang’s AST nodes model a class hierarchy, with most nodes deriving from three fundamental types:

  1. Stmt nodes, which represent control flow statements such as conditionals (if, switch/case), loops (while, do/while, for, range-based for), jumps (return, break, continue, goto), try/catch statements for exception handling, coroutines, and inline assembly
  2. Decl nodes, which represent declarations of struct/class types, functions (including function templates and specializations), namespaces, concepts, module import/export statements, and
  3. Expr nodes, which represent expressions such as function calls, type casts, binary and unary operators, initializers, literals, conditional expressions, lambdas, array subscripts, and class member accesses

Each AST node provides the source location and extent of the element they reference, as well as additional details depending on the specific type of node being processed. For example, a FunctionDecl node, which represents a function declaration or definition, exposes various properties such as the function’s name, return type information, references to parameters (if any, each represented by a ParmVarDecl node), and (of course) whether the referenced function is a definition or declaration. Additionally, it allows for checking:

  • Function attributes, such as [[noreturn]] and [[nodiscard]]
  • Whether the function is explicitly marked as static, constexpr, consteval, virtual (including pure virtual), and/or inline
  • Whether the function is explicitly (or implicitly) defaulted or deleted
  • Function exception specification (throw(...)/nothrow, noexcept, etc.)
  • Language linkage, or whether the function is nested within a C++ extern "C" or extern "C++" linkage
  • Whether the function is variadic
  • Whether the function represents a C++ overloaded operator, or a template (and, if so, what kind)
  • Whether it is a class member function defined out-of-line
  • And more!

This example merely scratches the surface. At the time of writing this post, there are over 300 different node types - the highly detailed and verbose nature of the Clang AST allows for extensive introspection into the structure of a C++ program.

I wanted to leverage the Clang AST to address the limitations of PrismJS and build a more robust and comprehensive syntax highlighting solution. For a given code snippet, my program generates and traverses the AST to identify various nodes and adds inline annotations based on the type of node being parsed. Before being rendered by the Markdown frontend, these annotations are extracted out and used to apply styling, similar to how PrismJS works.

Clang’s LibTooling API

Now that we are familiar with the structure of an AST, how do we traverse the one generated by Clang? While the process is a bit convoluted, we can set this up using Clang’s LibTooling library.

Creating an ASTFrontendAction

Tools built with LibTooling interact with Clang and LLVM by running FrontendActions over code. One such interface, ASTFrontendAction, provides an easy way to traverse the AST of a given translation unit. During traversal, we can extract relevant information about the AST nodes we care about and use it to add annotations for syntax highlighting.

Let’s start by defining our ASTFrontendAction:

cpp
1
class SyntaxHighlighter final : public clang::ASTFrontendAction {
2
public:
3
std::unique_ptr<clang::ASTConsumer> CreateASTConsumer(clang::CompilerInstance& compiler,
4
clang::StringRef file) override;
5
// ...
6
};

Creating an ASTConsumer

The ASTFrontendAction interface requires implementing the CreateASTConsumer() function, which returns an ASTConsumer instance. As the name suggests, the ASTConsumer is responsible for consuming (processing) the AST.

Our ASTConsumer is defined as follows:

cpp
1
class Consumer final : public clang::ASTConsumer {
2
public:
3
Consumer();
4
~Consumer() override;
5
6
private:
7
void HandleTranslationUnit(clang::ASTContext& context) override;
8
};

The ASTConsumer interface provides multiple entry points for traversal, but for our use case only HandleTranslationUnit() is necessary. This function is called by the ASTFrontendAction with an ASTContext for the translation unit of the file being processed.

The ASTContext is essential for retrieving semantic information about the nodes of an AST. It provides access to type details, declaration contexts, and utility classes like SourceManager, which maps nodes back to their source locations (as AST nodes do not store this information directly). As we will see, this information is crucial for inserting syntax highlighting annotations in the correct locations.

We simply instantiate and return an instance of our ASTConsumer from the CreateASTConsumer() function of the ASTFrontendAction.

cpp
1
std::unique_ptr<clang::ASTConsumer> SyntaxHighlighter::CreateASTConsumer(clang::CompilerInstance& compiler, clang::StringRef file) {
2
clang::ASTContext& context = compiler.getASTContext();
3
return std::make_unique<Consumer>(&m_annotator, m_tokenizer);
4
}

Creating a RecursiveASTVisitor

The final missing piece is the RecursiveASTVisitor, which handles visiting individual AST nodes. It provides Visit{NodeType} visitor hooks for most AST node types. Here are a few examples of visitor function declarations for common AST nodes:

cpp
bool VisitNamespaceDecl(clang::NamespaceDecl* node); // For visiting namespaces
bool VisitFunctionDecl(clang::FunctionDecl* node); // For visiting functions
bool VisitCXXRecordDecl(clang::CXXRecordDecl* node); // For visiting C++ class, struct, and union types
// etc.

The main exception to this pattern are TypeLoc nodes, which are passed by value instead of by pointer. The return value determines whether traversal of the AST should continue. By default, the implementation simply returns true, making it perfectly safe to omit Visit function definitions of any node types we are not interested in processing.

Our RecursiveASTVisitor is defined as follows:

cpp
1
class Visitor final : public clang::RecursiveASTVisitor<Visitor> {
2
public:
3
explicit Visitor(clang::ASTContext* context);
4
~Visitor();
5
6
// Visitor definitions here...
7
8
private:
9
clang::ASTContext* m_context;
10
};

It takes in the ASTContext from the ASTConsumer for retrieving node source locations during traversal. We will explore concrete visitor function implementations in more detail later on.

The traversal of the AST is kicked off in HandleTranslationUnit() from our ASTConsumer. By calling TraverseDecl with the root TranslationUnitDecl node (obtained from the ASTContext), we can traverse the entire AST:

cpp
1
void Consumer::HandleTranslationUnit(clang::ASTContext& context) {
2
// Traverse all the nodes in the translation unit using the C++ API, starting from the root
3
Visitor visitor { &context };
4
visitor.TraverseDecl(context.getTranslationUnitDecl());
5
}

Configuring the traversal behavior (optional)

The RecursiveASTVisitor also provides functions to control the behavior of the traversal itself. For example, overriding shouldTraversePostOrder() to return true switches the traversal from the default preorder to postorder.

cpp
1 
class Visitor final : public clang::RecursiveASTVisitor<Visitor> {
2 
public:
3 
explicit Visitor(clang::ASTContext* context);
4 
~Visitor();
5 
6+
bool shouldTraversePostOrder() const {
7+
// Configure the visitor to perform a postorder traversal of the AST
8+
return true;
9+
}
10 
11 
private:
12 
clang::ASTContext* m_context;
13 
};

Other functions modify traversal behavior in different ways. For example, shouldVisitTemplateInstantiations() enables visiting template instantiations, while shouldVisitImplicitCode() allows traversal of implicit constructors and destructors generated by the compiler.

Putting it all together

Finally, we invoke the tool using runToolOnCodeWithArgs(), specifying the ASTFrontendAction, source code, and any additional command line arguments:

cpp
1
int main(int argc, char* argv[]) {
2
if (argc < 2) {
3
utils::logging::error("no input file provided");
4
return 1;
5
}
6
7
const char* filepath = argv[1];
8
9
// Read file contents
10
std::ifstream file(filepath);
11
if (!file.is_open()) {
12
utils::logging::error("failed to open file {}", filepath);
13
return 1;
14
}
15
16
std::string content { std::istreambuf_iterator<char>(file), std::istreambuf_iterator<char>() };
17
18
std::vector<std::string> compilation_flags {
19
"-std=c++20",
20
"-fsyntax-only", // Included by default
21
22
// Project include directories, additional compilation flags, etc.
23
// ...
24
};
25
26
// runToolOnCodeWithArgs returns 'true' if the tool was successfully executed
27
return !clang::tooling::runToolOnCodeWithArgs(std::make_unique<SyntaxHighlighter>(content), content, compilation_flags, filepath);
28
}

Inserting annotations

One of the other responsibilities of our ASTConsumer is adding syntax highlighting annotations to the source code. An annotation follows the structure: [[{AnnotationType},{Tokens}]], where AnnotationType determines the CSS class applied to one or more Tokens. The annotations are embedded directly into the source code and later extracted from code blocks by a custom Markdown renderer, which applies CSS styles to transform them into styled elements.

Annotations provide hints to the Markdown renderer on how to apply syntax highlighting to symbols PrismJS cannot accurately identify. For example, the following snippet demonstrates a few common C++ annotation types: namespace-name for namespaces, class-name for classes, and function for functions.

text
[[keyword,namespace]] [[namespace-name,math]] {
[[keyword,struct]] [[class-name,Vector3]] {
// ...
};
[[keyword,float]] [[function,dot]]([[keyword,const]] [[class-name,Vector3]]& a, [[keyword,const]] [[class-name,Vector3]]& b) {
// ...
}
}

These annotations exist only in the Markdown source. When processed by the website frontend, they are removed, and the enclosed tokens are assigned the corresponding CSS styles. For the purposes of syntax highlighting, these styles simply correspond to the color these elements should have:

css
.language-cpp .function {
color: rgb(255, 198, 109);
}
.language-cpp .keyword {
color: rgb(206, 136, 70);
}
.language-cpp .class-name,
.language-cpp .namespace-name {
color: rgb(181, 182, 227);
}

As we traverse the AST, we will define additional annotation types as needed. It follows that there should be a corresponding CSS style for every annotation type.

If you are interested, I’ve written a short post about how this is implemented in the renderer itself (the same one being used for this post!).

The Annotator

All logic for inserting annotations is handled by the Annotator class:

cpp
1
struct Annotation {
2
Annotation(const char* name, unsigned offset, unsigned length);
3
~Annotation();
4
5
const char* name;
6
unsigned offset;
7
unsigned length;
8
};
9
10
class Annotator {
11
public:
12
explicit Annotator(std::string source);
13
~Annotator();
14
15
void insert_annotation(const char* name, unsigned line, unsigned column, unsigned length, bool overwrite = false);
16
void annotate();
17
18
private:
19
void compute_line_lengths();
20
[[nodiscard]] std::size_t compute_offset(unsigned line, unsigned column) const;
21
22
std::string m_source;
23
std::vector<unsigned> m_line_lengths;
24
std::vector<Annotation> m_annotations;
25
};

Annotations are registered through insert_annotation() and stored in the m_annotations vector. Annotations typically correspond to unique tokens in the source file. The insert_annotation() function calculates the character offset in the file for the given source location using compute_offset() and appends the annotation.

cpp
1
void Annotator::insert_annotation(const char* name, unsigned line, unsigned column, unsigned length, bool overwrite) {
2
std::size_t offset = compute_offset(line, column);
3
4
// Do not add duplicate annotations of the same name at the same location
5
for (Annotation& annotation : m_annotations) {
6
if (annotation.offset == offset) {
7
if (overwrite) {
8
annotation.name = name;
9
annotation.length = length;
10
}
11
12
return;
13
}
14
else if (offset > annotation.offset && offset < annotation.offset + annotation.length) {
15
utils::logging::error("Inserting annotation in the middle of another annotation");
16
return;
17
}
18
}
19
20
m_annotations.emplace_back(name, offset, length);
21
}

The overwrite flag allows for existing annotations to be overwritten when necessary. Multiple annotations cannot correspond to the same token as it would create ambiguity regarding which CSS style should be applied. Partial overlaps - where an annotation would sit inside another - are rejected entirely to preserve correctness. This is a rare condition, and typically indicates a problem with the code that’s calling insert_annotation().

The compute_offset() function calculates the absolute character position for a given line and column:

cpp
1
std::size_t Annotator::compute_offset(unsigned line, unsigned column) const {
2
std::size_t offset = 0;
3
for (std::size_t i = 0; i < (line - 1); ++i) {
4
offset += m_line_lengths[i];
5
}
6
return offset + (column - 1);
7
}

To support this, the length of each line (including newline characters) is precomputed during initialization:

cpp
1
Annotator::Annotator(std::string source) : m_source(std::move(source)) {
2
compute_line_lengths();
3
}
4
5
void Annotator::compute_line_lengths() {
6
std::size_t start = 0;
7
8
// Traverse through the string and count lengths of lines separated by newlines
9
for (std::size_t i = 0; i < m_source.size(); ++i) {
10
if (m_source[i] == '\n') {
11
m_line_lengths.push_back(i - start + 1);
12
start = i + 1;
13
}
14
}
15
16
// Add any trailing characters (if the file does not end in a newline)
17
if (start < m_source.size()) {
18
m_line_lengths.push_back(m_source.size() - start);
19
}
20
}

Newlines and carriage returns are included in the line lengths to ensure offsets remain accurate across different platforms.

The annotate() function

After AST traversal is complete, a call to annotate() generates the final annotated file. The full source for this function can be viewed here.

The first step is to sort the annotations by their offsets:

cpp
std::sort(m_annotations.begin(), m_annotations.end(), [](const Annotation& a, const Annotation& b) -> bool {
return a.offset < b.offset;
});

Since nodes in the AST are not guaranteed to be visited in the same order as their counterparts appear in the code, annotations are typically added to the Annotator out of order.

To avoid expensive reallocations, the final length of the file (including all annotations) is precomputed and allocated at once:

cpp
std::size_t length = m_source.length();
for (const Annotation& annotation : m_annotations) {
// Annotation format: [[{AnnotationType},{Tokens}]]
// '[[' + {AnnotationType} + ',' + {Tokens} + ']]'
length += 2 + strlen(annotation.name) + 1 + annotation.length + 2;
}
std::string file;
file.reserve(length);

This takes advantage of the fact that annotations follow a consistent pattern when inserted into the file: [[{AnnotationType},{Tokens}]], and is efficient because it avoids reallocating the file as the contents grow in size.

Annotations are then inserted sequentially:

cpp
std::size_t position = 0;
for (const Annotation& annotation : m_annotations) {
// Copy the part before the annotation
file.append(m_source, position, annotation.offset - position);
// Insert annotation
file.append("[[");
file.append(annotation.name);
file.append(",");
file.append(m_source, annotation.offset, annotation.length);
file.append("]]");
// Move offset into 'src'
position = annotation.offset + annotation.length;
}
// Copy the remaining part of the line
file.append(m_source, position, m_source.length() - position);

The annotated file is then written out and saved to disk. The generated code snippet can now be embedded directly into a Markdown source file, where annotations will be processed by the renderer for syntax highlighting.

Tokenization

With the logic to insert annotations into the source code implemented, the next task is to implement a way to retrieve a subset of tokens that are contained within a SourceRange. Why? In some cases, determining the exact location of a symbol is not always straightforward. In general, while we usually know what symbol(s) we are looking for, the corresponding AST node does not always provide a direct way to retrieve their location(s). It does, however, include a way to retrieve the range of the node - spanning from a start to an end SourceLocation - which greatly helps us narrow down our search. By tokenizing the source file and storing tokens in a structured manner, we can efficiently retrieve those that fall within the given SourceRange of an AST node without having to traverse every token of the file. We can then check against the spelling of the token until we find one that matches that of the symbol we are looking for.

For example, a CallExpr node only provides the location of the corresponding function invocation. If we want to annotate any namespace qualifiers on the function call, such as the std in std::sort, one possible workaround is to tokenize the node’s SourceRange and compare the tokens against the namespace names extracted from the function’s declaration. We will explore this in greater detail when we look at visitor function implementations.

Tokenization is handled by the Tokenizer class.

cpp
1
struct Token {
2
Token(std::string spelling, unsigned line, unsigned column);
3
~Token();
4
5
std::string spelling;
6
unsigned line;
7
unsigned column;
8
};
9
10
class Tokenizer {
11
public:
12
explicit Tokenizer(clang::ASTContext* context);
13
~Tokenizer();
14
15
[[nodiscard]] std::span<const Token> get_tokens(clang::SourceLocation start, clang::SourceLocation end) const;
16
[[nodiscard]] std::span<const Token> get_tokens(clang::SourceRange range) const;
17
18
private:
19
void tokenize();
20
21
clang::ASTContext* m_context;
22
std::vector<Token> m_tokens;
23
};

We can leverage Clang’s Lexer class from the LibTooling API to handle tokenization. The Lexer provides an API to process an input text buffer into a sequence of tokens based on a set of predetermined C/C++ language rules. Raw tokens for the code snippet are stored contiguously in m_tokens, allowing the get_tokens() functions to return a non-owning std::span instead of copying token data into a separate buffer. This helps avoid unnecessary allocations and provides a significant boost to performance, as the get_tokens() functions are called frequently during the traversal of the AST,

Tokenization is handled by the tokenize function:

cpp
1
void Tokenizer::tokenize() {
2
const clang::SourceManager& source_manager = m_context->getSourceManager();
3
4
clang::FileID file = source_manager.getMainFileID();
5
clang::SourceLocation file_start = source_manager.getLocForStartOfFile(file);
6
clang::LangOptions options = m_context->getLangOpts();
7
8
clang::StringRef source = source_manager.getBufferData(file);
9
10
// Tokenize with raw lexer
11
clang::Lexer lexer { file_start, options, source.begin(), source.begin(), source.end() };
12
clang::Token token;
13
while (true) {
14
lexer.LexFromRawLexer(token);
15
if (token.is(clang::tok::eof)) {
16
break;
17
}
18
19
clang::SourceLocation location = token.getLocation();
20
std::string spelling = clang::Lexer::getSpelling(token, source_manager, options);
21
unsigned line = source_manager.getSpellingLineNumber(location);
22
unsigned column = source_manager.getSpellingColumnNumber(location);
23
24
m_tokens.emplace_back(spelling, line, column, keywords.contains(spelling));
25
}
26
}

The heavy lifting in this function is done by Lexer::LexFromRawLexer, which returns the next token from the input buffer. Tokens are converted into lightweight Token instances and stored in m_tokens.

Since lexing happens after preprocessing, any whitespace tokens and comments are already removed. If desired, this behavior can be modified before lexing occurs: the SetKeepWhitespaceMode and SetCommentRetentionState functions from the Lexer enable the tokenization of whitespace and comments, respectively. Other properties, such as the source file to process and C/C++ language options, are specified on initialization and cannot be changed later. To keep things simple, these are retrieved directly from the ASTContext, which is configured by the arguments passed to runToolOnCodeWithArgs.

Now that the source file has been tokenized, let’s turn our attention to get_tokens. There are two version of this function: one takes a start and end SourceLocation, while the other accepts a SourceRange. This is done purely for convenience: internally, the SourceRange overload forwards the call to the other version, passing the start and end locations extracted using getBegin() and getEnd(), respectively.

cpp
1
std::span<const Token> Tokenizer::get_tokens(clang::SourceRange range) const {
2
return get_tokens(range.getBegin(), range.getEnd());
3
}

A key consideration when retrieving tokens is that the provided range may span multiple lines. A good example of this is a FunctionDecl node, which represents a multi-line function definition.

cpp
1
std::span<const Token> Tokenizer::get_tokens(clang::SourceLocation start, clang::SourceLocation end) const {
2
const clang::SourceManager& source_manager = m_context->getSourceManager();
3
unsigned start_line = source_manager.getSpellingLineNumber(start);
4
unsigned start_column = source_manager.getSpellingColumnNumber(start);
5
6
// Determine tokens that fall within the range defined by [start:end]
7
// Partial tokens (if the range start location falls within the extent of a token) should also be included here
8
9
unsigned offset = m_tokens.size(); // Invalid offset
10
for (std::size_t i = 0; i < m_tokens.size(); ++i) {
11
const Token& token = m_tokens[i];
12
13
// Skip any tokens that end before the range start line:column
14
if (token.line < start_line || (token.line == start_line && (token.column + token.spelling.length()) <= start_column)) {
15
continue;
16
}
17
18
offset = i;
19
break;
20
}
21
22
unsigned count = 0;
23
unsigned end_line = source_manager.getSpellingLineNumber(end);
24
unsigned end_column = source_manager.getSpellingColumnNumber(end);
25
26
for (std::size_t i = offset; i < m_tokens.size(); ++i) {
27
const Token& token = m_tokens[i];
28
29
// Skip any tokens that start after the range end line:column
30
if (token.line > end_line || token.line == end_line && token.column > end_column) {
31
break;
32
}
33
34
++count;
35
}
36
37
// Return non-owning range of tokens
38
return { m_tokens.begin() + offset, count };
39
}

The main challenge of get_tokens() is properly accounting for partial tokens - those that overlap the range specified by start and end - in the result. The function begins by locating the first token that starts at or after start. It then iterates through the tokens until it encounters one that begins after end, keeping track of all the tokens in between (those within the range). The resulting std::span contains a view of all tokens that overlap the given range. If start or end does not align with a token boundary, any tokens that straddle the range - either starting before but extending past start, or starting before but continuing past end - are also included.

Finally, we’ll also extend the Tokenizer with iterator support:

cpp
[[nodiscard]] std::vector<Token>::const_iterator Tokenizer::begin() const;
[[nodiscard]] std::vector<Token>::const_iterator Tokenizer::end() const;
[[nodiscard]] std::vector<Token>::const_iterator Tokenizer::at(clang::SourceLocation location) const;

The at() function returns the nearest token at the given SourceLocation using the same logic as get_tokens(). The underlying implementation of these functions simply returns at iterator to the beginning, one past the end, or the token corresponding to the requested location, respectively.

The Annotator and Tokenizer are added as member variables of the ASTFrontendAction class. Not all annotations we are interested are handled by the ASTFrontendAction. In this case, we’ll need to ability to pass references to the Annotator and Tokenizer around so that annotations are inserted into the same resulting file.

We’ve completed the necessary setup to be able to traverse the AST. In the <LocalLink text={“next post”} to={“Better C++ Syntax Highlighting - Part 2: Enums”}>, we’ll walk through a few basic visitor implementations to get familiarized with the process of extracting data from and annotating AST nodes. Thanks for reading!

Series: Better C++ Syntax Highlighting - Part 1 of 10
Part 2: Enums→