Paper: SemChunk-C: Semantic Segmentation for C Code

Four people at Huawei use a lightweight LLM to segment C code for
analysis.


Abstract
Semantic segmentation of code written in a C-family language remains a
challenging problem, due to the language's complex syntax, macro
expansion, and irregular structural patterns. Existing chunking methods,
such as fixed-sized windows, heuristic splitting, and syntax-based tools,
often fail to capture meaningful functional units, limiting the efficacy
of retrieval and other downstream LLM driven tasks.
In this paper, we address the problem of chunking in C-related languages.
First, we define a set of code chunk categories. Second, we train an
LLM-based classifier to a) identify chunk boundaries, and b) assign each
chunk a descriptive functional attribute (a category), which can be useful
for downstream tasks. By leveraging the LLM's ability to capture semantic
context within the code, we assume flexible chunk boundaries, allowing to
adapt to the specific structure and context of each instance. Third, we
introduce SemChunk-C, a family of lightweight language models for semantic
chunking of C-related files (.c, .cpp, .h, .cs, etc.). These models are
based on the first four Ettin encoders [1] with 17M, 32M, 68M, and 150M
parameters. Despite their relatively small size, they are capable of
identifying cohesive code units, such as data structures, interface
blocks, and other components. Furthermore, we demonstrate the robustness
of our approach on real-world code, including challenging constructs such
as nested definitions and macros.
We test our approach on various datasets, and show that it achieves high
boundary accuracy and semantic coherence, matching or outperforming
chunkers that are based on much larger code-oriented LLMs. We also
validate the improved performance of the downstream tasks on a few curated
benchmarks.


https://arxiv.org/abs/2606.23697


Regards,
John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

Return to the comp.compilers page.
Search the comp.compilers archives again.