Can Small Language Models Learn and Adapt for Software Engineering Tasks?

Large Language Models (LLMs) are widely used for code generation and other software engineering tasks. However, their size and resource demands often make them impractical for smaller-scale, domain-specific applications. Small Language Models (SLMs) provide a lightweight alternative but require task-specific fine-tuning and evaluation. This project explores whether SLMs can effectively learn and adapt to software engineering tasks.

This thesis will make two contributions. First, it will fine-tune a small language model (e.g., distilgpt2) for a specific software engineering task such as recognizing and unlearning misconceptions about design patterns. Second, it will evaluate the performance of the fine-tuned model, comparing it to state-of-the-art LLMs for similar tasks.

The contributions of this thesis will be:

• Fine-tuning SLMs for a software engineering task (e.g., understanding and correcting misconceptions about design patterns).
• Evaluating the fine-tuned model using metrics like accuracy, perplexity, and robustness.
• Investigating the practicality of SLMs for domain-specific software engineering applications.

The success of this project will depend on a thorough literature review and an understanding of state-of-the-art techniques. Suggested initial steps include exploring relevant studies [1], fine-tuning methods, and domain-specific benchmarks. The final phase will involve performance evaluation and comparison with large-scale LLMs.

Available spots: 2

Pointers to literature

[1] Hou, Xinyi, et al. “Large language models for software engineering: A systematic literature review.” ACM Transactions on Software Engineering and Methodology 33.8 (2024): 1-79..
[2] Ahmed, Toufique, et al. “Automatic semantic augmentation of language model prompts (for code summarization). In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE).” IEEE Computer Society (2024): 1004-1004.