\documentclass[12pt]{article}

%
%Margin - 1 inch on all sides
%
\usepackage[letterpaper]{geometry}
\usepackage{times}
\geometry{top=1.0in, bottom=1.0in, left=1.0in, right=1.0in}
\usepackage[style=mla,backend=biber]{biblatex}
\usepackage{comment}

%
%Doublespacing
%
\usepackage{setspace}
\doublespacing

%
%Rotating tables (e.g. sideways when too long)
%
\usepackage{rotating}


%
%Fancy-header package to modify header/page numbering (insert last name)
%
\usepackage{fancyhdr}
\usepackage{float}
\pagestyle{fancy}
\lhead{} 
\chead{} 
\rhead{Ayyalasomayajula \thepage} 
\lfoot{} 
\cfoot{} 
\rfoot{} 
\renewcommand{\headrulewidth}{0pt} 
\renewcommand{\footrulewidth}{0pt} 
%To make sure we actually have header 0.5in away from top edge
%12pt is one-sixth of an inch. Subtract this from 0.5in to get headsep value
\setlength\headsep{0.333in}

%
%Works cited environment
%(to start, use \begin{workscited...}, each entry preceded by \bibent)
% - from Ryan Alcock's MLA style file
%
\newcommand{\bibent}{\noindent \hangindent 40pt}
\newenvironment{workscited}{\newpage \begin{center} Works Cited \end{center}}{\newpage }

\addbibresource{references.bib}
%
%Begin document
%
\def\refname{}

\begin{document}
\begin{flushleft}

%%%%First page name, class, etc
Krishna Ayyalasomayajula\\
Ms. Horton\\
IB Extended Essay\\
February 28 2025\\


%%%%Title
\begin{center}
  \vspace{1em}
Rule-based Tensor Mutations Embedded within LLMs for Low-Cost Mathematical Comptuation
\end{center}


%%%%Changes paragraph indentation to 0.5in
\setlength{\parindent}{0.5in}
%%%%Begin body of paper here

{\raggedright \normalsize \textbf{Introduction}} 

Artificial Intelligence (AI) has surged in popularity and capability in recent years, leading to immense developments and investments into the technology. AI, since its modern inception, has been based on a Multi-Layer Perceptron (MLP), with the underlying structure being a Neural Network (NN) \parencite[1]{app14020744}. An MLP is a type of artificial neural network (ANN) that consists of a layering of neurons computed through the vector multiplication of weights and biases, resulting in propagation to a set of values that represents the states of the next layer. This structure, called a neural network is a computational model inspired by the human brain, borrowing from its chaining of neurons to create complex understanding from basic electrical impulses. The below expression details the computational process involved in evaluating progressive layers of neurons in a neural network. 

\[
\vec{y} = f\Bigl( \Bigl[ \sum_{j=1}^{n} w_{1j}x_j + b_1,\;\sum_{j=1}^{n} w_{2j}x_j + b_2,\;\ldots,\;\sum_{j=1}^{n} w_{mj}x_j + b_m \Bigr]^T \Bigr)
\]

The vector $\vec{y}$ is the output vector for any given preceding layer, resulting in an $m$ dimensional evaluation. Each connection between layers of an MLP has two values associated: a weight and a bias. Like the resistance across a synapse in the human brain, biases modulate the activation of each neuron. Mathematically, the biases for each connection are valued for each proceeding neuron, resulting in a $m\times 1$ shape. Weights are defined for each neuron, allowing the creation of a matrix $\mathbf{W}$, which is then computed against the activated values of the preceding layer in a matrix multiplication, then summed with the corresponding bias to generate a scalar for each neuron of the next layer. This process is called forward propagation, as compared to backpropagation. This entire propagation is composited inside a vector-valued activation function $\vec{f}$, which is essential in setting the bounds of the system. However, this can be problematic in complex situations. 

The end goal of any machine learning experiment is to function like a regression, except with many more parameters. Backpropogation is the process of updating these weights and biases using discrete evaluation to minimize a cost function $L$. Truly, MLPs are the largest optimization problem ever created. In other words, MLPs can be trained to create certain output for given input using the forward propagation to evaluation, a loss function to find error, and a backpropagation optimizer to update its variables. 

In recent years, a specialized kind of Machine Learning models have hit the market --- Large Language Models (LLMs). An LLM is a generative MLP that creates human-readable text given a prompt. They stem from early attempts in the 2000s to use neural networks with Recurrent Neural Networks to analyze sequences of words for sentiment, keywords, and grammar \parencite[2]{Wang2024}. With the rise of tokenizers, or models that convert words to vector embeddings that help encode the meaning thereof, the 2017 paper \textit{Attention is All You Need} changed the landscape of AI forever with the introduction of the self-attention transformer. This development allowed models to understand the relationships between words with far less training. While any task is possible --- theoretically --- with neural networks, optimizations such as these allow for lower error with far less training, making the while process more sustainable. The probability of reaching a minimum of the loss function is far greater with such improvements to MLP architecture \parencite[2]{vaswani2023attentionneed}. 

These techniques were later commercialized with the advent of GPT-2, GPT-3, and BERT from AI labs like OpenAI and Google's DeepMind \parencite[3]{Wang2024}. With increased supply of Graphical Processing Units (GPUs) and Tensor Processing Units (TPUs), these models began snowballing in scale. This was especially evident starting in 2019 with an iteration of GPT-2 being released with a production size of 1.5 billion parameters. In 2020, GPT-3 scaled up to 175 billion parameters --- achieving true coherence in reasoning for the first time ever for a machine. GPT-4 was released by OpenAI in 2023, with an undisclosed scale in the trillions of parameters. Development investment also climbed into the hundreds of billions of dollars, with new firms such as Anthropic, Grok, etc. Open sourced projects also gained popularity, some backed by multi-billion dollar R\&D teams such as Meta's Llama series. 

Functionally, there is no fundamental algorithmic difference between  generative and classification models. Indeed, most LLMs are initially trained to generate new sequences of words by setting the loss function to expect the next word in the series of an existing corpus, through a process known as Casual Language Modeling (CLM). For the purposes of commercialization, they have been re-purposed to be prompted as chat-bots by users. This is done by performing backpropagation based on the generation of conversational sequences, with the LLM often instructed to act as if filling out a conversation's transcript.

Several underlying technologies are involved in the lifecycle of an LLM. The process of creating one usually starts with the definition of a vocabulary. Sequences of language are broken into tokens by algorithms called tokenizers. Tokenizers split text into smaller units, which are then encoded into a vector by another MLP. This is done to develop a sense of meaning via the mathematical similarity of similar words. The similarity of two vectors can be calculated using the cosine-similarity formula, which calculates the angle $\phi$ between two vectors. 
\[
  \cos\phi=\frac{\vec{A}\cdot\vec{B}}{||\vec{A}||||\vec{B}||}
\]
Efforts to increase the performance of LLMs tend to include provisions for an increased vocabulary of cardinal tokens, leading to more efficient generation of text since more complex words, numbers, and symbols would normally need multiple tokens with the use of techniques like Byte Pair Encoding. 

%define LLM benchmarking

Benchmarks for evaluating Large Language Models (LLMs) assess their performance across various tasks, including reasoning, comprehension, generation, and factual accuracy. Standard benchmarks include GLUE and SuperGLUE for natural language understanding, MMLU (Massive Multitask Language Understanding) for evaluating knowledge across diverse subjects, and BIG-bench for measuring reasoning and generalization capabilities \parencite[8]{ivanov2024}. HELLASWAG and LAMBADA test commonsense reasoning and long-range dependency understanding, while TruthfulQA and BBQ assess biases, factual consistency, and ethical alignment \parencite[6]{ivanov2024}. Additionally, human evaluations and BLEU, ROUGE, and METEOR scores help measure text generation quality. As LLMs advance, new benchmarks continuously emerge to capture nuances in performance, efficiency, and ethical behavior.

Adding to the complexity of creating increasingly more performant are the computational and capital costs of building AI-capable supercomputers, clusters, and data centers for corpora, or CLM text databases. Improvements in model architecture are sought before attempts to increase the scale of models and their parameter counts because of the prohibitive scaling laws of neural networks. Experimentally, it has been found that increased parameter size has an exponential relationship with FLOPs of computational cost \parencite[2]{hoffmann2022trainingcomputeoptimallargelanguage}. This is seen in relation to the exponentially slowing gain in CLM accuracy with increased compute \parencite[5]{hoffmann2022trainingcomputeoptimallargelanguage}. This is taken to mean that there is a point at which scaling a model to gain accuracy is unsustainable. The Chinchilla scaling law is an experimentally conjectured hypothesis which states that an increase in model scale for a given architecture will tend to reducing model performance as the number of parameters tends to infinity. Although some teams claim to have statistically significant results to disprove it, these results have not been reaffirmed by third parties. 

{\raggedright \normalsize \textbf{Problem Statement}}
\begin{table}[H]
\centering
\caption{Comparison of LLM Sizes and Their Computational Requirements}
\label{tab:model-sizes}
\begin{tabular}{|l|r|r|r|r|}
\hline
\textbf{Model Name} & \textbf{Parameters} & \textbf{Training Compute} & \textbf{Inference Time} & \textbf{Memory Usage} \\
 & \textbf{(billions)} & \textbf{(PF-days)} & \textbf{(ms/token)} & \textbf{(GB)} \\
\hline
GPT-2 & 1.5 & 5.6 & 12 & 3 \\
\hline
GPT-3 & 175 & 3,640 & 75 & 350 \\
\hline
Llama-2-7B & 7 & 184 & 18 & 14 \\
\hline
Llama-2-13B & 13 & 368 & 32 & 26 \\
\hline
Llama-2-70B & 70 & 1,720 & 145 & 140 \\
\hline
Claude 2 & $\sim$100 & N/A & 82 & $\sim$200 \\
\hline
GPT-4 & $\sim$1,500 & $\sim$25,000 & 210 & $\sim$3,000 \\
\hline
\end{tabular}
\begin{flushleft}
\small{Note: Training compute is measured in petaflop-days. Inference time is measured for a single A100 GPU. Memory usage refers to the VRAM required during inference. Some values for proprietary models are estimated based on public information.}
\end{flushleft}
\end{table}


Despite the aforementioned significant advancements in LLMs and their increasingly large reasoning capabilities over time as seen in Figure \ref{tab:model-sizes}, the mathematical capabilities of models have increased in a sub-linear fashion. This also correlates with the decreasing performance as model sizes, even within the same generation, scale up. For example, Figure \ref{tab:model-sizes} displays the Llama-2 generation over 3 different sizes. Current LLMs approach mathematical operations through pattern recognition learned from their training data rather than through formal algorithmic processing, resulting in inconsistent performance when handling numerical calculations beyond simple arithmetic \parencite[3]{hendrycks2021measuringmathematicalproblemsolving}.

This research aims to investigate the potential integration of rule-based tensor mutations within an existing LLM architecture as a mechanism to enable low-cost, rule-driven (as opposed to pattern driven) mathematical computation. The intended outcome of the experiment detailed below include an increase in mathematical accuracy and a decrease in the Inference Time for prompts with necessary mathematical computation associated. 

\begin{quote}
  \textbf{RQ:} How can deterministic rule-based tensor mutations be embedded within LLM architectures to enable more accurate and efficient mathematical operations?
\end{quote}

%%%%Works cited
\newpage
\begin{center}
  \textbf{Works Cited}

\end{center}

\printbibliography[heading=none]
%\printbiblist
\end{flushleft}
\end{document}
\}
