Parser Refactoring: Separating Grammar from AST Building¶

Overview¶

This refactoring introduces the Builder pattern to separate parsing concerns from AST construction in the Onion compiler. This separation provides several benefits:

Testability: AST construction can be tested independently of parsing
Flexibility: Different AST builders can be used for different purposes
Maintainability: Grammar changes don't require AST construction changes and vice versa
Extensibility: New behaviors can be added without modifying the parser

Architecture¶

Before Refactoring¶

The original parser (JJOnionParser.jj) directly constructs AST nodes within the grammar rules:

AST.ClassDeclaration class_decl(int mset) : {
  // ... variable declarations ...
}{
  t1="class" t2=<ID> /* ... parsing ... */ {
    return new AST.ClassDeclaration(  // Direct AST construction
      p(t1), mset, t2.image, ty1, toList(ty2s), sec3, toList(sec2s)
    );
  }
}

After Refactoring¶

The refactored parser uses an ASTBuilder interface:

AST.ClassDeclaration class_decl(int mset) : {
  // ... variable declarations ...
}{
  t1="class" t2=<ID> /* ... parsing ... */ {
    return builder.createClassDeclaration(  // Delegated to builder
      p(t1), mset, t2.image, ty1, toList(ty2s), sec3, toList(sec2s)
    );
  }
}

Components¶

1. ASTBuilder Trait (`ASTBuilder.scala`)¶

Defines the interface for AST construction:

trait ASTBuilder {
  def createCompilationUnit(...): AST.CompilationUnit
  def createClassDeclaration(...): AST.ClassDeclaration
  def createMethodDeclaration(...): AST.MethodDeclaration
  // ... other AST node creation methods
}

2. DefaultASTBuilder (`ASTBuilder.scala`)¶

Provides the default implementation that simply constructs AST nodes:

class DefaultASTBuilder extends ASTBuilder {
  def createClassDeclaration(...) = {
    AST.ClassDeclaration(location, modifiers, name, ...)
  }
}

3. ASTBuilderAdapter (`ASTBuilderAdapter.java`)¶

Java adapter for seamless integration with JavaCC:

public class ASTBuilderAdapter {
  private final ASTBuilder builder;

  // Handles Java-Scala interop complexities
  public AST.ClassDeclaration createClassDeclaration(...) {
    return builder.createClassDeclaration(...);
  }
}

4. JJOnionParserRefactored (`JJOnionParserRefactored.jj`)¶

Modified JavaCC grammar that uses the builder pattern instead of direct AST construction.

Use Cases¶

1. Custom Analysis¶

class AnalyzingASTBuilder extends DefaultASTBuilder {
  var methodCount = 0

  override def createMethodDeclaration(...) = {
    methodCount += 1
    super.createMethodDeclaration(...)
  }
}

2. Validation¶

class ValidatingASTBuilder extends DefaultASTBuilder {
  override def createMethodDeclaration(...) = {
    if (args.length > 10) {
      throw new IllegalArgumentException("Too many parameters")
    }
    super.createMethodDeclaration(...)
  }
}

3. Transformation¶

class TransformingASTBuilder extends DefaultASTBuilder {
  override def createMethodDeclaration(...) = {
    val modifiedBody = addLogging(body)
    super.createMethodDeclaration(..., modifiedBody)
  }
}

4. Debugging¶

class LoggingASTBuilder extends DefaultASTBuilder {
  override def createClassDeclaration(...) = {
    println(s"Creating class: $name at $location")
    super.createClassDeclaration(...)
  }
}

Migration Strategy¶

Phase 1: Create builder infrastructure (completed)
ASTBuilder trait
DefaultASTBuilder implementation
ASTBuilderAdapter for Java interop
Phase 2: Refactor parser gradually
Start with simple constructs (literals, identifiers)
Move to complex constructs (classes, methods)
Maintain backward compatibility
Phase 3: Update existing code
Modify Parsing.scala to use new parser
Update tests to use refactored components
Phase 4: Leverage new capabilities
Add validation builders
Implement transformation builders
Create specialized builders for different compilation modes

Benefits Realized¶

Separation of Concerns: Grammar rules focus on syntax; builders focus on semantics
Testability: AST construction logic can be unit tested without parsing
Extensibility: New compilation features can be added via custom builders
Maintainability: Changes to AST structure don't require grammar modifications
Debugging: Logging/tracing can be added without touching the parser

Future Enhancements¶

Builder Composition: Chain multiple builders for complex transformations
Context-Aware Building: Builders that maintain compilation context
Error Recovery: Builders that can construct partial ASTs for better error messages
Optimization: Builders that perform early optimizations during parsing