Overview

Contract management is a labor-intensive process that requires reviewing and negotiating complex legal provisions, often under limited supervision and with large volumes of unstructured legacy data. Manual review is costly, slow, and prone to inconsistency, especially when determining whether proposed contract revisions are acceptable and how they should be corrected.

This work introduces a modular Retrieval-Augmented Generation (RAG) framework designed to streamline industrial contract management. The system automatically flags potentially unacceptable contract revisions, retrieves relevant historical clauses and contextual provisions, and generates improved alternatives that are more likely to be accepted. The framework is developed and evaluated in collaboration with Consolidated Edison Company of New York (Con Edison) on real-world internal contracts, demonstrating strong performance despite limited labeled data.

Methodology

The proposed system consists of four core components integrated into a clause-level RAG pipeline:

  1. Synthetic Data Generation
    Due to limited labeled supervision, the system augments training data by generating synthetic acceptable and unacceptable contract revisions using locally deployed LLaMA 3 models. Generated samples are filtered using embedding-based nearest-neighbor consistency checks.

  2. Semantic Retrieval
    A similarity retriever identifies:

    • Historically similar contract revisions from a database, and
    • Contextually related clauses within the same contract.
      Retrieval is performed using large-context embedding models (Qwen3-Embedding-4B) and improved via cross-encoder reranking.

  3. Acceptability Classification
    The system flags problematic revisions using an embedding-based binary classifier trained on synthetic data. An ensemble strategy with clustering improves robustness across different provision types.

  4. Revision Optimization via RAG
    For flagged revisions, the generator rewrites clauses using retrieved precedents and context. Two modes are explored:

    • Zero-shot RAG inference, and
    • Acceptability-based alignment, where a frozen classifier provides reward signals and the generator is refined using PPO-style reinforcement learning.

The system is designed as human-in-the-loop, allowing legal experts to retain control over ambiguous or high-risk decisions.

Dataset

The evaluation uses a proprietary industrial dataset provided by Con Edison, including:

  • 24 manually curated fallback revisions (acceptable/unacceptable),
  • 20 negotiated contracts with tracked edits,
  • 430 labeled real revisions (287 acceptable, 143 unacceptable), and
  • ~27,000 filtered synthetic revisions.

Weak labeling heuristics treat edited clauses as unacceptable and unchanged deviations as acceptable.

Results

Key empirical findings include:

  • High retrieval accuracy using general-purpose embedding models, significantly outperforming legal-domain embeddings.
  • Robust acceptability classification, with embedding-based ensemble classifiers achieving strong F1 scores on both synthetic and real data.
  • Effective revision optimization, where acceptability-aligned RAG increases the fraction of acceptable generated revisions to over 80%, outperforming zero-shot baselines.
  • Reduced manual workload, as the system automates routine clause review while escalating ambiguous cases to experts.

Contributions

This work makes the following contributions:

  1. A modular, retrieval-augmented LLM framework for contract revision analysis and optimization.
  2. A synthetic-data-driven approach for operating under extreme label scarcity.
  3. An acceptability-based alignment strategy that replaces costly human feedback with learned rewards.
  4. A real-world industrial evaluation demonstrating practical gains in efficiency and consistency.
  5. A scalable, customizable design adaptable across contracts, vendors, and legal regimes.

Limitations and Future Work

The system currently struggles with:

  • Fine-grained numerical or temporal changes (e.g., budgets, deadlines), and
  • Vendor-specific acceptability differences.

Future work may incorporate vendor identifiers, negotiation history, and expert-provided reward signals to further personalize optimization.