Overview

Large utility companies such as Consolidated Edison Company of New York, Inc. (Con Edison) must process vast volumes of regulatory documents issued by external authorities. These documents may introduce new operational requirements or policy changes that affect specific departments within the company. Traditionally, each document is manually reviewed by experts to determine its applicability and to identify the relevant internal departments, resulting in significant operational overhead.

This paper introduces an automatic document classification pipeline designed to improve efficiency and accuracy in regulatory compliance management. The proposed system adopts a bi-level architecture: a binary classification module first determines whether a regulation is applicable to the company, and a multi-label classification module then identifies the internal departments affected by applicable regulations. The approach combines classical machine learning models with transformer-based natural language processing techniques and is evaluated on large-scale, real-world regulatory data provided by Con Edison.

Methodology

The proposed pipeline consists of two sequential modules:

  1. Binary Classification (Applicability Filtering)

    • Purpose: Identify whether a regulatory document is applicable to the company.
    • Models: An ensemble of Naive Bayes (NB), Support Vector Machine (SVM), Random Forest (RF), and a fully connected Artificial Neural Network (ANN).
    • Features: Bag-of-Words representations with frequency-based vocabulary filtering.
    • Strategy: Soft-voting ensemble to improve robustness and generalization across time-shifted data distributions.

  2. Multi-label Classification (Department Routing)

    • Purpose: Predict the set of internal departments affected by an applicable regulation.
    • Model: Transformer-based DocBERT, augmented with a lightweight ANN classifier head.
    • Loss: Binary cross-entropy to handle multi-label outputs.
    • Evaluation: Top-k accuracy and ranking-based metrics to reflect operational relevance.

The modular design allows the binary classifier to act as a coarse filter, reducing the computational burden and error propagation for downstream multi-label classification.

Dataset

The system is evaluated on proprietary regulatory datasets provided by Con Edison, consisting of:

  • Regulations: Long-form documents labeled as Applicable or Not Applicable.
  • Obligations: Shorter, extracted text segments linked to applicable regulations and annotated with one or more department labels.

Key characteristics include:

  • Tens of thousands of regulatory documents,
  • Highly imbalanced department labels,
  • Distribution shifts between training data and held-out test data reflecting real-world temporal changes.

For confidentiality reasons, the datasets are not publicly available.

Results

Experimental results demonstrate that:

  • The binary classification ensemble achieves over 90% accuracy, with strong generalization to held-out datasets.
  • The DocBERT-based multi-label classifier significantly outperforms ANN and LSTM baselines, achieving over 80% Top-3 accuracy.
  • The full pipeline effectively handles distribution shifts and scales to real-world regulatory workloads.

The ensemble strategy improves robustness compared to single-model baselines, while transformer-based representations capture long-range semantic dependencies essential for regulatory text.

Contributions

This work makes the following key contributions:

  1. A bi-level document classification pipeline tailored for regulatory compliance in large enterprises.
  2. An ensemble-based binary filtering strategy that improves robustness and reduces overfitting.
  3. A DocBERT-based multi-label classification approach for accurate departmental routing.
  4. A large-scale industrial evaluation on real regulatory data from a major utility company.
  5. Practical insights into handling class imbalance and distribution shifts in enterprise NLP systems.

Limitations and Future Work

The current system is limited by the maximum input length of transformer models, as DocBERT processes only the final 512 tokens of long documents. Future work includes incorporating document chunking, hierarchical embeddings, or text abstraction methods to better handle long-form regulations.