IIT Madras researchers warn against bias in legal AI tools, propose ‘Legal Safety Score’

Researchers at the Indian Institute of Technology Madras (IIT Madras) have raised concerns about possible bias in Artificial Intelligence (AI) systems used in the legal field, warning that such tools could produce unfair or inconsistent outcomes if not properly monitored.

A study conducted by the Centre for Responsible AI (CeRAI) and the Wadhwani School of Data Science and AI (WSAI) at IIT Madras, along with Precog Labs at IIIT-Hyderabad, found that existing large language models (LLMs) can display social and cultural bias when applied to Indian legal problems.

To demonstrate this, researchers tested AI responses using similar case scenarios but with different social identifiers. For instance, in one experiment, the same robbery case was analysed twice once with a Hindu accused and once with a Christian accused. Surprisingly, the AI produced different judgements, exposing how stereotypes or hidden patterns in training data can influence legal reasoning.

To address such issues, the team has developed a new evaluation metric called the ‘Legal Safety Score’, which measures both fairness and accuracy in AI models. Unlike traditional benchmarks that only assess how correct an answer is, the Legal Safety Score also checks whether a model’s decisions remain consistent across religion, caste, gender, and other social factors.

The researchers found that by fine-tuning language models using India-specific legal datasets, they were able to reduce bias and improve reliability. This highlights the need for context-aware AI systems that reflect India’s legal diversity and social fabric.

The study—titled “InSaAF: Incorporating Safety through Accuracy and Fairness — Are LLMs Ready for the Indian Legal Domain?” emphasises that while AI can support legal professionals in research, drafting, and case analysis, it should never be treated as an unquestioned authority.

The researchers concluded that responsible use of AI, guided by fairness metrics and human oversight, is essential to ensure justice in an increasingly digital legal environment.