Socrates or Smarty Pants? Can LLMs really understand logic?

Despite the growing focus on evaluating LLM reasoning, no existing benchmark specifically targets  logic traps—often humorous yet deceptive statements common in English-speaking contexts. Moreover, no systematic survey has been conducted to assess how well LLMs navigate such reasoning challenges.

To fill this gap, SocratesEval introduces a benchmark with clear logical error labels, evaluating LLMs on two key tasks: (1) determining whether a given statement contains a logical error, and (2) classifying the type of fallacy. We introduce structured scoring to assess fallacy identification and overall reasoning quality, providing a detailed evaluation of LLM performance.

Our results offer the first comparative analysis of LLMs’ ability to handle logic traps, shedding light on their reasoning limitations and potential.

False Positive (FP) occurs when a logically correct sentence is misclassified as a logical fallacy, while False Negative (FN) means a logical fallacy is mistakenly labeled as correct.

F1 Score evaluates the model’s ability to distinguish logical errors, balancing precision and recall.

Fallacy Labelling Score measures how accurately the model classifies sentences into human-annotated and verified logical fallacy categories.

Explanation Score assesses how well the model's interpretation of logical errors aligns with human reasoning.