{"id":204,"date":"2014-04-28T18:00:26","date_gmt":"2014-04-28T18:00:26","guid":{"rendered":"http:\/\/neurecall.com\/blog\/?p=204"},"modified":"2025-06-03T02:03:18","modified_gmt":"2025-06-03T02:03:18","slug":"automated-scoring","status":"publish","type":"post","link":"https:\/\/www.syngli.com\/blog\/automated-scoring\/","title":{"rendered":"Automated scoring – Humans have become as bad as robots at grading essays"},"content":{"rendered":"\n

In an article<\/a> in The Chronicle of Higher Education<\/em>, author Steve Kolowich describes the satirical creation of the Babel (Basic Automatic B.S. Essay Language) Generator, by Les Perelman and his students at MIT. The software generates essays prompted only with keywords. The article gives a sample sentence generated as part of an essay based on the single keyword \u201cprivacy\u201d<\/p>\n\n\n\n

\n

Privateness has not been and undoubtedly never will be lauded, precarious, and decent\u2026 Humankind will always subjugate privateness.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Perelman’s intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d. Indeed, the \u201cprivacy\u201d essay scored 5.4\/6 according to the MY Access! online essay scoring software (also used as one of two readers by GMAT), with its \u201clanguage use and style\u201d and \u201cfocus and meaning\u201d scored as \u201cadvanced\u201d.<\/p>\n\n\n\n

\n
\"\"<\/a><\/figure>\n<\/figure>\n\n\n\n

Photo by Aneta Pawlik<\/a> on Unsplash<\/a>.<\/p>\n\n\n\n

The article does touch on the issue that the AES algorithms, since their introduction in 1973 (Ajay et al., 1973<\/a>), have now advanced to the point where they can provide ratings similar to those of human raters. It cites work by Shermis, who recently published a study that found reliably similar scores awarded by humans and automated algorithms among 22,029 essays by junior-high and highschool students (Shermis, 2014 Assessing Writing<\/em><\/a>). (Perelman published a critique<\/a> of an earlier version of the Shermis study in 2013 in The Journal of Writing Assessment<\/em>.)<\/p>\n\n\n\n

This phenomenon would seem to imply either that automated scoring algorithms have become very good at understanding language, or that human scorers of standardized tests treat these essays in much the same way as would a robot, e.g. by identifying keywords, complex sentences, and passage length. There\u2019s no doubt which interpretation is favored by Babel\u2019s creator. For assessments like the SAT writing test, writes Kolowich, \u201cMr. Perelman says the rubric is so rigid, and time so short, that they may as well be robots\u201d. The question of whether standardized exams can ever <\/em>feasibly assess high-level writing abilities remains the elephant in the room.<\/p>\n","protected":false},"excerpt":{"rendered":"In an article in The Chronicle of Higher Education, author Steve Kolowich describes the satirical creation of the Babel (Basic Automatic B.S. Essay Language) Generator, by Les Perelman and his students at MIT. The software generates essays prompted only with keywords. The article gives a sample sentence generated as part of an essay based on… View Article<\/a>","protected":false},"author":3,"featured_media":2011,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[115,11,114,21],"class_list":["post-204","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-graduate-management-admission-test-gmat","tag-language","tag-scholastic-aptitude-test-sat","tag-standardized-testing"],"yoast_head":"\nAutomated scoring - Humans have become as bad as robots at grading essays - Syngli - Blog<\/title>\n<meta name=\"description\" content=\"Perelman's intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.syngli.com\/blog\/automated-scoring\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Automated scoring - Humans have become as bad as robots at grading essays - Syngli - Blog\" \/>\n<meta property=\"og:description\" content=\"Perelman's intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.syngli.com\/blog\/automated-scoring\/\" \/>\n<meta property=\"og:site_name\" content=\"Syngli - Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-04-28T18:00:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-03T02:03:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"640\" \/>\n\t<meta property=\"og:image:height\" content=\"425\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Syngli\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Syngli\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/\",\"url\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/\",\"name\":\"Automated scoring - Humans have become as bad as robots at grading essays - Syngli - Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.syngli.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg\",\"datePublished\":\"2014-04-28T18:00:26+00:00\",\"dateModified\":\"2025-06-03T02:03:18+00:00\",\"author\":{\"@id\":\"https:\/\/www.syngli.com\/blog\/#\/schema\/person\/d4089eb970f09c370290004ed89a3b1e\"},\"description\":\"Perelman's intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.syngli.com\/blog\/automated-scoring\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/#primaryimage\",\"url\":\"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg\",\"contentUrl\":\"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg\",\"width\":640,\"height\":425},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.syngli.com\/blog\/automated-scoring\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.syngli.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Automated scoring – Humans have become as bad as robots at grading essays\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.syngli.com\/blog\/#website\",\"url\":\"https:\/\/www.syngli.com\/blog\/\",\"name\":\"Syngli - Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.syngli.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.syngli.com\/blog\/#\/schema\/person\/d4089eb970f09c370290004ed89a3b1e\",\"name\":\"Syngli\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.syngli.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/32a8fa5103b1eb131cfe25fa6c04b41997205891ac6fcb18cfd9a097320ed95e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/32a8fa5103b1eb131cfe25fa6c04b41997205891ac6fcb18cfd9a097320ed95e?s=96&d=mm&r=g\",\"caption\":\"Syngli\"},\"url\":\"https:\/\/www.syngli.com\/blog\/author\/overduin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Automated scoring - Humans have become as bad as robots at grading essays - Syngli - Blog","description":"Perelman's intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.syngli.com\/blog\/automated-scoring\/","og_locale":"en_US","og_type":"article","og_title":"Automated scoring - Humans have become as bad as robots at grading essays - Syngli - Blog","og_description":"Perelman's intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d.","og_url":"https:\/\/www.syngli.com\/blog\/automated-scoring\/","og_site_name":"Syngli - Blog","article_published_time":"2014-04-28T18:00:26+00:00","article_modified_time":"2025-06-03T02:03:18+00:00","og_image":[{"width":640,"height":425,"url":"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg","type":"image\/jpeg"}],"author":"Syngli","twitter_misc":{"Written by":"Syngli","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.syngli.com\/blog\/automated-scoring\/","url":"https:\/\/www.syngli.com\/blog\/automated-scoring\/","name":"Automated scoring - Humans have become as bad as robots at grading essays - Syngli - Blog","isPartOf":{"@id":"https:\/\/www.syngli.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.syngli.com\/blog\/automated-scoring\/#primaryimage"},"image":{"@id":"https:\/\/www.syngli.com\/blog\/automated-scoring\/#primaryimage"},"thumbnailUrl":"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg","datePublished":"2014-04-28T18:00:26+00:00","dateModified":"2025-06-03T02:03:18+00:00","author":{"@id":"https:\/\/www.syngli.com\/blog\/#\/schema\/person\/d4089eb970f09c370290004ed89a3b1e"},"description":"Perelman's intent is to expose the limitations of automated essay scoring (AES) algorithms, which he argues don\u2019t measure \u201cany of the real constructs that have to do with writing\u201d.","breadcrumb":{"@id":"https:\/\/www.syngli.com\/blog\/automated-scoring\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.syngli.com\/blog\/automated-scoring\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.syngli.com\/blog\/automated-scoring\/#primaryimage","url":"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg","contentUrl":"https:\/\/www.syngli.com\/blog\/wp-content\/uploads\/2014\/04\/aneta-pawlik-BBWmO2Xt-9s-unsplash.jpg","width":640,"height":425},{"@type":"BreadcrumbList","@id":"https:\/\/www.syngli.com\/blog\/automated-scoring\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.syngli.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Automated scoring – Humans have become as bad as robots at grading essays"}]},{"@type":"WebSite","@id":"https:\/\/www.syngli.com\/blog\/#website","url":"https:\/\/www.syngli.com\/blog\/","name":"Syngli - Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.syngli.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.syngli.com\/blog\/#\/schema\/person\/d4089eb970f09c370290004ed89a3b1e","name":"Syngli","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.syngli.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/32a8fa5103b1eb131cfe25fa6c04b41997205891ac6fcb18cfd9a097320ed95e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/32a8fa5103b1eb131cfe25fa6c04b41997205891ac6fcb18cfd9a097320ed95e?s=96&d=mm&r=g","caption":"Syngli"},"url":"https:\/\/www.syngli.com\/blog\/author\/overduin\/"}]}},"_links":{"self":[{"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/posts\/204","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/comments?post=204"}],"version-history":[{"count":4,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/posts\/204\/revisions"}],"predecessor-version":[{"id":2012,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/posts\/204\/revisions\/2012"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/media\/2011"}],"wp:attachment":[{"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/media?parent=204"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/categories?post=204"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.syngli.com\/blog\/wp-json\/wp\/v2\/tags?post=204"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}