Vol. 6, No. 3 Fall 2024
Brian Hotson, Editor, CWCA/RCCR
As I’ve written before here, writing support tools can be both helpful and harmful, just and unjust (See, Friends don’t let friends Studiosity (without reading the fine print) and Academic writing has completely changed: Turnitin forges ahead). Grammarly, a $13 billion USD company (2021), launched an AI detection tool in 2024, which comes bundled in with a Grammarly Premium subscription. Many universities and colleges offer Grammarly Premium without charge and promote the tool to students, faculty, and administrators. AI tools embedded into these writing support tools have changed the formulation of how these tools function. But, do they work as advertised?
Fooling the detector
Grammarly’s AI text detector tool, AI Detector by Grammarly, is described as “trained to identify AI-generated text.” Its interface is simple: cut-and-paste your text into the field provided, and the tool will tell you the percentage of that text that is generated by AI. To test AI Detector by Grammarly, I took the first paragraph of this article, which I wrote unaided, and checked it for AI-generated text.
The tool’s response, as indicated above, was that “0% of this text appears to be AI-generated.” Good news for me.
I then took the same paragraph, and I asked ChatGPT 4o to rewrite it using this prompt: “Rewrite the following passage, making it more academic, but keeping my voice and syntax.”
I then took the paragraph ChatGPT wrote, and entered it back into AI Detector by Grammarly. My instruction to write-like-me fooled the detector: the tool provided the same rating as my original paragraph, “0% of this text appears to be AI-generated” (see below).
I repeated this four times using the same ChatGPT generated text, to see if the detector had learned anything. The response was the same for each attempt: “0% of this text appears to be AI-generated.”
I then tried the process again. Using the text ChatGPT wrote, and not my original, I prompted ChatGPT to, “Rewrite this again, making it more academic sounding, but still sound like me.” This time, the tool response was “50% of this text appears to be AI-generated.”
Unfortunately for students, Grammarly promotes its detector as “an AI checker from a brand you already trust” and “a new standard” for AI detection: “Be sure that your work is one of a kind: Grammarly helps you check your content for potential plagiarism and AI use, so you can rest assured that what you’re submitting is your original work.” The failure of the tool in this case challenges this claim. The ramifications can be significant.
Next, I did a google search wit this prompt, “Is grammarly’s AI detector tool always right?” The first response was a Grammarly FAQ page, where they have published caveats, caveats that seem to hedge their landing page’s description of the tool’s effectiveness:
“What does the % score mean?
The score represents the % of scanned text that is likely AI-generated, based on our extensive machine-learning training model… like other AI text checkers, it cannot provide a definitive conclusion.”
“How accurate is the score?
The AI detection score is an averaged estimate of the amount of AI-generated text that is likely contained in a given document or piece of writing… Like all AI detectors, Grammarly’s AI detection is not 100% accurate and should not be used as a definitive assessment of whether AI-generated text is present.”
“If my professor uses a different AI detector to scan student writing, will Grammarly’s % score be the same?
Grammarly’s AI detection uses a proprietary in-house model, so the scores may differ from those of other solutions like Turnitin, GPTZero, Copyleaks, and others. You should not view certain score with Grammarly’s AI detection as a clear indication that your professor will see the same score when they run it through their preferred AI detector. However, Grammarly’s model is trained similarly to some of the other most accurate and used detectors, so we believe the scores should be directionally aligned.”
Which universities and colleges use Grammarly?
In a search of public-facing material from Canadian university and college websites, there are significant differences in the use, application, and regulation of Grammarly. What is being said to faculty and students on their public-facing website? Here are three examples.
USask
The University of Saskatchewan, for example, does not approve the use of Grammarly (See Grammarly is not approved (USask)). The reasons outlined are clear―student intellectual property is compromised. USask warns faculty: “Uploading your students’ work into Grammarly for plagiarism detection is not recommended. Grammarly may maintain a copy of a student’s intellectual property and use it for its own profit without benefit to, or permission from, a student.”
USask cites provincial rules regarding storage of personal data and information as well as intellectual property on foreign servers, servers that do not conform to provincial or federal regulations, such as provincial Freedom of Information and Protection of Privacy Acts. As USask warns,
Student identifiers, like name or email address, should not be held with student work on non-Canadian/European servers. When stored outside of Canada or Europe they may be searchable using the US’s Patriot Act (1), for example. Some ideas conveyed in student work are potentially objectionable to foreign governments.
Students cannot be compelled, for example with grades, to be required to use non-approved services for academic reasons. However, students can choose to use a service for their own research purposes. (Grammarly is not approved, 2024)
All Grammarly subprocessors, which store and process uploaded student intellectual property, to “operate, provide, improve, integrate, customize, and support our services” are housed in the U.S. (See, Does Grammarly use subprocessors?).
UPEI
For University of Prince Edward Island, a smaller university than USask, I dove deep into their policy making processes and reporting. To get an idea of its position and regulation on GenAI tool usage, intellectual property, and AI detection tools, specifically Grammarly, I reviewed UPEI’s Senate Committee minutes from November 2022, when ChatGPT was launched, to September 2024. There are two mentions of GenAI tools. The first, in 2023, is in regards to updating the undergrad academic integrity policies to restrict the use of GenAI tools, where senators also entered into the minutes, “[c]oncerns… around regulations discouraging the potential beneficial use of generative artificial intelligence (AI) in courses” (UPEI Senate Minutes – October 20, 2023). The other mention is from September 13, 2024, where a motion was passed “To update ‘Academic Regulation 20, Academic Integrity’, to prohibit the use of generative AI (genAI) detection tools to substantiate a claim of academic dishonesty.” This regulation does not appear on UPEI’s public facing website that I could find.
At the same time, a search for information on Grammaly on UPEI’s website reveals what seems to be contradictory information to the senate regulation. A UPEI’s Robertson Library has a page dedicated to registering, setting up, and using a Grammarly plagiarism detection tool,
We [UPEI] have an institutional subscription to Grammarly… Grammarly…provides a basic plagiarism tool that checks the open web for plagiarized text. Users can upload documents, turn on the plagiarism detection option and get a report on any text that may have been plagiarized. Remember that this tool will only report on open web content that it has in its index (Tools for Plagiarism Detection…, n.d.; link inserted in the original).
The library’s Grammarly login/signup page was last updated in 2018 (Grammarly, 2018).
UPEI is undertaking internal training for faculty. A two-hour GenAI PechaKucha symposium, Working with artificial intelligence: Two years of ethical AI for learning teaching and research [sic], is planned for later this month (link to the program).
Niagara College
UPEI’s uneven approach to GenAI and Grammarly is not uncommon for higher education institutions. A search of Niagara College’s public-facing website returned zero results for a search for Grammarly. In a google search of “grammarly” and “niagara college” returned a site for their Toronto campus, NCT offers Students Free Access to Writing Tool, Grammarly!. On the page, the Academic Dean, Shadya Mahfouz, for Niagara College—Toronto, is quoted,
Grammarly is a game-changing tool for students…We are always looking for ways to support our students in developing the communication skills they need to succeed in the classroom and beyond and we hope that all our students take full advantage of this opportunity to improve their writing.
Students are instructed to upload their “document or copy and paste your writing to the platform. You can also enable Grammarly wherever you type on your computer with the Chrome extension for Grammarly.” While a caveat on the same page mentions academic integrity, “As a Niagara College-Toronto student, you remain responsible to uphold the College’s standards for academic integrity and honesty,” there is no mention of student intellectual property or safety of student data uploaded and stored to a third party. The college’s own policies on “Disclosure of personal information to third parties” mentions the college’s responsibility for the data they maintain. What about the student data and their intellectual property that the college directs them to upload to a third party, for-profit, private company, based in the U.S. over which the college has no oversight?
Niagara College’s public-facing page, Academic Integrity and Artificial Intelligence Statement, directs readers to the college’s Academic Honesty Policy and Academic Misconduct Procedure, both of which were last updated in 2019.
As a note, the search prompt on google “University or Toronto” and “Grammarly” returns this link, Case No.: 1055, The university tribunal: The University of Toronto, second on the returned searches. Its placement there doesn’t seem to be a coincidence.
Students deserve better
Grammarly, with an estimated 30 million individual users (Ding & Zou, 2024), has become ubiquitous in higher education. Its reach is deep, right to the keyboard of each student, to their emails, assignments, and assessments. Students deserve fair and equal treatment. Faculty rely on its function for instruction and AI detection. Institutions should ensure that they provide services that work as claimed, as well as information that is accurate and non-contradictory. A student losing grades or failing a course because of third-party tools or contradictory information is a travesty that is preventable.
References
Grammarly. (2018). Robertson Library, University of Prince Edward Island. https://library.upei.ca/grammarly
Grammarly is not approved. (n.d.). LTE Toolkit, University of Saskatchewan. https://teaching.usask.ca/learning-technology/tools/grammarly.php
Tools for Plagiarism Detection: Plagiarism Detection with Grammarly. (n.d.). Robertson Library, University of Prince Edward Island. https://library.upei.ca/plagcheck
Ding, L., & Zou, D. (2024). Automated writing evaluation systems: A systematic review of Grammarly, Pigai, and Criterion with a perspective on future directions in the age of generative artificial intelligence. Educ Inf Technol, 29, 14151–14203. https://doi.org/10.1007/s10639-023-12402-3