The billions of pieces of digital content uploaded to online platforms and websites daily make content moderation both critical and challenging for tech firms to address. Although lots of content uploaded may be positive, there is a growing volume of harmful and illegal materials online – including violence, self-harm, extremist rhetoric, sexually explicit imagery and child sex abuse material (CSAM).
Tackling such large volumes of harmful content has become a defining challenge for businesses, as those unable (or unwilling) to do so may expose themselves to significant penalties and put their users at severe risk.
According to our research, more than a third (38%) of parents say their kids have experienced harmful or illegal contentonline. Many of them have accessed these materials—as graphic and as harmful as CSAM—within ten minutes of going online. It is, therefore, clear that the need for stronger content moderation measures has become more important than ever.
Businesses considering conventional or manual moderation methods may find them impractical or unscalable. Instead, leaders should opt for artificial intelligence’s (AI) complementary capabilities, which are transforming the landscape of content moderation through automation, scalability, and enhanced accuracy.
The decisions companies make today will massively impact their future operations. Therefore, like any emerging innovation, companies interested in using AI must ensure they deploy the technology appropriately to meet regulatory compliance.
Using AI in content moderation
AI has significantly transformed content moderation by automatically scanning images, pre-recorded videos, and live streams in real-time. It works by being trained on large volumes of “ground truth data” by collecting and analysing insights from archives of tagged imagery and videos.
It can identify issues such as underage presence in adult content or nudity, sexual activity, extreme violence, self-harm, and hate symbols across user-generated content platforms, including social media. The accuracy and efficacy of AI systems directly correlate to the quality and quantity of this data, and once trained, AI can effectively detect various forms of harmful content. This is especially important during live streaming scenarios, where content moderation needs to be available in real-time, as well as viable across diverse platforms with varying legal and community standards.
An automated approach accelerates the content moderation process and provides scalability. Given the vast amounts of content online, this is a vital feature in a time when human moderation alone wouldn’t be possible.
AI and human collaboration
AI automation offers significant benefits to organisations, allowing them to both moderate content at scale and reduce costs by removing the need for large moderator teams. However, AI is far from perfect and requires human judgement to accompany it, especially when identifying specific cues and contextual nuances that can sometimes confuse systems and produce inaccurate results. For example, AI may be unable to distinguish between a kitchen knife used in a cooking video and a weapon used in an act of violence, or even mistake a toy gun in a kid’s advertisement with an actual handgun.
So, when AI alerts to content it deems potentially harmful or in violation of community guidelines, human moderators can jump in to examine it and make the final decision. This hybrid approach ensures that while AI extends the scope of content moderation and streamlines the process, humans retain the ultimate authority, especially in complex cases.
In the coming years, the sophistication level of AI identification and verification methods will continue to improve. This includes improving the accuracy of matching individuals with their identity documents across different forms of digital content—a continued step in the right direction for ensuring consent and mitigating unauthorised content distribution.
With its learning capabilities, AI will also continue improving its accuracy and efficiency, potentially replacing the need for human involvement as it continues to evolve. However, humans will continue to be a necessity, especially during appeals and dispute resolutions related to content moderation decisions. Not only do current AI technologies lack nuanced perspective and understanding, but humans can also serve as a check against potential algorithmic biases or errors.
Understanding AI regulation across the globe
As AI evolves, businesses of all sizes will look to regulators to outline their plans to govern AI applications. The European Union (EU) is at the forefront of this legislation, as its Artificial Intelligence Act came into force in August 2024. Positioned as a guide in the regulatory field, the Act categorises AI systems into three types: those posing an unacceptable risk, those of high risk, and a final category with fewer regulations.
As a result, an AI office has been established to oversee the implementation of the Act. It consists of five units: safety, regulation and compliance, AI innovation and policy coordination, excellence in AI robotics, and AI for societal good. This office will also oversee the deadlines for certain businesses to comply with the new regulations, ranging between six months for prohibited AI systems and 36 months for higher-risk AI systems.
With this, businesses in the EU are recommended to stay abreast of the legislative landscape closely to assess the potential effects on their operations and ensure their AI systems are compliant within established deadlines.
The legislation is expected to inform policies not just within the EU but also in the UK, the US, and other regions. UK and US-based AI regulations will follow suit, so businesses must ensure they have their finger on the pulse and that any tools they implement now will likely meet the compliance guidelines rolled out by these countries in the future.
Working together to make the internet safer
The successful deployment of AI in content moderation tools will require a solid commitment to innovation and continuous improvements. Solutions are typically developed before regulations come into effect – therefore, it is essential that businesses proactively audit solutions to mitigate potential biases, protect user privacy and ensure fairness. Businesses must also invest in year-round human moderation training to effectively manage nuanced cases raised by AI for review.
Given the psychological toll of content moderation, it is equally important that providers take the mental health of their human moderators seriously. This can be done by offering robust psychological support, wellness resources, and strategies to mitigate extended exposure to disturbing content.
By adopting a proactive yet responsible approach to AI-powered content moderation, digital platforms can cultivate a community that encourages connection, creativity, and constructive discussion while protecting users from harm.
Ultimately, AI-driven content moderation tools provide organisations with a comprehensive toolkit needed to overcome challenges brought on by the digital age. By monitoring and filtering massive volumes of user-generated content in real-time, these innovations help humans create a content moderation solution that ensures the protection of kids as they access the internet and acts as the foundation for creating a safe online ecosystem.
Andy Lulham
Andy Lulham is Chief Operating Officer at VerifyMy. Andy has over 15 years’ experience in senior marketing, commercial, operations and international roles, mainly in eGaming at brands including Betfair and Oddschecker.
