ValX Package Documentation

Package documentation for ValX, a Python package for handling text-cleaning tasks, including profanity and PII. Now includes hate speech detection and offensive language detection using AI.

Changelog

  • 0.2.5 (Latest): Introduced enhanced flexibility for profanity filtering:

    • Added custom_words_list parameter to detect_profanity and remove_profanity for user-defined profanity lists.

    • Support for standalone custom lists by setting language=None.

    • Support for combined lists (built-in language + custom list).

    • New helper function load_custom_profanity_from_file(filepath) to load custom words from a file.

    • detect_profanity output now specifies profanity source (e.g., "Custom", "Custom + English").

  • 0.2.4: Fixed compatibility issues with scikit-learn versions 1.3.0 and up. Also removed dependency for scikit-learn 1.2.2 as it is no longer needed, older versions and newer versions are now compatible. Please read this issue for more information: https://github.com/Infinitode/ValX/issues/1

  • 0.2.3: Created new detection patterns for sensitive information, and created a new optional info_type parameter to control sensitive information detection and removal.

  • 0.2.2: Refactored detect_profanity function to return more information about the found profanities. Also removed unnecessary printing in functions.

  • 0.2.1: Updated project PYPI description.

  • 0.2.0: Created a new function, to automatically remove detected hate speech or offensive speech from text.

  • 0.1.8 - 0.1.9: Updated docstrings.

  • 0.1.7: Added AI models to ValX for hate speech detection.

  • 0.1.1 - 0.1.6: Fixed errors in code, and created several functions for text cleaning.

  • 0.1.0: Initial release.

Installation

You can install ValX using PyPi, please make sure that you are using Python 3.6 or later before installing ValX:

pip install valx

List of supported languages for profanity detection and removal

Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for language:

  • All

  • Arabic

  • Czech

  • Danish

  • German

  • English

  • Esperanto

  • Persian

  • Finnish

  • Filipino

  • French

  • French (CA)

  • Hindi

  • Hungarian

  • Italian

  • Japanese

  • Kabyle

  • Korean

  • Dutch

  • Norwegian

  • Polish

  • Portuguese

  • Russian

  • Swedish

  • Thai

  • Klingon

  • Turkish

  • Chinese

Example Usage

Profanity Detection

from valx import detect_profanity

sample_text = [
    "This is a sample text containing some profanity like bad word 1, bad word 2, and bad word 3.",
    "This line doesn't contain any profanity.",
    "But this one has another, just in another language: bad word 4."
]

# Detect profanity
results = detect_profanity(sample_text, language='English')
print("Profanity Evaluation Results", results)

Profanity Removal

from valx import remove_profanity

sample_text = [
    "This is a sample text containing some profanity like bad word 1, bad word 2, and bad word 3.",
    "This line doesn't contain any profanity.",
    "But this one has another, just in another language: bad word 4."
]

# Remove profanity
removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")

PII Detection

from valx import detect_sensitive_information

sample_text = [
        "Please contact john.doe@example.com or call 555-123-4567 for more information.",
        "We will need your credit card number to complete the transaction: 1234-5678-9012-3456.",
        "My social security number is 123-45-6789 and my ID number is AB123456.",
        "Our office address is 123 Main St, Anytown, USA. Please visit us!",
        "Your IP address is 192.168.1.1. Please don't share it with anyone."
]

# Detect sensitive information
detected_information = detect_sensitive_information(sample_text)

PII Removal

from valx import remove_sensitive_information

sample_text = [
        "Please contact john.doe@example.com or call 555-123-4567 for more information.",
        "We will need your credit card number to complete the transaction: 1234-5678-9012-3456.",
        "My social security number is 123-45-6789 and my ID number is AB123456.",
        "Our office address is 123 Main St, Anytown, USA. Please visit us!",
        "Your IP address is 192.168.1.1. Please don't share it with anyone."
]

# Detect sensitive information
cleaned_information = remove_sensitive_information(sample_text)

Hate Speech Detection

from valx import detect_hate_speech

# Detect hate speech or offensive language
outcome_of_detection = detect_hate_speech("You are stupid.")

Remove Hate Speech

from valx import detect_hate_speech

sample_text = [
    "This is a sample text containing some profanity like bad word 1, bad word 2, and bad word 3.",
    "This line doesn't contain any profanity.",
    "But this one has another, just in another language: bad word 4."
]
# Remove hate speech or offensive language
cleaned_text = remove_hate_speech(sample_text)

Custom Profanity Filtering

ValX allows for flexible profanity filtering using custom word lists.

from valx import detect_profanity, remove_profanity, load_custom_profanity_from_file

# Example: Create a dummy custom profanity file
with open("custom_profanity.txt", "w") as f:
    f.write("# This is a comment and will be ignored\n")
    f.write("sillyword1\n")
    f.write("    \n") # Empty line, ignored
    f.write("sillyword2\n")
    f.write("custombadword\n")

sample_text_custom = [
    "This text contains a sillyword1.",
    "Another line with sillyword2 and also a standard badword.",
    "This custombadword should be caught."
]

# 1. Using custom_words_list directly (standalone)
custom_list_direct = ["sillyword1", "custombadword"]
detected_custom_direct = detect_profanity(sample_text_custom, language=None, custom_words_list=custom_list_direct)
print("Detected with direct custom list (standalone):", detected_custom_direct)
# Expected: Detects "sillyword1" and "custombadword" with source "Custom"

# 2. Loading custom words from a file
loaded_custom_list = load_custom_profanity_from_file("custom_profanity.txt")
print(f"Loaded custom words: {loaded_custom_list}") # Expected: ['sillyword1', 'sillyword2', 'custombadword']

# 3. Using loaded custom list with a built-in language (English)
detected_custom_combined = detect_profanity(sample_text_custom, language="English", custom_words_list=loaded_custom_list)
print("Detected with loaded custom list + English:", detected_custom_combined)
# Expected: Detects "sillyword1", "sillyword2", "custombadword" (Source: Custom + English or Custom)
# and "badword" (Source: English)

# 4. Removing profanity using a custom list
cleaned_text_custom = remove_profanity(sample_text_custom, language=None, custom_words_list=loaded_custom_list)
print("Text after removing profanity with custom list:", cleaned_text_custom)
# Expected: "sillyword1", "sillyword2", "custombadword" are removed.

# Clean up dummy file (optional)
import os
os.remove("custom_profanity.txt")
PreviousPyWebScrapr FunctionsNextValX Reference

Last updated

Was this helpful?