> For the complete documentation index, see [llms.txt](https://infinitode-docs.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://infinitode-docs.gitbook.io/documentation/package-documentation/duplipy-package-documentation/duplipy-reference/formatting-functions.md).

# Formatting Functions

**Available functions:**

* [`remove_stopwords`](#remove-stopwords)`(text)`: Remove stopwords from the input text using NLTK's stopwords.
* [`remove_numbers`](#remove-numbers)`(text)`: Remove numbers from the input text.
* [`remove_whitespace`](#remove-whitespace)`(text)`: Remove excess whitespace from the input text.
* [`normalize_whitespace`](#normalize-whitespace)`(text)`: Normalize multiple whitespaces into a single whitespace in the input text.
* [`seperate_symbols`](#seperate-symbols)`(text)`: Separate symbols and words with a space to ease tokenization.
* [`remove_special_characters`](#remove-special-characters)`(text)`: Remove special characters from the input text.
* [`standardize_text`](#standardize-text)`(text)`: Standardize the formatting of the input text.
* [`tokenize_text`](#tokenize-text)`(text)`: Tokenize the input text into individual words.
* [`stem_words`](#stem-words)`(words)`: Stem the input words using Porter stemming algorithm.
* [`lemmatize_words`](#lemmatize-words)`(words)`: Lemmatize the input words using WordNet lemmatization.
* [`pos_tag`](#pos-tag)`(text)`: Perform part-of-speech (POS) tagging on the input text.
* [`remove_profanity_from_text`](#remove-profane-words-from-text)`(text)`: Remove profane words from the input text.
* [`remove_sensitive_info_from_text`](#remove-sensitive-information-from-text)`(text)`: Remove sensitive information from the input text.
* [`remove_hate_speech_from_text`](#remove-hate-speech-from-text-using-ai)`(text)`: Remove hate speech or offensive speech from the input text.
* [`post_format_text`](#post-format-text-using-regex)`(text)`: Post-format the text using regex.

***

### Remove stopwords

Remove stopwords from the input text using NLTK's stopwords.

```
Parameters:
- `text` (str): The input text from which stopwords should be removed.

Returns:
- `str`: The text without stopwords.
```

### Remove numbers

Remove numbers from the input text.

```
Parameters:
- `text` (str): The input text from which numbers should be removed.

Returns:
- `str`: The text without numbers.
```

### Remove whitespace

Remove excess whitespace from the input text.

```
Parameters:
- `text` (str): The input text from which excess whitespace should be removed.

Returns:
- `str`: The text with the removed excess whitespace.
```

### Normalize whitespace

Normalize multiple whitespaces into a single whitespace in the input text.

```
Parameters:
- `text` (str): The input text from which whitespace should be normalized.

Returns:
- `str`: The text with normalized whitespace.
```

### Seperate symbols

Separate symbols and words with a space to ease tokenization.

```
Parameters:
- `text` (str): The input text from which symbols needs to be seperated.

Returns:
- `str`: The text from which symbols have been seperated.
```

### Remove special characters

Remove special characters from the input text.

```
Parameters:
- `text` (str): The input text from which special characters should be removed.

Returns:
- `str`: The text with special characters removed.
```

### Standardize text

Standardize the formatting of the input text.

```
Parameters:
- `text` (str): The input text which needs to be standardized.

Returns:
- `str`: The standardized text.
```

### Tokenize text

Tokenize the input text into individual words.

```
Parameters:
- `text` (str): The input text to be tokenized.

Returns:
- `list`: A list of tokens (words) from the input text.
```

### Stem words

Stem the input words using the Porter stemming algorithm.

```
Parameters:
- `words` (list): A list of words to be stemmed.

Returns:
- `list`: A list of stemmed words.
```

### Lemmatize words

Lemmatize the input words using WordNet lemmatization.

```
Parameters:
- `words` (list): A list of words to be lemmatized.

Returns:
- `list`: A list of lemmatized words.
```

### POS tag

Perform part-of-speech (POS) tagging on the input text.

```
Parameters:
- `text` (str): The input text to be POS tagged.

Returns:
- `list`: A list of tuples containing (word, tag) pairs.
```

### Remove profane words from text

This ensures that the text is clean and does not contain inappropriate language.

```
Parameters:
- `text` (str): The input text to remove profanity from.

Returns:
- `text` (str): The cleaned output text.
```

### Remove sensitive information from text

This can be useful for depersonalization of text data.

```
Parameters:
- `text` (str): The input text to remove sensitive information from.

Returns:
- `text` (str): The cleaned output text.
```

### Remove hate speech from text using AI

This function removes sentences, and not just a certain word because it is context-relevant.

```
Parameters:
- `text` (str): The input text to remove hate speech and offensive speech from.

Returns:
- `text` (str): The cleaned output text.
```

### Post-format text using regex

This function post-formats the text after DupliPy's augmentation or other processes.

```
Parameters:
- `text` (str): The input text to be post-formatted.
    
Returns:
- `str`: The post-formatted text.
```