![]() Here we have encoded our text in batches of 16. To initialize the model and encode our Reddit topics data, we first pip install sentence-transformers and then write: The first result of this search is sentence-transformers/all-MiniLM-L6-v2, this is a popular high-performing model that creates 384-dimensional sentence embeddings. We can find official sentence transformer models by searching for “sentence-transformers” on HuggingFace Hub. They can be found on HuggingFace Hub by searching for “sentence-transformers”. Of the above, the Sentence Transformers library provides the most extensive library of high-performing sentence embedding models. If we build poor quality embeddings, nothing we do in the other steps will be able to help us, so it is very important that we choose a suitable embedding model from one of the supported libraries, which include: Transformer EmbeddingīERTopic supports several libraries for encoding our text to dense vector embeddings. This section will work through each component without BERTopic, and learn how they work before returning to BERTopic at the end. However, we can optimize the process by understanding the essentials of each component. We already did all of this in those few lines of BERTopic code everything is just abstracted away. There are four key components used in BERTopic, those are: Instead, let’s try and understand how BERTopic works. These represent the surface level of the BERTopic library, which has excellent documentation, so we will not rehash that here. The library has several built-in visualization methods like visualize_topics, visualize_hierarchy, and visualize_barchart.īERTopic’s visualize_hierarchy visualization allows us to view the “hierarchy” of topics. However, we removed stop words via the vectorizer_model argument, and so it shows us the “most generic” of topics like “Python”, “code”, and “data”. The top -1 topic is typically assumed to be irrelevant, and it usually contains stop words like “the”, “a”, and “and”. We can then view the topics using get_topic_info. ![]() probs contains a list of probabilities that an input belongs to their assigned topic.topics contains a one-to-one mapping of inputs to their modeled topic (or cluster).The “basic” approach requires just a few lines of code.įrom model.fit_transform we return two lists: We perform topic modeling using the BERTopic library. Some are empty or short, so we remove them with: Reddit thread contents are found in the selftext feature. The code used for this (and all other examples) can be found here. The dataset contains data extracted using the Reddit API from the /r/python subreddit. We can download the dataset from HuggingFace datasets with: We will dive into the details behind BERTopic, but before we do, let us see how we can use it and take a first glance at its components. In machine learning, we refer to this task as topic modeling, the automatic clustering of data into particular topics.īERTopic takes advantage of the superior language capabilities of these (not yet sentient) transformer models and uses some other ML magic like UMAP and HDBSCAN (more on these later) to produce what is one of the most advanced techniques in language topic modeling today. Machines with a human-like comprehension of language are pretty helpful for organizing masses of unstructured text data. Our NLP transformers lie somewhere in the middle, they’re not sentient Autobots (yet), but they can understand language in a way that existed only in sci-fi until a short few years ago. These transformers are (unfortunately) not Michael Bay’s Autobots and Decepticons and (fortunately) not buzzing electrical boxes. We can now search text based on meaning, identify the sentiment of text, extract entities, and much more. More and more of this unstructured text is becoming accessible and understood by machines. Organization is complicated because unstructured text data is not intended to be understood by machines, and having humans process this abundance of data is wildly expensive and very slow.įortunately, there is light at the end of the tunnel. That’s great for human consumption, but it is very hard to organize when we begin dealing with the massive amounts of data abundant in today’s information age.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |