Introducing Semantic Search 9/15/22

Written by: Exile

Hello maplers, and welcome to our first dev blog!

We wanted to share with you a project we've been working on to bring advanced search capabilities to MapleStory Network. Put simply, the semantic search engine is a tool to help you discover items (and eventually all game data, coming soonTM) by offering a description of what you want, instead of the text-matching search that we currently provide. For example, searching with the text "black bandana with red maple leaf on it" might dig up the legendary Mark of the Beta in the results, despite the search query not being a textual "match" with the item name/description. This is a new paradigm for finding (or freely exploring) data which we hope will create new ways to design characters and learn about the vast world of MapleStory. This blog will show off some intriguing and fun capabilities of this approach, and how it stacks up against our existing search feature.

This project has been a work-in-progress for several months and we are proud to announce that it will be partially available today in an early beta state, with a full release on the horizon. Please note that due to performance/cost constraints, certain advanced features, marked with a⭐, will only be available to supporter users. Also, semantic search will not be available through an API on MapleStory:Develop; if you are a developer interested in having access in this way, please contact us directly. That said, in the spirit of keeping things free and accessible, our goal is still to offer as much access to the tool as we can to free users. In the process we hope to address the feedback and suggestions we have received on improving our search/filtering capabilities, such as filtering items by color or other criteria.

So, How Does it Work?

Without getting too technical (leave feedback if you're interested in more technical dev blogs!), the system works as follows: given a text query, the search engine finds the item whose icon sprite looks closest to what is described by the text. This is why querying for a visual description of the object works. At the core of the engine is a neural network(s) that can intelligently interpret images and text so that conceptual text-image similarity can be calculated. There are some extremely interesting and useful consequences (as well as limitations) when doing things in this way, which we will now explore.

Basic Filtering

The simplest (and probably most common) use case one might imagine is using a text query to filter for some basic aspect of a set of items. For example, a commonly requested feature has been to filter items by color. With semantic search you can simply type a query like "<color> t-shirt" and get t-shirts of the desired color. What's interesting is that because the search engine is intelligent, this allows for searching for colors that aren't usually mentioned in-game. For example, many in-game items are labelled as red, yellow, green, black, etc. To get our feet wet with this new approach, let's instead search for something more exotic: magenta items. Running the search on MapleStory.net (left) yields only a few items with "magenta" in the name or description, while the semantic search (right) returns a broader, more relevant, set of results:

MapleStory.net query: "magenta"	Semantic search query: "a magenta wizard staff"

Already, we see the strengths and weaknesses of this new system: it managed to find many more objects with that rich purple-red tint to them, but not all are staffs. This happens because the engine looks for objects that look like staffs, rather than objects that actually are staffs in the game. So there is an assortment of staff-shaped objects like spears and weirdly shaped swords. This is an area we are going to improve before the feature's full launch - but this simple example should show that this isn't always a bad thing: using semantic search, you can search for things that look like what you ask for, even if what you ask for is not exactly accurate. This looser interpretation of your prompt also allows for more explorative searches, where you might be looking to discover something new with a very "loose" search. Part of this spear-sword-staff confusion also stems from the fact that the model is equipped with knowledge of MapleStory's data, and there are many items in MapleStory (notably cash shop items) that may be equipped over many different other kinds of items. Nevertheless, it is nice to use this kind of filtering to have better access to the game's magenta items when the regular search fails. Also, we plan to combine the existing filters with semantic search, which would, for example, allow for filtering search results by the usual item types and keywords.

For the examples from now on, we encourage you to try searching on MapleStory.net using the traditional approach yourself for comparison. Here's another example of a descriptive search:

Semantic search query: "object with a picture of a person on it"

Conceptual Search

Now let's try a more advanced query. Let's say winter is on its way and your character needs to keep themselves warm. No problem, let's find some nice winter gear:

Semantic search query: "wintertime clothing"

Clearly, the engine understands the concept of winter and what clothes one might wear during it. This is a step up from just searching based on a visual description of an object. But maybe you are going after a particular style. Maybe you wanted a fancy outfit with a fur coat. Can we find clothes made of that material?

Semantic search query: "clothing made with fur"

Nice. Here we see another limitation: some items are not equipment! This will not be the case in the full release, though, and etc/equipment/consumable items will be in their own categories to be searched independently. What is really nice, though, is that it by and large still worked - the search engine was able to find objects made with a particular material or texture.

Genie in a Lamp

One weakness of the system is that it sometimes returns the "correct" results in a way that you aren't expecting. For example, when searching for "item used for shooting arrows", one might expect the result to contain bows and crossbows. Instead, the search engine appears to understand that to shoot arrows, one needs to use arrows, and thus returns sharp objects that look arrow-like:

Semantic search query: "item used for shooting arrows"

This kind of behavior is like a genie in a lamp who grants your wishes, but in wonky ways that are technically "correct" yet not what you actually want. To be fair, there are some bows in there...

Search Based on Cultural Contexts and Intent

Oftentimes words can take on several different meanings, and so a normal textual search may not understand your intent when you search for something. Take for example the concept of pride - in MapleStory, "pride" mostly refers to a medallion object, perhaps to be proud of one's accomplishments. But in real life, "pride" is often used in the context of the LGBT movement, or more generically the idea of cultural pride, and this more relevant interpretation is understood by the semantic search engine. This is probably the most stark difference yet between the two search methods:

MapleStory.net query: "pride"	Semantic search query: "pride"

This is an exceptional demonstration of how semantic search might better understand the intent behind your request, or cultural information.

A very tasty example of this is the system's knowledge of different kinds of food:

MapleStory.net query: "street food"	Semantic search query: "fancy restaurant food"

Delicious. A cute result is the fact that "street food" turns up icons of food vendors, like trucks and stands, since those are objects associated with the concept of street food.

A note and warning on cultural bias:

Bias can be a siginificant issue in systems like this, and we are taking steps to mitigate it. Models like these are excellent at understanding cultural concepts and this can make them powerful and fascinating, but they may be prone to picking up racial/cultural stereotypes and other offensive things. MapleStory itself also introduces certain biases and stereotypes. A mostly benign example is the gender-specific constraints tied to particular items' themes and styles (you may have seen in-game items marked with "(M)" or "(F)", for example). The game explores concepts like this, so its biases, worldview, and sense of humor are reflected in the search results. We consider these to be mostly appropriate; they exist within the artistic context that is MapleStory, as created by its developers. Our stance on all of this is subject to change as the situation evolves and we become smarter and revise our understanding of bias in systems like this.

We ask that users use this feature respectfully and appropriately, and because offensive (or inoffensive but irrelevant) biases are difficult to eliminate entirely automatically, we want to make it clear that usage of this system is subject to our usual rules and terms of service regarding offsensive behavior/language, harassment, and discrimination of any kind.

⭐ Prompt Averaging

One way to boost search accuracy is to take a query, and then generate multiple similar queries using templates, run them, and average the resulting scores before building the final ranking of found items. This is a work in progress, but in the future it will be possible to "retry" a query with a different template, or to simply run a batch for the best results. Because it is expensive to do this, we tentatively plan this as a premium feature for supporters.

Finding Related Items With Image-Image Searching

Here's another use: instead of matching images to text, match images to other images. While we don't have a nice screenshot of this in action for the blog here, we are working on this now: imagine clicking on an item, and then having suggestions of other items that look like it. With a tool like this, one could first find an interesting item via a textual search, and then navigate through related items to refine the result in order to discover the item they are most interested in. This will be great for designing that dream character with the perfect outfit.

⭐ Let's Play Pictionary

(Note: this feature is still a work-in-progress) Maybe you're looking for an item you saw in the game, but can't quite describe it well enough in words. Instead, you can just draw your best interpretation of it, and the search engine will dig up the existing game items that are the most similar. This might also be useful in cases like these:

• you want to know what items are in a screenshot - submit the screenshot and see the results
• you have a dream item in mind that you wish to have for your character, but don't know if it actually exists in-game - just draw it to check
• you are looking for a certain color/pattern arrangement - you can illustrate the desired texture/pattern and find objects with it
• this feature is just genuinely fun to play around with, even when you have no practical use for it

Here are some examples:

Drawing	Top 5 Results

In the first example, I wanted to see if I could find the Fairfrozen , since proper-noun names like this make items hard to find in a normal text search if you don't know the name already (you can also try to use semantic search for "a spear made out of ice" and see what happens). My memory of the item's appearance was not perfect (and my drawing skill not quite the greatest) but it found the correct item as the first result, and showed meaningfully relevant items for the other 4 results. In the second one, I drew a mysterious looking cube, and the results are as expected. By the last example it is clear what this approach is or isn't capable of.

Conclusion

We hoped you liked this preview of the new search functionality coming to MapleStory.net, and inspired you in coming up with clever ways to use it. What we have shown here is just scratching the surface of what is possible: we are tuning the output to be more accurate before a final "1.0" release, and there are likely many techniques for crafting queries that we have yet to discover. As a reminder, we will be releasing an experimental and limited version of this feature when or shortly after this post goes live, so feel free to head over, check it out, and let us know what you think if you have comments, questions, or suggestions!

Acknowledgements

We'd like to thank each and every one of your for your support and interest in MapleStory network. For our supporters: projects like this would not be possible without you, and we are incredibly grateful for your contributions to our continued development efforts. If you are interested in becoming a supporter and the perks that come with it, please visit https://maplestory.net/supporter. Money from donations is used to compensate staff for their hard work, for server maintenance/other project costs, and potentially for future advertising efforts.