Finding relevant research papers through traditional keyword search can be challenging because different papers often describe the same concept using different terminology.

This project solves that problem by implementing a Semantic Research Paper Search Engine powered by Natural Language Processing (NLP) and Transformer-based sentence embeddings.

Instead of matching exact words, the system understands the context and semantic meaning of a user's query, allowing it to retrieve the most relevant research papers—even when the query and the paper use different vocabulary.

The project was developed as part of the CBSOT Summer Internship 2026 to explore modern Information Retrieval techniques using state-of-the-art NLP models.

• 🔍 Context-aware semantic search
• 🤖 Transformer-based text embeddings
• ⚡ Ultra-fast similarity search with FAISS
• 📚 Research paper retrieval using Natural Language queries
• 📊 Data preprocessing and exploratory data analysis
• 💾 Efficient embedding storage for faster searches
• 📈 Scalable architecture for large document collections
• 🧠 Uses pretrained Sentence Transformer models

Semantic-Research-Paper-Search/
│
├── EDA.ipynb
├── Search_Engine.ipynb
├── dataset.md
├── requirements.txt
├── .gitignore
└── README.md

Research Paper Dataset
          │
          ▼
Data Cleaning & Preprocessing
          │
          ▼
Sentence Transformer
(all-MiniLM-L6-v2)
          │
          ▼
Generate Vector Embeddings
          │
          ▼
Store Embeddings
          │
          ▼
Build FAISS Index
          │
          ▼
User Natural Language Query
          │
          ▼
Convert Query to Embedding
          │
          ▼
Similarity Search
          │
          ▼
Top Relevant Research Papers

This project uses the ML-ArXiv-Papers dataset available on Hugging Face.

The dataset contains thousands of Machine Learning research papers, including:

• Research Paper Title
• Abstract
• Authors
• Categories

The complete dataset reference is available in dataset.md.

git clone https://github.com/your-username/semantic-research-paper-search.git
cd semantic-research-paper-search

pip install -r requirements.txt

Run

EDA.ipynb

This notebook:

• Loads the dataset
• Cleans the data
• Performs Exploratory Data Analysis
• Generates semantic embeddings

Run

Search_Engine.ipynb

This notebook:

• Loads saved embeddings
• Creates (or loads) the FAISS index
• Accepts a natural language query
• Returns the most semantically similar research papers

Deep learning techniques for image classification

✔ Most Relevant Research Papers

✔ Similarity Scores

✔ Paper Titles

✔ Research Abstracts

Unlike traditional keyword search:

❌ Keyword Search

• Matches exact words
• Misses related concepts
• Sensitive to wording

✅ Semantic Search

• Understands context
• Finds conceptually similar papers
• Uses dense vector embeddings
• Produces more meaningful search results

Through this project I gained practical experience with:

• Natural Language Processing (NLP)
• Semantic Search
• Transformer Models
• Sentence Embeddings
• Vector Databases
• FAISS Indexing
• Information Retrieval
• Research Paper Retrieval Systems
• Data Preprocessing
• Exploratory Data Analysis

• 🌐 Streamlit Web Application
• 📄 PDF Upload & Search
• 🤖 Research Paper Recommendation System
• 🔍 Hybrid Keyword + Semantic Search
• 📊 Interactive Search Dashboard
• 🐳 Docker Support
• ☁ Cloud Deployment
• 🔌 REST API Integration

Contributions are always welcome!

If you have ideas to improve this project, feel free to:

• Fork the repository
• Create a new branch
• Submit a Pull Request

If you found this project useful, please consider giving it a ⭐ Star on GitHub.

It helps support the project and motivates future improvements.

Vaishnavi Kushwah

Developed during the CBSOT Summer Internship 2026.

Let's connect and collaborate on AI, Machine Learning, NLP, and Open Source projects!