Shubham Sharma

This work addresses the challenge of creating interactive learning environments by using Multimodal Large Language Models (MLLMs) to automatically answer student questions derived from online lecture videos. We introduce the EduVidQA dataset containing 5,252 question-answer pairs curated from 296 computer science lecture videos. We built a data generation pipeline combining manual annotation with LLM-assisted question synthesis (Gemini 1.5 Flash), producing 4,300+ Q&A pairs. We benchmark six state-of-the-art multimodal LLMs (Video-LLaMA, mPLUG-Owl, etc.) across zero-shot, few-shot, and fine-tuned settings, evaluating them using both text-based metrics (BLEU, ROUGE, METEOR) and qualitative student feedback.

We introduce the task of Aspect-Based Complaint Detection (ABCD) and propose a deep-learning-based multi-modal framework that processes both text and visual information from Amazon product reviews. The framework operates in two stages: (1) Aspect Prediction to identify targeted product features, and (2) Complaint Categorization to classify complaint severity. We collect a multimodal dataset from Amazon, extract textual embeddings using BERT and visual representations using pre-trained CNNs (VGG16, ResNet), and develop a multimodal interaction model to learn the relationship between text, image, and product aspects for identifying complaint eligibility.

We develop an effective pneumonia detection system by combining advanced image analysis with ensemble learning. We compile and preprocess a Mendeley dataset with three classes (Covid-19, Pneumonia, Normal) and apply data augmentation to address overfitting. Image features are extracted using Vision Transformer (ViT) and ResNet50 models, then combined into an ensemble. XGBoost is used as a meta-learner classifier on the concatenated features, achieving 96.19% accuracy on the pneumonia detection task.

We develop a deep learning-based Intrusion Detection System using the NSL-KDD dataset. Feature selection is optimized using Recursive Feature Elimination (RFE) with a Decision Tree classifier. The hybrid architecture combines two sets of 1D convolutional layers (64 and 128 filters) with max-pooling and batch normalization for spatial feature extraction, followed by a 100-unit LSTM layer for capturing temporal dependencies. The model achieves 95% accuracy and 0.94 F1-score for binary classification of network traffic as normal or intrusive.

We develop IndicBART, a multimodal summarization framework that integrates text and image data to produce regionally contextualized summaries in Indian languages. We collect a multimodal dataset (Hindi, Tamil, Bengali, Marathi) with paragraphs, summaries, and images from the Large Scale Multi-Lingual Multi-Modal Summarization Dataset (M3LS). We extend the BART model with a visual-aware encoder and achieve a ROUGE-1 score of 0.266 on the Hindi dataset and 44.1% image precision using an image pointer for multi-output prediction.

We perform a comparative evaluation of various machine learning and deep learning approaches for crop yield forecasting. The study evaluates Random Forest, Extra Trees, Artificial Neural Networks, SVR, and Decision Tree Regression on agricultural datasets incorporating climatic parameters (rainfall, temperature), soil composition, pesticide usage, and historical yield data. Models are validated using RMSE, MAE, and R-squared scores with cross-validation, providing practitioners with insights into selecting the most robust models for precision farming and resource allocation.