Nashaat (نشاط) — Government Analytics & Intelligence Platform
Government analytics platform replacing manual spreadsheet tabulation with automated decision intelligence across 100+ schools.
What was broken before this system existed?
Educational leadership was burdened with 13+ disparate, unstandardized Arabic Excel spreadsheets from Microsoft Forms and manual administrative files across 100+ schools. Generating district-level quarterly reports required 4 full weeks of manual data cleaning and cross-tabulation, riddled with 200+ free-text spelling variations of school names, missing records, and inconsistent column headers.
What did I actually engineer and own?
As the Full-Stack Developer & Data Engineer (Lead), I owned the system end-to-end: from initial requirements definition and architectural design to database schema modeling, backend pipeline development, and final production deployment.
Engineered a centralized, automated 3-tier ETL and analytics platform in Django. Built an intelligent parsing pipeline capable of ingesting 13+ distinct Excel layouts without per-file configuration, a custom Arabic NLP normalization engine to unify noisy text inputs, an automated KPI aggregation core with severity-graded recommendations, and a responsive RTL executive command dashboard.
How the system is structured
System Design
Built a robust three-tier Django architecture to process and visualize extensive educational survey data. The data layer handles dynamic ingestion and normalization across multiple operational categories. The presentation layer features a comprehensive custom analytics engine responsible for processing Likert scales, Arabic NLP, barrier classification, trend analysis, and automated recommendation generation. The system is secured by a custom authentication flow with strict role-based access control.
Architecture & Database Schema
Engineered a relational database schema utilizing dynamic field patterns to enable automated column mapping during data ingestion. Implemented advanced UPSERT logic using hash-based unique constraints to ensure data integrity during continuous operational tracking updates.
Data Pipeline
Designed an automated ETL (Extract, Transform, Load) pipeline: Unstructured File Ingestion → Multi-Strategy Header Detection → Dynamic Field Mapping → Arabic Text Normalization → Database UPSERT → Analytics Engine → JSON Payload Generation → Client-Side Rendering.
Frontend Architecture
Developed a server-rendered Django application paired with Tailwind CSS for a fully responsive, RTL Arabic layout. Integrated ApexCharts.js for dynamic, client-side data visualization. Extended the UI with Flowbite components and custom typography optimized for formal Arabic governmental standards.
Key Technical Challenges Solved
Unstructured Data Ingestion: Engineered a robust parsing service capable of handling complex, unstructured spreadsheets with merged cells and embedded metadata, utilizing multi-strategy header detection and forward-filling algorithms.
Advanced Arabic NLP & Normalization: Resolved massive data fragmentation (hundreds of spelling variations across tens of thousands of records) by building a keyword-based canonical mapping algorithm that normalizes Arabic text without requiring exact string matches.
Accessible UX for Administrative Staff: Built a zero-configuration upload flow with automated file-type identification and parsing, allowing administrative teams to securely ingest multiple data formats without requiring technical intervention.
Automated Insights: Transformed raw survey data into automated, Arabic-language recommendations with algorithmic severity grading, instantly providing actionable intelligence to decision-makers.
Why this architecture? Tradeoffs & judgment
1. Why Relational Schema with Dynamic Model Mapping: Avoided document stores because reporting required strict relational integrity, cross-semester comparative joins, and deterministic aggregation across schools.
2. Why Custom Keyword Canonicalization over Embeddings: Production constraint required sub-100ms instant normalization for hundreds of free-text school names without external cloud LLM latency or non-deterministic token drift.
3. What happens at 10x scale: Ingestion pipeline was designed with MD5-hash idempotent UPSERT logic and chunked Pandas streaming; scaling to 200k+ records would move task processing to Celery workers with PostgreSQL partitioning.
4. What was deliberately NOT built: Did not build a heavy drag-and-drop spreadsheet builder or complex microservices; a clean modular Django monolith satisfied strict government on-premise security and single-operator maintainability.
Hardest engineering hurdles
Handling merged cells, inconsistent header rows, and unaligned tabular data in legacy administrative spreadsheets without human intervention. Solved by developing forward-fill heuristic scanners that detect title blocks and dynamic table boundaries automatically.
Intelligent Multi-Format Data Ingestion
Built a three-tier Excel parsing pipeline capable of handling 13+ distinct Arabic-language data file types — from structured MS Forms exports (up to 5,000 records each) to completely unstructured, merged-cell management spreadsheets. The system auto-detects header rows, dynamically maps Excel columns to database fields using Django model introspection, and handles semester-aware data tagging without requiring any configuration per file type.
Custom Arabic NLP & Text Normalization Engine
Developed a comprehensive Arabic text processing pipeline to process high-variance, unstructured free-text survey responses:
Letter form unification: Normalizes أ/إ/آ → ا, ة → ه, ى → ي across all input
School name canonicalization: A 140+ entry keyword-matching system that resolves 200+ unique free-text spelling variations of school names into canonical forms — handling typos, missing letters, and alternative Arabic spellings without exact matching
Barrier categorization: Automatically classifies free-text non-participation reasons into actionable categories (timing, transportation, awareness, etc.)
Interest mapping: Groups diverse activity suggestions into strategic categories (sports, technology, arts, religious, scientific)
4,700+ Line Analytics Engine with Decision Support
The analytics core processes every data category with its own specialized analytical pipeline. For student satisfaction alone, it:
Computes weighted averages across 9 Likert-scale dimensions
Generates per-school and per-grade satisfaction rankings
Performs semester-over-semester comparative analysis detecting dimension-level regressions
Auto-generates severity-graded decision recommendations (critical/warning/excellent) based on computed thresholds
Outputs radar charts, treemaps, horizontal bars, grouped comparisons, radial gauges, and participation donuts — all from real data, zero hardcoded values
Executive Dashboard with Real-Time KPIs
A command-center style overview showing total system records, meeting completion rates, satisfaction survey volumes, field visit tracking, and event report counts — with dynamic year-based filtering, contextual alert banners, and drill-down navigation to 16 specialized analytics pages.
Operational Task Management with UPSERT Logic
Implemented an MD5 hash-based deduplication system for operational plan tasks that supports idempotent re-imports — uploading the same tracking spreadsheet multiple times updates existing records rather than creating duplicates, enabling weekly status refresh workflows.
Event Image Gallery
A searchable, paginated image management system for documenting events with per-event and per-date filtering, supporting bulk upload with automatic metadata association.
Premium Arabic RTL Dashboard
A production-quality, fully right-to-left Arabic interface featuring:
Hover-expanding sidebar (80px → 256px with animated icon-to-label transitions)
Glassmorphism navigation with backdrop blur
Custom color palette (Burgundy, Gold, Navy) reflecting governmental branding
Responsive mobile-first design with drawer navigation
Micro-animations (fadeInUp, slideInRight, hover-lift)
Quantified results & production impact
• Reduced quarterly ministerial reporting turnaround from 4 weeks of manual tabulation to under 3 minutes of automated processing.
• Scaled to 20,000+ survey records across two academic semesters, generating actionable insights for 100+ schools.
• Normalized 200+ free-text spelling variations of 140+ school names into canonical entities with deterministic rule-based algorithms.
• Automated executive decision support: autonomously flags low participation rates and dimension-level satisfaction drops with Arabic-language severity alerts.
What I would change at 10x scale
Real-world data is infinitely dirtier than benchmark datasets. Investing in deterministic validation and canonical mapping dictionaries upfront saved weeks of downstream debugging in aggregate calculations.