Nashaat (نشاط) — Government Analytics & Intelligence Platform
Government analytics platform for turning fragmented education data into actionable decision intelligence.
What was broken before this system existed?
Educational leadership was burdened with 13+ disparate, unstandardized Arabic Excel spreadsheets from Microsoft Forms and manual administrative files across 100+ schools. Generating district-level reports took days of manual effort, riddled with 200+ free-text spelling variations of school names, missing records, and conflicting column schemas.
What did I actually engineer and own?
As the Full-Stack Developer & Data Engineer, I owned the system end-to-end: from initial requirements definition and architectural design to database schema modeling, backend pipeline development, and final production deployment.
Engineered a centralized, automated 3-tier ETL and analytics platform. Built a custom Arabic NLP normalization engine to unify noisy text inputs, automated header-detection algorithms for zero-configuration ingestion, an asynchronous calculation engine with severity-graded decision intelligence, and a responsive RTL executive command dashboard.
How the system is structured
System Design
Built a robust three-tier Django architecture to process and visualize extensive educational survey data. The data layer handles dynamic ingestion and normalization across multiple operational categories. The presentation layer features a comprehensive custom analytics engine responsible for processing Likert scales, Arabic NLP, barrier classification, trend analysis, and automated recommendation generation. The system is secured by a custom authentication flow with strict role-based access control.
Architecture & Database Schema
Engineered a relational database schema utilizing dynamic field patterns to enable automated column mapping during data ingestion. Implemented advanced UPSERT logic using hash-based unique constraints to ensure data integrity during continuous operational tracking updates.
Data Pipeline
Designed an automated ETL (Extract, Transform, Load) pipeline: Unstructured File Ingestion → Multi-Strategy Header Detection → Dynamic Field Mapping → Arabic Text Normalization → Database UPSERT → Analytics Engine → JSON Payload Generation → Client-Side Rendering.
Frontend Architecture
Developed a server-rendered Django application paired with Tailwind CSS for a fully responsive, RTL Arabic layout. Integrated ApexCharts.js for dynamic, client-side data visualization. Extended the UI with Flowbite components and custom typography optimized for formal Arabic governmental standards.
Key Technical Challenges Solved
Unstructured Data Ingestion: Engineered a robust parsing service capable of handling complex, unstructured spreadsheets with merged cells and embedded metadata, utilizing multi-strategy header detection and forward-filling algorithms.
Advanced Arabic NLP & Normalization: Resolved massive data fragmentation (hundreds of spelling variations across tens of thousands of records) by building a keyword-based canonical mapping algorithm that normalizes Arabic text without requiring exact string matches.
Accessible UX for Administrative Staff: Built a zero-configuration upload flow with automated file-type identification and parsing, allowing administrative teams to securely ingest multiple data formats without requiring technical intervention.
Automated Insights: Transformed raw survey data into automated, Arabic-language recommendations with algorithmic severity grading, instantly providing actionable intelligence to decision-makers.
Why this architecture? Tradeoffs & judgment
1. Why Relational Schema with Dynamic Model Mapping: Avoided document stores because reporting required strict relational integrity, cross-semester comparative joins, and deterministic aggregation across schools.
2. Why Custom Keyword Canonicalization over Embeddings: Production constraint required sub-100ms instant normalization for hundreds of free-text school names without external cloud LLM latency or non-deterministic token drift.
3. What happens at 10x scale: Ingestion pipeline was designed with MD5-hash idempotent UPSERT logic and chunked Pandas streaming; scaling to 200k+ records would move task processing to Celery workers with PostgreSQL partitioning.
4. What was deliberately NOT built: Did not build a heavy drag-and-drop spreadsheet builder or complex microservices; a clean modular Django monolith satisfied strict government on-premise security and single-operator maintainability.
Hardest engineering hurdles
Handling merged cells and unaligned headers in legacy administrative spreadsheets without human intervention. Solved by developing forward-fill heuristic scanners that detect title blocks and dynamic table boundaries automatically.
Intelligent Multi-Format Data Ingestion
Built a three-tier Excel parsing pipeline capable of handling 13+ distinct Arabic-language data file types — from structured MS Forms exports (up to 5,000 records each) to completely unstructured, merged-cell management spreadsheets. The system auto-detects header rows, dynamically maps Excel columns to database fields using Django model introspection, and handles semester-aware data tagging without requiring any configuration per file type.
Custom Arabic NLP & Text Normalization Engine
Developed a comprehensive Arabic text processing pipeline to process high-variance, unstructured free-text survey responses:
Letter form unification: Normalizes أ/إ/آ → ا, ة → ه, ى → ي across all input
School name canonicalization: A 140+ entry keyword-matching system that resolves 200+ unique free-text spelling variations of school names into canonical forms — handling typos, missing letters, and alternative Arabic spellings without exact matching
Barrier categorization: Automatically classifies free-text non-participation reasons into actionable categories (timing, transportation, awareness, etc.)
Interest mapping: Groups diverse activity suggestions into strategic categories (sports, technology, arts, religious, scientific)
4,700+ Line Analytics Engine with Decision Support
The analytics core processes every data category with its own specialized analytical pipeline. For student satisfaction alone, it:
Computes weighted averages across 9 Likert-scale dimensions
Generates per-school and per-grade satisfaction rankings
Performs semester-over-semester comparative analysis detecting dimension-level regressions
Auto-generates severity-graded decision recommendations (critical/warning/excellent) based on computed thresholds
Outputs radar charts, treemaps, horizontal bars, grouped comparisons, radial gauges, and participation donuts — all from real data, zero hardcoded values
Executive Dashboard with Real-Time KPIs
A command-center style overview showing total system records, meeting completion rates, satisfaction survey volumes, field visit tracking, and event report counts — with dynamic year-based filtering, contextual alert banners, and drill-down navigation to 16 specialized analytics pages.
Operational Task Management with UPSERT Logic
Implemented an MD5 hash-based deduplication system for operational plan tasks that supports idempotent re-imports — uploading the same tracking spreadsheet multiple times updates existing records rather than creating duplicates, enabling weekly status refresh workflows.
Event Image Gallery
A searchable, paginated image management system for documenting events with per-event and per-date filtering, supporting bulk upload with automatic metadata association.
Premium Arabic RTL Dashboard
A production-quality, fully right-to-left Arabic interface featuring:
Hover-expanding sidebar (80px → 256px with animated icon-to-label transitions)
Glassmorphism navigation with backdrop blur
Custom color palette (Burgundy, Gold, Navy) reflecting governmental branding
Responsive mobile-first design with drawer navigation
Micro-animations (fadeInUp, slideInRight, hover-lift)
Quantified results & production impact
Eliminated manual reporting: Replaced 13+ separate Excel-based reporting workflows with a single automated platform, reducing report generation from days to seconds
Scaled to 20,000+ records: Ingested and analyzed 10,000+ student satisfaction surveys + 10,000+ parent/staff feedback responses across two academic semesters, providing insights across 100+ schools
Automated decision intelligence: The system autonomously identifies critical issues (e.g., satisfaction drops, low participation rates, underperforming schools) and generates Arabic-language recommendations with severity levels — previously requiring manual expert analysis
School-level accountability: Enabled granular satisfaction scoring for every school in the system, identifying specific institutions requiring intervention based on data rather than anecdotal reports
Edited:
School-level analytics: Enabled granular satisfaction scoring for every school in the system, identifying specific institutions requiring targeted intervention based on standardized data models rather than subjective reporting.
140+ school name normalizations: Resolved the data quality problem of 200+ unique free-text spellings for school names, making school-level analytics possible for the first time
Edited:
140+ school name normalizations: Resolved high-variance unstructured text inputs by normalizing 200+ unique spelling variations of school names, enabling precise, centralized school-level analytics.
Zero-configuration file types: Adding a new MS Forms survey type requires only defining a Django model — the import pipeline, field mapping, and analytics adapt automatically
What I would change at 10x scale
Real-world data is infinitely dirtier than clean benchmark datasets. Investing in deterministic validation rules upfront saved weeks of downstream debugging in aggregate calculations.