Schools Database
A comprehensive system for managing and analyzing Australian school data. Import CSV files, run scraping tasks, and search or export results.
Getting Started
1. Request Access
To use the Schools Database, you need to be logged in as a superuser.
-
Click the "Login" link in the top navigation
- If you don't have an account, contact the AFR datajournalism team
- Once logged in, you'll have access to all features of the application
2. Import Schools Data
Start by importing school data from the Acara website.
-
Navigate to Import in the main menu
- Select a year you want to import for. It should be a year that has not already been imported.
- The system will import the current list of Australian schools from Acara.
- Monitor the import progress in real-time using the task panel
3. Scrape School Data
After importing, run scraping tasks to fetch the latest school information.
- Click on the year you want to scrape, from the table of imported years. The row should open up to reveal the scrape tools.
- Run the scraping tasks for Profile, Finances, and NAPLAN pages
- Monitor progress and view screenshots for each scraping step
4. Search and Export
Use the SQL Query Tool to analyze your processed data.
-
Navigate to Query Tool in the main menu
- Write and execute SELECT queries against your data
- View results in a paginated table format
- Export query results to CSV for further analysis
- Use the query history to reuse previous queries
Security & Performance Notes
- Only SELECT queries are allowed in the Query Tool
- Queries are limited to 45 seconds execution time and 100,000 rows
- All data processing happens asynchronously in the background
- Real-time progress updates are provided via WebSocket connections
Quick Links
Data Tables
The system manages the following main data entities:
-
Schools – Core table for school records and metadata (name, IDs, type, state, etc.)
-
Addresses – Physical addresses for each school (city, state, postal code, country, coordinates)
-
Import Events – Tracks each CSV import batch (year, timestamp, file, scrape options)
-
Import Queue – Queue of scraping tasks for each school in an import (status, progress, retries, notes)
-
School Profiles – Yearly profile data for each school (sector, type, staff, enrolments, student background, etc.)
-
School Finances – Yearly finances for each school (income, funding, capital expenditure, per-student amounts)
-
School NAPLAN – Yearly NAPLAN results for each school (scores, participation rates, national averages, error margins)
System Architecture
The Schools Database is built on Django with the following key components:
-
PostgreSQL - Primary data storage
-
Redis - Task queue and caching
-
Celery - Background task processing
-
Django Channels - Real-time WebSocket updates
-
Bootstrap - Responsive UI framework
All data processing tasks run asynchronously, providing real-time progress updates through WebSocket connections. The system is designed for high performance and reliability when handling large datasets.
Version Information
- App Version
- 2.1.0
- Core Version
- 1.1.8
- Git Commit
f9f17f7
- Dirty State