ADMA System Documentation

Welcome to the Agricultural Data Management & Analytics (ADMA) system documentation. This comprehensive guide covers all aspects of the platform designed for managing, visualizing, and analyzing agricultural data.

File Management

Upload, organize, and manage files with hierarchical folder structures and advanced metadata handling.

GIS Integration

Visualize spatial data with integrated GeoServer, supporting multiple coordinate systems and file formats.

Data Sharing

Share files and folders publicly or privately with granular access controls and collaboration features.

Third-Party Integration

Automatic sync with John Deere Operations Center and Realm5 weather stations for comprehensive farm data.

Analysis Tools

Specialized agricultural analysis tools including seeding analysis, format conversion, and spectral indices.

Custom Maps

Create multi-layer maps by combining GIS files for comprehensive spatial visualization and analysis.

Key Features

Cloud File Storage GIS Data Processing Map Visualization Public/Private Sharing Folder Hierarchies Duplicate Handling Pagination Support Token-Based API ZIP Download Bulk Upload API Authentication John Deere Integration Realm5 Weather Data Agricultural Analysis Tools Custom Map Creation

Getting Started

System Requirements

Component	Requirement	Notes
Web Browser	Chrome 90+, Firefox 88+, Safari 14+	JavaScript must be enabled
Internet Connection	Broadband recommended	For file uploads and map tiles
Screen Resolution	1024x768 minimum	Responsive design supports mobile

User Registration & Login

1 Registration: Click "Register" in the top navigation to create a new account with username, email, and password.

2 Login: Use your credentials to access the dashboard and all authenticated features.

3 Dashboard: Your personalized workspace for managing files, folders, and data visualization.

Tip: Use a strong password and remember your credentials. Password recovery features are available through the admin interface.

File Management

Supported File Types

GIS Files

GeoPackage (.gpkg) - Complete spatial databases
GeoJSON (.geojson) - Vector data format
ESRI Shapefile (.shp + components) - Industry standard vector format
KML/KMZ (.kml, .kmz) - Google Earth formats
GeoTIFF (.tif, .tiff) - Raster data with georeference
CSV with coordinates - Tabular spatial data
3D Tiles - 3D visualization data

General Files

Documents: PDF, DOC, DOCX, TXT, MD
Spreadsheets: XLS, XLSX, CSV
Images: JPG, PNG, GIF, BMP, SVG
Archives: ZIP, RAR, 7Z
Videos: MP4, AVI, MOV
Other: JSON, XML, LOG

File Upload Methods

Individual File Upload

1 Click "Upload Files" button from dashboard or within any folder

2 Drag and drop files or click to browse and select multiple files

3 Choose public/private visibility setting

4 Click "Upload Files" to start the upload process

Folder Upload

1 Click "Upload Folders" button (available in dashboard and all folders)

2 Select entire folder structures with all nested files and subfolders

3 Preview folder structure and file count before upload

4 Upload preserves complete directory hierarchy

File Operations

Operation	Location	Description
Download	File actions menu (⋮)	Download original file to your device
Delete	File actions menu (⋮)	Permanently remove file from system
View Details	Click file name	See metadata, properties, and spatial information
Toggle Visibility	File detail page	Switch between public and private access
View on Map	File actions menu (⋮)	Visualize GIS files on interactive map

Duplicate File Handling

When uploading files with names that already exist, the system automatically renames them:

Original: document.pdf
Duplicate 1: document_1.pdf
Duplicate 2: document_2.pdf
Duplicate 3: document_3.pdf
                

Note: File renames preserve the original extension and maintain uniqueness within each folder context.

Folder Management

Folder Operations

Creating Folders

1 Click "New Folder" button from dashboard or within any existing folder

2 Enter folder name and choose visibility (public/private)

3 Folder is created in current location with unique name if duplicates exist

Folder Hierarchy

ADMA supports unlimited nested folder structures:

📁 Research Project 2024/
  ├── 📁 Data Collection/
  │   ├── 📁 Field Surveys/
  │   │   ├── 📄 survey_data.csv
  │   │   └── 📄 gps_coordinates.gpkg
  │   └── 📁 Satellite Images/
  │       ├── 📄 landsat_2024.tif
  │       └── 📄 ndvi_analysis.tif
  ├── 📁 Analysis Results/
  │   ├── 📄 statistical_summary.pdf
  │   └── 📄 yield_prediction.xlsx
  └── 📁 Reports/
      ├── 📄 final_report.docx
      └── 📄 presentation.pptx
                

Navigation Features

Breadcrumb Navigation: Shows current location path with clickable links
Folder Statistics: Displays file count and subfolder count for each folder
View Modes: Toggle between list view and panel view for optimal browsing
Pagination: Large folders display 100 items per page with navigation controls

Duplicate Folder Handling

When creating or uploading folders with existing names:

Entire folder structure is renamed (not merged)
Original folder contents remain unchanged
New folder gets incremental suffix: MyFolder → MyFolder_1
All files within renamed folders maintain their original names

GIS Features

Automatic GIS Processing

ADMA automatically detects and processes GIS files upon upload:

1 File Detection: System identifies spatial file formats automatically

2 Background Processing: Celery workers process files asynchronously

3 GeoServer Integration: Processed files are published to GeoServer for visualization

4 Metadata Extraction: Coordinate system, extent, and properties are extracted

Supported Coordinate Systems

EPSG Code	Name	Usage
EPSG:4326	WGS 84	Global geographic coordinates (lat/lon)
EPSG:3857	Web Mercator	Web mapping applications
EPSG:32614	UTM Zone 14N	Nebraska region projections
Custom	Auto-detected	System detects and handles various projections

GIS File Status

Status	Description	Actions Available
Pending	File uploaded, queued for processing	View details
Processing	File being analyzed and prepared	View details
Processed	Analysis complete, ready for publishing	View details, manual publish
Published	Available in GeoServer for visualization	View details, view on map
Error	Processing failed	View error log, retry

Shapefile Processing

Shapefiles require multiple component files and receive special handling:

Required Components: .shp, .shx, .dbf files must all be present
Optional Components: .prj (projection), .cpg (encoding), .sbn/.sbx (spatial index)
Automatic Bundling: System groups scattered components and creates proper shapefile packages
Layer Naming: Published layers use original shapefile name for consistency

Map Visualization

Map Interface Features

Base Layers

OpenStreetMap (default)
Satellite imagery
Topographic maps
Custom base layers

Map Controls

Zoom in/out controls
Pan and navigate
Full screen mode
Layer selector dropdown

Data Visualization

ADMA uses OpenLayers for robust map visualization with the following capabilities:

Vector Data: Points, lines, and polygons from shapefiles and GeoJSON
Raster Data: Satellite imagery, DEMs, and analytical rasters from GeoTIFF
Dynamic Extent: Map automatically zooms to data bounds
Coordinate Transformation: Automatic reprojection to web display coordinates
Attribution: Proper data source attribution displayed

Map Information Panel

The right sidebar provides detailed information:

1 File Information: Name, size, type, owner, creation date

2 Spatial Information: Coordinate system, spatial extent, layer details

3 Processing Status: Current status and processing log

4 Actions: Download, toggle visibility, file management

Tip: Use the "Test WMS" button to verify that your GIS data is properly served by GeoServer and displaying correctly.

User Management

User Roles & Permissions

Permission	Owner	Public User	Anonymous User
Upload files/folders	✅	✅ (own content)	❌
Create folders	✅	✅ (own content)	❌
Delete content	✅ (own content)	✅ (own content)	❌
View public content	✅	✅	✅
View private content	✅ (own content)	❌	❌
Toggle visibility	✅ (own content)	✅ (own content)	❌

User Dashboard

Each authenticated user has access to a personalized dashboard featuring:

File Statistics: Total files, folders, storage used, public files count
Quick Actions: Create folder, upload folders, upload files
Recent Files: Latest uploaded or modified files
Folder Navigation: Browse personal folder hierarchy
View Toggle: Switch between list and panel view modes

Admin Features

System administrators have additional capabilities through Django Admin:

User account management
System-wide file and folder management
GeoServer configuration
Database maintenance and monitoring
Celery task monitoring

Third-Party Integration

ADMA integrates with external agricultural data platforms to automatically sync field data, boundaries, and sensor observations.

John Deere Operations Center

John Deere Integration

ADMA connects to John Deere Operations Center API to automatically sync field data, boundaries, and field operations.

Data Synced from John Deere

Field Data

Field Metadata: Field name, ID, and properties saved as JSON
Field Boundaries: Geographic boundaries saved as shapefiles (.shp, .shx, .dbf, .prj, .cpg)
Automatic GIS Processing: Boundaries are automatically published to GeoServer for visualization

Field Operations

Operation Metadata: Operation type, dates, and details saved as JSON
Operation Boundaries: Coverage areas saved as shapefiles
Operation Types: Planting, harvesting, tillage, application operations

Folder Structure

John Deere data is organized in a structured folder hierarchy:

📁 Third Party/
  └── 📁 John Deere/
      └── 📁 {Field Name}/
          ├── 📄 {Field Name}_metadata.json
          ├── 📁 {Field Name}_boundary/
          │   ├── 📄 {Field Name}_boundary.shp
          │   ├── 📄 {Field Name}_boundary.shx
          │   ├── 📄 {Field Name}_boundary.dbf
          │   ├── 📄 {Field Name}_boundary.prj
          │   └── 📄 {Field Name}_boundary.cpg
          └── 📁 field_operations/
              └── 📁 {Operation Type}_{Date}/
                  ├── 📄 operation_metadata.json
                  └── 📁 {Operation}_boundary/
                      └── 📄 {Operation}_boundary.shp (+ components)
                

Sync Schedule

John Deere data sync runs automatically on a daily schedule via Celery Beat. The sync is incremental - only new fields and operations are added.

Realm5 Weather Stations

Realm5 Integration

ADMA connects to Realm5 API to sync weather station data and daily observations for agricultural monitoring.

Data Synced from Realm5

Device Information: Weather station metadata and configuration
Daily Observations: Weather data saved as JSON files (one per day)
Historical Data: Syncs observations starting from January 1, 2024
Sensor Readings: Temperature, humidity, precipitation, wind, and other environmental data

Folder Structure

📁 Third Party/
  └── 📁 Realm5/
      └── 📁 {Device Serial Number}/
          ├── 📄 device_info.json
          └── 📁 observations/
              ├── 📄 2024-01-01.json
              ├── 📄 2024-01-02.json
              └── 📄 ... (daily observation files)
                

Sync Schedule

Realm5 data syncs daily via Celery Beat. The sync fetches all historical data from January 1, 2024 onwards and continues to add new daily observations.

Note: Third-party sync tasks are configured by administrators. The sync respects API rate limits and may take several minutes to complete for large data sets.

Analysis Tools

ADMA provides specialized agricultural analysis tools for processing GIS data and generating insights.

Seeding Tool

Seeding Analysis Tool

Processes as-applied seeding data to generate polygon maps and summary statistics.

Features

Polygon Generation: Creates polygon shapefiles from seeding point data
Boundary Extraction: Generates field boundary shapefiles
Summary Statistics: Produces CSV summary reports with key metrics
Background Processing: Runs asynchronously via Celery for large datasets

Output Files

Output	Format	Description
Polygons	Shapefile (.shp + components)	Polygon representation of seeding data
Boundary	Shapefile (.shp + components)	Field boundary extracted from data
Summary	CSV	Statistical summary and metrics

How to Use

1 Navigate to a compatible GIS file (shapefile with seeding data)

2 Click the "Seeding Tool" button in the file actions panel

3 Select the output folder for generated files

4 The tool runs in the background and generates output files automatically

Shape to JSON Converter

Shapefile to GeoJSON Conversion

Converts ESRI Shapefile format to GeoJSON for web-friendly data interchange.

Features

Format Conversion: Transforms .shp files to .geojson
Attribute Preservation: Maintains all feature attributes
Coordinate Systems: Handles coordinate transformations
Web Compatible: Output is directly usable in web mapping applications

SI Tool (Spectral Index Tool)

Spectral Index Analysis

Calculates spectral indices from multispectral imagery for crop health monitoring.

Features

Index Calculation: Computes vegetation indices (NDVI, etc.)
Raster Processing: Handles GeoTIFF input files
Output Generation: Creates indexed raster outputs

Tip: Analysis tools require user authentication. The "Run Tool" buttons appear on compatible file detail pages.

Custom Maps

ADMA allows you to create custom maps by combining multiple GIS layers into a single interactive visualization.

Creating Custom Maps

Multi-Layer Map Creation

Combine multiple published GIS files into a single map view for comprehensive data visualization.

How to Create a Map

1 Navigate to Create Map: Click "Create Map" in the navigation menu

2 Enter Map Name: Provide a descriptive name for your map

3 Select Layers: Browse your folders and select published GIS files to include

4 Create Map: Click the "Create Map" button to generate the map

5 View Map: The system redirects to the interactive map viewer

Layer Selection

Folder Navigation: Browse through your folder hierarchy to find layers
Published Files Only: Only GIS files with "Published" status can be added as layers
Multiple Selection: Select multiple files by clicking the checkboxes
Layer Count: The interface shows how many layers are currently selected

Supported Layer Types

Layer Type	File Formats	Description
Vector Layers	.shp, .geojson, .gpkg, .kml	Points, lines, and polygon features
Raster Layers	.tif, .tiff (GeoTIFF)	Satellite imagery, DEMs, analysis outputs

Map Features

Layer Toggle: Show/hide individual layers on the map
Layer Ordering: Layers are displayed in the order they were added
Auto Extent: Map automatically zooms to fit all layer bounds
Base Maps: Multiple base map options (OpenStreetMap, satellite, etc.)

Managing Maps

Your created maps are accessible from the Maps section:

View Maps: Open any saved map in the interactive viewer
Map List: Browse all your created maps
Public Maps: Maps can be made public for sharing

Tip: When creating a map, a confirmation message appears showing the map name and layer count. The map is created in the background and you'll be redirected to view it once ready.

API Endpoints

Token-Based REST API

ADMA provides comprehensive token-based REST APIs for programmatic access:

Authentication: All API endpoints use token authentication. Include Authorization: Token your_token_here in request headers.

Authentication Endpoints

Endpoint	Method	Purpose	Authentication
/api/v1/auth/token/	POST	Create authentication token	Username/Password

File Management APIs

Endpoint	Method	Purpose	Parameters
/api/v1/files/upload/	POST	Upload multiple files	files, folder_id, is_public
/api/v1/files/	GET	List user's files	folder_id, is_public, file_type
/api/v1/files/{id}/download/	GET	Download specific file	None

Folder Management APIs

Endpoint	Method	Purpose	Parameters
/api/v1/folders/upload/	POST	Upload folder structure	files, file_paths, folder_id, is_public
/api/v1/folders/	GET	List user's folders	parent_id, is_public
/api/v1/folders/{id}/info/	GET	Get folder information	None
/api/v1/folders/{id}/download/	GET	Download folder as ZIP	include_subfolders

API Usage Examples

# Get authentication token
curl -X POST /api/v1/auth/token/ \
  -H "Content-Type: application/json" \
  -d '{"username": "user", "password": "pass"}'

# Upload files with token
curl -X POST /api/v1/files/upload/ \
  -H "Authorization: Token your_token_here" \
  -F "files=@document.pdf" \
  -F "is_public=true"

# Download folder as ZIP
curl -X GET /api/v1/folders/{folder_id}/download/ \
  -H "Authorization: Token your_token_here" \
  -o folder.zip
                

Web Interface AJAX Endpoints

Internal endpoints used by the web interface:

Endpoint	Method	Purpose	Authentication
/api/folder/create/	POST	Create new folder (web UI)	Session
/api/files/upload/	POST	Upload files (web UI)	Session
/api/folders/upload/	POST	Upload folders (web UI)	Session
/api/item/delete/	POST	Delete files or folders	Session
/api/item/toggle-visibility/	POST	Change public/private status	Session
/api/search/	GET	Search files and folders	Session

GeoServer Integration

GIS data is served through GeoServer REST API:

WMS URL: https://adma.aisoup.net/geoserver/wms
Workspace: adma_geo
Layer Format: {workspace}:{layer_name}
GetCapabilities: ?service=WMS&request=GetCapabilities
                

File Download URLs

File Detail: /file/{file_id}/
File Download: /file/{file_id}/download/
Map Viewer: /file/{file_id}/map/
Public File: /public/file/{file_id}/
Public Map: /public/file/{file_id}/map/
                

API Documentation

Complete API Reference

ADMA provides comprehensive token-based REST APIs for programmatic access to all system features. The complete API documentation includes detailed examples, error handling, and integration guides.

API Documentation File

The complete API documentation is available as a detailed markdown file in the project repository:

Location: /adma_geo/API_DOCUMENTATION.md
Format: Markdown with code examples
Content: 500+ lines of comprehensive documentation
                    

Key API Features

Token Authentication

Secure token-based authentication
Create tokens via username/password
Long-lived persistent tokens
Management commands for token creation

File Operations

Upload single or multiple files
Upload complete folder structures
Download individual files
Download folders as ZIP archives

Folder Management

List folders with filtering
Get folder information and statistics
Preserve folder hierarchy on upload
Recursive subfolder inclusion

Advanced Features

Comprehensive error handling
Query parameter filtering
JSON response format
RESTful design principles

API Usage Workflow

1 Authentication: Obtain API token using username and password via POST /api/v1/auth/token/

2 File Upload: Upload files using POST /api/v1/files/upload/ with token authentication

3 File Management: List, filter, and manage files using GET /api/v1/files/

4 Download: Download files or folders using respective download endpoints

Python Integration Example

import requests

# Get API token
response = requests.post('http://localhost/api/v1/auth/token/', json={
    'username': 'your_username',
    'password': 'your_password'
})
token = response.json()['token']

# Set headers for API calls
headers = {'Authorization': f'Token {token}'}

# Upload files
with open('data.csv', 'rb') as f:
    files = {'files': f}
    data = {'is_public': 'true'}
    response = requests.post(
        'http://localhost/api/v1/files/upload/',
        headers=headers,
        files=files,
        data=data
    )

# List files
response = requests.get(
    'http://localhost/api/v1/files/',
    headers=headers
)
files = response.json()['files']

# Download folder as ZIP
response = requests.get(
    'http://localhost/api/v1/folders/{folder_id}/download/',
    headers=headers
)
with open('folder.zip', 'wb') as f:
    f.write(response.content)
                

Complete Documentation: For detailed API reference including all parameters, response formats, error codes, and advanced examples, refer to the API_DOCUMENTATION.md file in the project repository.

API Management Commands

Command	Purpose	Example
create_api_token	Create token for user	`python manage.py create_api_token username`
create_api_token --regenerate	Regenerate existing token	`python manage.py create_api_token username --regenerate`

System Architecture

Technology Stack

Backend Technologies

Django 4.2: Web framework and ORM
PostgreSQL: Primary database with PostGIS extension
Redis: Caching and Celery message broker
Celery: Asynchronous task processing
GeoServer: Spatial data server and WMS
GDAL/OGR: Geospatial data processing libraries

Frontend Technologies

Bootstrap 5: UI framework and responsive design
OpenLayers: Interactive mapping library
Font Awesome: Icons and visual elements
JavaScript ES6+: Modern client-side scripting
AJAX/Fetch: Asynchronous data operations
CSS3: Advanced styling and animations

Service Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│     Nginx       │    │     Django      │    │   PostgreSQL    │
│  Load Balancer  │◄──►│   Web Server    │◄──►│    Database     │
│   Static Files  │    │   Application   │    │   + PostGIS     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                │
                                ▼
                       ┌─────────────────┐    ┌─────────────────┐
                       │     Celery      │    │     Redis       │
                       │  Task Workers   │◄──►│ Message Broker  │
                       │   GIS Processor │    │     Cache       │
                       └─────────────────┘    └─────────────────┘
                                │
                                ▼
                       ┌─────────────────┐
                       │   GeoServer     │
                       │   WMS Server    │
                       │ Spatial Engine  │
                       └─────────────────┘
                

Data Flow

1 File Upload: User uploads files through Django web interface

2 Storage: Files stored in filesystem, metadata in PostgreSQL

3 GIS Detection: Django detects spatial files and queues processing

4 Background Processing: Celery workers process GIS files

5 GeoServer Publishing: Processed files published to GeoServer

6 Map Visualization: OpenLayers displays data via WMS

Deployment

ADMA is containerized using Docker Compose for easy deployment:

services:
  - django: Web application server
  - postgres: Database with PostGIS
  - redis: Cache and message broker
  - celery: Background task processor
  - nginx: Web server and reverse proxy
  - geoserver: Spatial data server
                

Troubleshooting

Common Issues

File Upload Problems

Issue: "Error uploading folders: Unexpected token"
Solution: Reduce number of files per upload or check server file limits

Issue: Upload button not working in folders
Solution: Refresh page, check browser console for JavaScript errors

GIS File Processing

Issue: Shapefile stuck in "Pending" status
Solution: Ensure all required components (.shp, .shx, .dbf) are uploaded

Issue: Map not displaying data
Solution: Verify file is in "Published" status and coordinate system is supported

Map Visualization Issues

Issue: Map tiles not loading
Solution: Check internet connection and GeoServer availability

Issue: Attribution text in wrong position
Solution: Refresh page, browser may need time to position elements

Performance Optimization

Large Files: Upload large GIS files during off-peak hours
Batch Uploads: Use folder upload for multiple related files
Browser Cache: Clear browser cache if interface appears outdated
Pagination: Use page navigation for folders with many items

Getting Help

Check browser console (F12) for JavaScript errors
Verify file formats are supported
Ensure adequate internet connection for uploads
Contact system administrator for persistent issues

Frequently Asked Questions

General Usage

Q: What's the maximum file size I can upload?

A: The system supports large file uploads, but practical limits depend on your internet connection and server configuration. For very large GIS files (>100MB), consider compressing or splitting the data.

Q: Can I upload folders with nested subfolders?

A: Yes, the folder upload feature supports unlimited nesting levels and preserves the complete directory structure.

Q: How do I share my data with others?

A: Set files or folders to "Public" in their detail pages. Public content appears on the home page and can be accessed via direct links without authentication.

GIS and Mapping

Q: Which coordinate systems are supported?

A: ADMA automatically handles most common coordinate systems including WGS84, Web Mercator, and UTM zones. The system detects and transforms coordinates as needed.

Q: Why is my shapefile not displaying on the map?

A: Ensure all required shapefile components are uploaded (.shp, .shx, .dbf minimum) and wait for processing to complete. Check that the file status shows "Published" before attempting map visualization.

Q: Can I view my data in other mapping applications?

A: Yes, published GIS data is served via standard WMS protocols and can be accessed by QGIS, ArcGIS, and other GIS software using the provided WMS URLs.

Technical Questions

Q: Is my data secure?

A: Private files are only accessible to their owners. The system uses Django's built-in authentication and authorization. Public files are intentionally accessible to anyone with the link.

Q: How long does GIS processing take?

A: Processing time depends on file size and complexity. Simple files process within seconds, while large or complex datasets may take several minutes. The system provides real-time status updates.

Q: Can I use the system on mobile devices?

A: Yes, ADMA features a responsive design that works on tablets and smartphones. Some features like folder uploads work best on desktop browsers.

Table of Contents

ADMA System Documentation

File Management

GIS Integration

Data Sharing

Third-Party Integration

Analysis Tools

Custom Maps

Key Features

Getting Started

System Requirements

User Registration & Login

File Management

Supported File Types

GIS Files

General Files

File Upload Methods

Individual File Upload

Folder Upload

File Operations

Duplicate File Handling

Folder Management

Folder Operations

Creating Folders

Folder Hierarchy

Navigation Features

Duplicate Folder Handling

GIS Features

Automatic GIS Processing

Supported Coordinate Systems

GIS File Status

Shapefile Processing

Map Visualization

Map Interface Features

Base Layers

Map Controls

Data Visualization

Map Information Panel

User Management

User Roles & Permissions

User Dashboard

Admin Features

Public Sharing

Visibility Settings

Private Content

Public Content

Sharing Workflows

Making Content Public

Public Home Page

Search & Filter

Pagination System

View Modes

List View

Panel View

Filter Options

Third-Party Integration

John Deere Operations Center

John Deere Integration

Data Synced from John Deere

Field Data

Field Operations

Folder Structure

Sync Schedule

Realm5 Weather Stations

Realm5 Integration

Data Synced from Realm5

Folder Structure

Sync Schedule

Analysis Tools

Seeding Tool

Seeding Analysis Tool

Features

Output Files

How to Use

Shape to JSON Converter

Shapefile to GeoJSON Conversion

Features

SI Tool (Spectral Index Tool)

Spectral Index Analysis

Features