Government

Employment Data Sync with Hunan’s Provincial Job Platform

A side-car sync hub that pushes increments by watermark without touching the original site; failures are searchable and resumable.

LiveGovernmentFocused projectPC console

Client: City-level public HR marketplace operatorUndisclosed by agreement
Industry: Government
Platforms: PC console
Scale: Focused project · 2020

01Background

The client operates a city-level public HR marketplace whose employer onboarding, job postings and job-fair registration all run on a legacy PHP recruitment site. Provincial rules also require employer, job and job-fair data to be reported to Xiang Jiuye, Hunan’s provincial employment platform — until now done by hand or with one-off scripts, so the two datasets kept drifting apart. The goal was to turn reporting into a timely, compliant and traceable process without touching the existing site.

Client: City-level public HR marketplace operator
Synced data: Employers, jobs, job fairs + reference dictionaries
Users: Operations admins, developers
Platform: Web admin console (desktop)

02Challenges

1Manual dual-track ops

The local site kept changing while provincial reports were patched by hand or ad-hoc scripts.

2Mismatched field formats

Internal category IDs, Unix timestamps and free-text addresses didn’t meet provincial code and date standards.

3No clear sync boundary

Nothing recorded where the last sync stopped; full re-pushes were slow, duplicated data and couldn’t resume.

4Invisible failures

API errors stayed in server logs — ops had no idea which employer or job failed to report.

The legacy site couldn’t change, yet the province needed standardized, timely, auditable data.

03Solution

Shanku built a standalone sync middle layer that sits beside the legacy site, reading its database without altering any front-end or review workflow. It handles incremental extraction, field mapping, batched delivery and per-record logging, so ops can see in the admin console how far each dataset has synced, which records failed and why.

Read-only, side-by-side

The sync service reads the legacy MySQL database cross-schema; all data entry and review stay on the original site.

Watermark-based sync

Employers, jobs and job fairs each keep their own high-water mark; only newer changes are pulled, in time order.

Mapping in SQL

Dictionary joins convert categories, education and districts to provincial codes; dates become YYYYMMDD, addresses are trimmed.

Business rules upfront

Four employer verification states map to the province’s valid/invalid; only jobs within three months of posting are sent.

Batched, logged per record

Records go out 10 per batch, each batch with its own ID; every result code is logged and the watermark advances per record.

Dictionaries first

Industry, education and region dictionaries are mirrored in full; each dataset runs on its own schedule or on demand.

04Core workflow

Watermark-based incremental sync

1
Read mark
Load the last synced timestamp
2
Extract
Query changes past the watermark
3
Map & clean
Apply provincial codes and formats
4
Batch
10 records per batch, unique batch ID
5
Sign & send
Signed call via the city exchange hub
6
Log
Store result code per record
7
Advance
Move the watermark, fetch next batch

The watermark advances after every record, so an interrupted run resumes from the last success without re-sending anything.

05Results

Zero legacy changes

The recruitment site and its review flow stay as-is; business tables only maintain a change timestamp.

Every record traceable

Each push carries a batch ID, record ID, result code and message — “was it sent, why did it fail” is one search away.

Resumable runs

Per-dataset watermarks advance record by record, so interrupted jobs resume instead of re-pushing everything.

Tunable by config

Batch size and schedules are configurable, test and production credentials are separate; rate-limit changes need no SQL edits.

06Screens

Screens are prototypes or rebuilt with sample data. They contain no real client data, amounts or accounts.

07Tech stack

Backend: Java 17Spring BootMyBatis-FlexCool-Admin
Frontend: Vue 3Element Plus
Data & integration: MySQL (cross-schema, read-only)Quartz schedulerApache HttpClientHMAC-SHA1 signing
Deployment: Standalone service, co-located with legacy siteSeparate test / production config