Career Pathfinder Engine PoC

This is the companion notebook to the Pathfinder app. It is meant to give some insights about the nature and shape of the data and give some cluse about the design decisions that have been taken along the way.

A brief look at the raw data

We basically use 4 datasets coming from the usual places: Kaggle, Huggingface, StackOverflow. The jobs, the courses, world cities geo information, and the StackOverflow 2025 survey.

The job offers are from Kaggle

https://www.kaggle.com/datasets/asaniczka/linkedin-data-engineer-job-postings

(def jobs
  (load/load-dataset (jobs/job-postings-path)))

(ds-info jobs)

here is an example

(-> (tc/random jobs 5)
    (tc/drop-columns [:job-summary]))

The jobs contain synthetic data that doesn't always make much sense for our specific presentation:

(-> (load/load-dataset (jobs/job-postings-path) :column-whitelist ["job_location" "search_city" "search_country"])
    (tc/rename-columns {:job-location :location})
    (tc/drop-missing [:location])
    (tc/group-by [:location])
    (tc/aggregate {:count tc/row-count})
    (tc/order-by :count :desc))

We'll take care of that, augmenting this dataset with European cities and geo locations, below.

The courses are from Kaggle

(def courses
  (load/load-dataset (courses/courses-path)))

(ds-info courses)

here is an example

(tc/random courses 5)

The cities from SimpleMaps

They will "complete" the job offers and let us map between different naming conventions https://simplemaps.com/data/world-cities

(def cities
  (load/load-dataset (cities/worldcities-path)))

(ds-info cities)

here's an example

(tc/random cities 10)

The market trends from StackOverflow

We gather insights on the hottest technologies from StackOverflow, which famously performs a yearly survey.

(def survey
  (load/load-dataset (trends/tech-survey-path) :num-rows 20))

(ds-info survey)

here's an example

(tc/random survey 5)

The data will need major massaging before being useful to Pathfinder.

What ends up in the database

Some of this data may conceivably grow a bit too large to keep in memory, so we store the clean and adjusted data in DuckDB. DuckDB has been chosen for its native support of vector embeddings.

This is how the jobs table looks like

(count-table-rows "jobs")

6025

(-> (jdbc/execute "select * from jobs limit 5")
    tc/dataset
    (tc/drop-columns [:embedding])
    (tc/map-columns :description
                    [:description]
                    (fn [desc]
                      (if (> (count desc) 200)
                        (str (subs desc 0 200) " [...]")
                        desc))))

The job offers have been relocated. That is, they have been assigned a major European city according to its population. This gives predicatble distribution across familiar places.

(-> (jdbc/execute "select location,latitude,longitude from jobs")
    tc/dataset
    jobs/plot-job-offers)

The courses data looks like this

(count-table-rows "courses")

6403

(-> (jdbc/execute "select * from courses limit 5")
    tc/dataset
    (tc/drop-columns [:embedding]))

The technology trends is a distillation of the StackOverflow survey data. Once reduced to it's bare essentials it isn't that unwieldy any more.

(ds-info @trends/trends-ds)

here is an example

(tc/random @trends/trends-ds 10)

How to interpret the data

The desirability index may require some explanation. This index tells how a technology is desired (sparks geunine user interest) compared to how often people are paid to use it:

• The Growth Drivers (Ratio > 1.1): Languages like Rust, Go, or specialized AI frameworks. These represent technologies where community excitement outstrips current enterprise adoption. This has likely a high future demand and talent scarcity, which typically commands a salary premium.

• The Market Anchors (Ratio 0.4 – 0.7): The baseline industry giants like JavaScript, Java, and SQL. They have massive have-count values but lower ratios. This demonstrates the "Enterprise Lock-in" concept; these are the bedrock of the job market where real-world hiring volume is guaranteed, even if the developer "hype" has moved on.

• The Atrophy Zone (Ratio < 0.3): The legacy stacks or fading frameworks. If a technology has a low count, low salary, and a plunging ratio, it tells a clear story of a shrinking ecosystem where finding modern, lucrative roles will become increasingly difficult.

The Tech Landscape

The StackOverflow's survey covers a whole lot of technologies across several countries. We take care of taking a picture of a localised reality as to avoid bunching together wildly different data groups.

For Germany:

(trends/plot-trends
 (tc/select-rows @trends/trends-ds
                 (fn [row]
                   (and (= (:year row) 2025)
                        (> (:have-count row) 400)
                        (= "Germany" (:country row))))))

For USA:

(trends/plot-trends
 (tc/select-rows @trends/trends-ds
                 (fn [row]
                   (and (= (:year row) 2025)
                        (> (:have-count row) 900)
                        (= "United States of America" (:country row))))))

Which exposes the typical IT conundrum that sees fun techonology as the least in demand (by employers) and the most adopted technology among the least desirable (by developers).

The Pay Gap

Her is how the surveys distribute incomes across the globe:

(trends/plot-survey-geo-distribution
 (trends/join-so-data-with-centroids
  (trends/extract-country-overview (trends/tech-survey-path))))

Implementation details

StackOverflow's data covers a large number of countries and it looks different from region to region. For this reason the indices calculated by the app are limited to the area where the user looks for job opportunities. For each skill that is recognised in the course a number of indices are calculated. Eventually those indices are compounded (averaged) for the course.

The calculated indexed are:

• desirability (appeal)
• use in actual job setting (adoption)
• expected salary or, rather, the median salary of people using the same skill professionally

Architecture diagrams

Odd looking boxes connected by squiggly lines

Data flow

Although the data comes from disparate sources, it is all ingested in DuckDB tables for easy access.

pipeline-diagram

User Interface

How the app should behave, when it behaves.

ui-diagram

EOF

source: notebooks/notebook.clj