In [2]:
IPython.display.HTML('''<script>
code_show=true;
function code_toggle() {
if (code_show){
$('div.input').hide();
} else {
$('div.input').show();
}
code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click <a href="javascript:code_toggle()">here</a>.''')
Out[2]:
Outline¶
- Context
- Transforming data into a form for analysis
- Understanding the data
Context¶
Why were we doing this?¶
- City of Cape Town has a Data Strategy.
- City-wide initiative to improve how the City works with data.
- One part of the strategy (Data Capabilities) concerns City employees
- Need to understand "data-intensivity" of work
Caveats¶
- Use of Formal HR data
- Use of pre-trained models
* Qualification: For the purposes of brevity, administrative points have been removed
Transforming Data¶
In [3]:
IPython.display.HTML('./gordon_source_df.html')
Out[3]:
Cleaning¶
First, loading the model:
nlp = spacy.load('en_core_web_lg')
stop_words = {
"service", "delivery",
"function", "functions",
"orientation", "orientations",
"problem", "solving",
"cfadm", "cfpro", "cfuni", "cfsup", "cfart", "cfman", "cfart", "cftec",
"kpaa", "kpan",
"l1", "l2", "l3", "l4", "l5"
}
nlp.Defaults.stop_words |= stop_words
Embedding¶
hr_df.RowVector = hr_df.Row.apply(
lambda row: nlp(row).vector
)
- Takes $\approx$ 2 mins using 16 cores.
- 4 chunks per core, at least 10k entries per core
In [4]:
IPython.display.HTML('./gordon_source_wv_df.html')
Out[4]:
Reducing critera -> positions¶
Using centre of mass formula:
$$C = \frac{\sum_i^N{W_i X_i}}{\sum_i^N{W_i}}$$- $C$ - new position
- $N$ - Number of entries in row $i$
- $W_i$ - row $i$'s weight
- $X_i$ - row $i$'s vector
a few weighted averages later...
In [5]:
IPython.display.HTML('./gordon_cg_df.html')
Out[5]:
and a few more...
In [6]:
IPython.display.HTML('./gordon_position_df.html')
Out[6]:
But what does this actually look like?!?¶
In [7]:
IPython.display.HTML('./hr_translation_I.html')
Out[7]:
In [8]:
IPython.display.HTML('./hr_translation_II_na.html')
Out[8]:
Data-relevance Scoring¶
Relationship to data-intensive work¶
data_words = [
"data",
"gathering",
"processing",
"analysis",
"dissemination"
]
data_word_vectors = {
word: nlp(word.lower()).vector
for word in data_words
}
for word, word_vector in data_word_vectors.items():
score_df[f"{word.title()}Score"] = sklearn.metrics.pairwise.cosine_similarity(
numpy.vstack(score_df.PositionVector.values),
numpy.array([word_vector])
)
- Faily fast - few seconds at most
In [9]:
IPython.display.HTML('./data_score_df.html')
Out[9]:
But what does this actually look like?!?¶
In [10]:
IPython.display.HTML('./na_data_scoring.html')
Out[10]:
On validation...¶
- Those affliated to the Data Strategy are probably doing data-related work...
In [13]:
IPython.display.HTML('./data_scoring_comparison.html')
Out[13]:
Conclusion¶
Key Findings¶
- City job description data appears amenable to NLP analysis
- City positions seem to have three groupings in relation to data key words:
- Intensive workers (the green band)
- Majority in the middle (the grey band)
- Low intensity/bad data (the red band)
- 'Processing' and 'Analysis' terminology is more prevelant than 'Gathering' and 'Dissemination'.
Recommendations¶
- Analysis is validated, qualitatively
- Use the 'green band' as beta testers for Data Strategy initiatives
- Data Strategy leaderships needs to reflect on absence of 'gathering' and 'processing' intensive positions.
- ?
Bonus Slides¶
Investigating Dynamics¶
- Principal Component Analysis - tries to explain variance (difference in the dataset).
- Remaps data into new, reduced dimension form.
- Sometimes, these dimensions have meanings.
In [12]:
IPython.display.HTML('./data_scoring_pca.html')
Out[12]: