05_sliding_windows.ipynb¶
words_for_analysis and stopwords¶
I'm using the standard (i.e., modern) set of nltk english stopwords. It's not the best set, perhaps, although I don't think it makes a lot of difference . . . I also added "u" to the list.
In [1]:
import codecs, re, textwrap
from nltk.corpus import stopwords
#words_for_analysis = ['eye', 'look', 'see', 'observe', 'witness']
words_for_analysis = ['eye', 'look', 'see']
print 'words_for_analysis:',
print
print '\n' + '\n'.join(textwrap.wrap(' '.join(sorted(list(words_for_analysis))), 80))
sw = set(stopwords.words('english') + [])
wrapper = textwrap.TextWrapper(width=60)
print
print 'stopwords:',
print
print '\n' + '\n'.join(textwrap.wrap(' '.join(sorted(list(sw))), 80))
Load spacy¶
In [2]:
import spacy
nlp = spacy.load('en')
Load the text, etc¶
In [3]:
CORPUS_FOLDER = '/home/spenteco/0/corpora/muncie_public_library_corpus/PG_no_backmatter_fiction/'
text = codecs.open(CORPUS_FOLDER + 'Bront_Charlotte_Jane_Eyre_An_Autobiography_PG_1260.txt',
'r', encoding='utf-8').read()
text = re.sub('\s+', ' ', text).strip()
doc = nlp(text)
tokenization, word counting, etc¶
In [4]:
import string
from collections import defaultdict
original_tokens = []
tokens_for_shingles = []
pos_lemma_counts = defaultdict(lambda : defaultdict(int))
for token in doc:
original_tokens.append(token.text)
if token.pos_ == 'PUNCT':
tokens_for_shingles.append('')
elif token.pos_ == 'PRON':
tokens_for_shingles.append(token.text.lower())
else:
tokens_for_shingles.append(token.lemma_)
pos_lemma_counts[token.pos_][token.lemma_] += 1
word_counts = defaultdict(int)
for token in tokens_for_shingles:
word_counts[token] += 1
Helper functions¶
The next three cells contain functions called from the "main process loop" (see below).
get_shingles breaks the poem into overlapping "shingles" (shingles are like chunks, except that they overlap). Note that shingle_size and shingle_overlap as passed into this routine as parameters, so it's very easy to change them, and to run this notebook with different settings. Interestingly enough, if shingle_size = shingle_overlap, then this routine will produce non-overlapping shingles (i.e., "chunks" as we usually have understood them).
graph_word produces the bar plots that appear below.
find_local_maximums locates the "peak" or "peaks" in the bar plots. It works with the shingle_size and shingle_overlap settings which produced the bar plots below; however, this line of code:
window_size = int(len(shingle_scores) * 0.25)
may cause the function to not work correctly with other shingle_size and shingle_overlap settings, the problem being the fixed 0.25 factor used to set window_size.
In [5]:
import itertools
def get_shingles(words, shingle_size, shingle_overlap):
shingles = []
shingle_offsets = []
a = 0
keep_looping = True
while keep_looping == True:
to_a = a + shingle_size
if to_a > len(words):
to_a = len(words)
keep_looping = False
shingles.append(words[a: to_a])
shingle_offsets.append([a, to_a])
a = a + shingle_size - shingle_overlap
return shingles, shingle_offsets
In [6]:
!conda uninstall -y pandas seaborn
In [7]:
%matplotlib inline
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
import unicodecsv as csv
from pylab import rcParams
rcParams['figure.figsize'] = 25, 3
#import seaborn as sns
def graph_word(variance, word, n_occurences, shingle_scores, high_score, local_maximums):
print
print word, 'n_occurences', n_occurences, \
'variance', variance, \
'local_maximums', local_maximums
plt.bar(range(len(shingle_scores)), shingle_scores.values(), align='center', color='#98AFC7', alpha=1.0)
plt.title(word)
plt.xlabel('shingle')
plt.ylabel('n words')
plt.ylim(0, high_score)
plt.show()
In [8]:
def find_local_maximums(shingle_scores):
window_size = int(len(shingle_scores) * 0.25)
local_maximums = []
for a in range(0, len(shingle_scores)):
slice_start = a - window_size
slice_end = a + window_size
a_is_local_max = True
for b in range(slice_start, slice_end):
if b != a and b >= 0 and b < len(shingle_scores.values()):
if shingle_scores.values()[b] > shingle_scores.values()[a]:
a_is_local_max = False
if a_is_local_max == True and shingle_scores.values()[a] != 0:
local_maximums.append((a, shingle_scores.values()[a]))
local_maximums = sorted(list(set(local_maximums)))
return local_maximums
Main process loop¶
This cell, which calls the functions listed in the previous three cells, produces two outputs:
For every word in words_for_analysis (159 words, in this run), output a line of text listing its number of occurrencs, its variance (i.e., its amount of "clumpiness"), and its local maximums. For example:
gott n_occurences 228 variance 15.6544929059 local_maximums [(0, 69), (10, 130)]
Local maximums are expressed as pairs (shingle_number, number of occurrences). "gott", for example, has two local maximums, one in shingle 0 (69 occurences; shingle counting starts with zero, not one), and one in shingle 10 (130 occurrences). So "gott" is clumped at the beginning and the end of the poem.
Words are listed in variance ("clumpiness") order, high to low.
- After the graphs, the process lists for each shingle the words which have a local maximum in that shingle. In other words, the list shows which words are clumped where. I include only words which a variance > 1.0, and with three or fewer local maximum; i.e., the list includes only clearly "clumpy" words. This section (scroll to the bottom) would seem to be the most critically interesting. The graphs serve two purposes: one, to demonstrate the method; and two, to provide background for the list of "clumpy" words. For example, "türken" appears in the list at shingle 5; however, if you look at its graph, you'll see that shingle 5 is its peak, and that it also occurs frequently in shingles before and after 5.
This cell contains a lot of commented-out code (the lines prefixed with "#"), where I experiment with different shingle sizes, check the number of words in the resulting shingles, etc.
Bottom line?¶
There's a lot of clumping, and a lot of similar words clumping, in shingles 0 and 10 (i.e., at the beginning and end of the poem). Does the poem begin and end with similar concerns?
There's significant clumping in shingles 4, 5 and 6, although not as much as in 0, and 10. One of these shingles (5, the middle of poem) shows that "turk", etc appears there, much as we expected. Interesting, and unlike, 0 and 10, the clumpy words in 4, 5 and 6 are different.
In [9]:
from gensim import corpora, models
import numpy as np
highest_score = -1.0
#for SHINGLE_SIZE, SHINGLE_OVERLAP, HIGH_SCORE in [[1000, 200, 76], [2000, 400, 125]]:
#for SHINGLE_SIZE, SHINGLE_OVERLAP, HIGH_SCORE in [[250, 75, 10]]:
for SHINGLE_SIZE, SHINGLE_OVERLAP, HIGH_SCORE in [[250, 0, 10]]:
shingles, shingle_offsets = get_shingles(tokens_for_shingles, SHINGLE_SIZE, SHINGLE_OVERLAP)
#for sn, s in enumerate(shingles):
# print 'shingle number', sn, 'number of words', len(s)
#print
print
print '************************************************************'
print 'SHINGLE_SIZE', SHINGLE_SIZE, 'SHINGLE_OVERLAP', SHINGLE_OVERLAP, 'len(shingles)', len(shingles)
print '************************************************************'
dictionary = corpora.Dictionary(shingles)
corpus = [dictionary.doc2bow(doc) for doc in shingles]
#tfidf = models.TfidfModel(corpus)
#corpus_tfidf = tfidf[corpus]
#corpus_tf = []
#for a in range(0, len(corpus)):
# new_row = []
# for b in corpus[a]:
# new_row.append([b[0], float(b[1]) / float(len(shingles[a]))])
# corpus_tf.append(new_row)
doc_word_scores = []
#for doc in corpus_tfidf:
#for doc in corpus_tf:
for doc in corpus:
word_scores = {}
for id, value in doc:
word = dictionary.get(id)
if word in words_for_analysis:
word_scores[word] = value
if value > highest_score:
highest_score = value
doc_word_scores.append(word_scores)
for doc_word_score in doc_word_scores:
total_doc_word_score = 0
for k, v in doc_word_score.iteritems():
total_doc_word_score += v
doc_word_score['ALL'] = total_doc_word_score
current_shingle_value = -1
current_shingle_numbers = []
print_these = []
print
for shingle_n, doc_word_score in enumerate(doc_word_scores):
current_value = doc_word_score['ALL']
if current_value > 0:
current_value = 1
if current_value != current_shingle_value:
#print current_shingle_value, current_shingle_numbers
print_these.append([current_shingle_value, current_shingle_numbers])
current_shingle_numbers = []
current_shingle_value = current_value
current_shingle_numbers.append(shingle_n)
if len(current_shingle_numbers) > 0:
#print current_shingle_value, current_shingle_numbers
print_these.append([current_shingle_value, current_shingle_numbers])
print
scores_by_variance = []
for word in words_for_analysis + ['ALL']:
plot_results = {}
for dn, d in enumerate(doc_word_scores):
plot_results[dn] = 0.0
try:
plot_results[dn] = d[word]
except KeyError:
pass
plot_results_total = 0.0
for v in plot_results.values():
plot_results_total += v
plot_results_scaled = []
for v in plot_results.values():
plot_results_scaled.append(v / plot_results_total)
# COMPUTE VARIANCE USING THE RAW DF SCORES, OR SCALED SCORES?
#scores_by_variance.append([np.var(plot_results_scaled), word, len(word_lines[word]), plot_results])
#scores_by_variance.append([np.var(plot_results.values()), word, len(word_lines[word]), plot_results])
scores_by_variance.append([(np.var(plot_results.values()) / np.mean(plot_results.values())),
word, word_counts[word], plot_results])
all_local_maximums = {}
scores_by_variance.sort(reverse=True)
print
print 'ALL ********************************************************'
#print 'HIGH *******************************************************'
#print 'LOW ********************************************************'
for s in scores_by_variance:
#for s in scores_by_variance[:10]:
#for s in scores_by_variance[-10:]:
local_maximums = find_local_maximums(s[3])
if s[0] > 1.0 and len(local_maximums) <= 3:
for l in local_maximums:
try:
all_local_maximums[l[0]].append([s[1], l[1]])
except KeyError:
all_local_maximums[l[0]] = [[s[1], l[1]]]
graph_word(s[0], s[1], s[2], s[3], HIGH_SCORE, local_maximums)
print
print 'LOCAL MAXIMUMS **********************************************'
print
for shingle_n in sorted(all_local_maximums.keys()):
print 'shingle', shingle_n, 'words:',
for wn, w in enumerate(all_local_maximums[shingle_n]):
if wn == len(all_local_maximums[shingle_n]) - 1:
print w[0] + ' ' + str(w[1])
else:
print w[0] + ' ' + str(w[1]) + ',',
print
print
print 'highest_score', highest_score
In [10]:
import tabletext
# shingle_offsets
# original_tokens
print_results = [['', 'n tokens', 'n quotes', 'n quotes/n tokens'],]
for a in print_these[1:]:
shingle_type = 'NOT SEEING'
if a[0] == 1:
shingle_type = 'SEEING'
from_offset = shingle_offsets[a[1][0]][0]
to_offset = shingle_offsets[a[1][-1]][-1]
n_quotes = 0
for token in original_tokens[from_offset: to_offset]:
if token == '"':
n_quotes += 1
ratio_quotes = '%.5f' % (float(n_quotes) / float(len(original_tokens[from_offset: to_offset])))
#print
#print shingle_type, from_offset, to_offset, len(original_tokens[from_offset: to_offset])
#print shingle_type, len(original_tokens[from_offset: to_offset]), n_quotes, ratio_quotes
print_results.append([shingle_type, len(original_tokens[from_offset: to_offset]), n_quotes, ratio_quotes])
print tabletext.to_text(print_results)
In [11]:
from collections import defaultdict, Counter
from textblob import Word
from textblob.wordnet import VERB
from textblob.wordnet import NOUN
In [12]:
hyper = lambda s: s.hypernyms()
def matches_beginnings(synset, beginnings):
result = False
for b in beginnings:
if synset.startswith(b + '.'):
result = True
break
return result
verb_synset_beginnings = [
'experience',
'see',
'perceive',
'look',
'read',
'examine',
'interpret',
'spy',
'witness',
'visualize'
]
noun_synset_beginnings = [
'writing',
'written_communication',
'light',
'eye',
'fire',
'written_symbol',
'communication',
'body_part',
'feeling',
'phenomenon',
'perception',
'visual_communication',
'visual_property',
]
verb_synset_words = defaultdict(list)
noun_synset_words = defaultdict(list)
for pos in pos_lemma_counts.keys():
synset_counts = defaultdict(int)
if pos == 'NOUN':
for pos_lemma_count in Counter(pos_lemma_counts[pos]).most_common():
word_synsets = Word(pos_lemma_count[0]).get_synsets(pos=NOUN)
for w in word_synsets:
h = list(w.closure(hyper, depth=10))
synset_counts[w.name().split('.')[0]] += pos_lemma_count[1]
if matches_beginnings(w.name(), noun_synset_beginnings) == True:
noun_synset_words[w.name()].append(pos_lemma_count[0])
for s in h:
synset_counts[s.name().split('.')[0]] += pos_lemma_count[1]
if matches_beginnings(s.name(), noun_synset_beginnings) == True:
noun_synset_words[s.name()].append(pos_lemma_count[0])
if pos == 'VERB':
pass
for pos_lemma_count in Counter(pos_lemma_counts[pos]).most_common():
#if w[1] < 100:
# break
word_synsets = Word(pos_lemma_count[0]).get_synsets(pos=VERB)
for w in word_synsets:
h = list(w.closure(hyper, depth=10))
synset_counts[w.name().split('.')[0]] += pos_lemma_count[1]
if matches_beginnings(w.name(), verb_synset_beginnings) == True:
verb_synset_words[w.name()].append(pos_lemma_count[0])
for s in h:
synset_counts[s.name().split('.')[0]] += pos_lemma_count[1]
if matches_beginnings(s.name(), verb_synset_beginnings) == True:
verb_synset_words[s.name()].append(pos_lemma_count[0])
#for w in Counter(synset_counts).most_common():
# print w[0], w[1]
#print
#for k in sorted(verb_synset_words.keys()):
# print k, sorted(list(set(verb_synset_words[k])))
print
for k in sorted(noun_synset_words.keys()):
print k, sorted(list(set(noun_synset_words[k])))
In [ ]: