{"id":608,"date":"2016-11-18T15:22:59","date_gmt":"2016-11-18T20:22:59","guid":{"rendered":"http:\/\/blogs.shu.edu\/digitalhumanities\/?p=608"},"modified":"2019-04-16T11:29:53","modified_gmt":"2019-04-16T15:29:53","slug":"text-processing-and-analysis","status":"publish","type":"post","link":"https:\/\/blogs.shu.edu\/digitalhumanities\/2016\/11\/18\/text-processing-and-analysis\/","title":{"rendered":"Text Processing And Analysis"},"content":{"rendered":"

In working with Nathan Kahl on his Digital Humanities Faculty Fellows<\/a> project, I came across an online service from Meaning Cloud<\/a> that was really helpful. Here’s a primer on how you can get started, and some of the kinds of results you can get back from it.<\/p>\n

Besides simple web forms to process pieces of text, they offer a whole set of APIs that let you automate the process for larger sets. Nate’s project, for example, used the lyrics of 4,000 popular songs. And in addition to the APIs themselves, they have sample code that helps you cut through the hard-for-humans-to-read XML or JSON<\/a> results.<\/p>\n

The kinds of processing you can do include topics extraction, text classification, sentiment analysis, language identification, lemmatization part-of-speech and parsing, corporate reputation and text clustering.<\/p>\n

Getting Started<\/h3>\n

You’ll need to open an account. The basic level is free, and it gives you something like 40,000 individual submissions per month. You’ll get a logon and password, and also an API key which you will use in all your requests. Be sure to copy it someplace safe! Once you have it, you can right away start to work with some sample texts.<\/p>\n

\"lemmatization-pos-and-parsing-2-0-console-meaningcloud\"<\/a><\/p>\n

For Example<\/h3>\n

Lemmatization, POS and parsing was the most relevant tool for our project, which involved simple word counts (similar to n-grams) in Billboard top-100 songs over a 40 year period. Let’s take a small excerpt from a song, which should fit in under fair use, and run it through the processor.<\/p>\n

Mostly you can get by with the default settings. But you’ll need to specify your key, output format, language and text. You can paste your text in, as I did, upload a file, or point to a URL.<\/p>\n

I’ve gone with a JSON result, because it’s easy to use with PHP, JavaScript and other tools. The result is what in programming language is called an “array<\/a>.”<\/p>\n

If you want to go in for a deep dive, the resulting array is here<\/a>; but let’s break out a few sections so you can see what you’re getting.<\/p>\n

The text is split up into “tokens,” which is exactly what we needed. Top level tokens are sentences, second level tokens are phrases, and finally you get down to the level of an individual word. (This explanation is all a really simple gloss of course; language processing is quite complex and I don’t pretend to understand more than a fraction of it.) The array contains the type (phrase, sentence) along with the form (the original text). Formatting is indicated — if the original was bold, italic, underlined, or a title — whether it’s affected by a negation, and each atom gets an ID number so you can reference them.<\/p>\n

This is a result for the first word, “early.”
\n<\/code><\/p>\n

token_list: [\r\n  {\r\n    form: \"Early\",\r\n    id: \"1\",\r\n    inip: \"0\",\r\n    endp: \"4\",\r\n    style: {\r\n      isBold: \"no\",\r\n      isItalics: \"no\",\r\n      isUnderlined: \"no\",\r\n      isTitle: \"no\"\r\n    },\r\n    separation: \"_\",\r\n    quote_level: \"0\",\r\n    affected_by_negation: \"no\",\r\n    analysis_list: [\r\n      {\r\n      tag: \"ENMPN6\",\r\n      lemma: \"early\",\r\n      original_form: \"early\"\r\n      }\r\n    ]\r\n  }\r\n]\r\n<\/pre>\n
The part we were interested in is at the end, where we get the tag, lemma and original form.<\/p>\n
The tag is a coding of the type of speech and context of the word. The lemma is similar to word-stemming, the difference being that instead of just capturing “big,” “bigger” and “biggest” as forms of the same word it also captures “good,” “better” and “best.” Or verb forms like “was shining” reduce down to “shine.” All three of these go into our database.<\/p>\n
The individual characters of the tags correspond to a morphosyntactic tagset<\/a>. So we can look up our tag for “early” and derive the following:<\/p>\n
\n
E<\/dt>\n
Adverb<\/a><\/dd>\n
N<\/dt>\n
Normal<\/dd>\n
M<\/dt>\n
Adverb of manner<\/dd>\n
P<\/dt>\n
Positive<\/dd>\n
N<\/dt>\n
Normal word<\/dd>\n
6<\/dt>\n
Medium-high frequency<\/dd>\n<\/dl>\n
In a similar way it analyzes the individual clauses or phrases. So “wondering if she’d changed at all” returns a tag of “ZA—XPP—-“, which can be analyzed from the clause table<\/a> as:<\/p>\n
\n
Z<\/dt>\n
Clause<\/a><\/dd>\n
A<\/dt>\n
Adjectival<\/dd>\n
–<\/dt>\n
Gender (not applicable in this case)<\/dd>\n
–<\/dt>\n
Number (also not applicable)<\/dd>\n
–<\/dt>\n
Person (n\/a)<\/dd>\n
X<\/dt>\n
Syntactic Function (in this case, Non-restrictive apposition)<\/dd>\n
P<\/dt>\n
Mode (participle)<\/dd>\n
P<\/dt>\n
Tense (past)<\/dd>\n
–<\/dt>\n
n\/a<\/dd>\n
–<\/dt>\n
n\/a<\/dd>\n
–<\/dt>\n
n\/a<\/dd>\n
–<\/dt>\n
n\/a<\/dd>\n<\/dl>\n
As you browse the code tables you see there’s a very rich analysis available.<\/p>\n
The database stores the song titles, years, performers, and chart position; as well as a table of words; a table that maps words to songs along with a total count for each; the same for lemmas; and the same for tags. Using these indexed tables you can search very quickly for the cumulative data.<\/p>\n
What Does All This Do?<\/h3>\n
Our first implementation was a page that graphs the frequency of usage for words, year-by-year, over the 40 year period from 1960 to 2000. Here’s an example, comparing the number of songs that used “man” and “woman”:<\/p>\n
\"word-count-examples\"<\/a><\/p>\n
You can also get the number of times in total the words were used:<\/p>\n
\"word-count-examples-with-counts\"<\/a><\/p>\n
Wasn’t it James Brown who said, “it’s a man’s world?” But I digress.<\/p>\n
There is another version that also returns a table, which you could import into Excel and do further analysis.<\/p>\n
Another view shows you the lemmatization of a word, in this case, all the various word-forms for “love.”<\/p>\n
\"screenshot_1\"<\/a><\/p>\n
You can play with the whole package on the project page<\/a>. There are also a couple of APIs that return the song counts along with a list of the songs where the word is used.<\/p>\n
\n
Meaning Cloud’s JSON Return Array<\/h3>\n
<\/code><\/p>\n
{\r\n  status: {\r\n  code: \"0\",\r\n  msg: \"OK\",\r\n  credits: \"1\",\r\n  remaining_credits: \"39993\"\r\n  },\r\ntoken_list: [\r\n{\r\ntype: \"sentence\",\r\nid: \"36\",\r\ninip: \"0\",\r\nendp: \"120\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"A\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\ntoken_list: [\r\n{\r\ntype: \"phrase\",\r\nform: \"Early one morning the sun was shining, I was laying in bed, wondering if she'd changed at all, if her hair was still red\",\r\nid: \"53\",\r\ninip: \"0\",\r\nendp: \"119\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"_\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"Z-----------\",\r\nlemma: \"*\",\r\noriginal_form: \"Early one morning the sun was shining, I was laying in bed, wondering if she'd changed at all, if her hair was still red\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\ntype: \"phrase\",\r\nform: \"Early\",\r\nid: \"49\",\r\ninip: \"0\",\r\nendp: \"4\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"_\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"1\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"35\",\r\ntype: \"isMannerComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GEM--M--\",\r\nlemma: \"early\",\r\noriginal_form: \"Early\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"Early\",\r\nid: \"1\",\r\ninip: \"0\",\r\nendp: \"4\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"_\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"ENMPN6\",\r\nlemma: \"early\",\r\noriginal_form: \"early\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"one morning\",\r\nid: \"39\",\r\ninip: \"6\",\r\nendp: \"16\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"3\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"35\",\r\ntype: \"isTimeComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GN-S3T--\",\r\nlemma: \"morning\",\r\noriginal_form: \"one morning\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"one\",\r\nnormalized_form: \"numeric@1\",\r\nid: \"2\",\r\ninip: \"6\",\r\nendp: \"8\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"MD-SCN7\",\r\nlemma: \"one\",\r\noriginal_form: \"one\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"e01b5f50fb\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"e01b5f50fb\",\r\nform: \"one\",\r\ninfo: \"sementity\/class=class@fiction=nonfiction@id=ODENTITY_NUMEX@type=Top>Numex semld_list=sumo:Quantity\"\r\n}\r\n]\r\n},\r\n{\r\nform: \"morning\",\r\nid: \"3\",\r\ninip: \"10\",\r\nendp: \"16\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"NC-S-N6\",\r\nlemma: \"morning\",\r\noriginal_form: \"morning\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"the sun\",\r\nnormalized_form: \"date@|||d|||||||\",\r\nid: \"40\",\r\ninip: \"18\",\r\nendp: \"24\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"5\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"35\",\r\ntype: \"isSubject\"\r\n},\r\n{\r\nid: \"45\",\r\ntype: \"iof_isAnaphora\"\r\n},\r\n{\r\nid: \"42\",\r\ntype: \"iof_isPossessor\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GN-S3S--\",\r\nlemma: \"sun\",\r\noriginal_form: \"the sun\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"the\",\r\nid: \"4\",\r\ninip: \"18\",\r\nendp: \"20\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"TD-SN9\",\r\nlemma: \"the\",\r\noriginal_form: \"the\"\r\n}\r\n]\r\n},\r\n{\r\nform: \"sun\",\r\nnormalized_form: \"date@|||d|||||||\",\r\nid: \"5\",\r\ninip: \"22\",\r\nendp: \"24\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"NC-S-N5\",\r\nlemma: \"sun\",\r\noriginal_form: \"sun\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"2245de2af4\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"2245de2af4\",\r\nform: \"sun\",\r\ninfo: \"sementity\/class=class@fiction=nonfiction@id=ODENTITY_ASTRAL_BODY@type=Top>Location>AstralBody semld_list=sumo:AstronomicalBody semtheme_list\/id=ODTHEME_ASTRONOMY@type=Top>NaturalSciences>Astronomy\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\ntype: \"multiword\",\r\nform: \"was shining\",\r\nid: \"35\",\r\ninip: \"26\",\r\nendp: \"36\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"6\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"49\",\r\ntype: \"iof_isMannerComplement\"\r\n},\r\n{\r\nid: \"39\",\r\ntype: \"iof_isTimeComplement\"\r\n},\r\n{\r\nid: \"40\",\r\ntype: \"iof_isSubject\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"VI-S3ACA-N-N9\",\r\nlemma: \"shine\",\r\noriginal_form: \"was shining\"\r\n}\r\n]\r\n},\r\n{\r\nform: \",\",\r\nid: \"8\",\r\ninip: \"37\",\r\nendp: \"37\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"A\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"1D--\",\r\nlemma: \",\",\r\noriginal_form: \",\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"I\",\r\nid: \"46\",\r\ninip: \"39\",\r\nendp: \"39\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"9\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"27\",\r\ntype: \"isSubject\"\r\n},\r\n{\r\nid: \"34\",\r\ntype: \"isSubject\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GN-S1S--\",\r\nlemma: \"I\",\r\noriginal_form: \"I\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"I\",\r\nid: \"9\",\r\ninip: \"39\",\r\nendp: \"39\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"PP-S1NN-\",\r\nlemma: \"I\",\r\noriginal_form: \"I\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"PRONHUMAN\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"PRONHUMAN\",\r\nform: \"I\",\r\ninfo: \"semhum=human\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\ntype: \"multiword\",\r\nform: \"was laying\",\r\nid: \"34\",\r\ninip: \"41\",\r\nendp: \"50\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"10\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"46\",\r\ntype: \"iof_isSubject\"\r\n},\r\n{\r\nid: \"48\",\r\ntype: \"iof_isComplement\"\r\n},\r\n{\r\nid: \"50\",\r\ntype: \"iof_isComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"VI-S1ACA-N-N9\",\r\nlemma: \"lay\",\r\noriginal_form: \"was laying\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"in bed, wondering if she'd changed at all\",\r\nid: \"48\",\r\ninip: \"52\",\r\nendp: \"92\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"12\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"34\",\r\ntype: \"isComplement\"\r\n},\r\n{\r\nid: \"48\",\r\ntype: \"isNonRestrictiveApposition\"\r\n},\r\n{\r\nid: \"48\",\r\ntype: \"iof_isNonRestrictiveApposition\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GY---C--\",\r\nlemma: \"in\",\r\noriginal_form: \"in bed\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"in\",\r\nid: \"12\",\r\ninip: \"52\",\r\nendp: \"53\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"YN6\",\r\nlemma: \"in\",\r\noriginal_form: \"in\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"bed\",\r\nid: \"41\",\r\ninip: \"55\",\r\nendp: \"57\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"13\",\r\nanalysis_list: [\r\n{\r\ntag: \"GN-S3---\",\r\nlemma: \"bed\",\r\noriginal_form: \"bed\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"bed\",\r\nid: \"13\",\r\ninip: \"55\",\r\nendp: \"57\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"NC-S-N6\",\r\nlemma: \"bed\",\r\noriginal_form: \"bed\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"19b7b05b8c\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"19b7b05b8c\",\r\nform: \"bed\",\r\ninfo: \"sementity\/class=class@fiction=nonfiction@id=ODENTITY_CHANNEL@type=Top>Location>GeographicalEntity>WaterForm>Channel semld_list=sumo:Channel\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\nform: \",\",\r\nid: \"14\",\r\ninip: \"58\",\r\nendp: \"58\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"A\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"1D--\",\r\nlemma: \",\",\r\noriginal_form: \",\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"wondering if she'd changed at all\",\r\nid: \"52\",\r\ninip: \"60\",\r\nendp: \"92\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"ZA---XPP----\",\r\nlemma: \"*\",\r\noriginal_form: \"wondering if she'd changed at all\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"wondering\",\r\nid: \"15\",\r\ninip: \"60\",\r\nendp: \"68\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"51\",\r\ntype: \"iof_isComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"VP---PSA-N-N4\",\r\nlemma: \"wonder\",\r\noriginal_form: \"wondering\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"ODENTITY_INTENTIONAL_PSYCHOLOGICAL_PROCESS\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"ODENTITY_INTENTIONAL_PSYCHOLOGICAL_PROCESS\",\r\nform: \"wonder\",\r\ninfo: \"sementity\/id=ODENTITY_INTENTIONAL_PSYCHOLOGICAL_PROCESS@type=Top>Process>IntentionalProcess>IntentionalPsychologicalProcess semld_list=sumo:IntentionalProcess\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"if she'd changed at all\",\r\nid: \"51\",\r\ninip: \"70\",\r\nendp: \"92\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"15\",\r\ntype: \"isComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"ZE---CIA----\",\r\nlemma: \"*\",\r\noriginal_form: \"if she'd changed at all\"\r\n},\r\n{\r\ntag: \"ZE---CO-----\",\r\nlemma: \"*\",\r\noriginal_form: \"if she'd changed at all\"\r\n},\r\n{\r\ntag: \"ZE---CC-----\",\r\nlemma: \"*\",\r\noriginal_form: \"if she'd changed at all\"\r\n},\r\n{\r\ntag: \"ZE---CA-----\",\r\nlemma: \"*\",\r\noriginal_form: \"if she'd changed at all\"\r\n},\r\n{\r\ntag: \"ZE---CO-----\",\r\nlemma: \"*\",\r\noriginal_form: \"if she'd changed at all\"\r\n},\r\n{\r\ntag: \"ZE---CC-----\",\r\nlemma: \"*\",\r\noriginal_form: \"if she'd changed at all\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"if\",\r\nid: \"16\",\r\ninip: \"70\",\r\nendp: \"71\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"CSBN7\",\r\nlemma: \"if\",\r\noriginal_form: \"if\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"she\",\r\nid: \"45\",\r\ninip: \"73\",\r\nendp: \"75\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"17\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"40\",\r\ntype: \"isAnaphora\"\r\n},\r\n{\r\nid: \"33\",\r\ntype: \"isSubject\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GNFS3S--\",\r\nlemma: \"she\",\r\noriginal_form: \"she\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"she\",\r\nid: \"17\",\r\ninip: \"73\",\r\nendp: \"75\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"PPFS3NN8\",\r\nlemma: \"she\",\r\noriginal_form: \"she\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"PRONHUMAN\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"PRONHUMAN\",\r\nform: \"she\",\r\ninfo: \"semhum=human\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\ntype: \"multiword\",\r\nform: \"'d changed\",\r\nid: \"33\",\r\ninip: \"76\",\r\nendp: \"85\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"A\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"32\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"45\",\r\ntype: \"iof_isSubject\"\r\n},\r\n{\r\nid: \"47\",\r\ntype: \"iof_isComplement\"\r\n},\r\n{\r\nid: \"50\",\r\ntype: \"iof_isComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"VI-UUASA-N-E-\",\r\nlemma: \"change\",\r\noriginal_form: \"'d changed\"\r\n},\r\n{\r\ntag: \"VI-UUCSA-N-E8\",\r\nlemma: \"change\",\r\noriginal_form: \"'d changed\"\r\n},\r\n{\r\ntag: \"VW-UUOSA-N-E7\",\r\nlemma: \"change\",\r\noriginal_form: \"'d changed\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"at all\",\r\nid: \"47\",\r\ninip: \"87\",\r\nendp: \"92\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"21\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"33\",\r\ntype: \"isComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GY---C--\",\r\nlemma: \"at\",\r\noriginal_form: \"at all\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"at\",\r\nid: \"21\",\r\ninip: \"87\",\r\nendp: \"88\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"YN8\",\r\nlemma: \"at\",\r\noriginal_form: \"at\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"all\",\r\nid: \"44\",\r\ninip: \"90\",\r\nendp: \"92\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"22\",\r\nanalysis_list: [\r\n{\r\ntag: \"GN--3---\",\r\nlemma: \"all\",\r\noriginal_form: \"all\"\r\n},\r\n{\r\ntag: \"GN-S3---\",\r\nlemma: \"all\",\r\noriginal_form: \"all\"\r\n},\r\n{\r\ntag: \"GN-P3---\",\r\nlemma: \"all\",\r\noriginal_form: \"all\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"all\",\r\nid: \"22\",\r\ninip: \"90\",\r\nendp: \"92\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"QP--PN6\",\r\nlemma: \"all\",\r\noriginal_form: \"all\"\r\n},\r\n{\r\ntag: \"QP-SPN6\",\r\nlemma: \"all\",\r\noriginal_form: \"all\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"PRONNONHUMAN\"\r\n}\r\n]\r\n},\r\n{\r\ntag: \"QP-PPN6\",\r\nlemma: \"all\",\r\noriginal_form: \"all\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"PRONNONHUMAN\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"PRONNONHUMAN\",\r\nform: \"all\",\r\ninfo: \"semhum=nonhuman\"\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\nform: \",\",\r\nid: \"23\",\r\ninip: \"93\",\r\nendp: \"93\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"A\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"1D--\",\r\nlemma: \",\",\r\noriginal_form: \",\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"if her hair was still red\",\r\nid: \"50\",\r\ninip: \"95\",\r\nendp: \"119\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"34\",\r\ntype: \"isComplement\"\r\n},\r\n{\r\nid: \"33\",\r\ntype: \"isComplement\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"ZE---CIA----\",\r\nlemma: \"*\",\r\noriginal_form: \"if her hair was still red\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"if\",\r\nid: \"24\",\r\ninip: \"95\",\r\nendp: \"96\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"CSBN7\",\r\nlemma: \"if\",\r\noriginal_form: \"if\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"her hair\",\r\nid: \"42\",\r\ninip: \"98\",\r\nendp: \"105\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"26\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"40\",\r\ntype: \"isPossessor\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GN-S3---\",\r\nlemma: \"hair\",\r\noriginal_form: \"her hair\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"her\",\r\nid: \"25\",\r\ninip: \"98\",\r\nendp: \"100\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"SD-SFS3N7\",\r\nlemma: \"her\",\r\noriginal_form: \"her\",\r\nsense_id_list: [\r\n{\r\nsense_id: \"PRONHUMAN\"\r\n}\r\n]\r\n}\r\n],\r\nsense_list: [\r\n{\r\nid: \"PRONHUMAN\",\r\nform: \"her\",\r\ninfo: \"semhum=human\"\r\n}\r\n]\r\n},\r\n{\r\nform: \"hair\",\r\nid: \"26\",\r\ninip: \"102\",\r\nendp: \"105\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"NC-S-N6\",\r\nlemma: \"hair\",\r\noriginal_form: \"hair\"\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\nform: \"was\",\r\nid: \"27\",\r\ninip: \"107\",\r\nendp: \"109\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"46\",\r\ntype: \"iof_isSubject\"\r\n},\r\n{\r\nid: \"43\",\r\ntype: \"iof_isAttribute\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"VI-S1ASA-N-N9\",\r\nlemma: \"be\",\r\noriginal_form: \"was\"\r\n},\r\n{\r\ntag: \"VI-S3ASA-N-N9\",\r\nlemma: \"be\",\r\noriginal_form: \"was\"\r\n}\r\n]\r\n},\r\n{\r\ntype: \"phrase\",\r\nform: \"still red\",\r\nid: \"43\",\r\ninip: \"111\",\r\nendp: \"119\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nhead: \"29\",\r\nsyntactic_tree_relation_list: [\r\n{\r\nid: \"27\",\r\ntype: \"isAttribute\"\r\n}\r\n],\r\nanalysis_list: [\r\n{\r\ntag: \"GA---A--\",\r\nlemma: \"red\",\r\noriginal_form: \"still red\"\r\n}\r\n],\r\ntoken_list: [\r\n{\r\nform: \"still\",\r\nid: \"28\",\r\ninip: \"111\",\r\nendp: \"115\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"ENTPN4\",\r\nlemma: \"still\",\r\noriginal_form: \"still\"\r\n}\r\n]\r\n},\r\n{\r\nform: \"red\",\r\nid: \"29\",\r\ninip: \"117\",\r\nendp: \"119\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"1\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"AP-N4\",\r\nlemma: \"red\",\r\noriginal_form: \"red\"\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n},\r\n{\r\nform: \".\",\r\nid: \"30\",\r\ninip: \"120\",\r\nendp: \"120\",\r\nstyle: {\r\nisBold: \"no\",\r\nisItalics: \"no\",\r\nisUnderlined: \"no\",\r\nisTitle: \"no\"\r\n},\r\nseparation: \"A\",\r\nquote_level: \"0\",\r\naffected_by_negation: \"no\",\r\nanalysis_list: [\r\n{\r\ntag: \"1D--\",\r\nlemma: \".\",\r\noriginal_form: \".\"\r\n}\r\n]\r\n}\r\n]\r\n}\r\n]\r\n}<\/pre>\n","protected":false},"excerpt":{"rendered":"
Part one of what will hopefully be a series about language-processing tools you can use in your projects.<\/p>\n","protected":false},"author":1,"featured_media":613,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[9],"tags":[101,104,182,102,103,100],"class_list":["post-608","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tools","tag-json","tag-lemma","tag-meaning-cloud","tag-natural-language","tag-text-analysis","tag-xml"],"_links":{"self":[{"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/posts\/608","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/comments?post=608"}],"version-history":[{"count":4,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/posts\/608\/revisions"}],"predecessor-version":[{"id":1226,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/posts\/608\/revisions\/1226"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/media\/613"}],"wp:attachment":[{"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/media?parent=608"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/categories?post=608"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.shu.edu\/digitalhumanities\/wp-json\/wp\/v2\/tags?post=608"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}